CRM and IIS Machine Keys

When you have multiple CRM web (front-end) servers splitting the load, requests can bounce between servers or they can be set to remain on the same server through the NLB/Firewall settings. The latter option is commonly referred to as “Sticky Sessions” or “Persistence”.

If requests bounce between servers (whether by design or by fault), you may notice issues when accessing certain functionality within CRM. Often it will be accompanied by the following error message in the event viewer:

Event code: 3012 
Event message: An error occurred processing a web or script resource request. The resource identifier failed to decrypt. 

Exception information: 
    Exception type: HttpException 
    Exception message: Unable to validate data.
   at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType, Boolean signData)
   at System.Web.UI.Page.DecryptString(String s, Purpose purpose)
   at System.Web.Handlers.ScriptResourceHandler.ProcessRequest(HttpContextBase context, VirtualFileReader fileReader, Action`2 logAction, Boolean validatePath)

Basically what is happening is one server encrypted (validated) some of the traffic with an automatically generated key and another server received it but cannot decrypt it because it has different keys. These keys are known as “machine keys” and are found within IIS. By default, these keys are all set to generate automatically and no two are the same. When you are using multiple web servers, the machine keys need to be set statically and shared amongst them. Here is how to do this, starting on the first web server:

1. Open IIS, select the Microsoft Dynamics CRM Website, and double-click “Machine Key”
   
   
2. Uncheck the “Automatically generate at runtime” and “Generate a unique key for each application” boxes under both “Validation Key” and “Decryption Key”.
   
   
3. Click “Generate Keys” on the right side of the screen.
   
   
4. This will create new, random keys.
   
   
5. Click apply.
   
   
6. From here, repeat steps 1 and 2 on the remaining web servers. Instead of generating new keys as in step 3, copy and paste the keys generated from earlier into the remaining servers so that all servers have the same set of keys.
7. Once keys on all servers are set and applied, reset IIS on all the boxes. That should do it.

Why You Should Have TCP Port 80 Open Outbound On Your ADFS Servers

Active Directory Federation Services (ADFS) performs a lot of tasks when it comes to authenticating users into CRM securely. One of those tasks in particular is a certification revocation check to validate that the certificates being used are still valid. ADFS completes this process by reaching out to certification revocation lists (CRLs) over TCP port 80 – basic HTTP communication.

What we’ve seen is that businesses will want to lock down their ADFS servers just to be on the “safe side” and that includes closing TCP Port 80 outbound (e.g. no internet access). If left in its default state, ADFS will break and cause authentication to fail as it knows that it is supposed to check the CLRs to validate the certificate before issuing a token to allow a user into CRM. If it cannot do this, it will not issue a token. You may see an error similar to the following in the ADFS event viewer logs after a failed authentication attempt:

Event ID: 364
Microsoft.IdentityServer.AuthenticationFailedException: MSIS3014: The encryption certificate of the relying party trust 'https://crm.domain.com/' identified by thumbprint '01DEDF6E6F532BF7357457EBEC31DA82SFDA1234' is not valid. It might indicate that the certificate has been revoked, has expired, or that the certificate chain is not trusted. ---> Microsoft.IdentityServer.Service.SecurityTokenService.RevocationValidationException: MSIS3014: The encryption certificate of the relying party trust 'https://crm.domain.com/' identified by thumbprint '01DEDF6E6F532BF7357457EBEC31DA82SFDA1234' is not valid. It might indicate that the certificate has been revoked, has expired, or that the certificate chain is not trusted.

So what are your options?

1. Have your networking team open TCP 80 outbound on the ADFS server(s). This would also apply to all ADFS Proxies or WAP servers. While opening a port might seem less secure at face value it would actually be the opposite as ADFS is able to validate the certificates being used. 
2. The less preferred, but still acceptable, method would be turning off the Certificate Revocation Check of ADFS. The check is controlled individually for each relying party in ADFS so it would need to be turned off for all one by one. To do this open an admin PowerShell prompt and issue the following command:

                             Set-ADFSRelyingPartyTrust  -TargetName <relyingpartytrustName>  

                             -EncryptionCertificateRevocationCheck None