HTTP to HTTPS Redirect for CRM using Claims-Based Authentication and IFD

In some cases, our customers would like HTTP to HTTPS redirection setup for CRM when using Claims-Based Authentication and IFD. There are a number of articles and blogs out on the internet that use various methods to accomplish this (e.g. URL Rewriting) but I find that a lot are overly complicated if all you are looking to do is have the redirect functioning internally (you do not want port 80 open on your firewall anyways). Below is what I perceive to be the easiest method to achieving this – using the HTTP Redirection capabilities of IIS:

1. First we will need to install HTTP Redirection on the CRM Web Server if it is not installed already. In Server Manager, go to Add Roles and Features and choose to add the HTTP Redirection feature under the Web Server role.

2. Create two folders in a directory that you know will not be deleted. Name these folders something along the lines of “CRM Internal Redirect” and “CRM External Redirect”. Folder names and paths aren’t too important but in my example I just put them in the inetpub directory.

3. Next, go into IIS. Make sure to remove the HTTP (port 80) binding from the CRM site before proceeding with the next steps.

4. Since two URLs can be used to be accessed CRM when it is setup for Claims and IFD, we will need to create the redirects for both URLs. Begin by creating a new website in IIS for the Internal URL.

• Site Name: Something descriptive such as “CRM Internal Redirect”
• Physical Path: Browse to the CRM Internal Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your internal CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, open the CRM Deployment Manager and go to properties. Select the Web Addresses tab and you should see your internal URL.

5. Just like step 4, create another new site in IIS – this time for the external URL:

• Site Name: Something descriptive such as “CRM External Redirect”
• Physical Path: Browse to the CRM External Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your exteransl CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, this is the unique name of your CRM organization followed by the domain (e.g. orgname.domain.com).

6. You should now have IIS looking similar to this:

7. Click on your Internal Redirect site and open the HTTP Redirect module:

8. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your internal URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

9. Now do the same for the External redirect site. Click on your External Redirect site and open the HTTP Redirect module. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your external URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

That’s it. You should now be able to enter either of the CRM URLs with HTTP into a browser and have it redirected to HTTPS appropriately.

Insufficient Winsock Resources Available to Complete Socket Connection Initiation

Seemingly out of the blue, one of our customers could no longer log in to their CRM. As it appeared to be an authentication issue, I began troubleshooting on their ADFS server. Upon opening the ADFS Management Console, I was greeted with the following error message:

In the event viewer were a number of errors similar to this:

Microsoft.IdentityServer.Configuration.ReadServiceConfigFailedException: MSIS2001: Configuration service URL is not configured. ---> System.InsufficientMemoryException: Insufficient winsock resources available to complete socket connection initiation. ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 127.0.0.1:1500

Well, this was a new one for me but at least the error message actually gave a description instead of something generic. Logic dictates that if there isn’t enough of something, give it more. In this case, it appeared to me that ADFS was complaining about not having enough ephemeral ports so I figured I would increase the number available.

1. Start Registry Editor.
2. Locate the following subkey in the registry, and then click Parameters:  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. On the Edit menu, click New, and then add the following registry entry:
   Value Type: DWORD
   Value Name: MaxUserPort
   Value Data: 65534 (decimal)
4. Reboot server

After completing the above, ADFS was able to open the connection to its database and users were able to log back in to CRM. Now, it may be worthwhile to investigate exactly why ports suddenly became busy… perhaps there is a socket leak or new applications were installed… but this should at least get you up and running in the meantime.

Calling LoadLibraryEx on ISAPI filter DefaultAddonFilter.dll failed

I recently had an interesting issue with a client’s CRM 2011 to 2016 go-live. Following all of my upgrade work and initial testing, I typically like to reboot the CRM and SQL servers and then test again. In this particular instance, when the servers came back online I tested CRM again only to receive a lovely error message from Internet Explorer:

HTTP Error 500.0 – Internal Server Error

Calling LoadLibraryEx on ISAPI filter C:\Program Files\Microsoft Dynamics CRM\Server\bin\DefaultAddonFilter.dll failed.

After a fair amount of digging I noticed that the Visual C++ 2013 Redistributable (64-bit) was not present in the add/remove programs list on the CRM server. It seemingly uninstalled itself during the reboot because otherwise I would have had this issue pre-reboot. So, I went out and grabbed the installer package for Visual C++ 2013 Redistributable (64-bit) from MS (http://go.microsoft.com/fwlink/p/?LinkId=402059) and reinstalled it. After that and an IISReset, CRM began functioning as normal and everything retested everything successfully.

Unfortunately, I am at a loss as to why this happened – perhaps something to do with server patching. Also of note is that the redistributable did not remove itself from the SQL server.

Reports in CRM not working - The type initializer for 'Microsoft.Crm.LocatorService' threw an exception.

We had a customer running into an issue while trying to run reports from CRM. Of course, the error in the CRM UI was the usual rsProcessingAborted error - Cannot create a connection to Data Source. So as I always need to do, I dug into the SSRS logs. The errors I could extract from there were:

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The type initializer for 'Microsoft.Crm.LocatorService' threw an exception. ---> 

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: Cannot load Counter Name data because an invalid index 'W3SVC_W3WP' was read from the registry.

From what I could surmise, there was something wrong with the performance counters on the server(s). I did find that Microsoft does have a KB for this error but it is in reference to update rollups on the CRM Outlook Client. Luckily for me, the fix turned out to be similar!

1. On the CRM Server, open an administrative command prompt.
2. Type “lodctr /r” and press enter.
3. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
4. Restart CRM Services and run an IISReset
5. On the SSRS Server, open an administrative command prompt.
6. Type “lodctr /r” and press enter.
7. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
8. Restart SSRS Services.

After following the above process, reports in CRM began working again. The commands run rebuild the performance counters.

An attempt was made to access a socket in a way forbidden by its access permissions

When setting up the Microsoft Dynamics Email Router, there can be a lot of obstacles in getting emails to send and receive properly. One of which turns out to be McAfee VirusScan. If you get the following error when testing the connection on the Outgoing profile, you may want to see if McAfee (or any other virus protection software) is running on the server:

Outgoing Status: Failure – An error occurred while checking the connection to the email server {smtp server name}. An attempt was made to access a socket in a way forbidden by its access permissions.

In order to fix this, right click on the McAfee icon in the system tray and open up the VirusScan Console.

In the VirusScan Console, double-click “Access Protection”.

Under Categories, click on “Anti-virus Standard Protection” and then check to see if there are any checkmarks next to “Prevent mass mailing worms from sending mail”. If there are, simply uncheck them.

Click OK and close out of the VirusScan Console. Open the email router and try testing again. You should now be seeing green!

Report Server cannot load the SQLPDW extension

While troubleshooting SSRS issues, you may run into the following error messages in logs and traces:

Report Server (MSSQLSERVER) cannot load the SQLPDW extension.

Report Server (MSSQLSERVER) cannot load the TERADATA extension.

On numerous occasions I have seen both customers and colleagues believe that these errors were the source of their problems. After which, the wild goose chase to resolve these errors ensues. I am here to tell you that it is highly unlikely that they are related to your problem at hand (especially if you are only using SSRS for CRM reporting).

These errors occur because the Teradata and SQLPDW extensions are registered in the Reporting Services configuration file by default but the assemblies for these extensions do not get installed in the same manner. Typically these extensions are used with big data and business analytics.

If the errors do not bother you, simply ignore them and track down other error messages. However, if you are the type that prefers clean logs you can take the following steps to alleviate the error messages from being logged:

• Open the rsreportserver.config file. This can usually be found in \Program Files\Microsoft SQL Server\MSRSXX_X.MSSQLSERVER\Reporting Services\ReportServer.
• In this config file, simply search for and comment out the entries related to SQLPDW and TERADATA. There should be 2 entries for SQLPDW and 3 for TERADATA. Below is an example:

<!--Extension Name="SQLPDW" Type="Microsoft.ReportingServices.DataExtensions.SqlDwConnectionWrapper,Microsoft.ReportingServices.DataExtensions"/-->

<!--Extension Name="SQLPDW" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.MSSQLADW.MSSqlAdwSQCommand,Microsoft.ReportingServices.SemanticQueryEngine">

    <Configuration>

        <EnableMathOpCasting>False</EnableMathOpCasting>

    </Configuration>

</Extension-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.DataExtensions.TeradataConnectionWrapper,Microsoft.ReportingServices.DataExtensions"/-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.Teradata.TdSqlSQCommand,Microsoft.ReportingServices.SemanticQueryEngine">

    <Configuration>

        <EnableMathOpCasting>True</EnableMathOpCasting>

        <ReplaceFunctionName>oREPLACE</ReplaceFunctionName>

    </Configuration>

</Extension-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.Teradata.TdSqlModelGenerator,Microsoft.ReportingServices.SemanticQueryEngine"/-->

• Save and close the config file.
• Restart the SSRS service.

The private key does not support the exchange KeySpec.

When using Active Directory Federation Services (ADFS) for claims-based authentication with Dynamics CRM, one of the requirements is a SSL certificate. If a new certificate has to be procured, it is imperative to make sure the certificate request (CSR) is being generated with the correct KeySpec, if required.

There are two options for KeySpec:

• “1” or “XCN_AT_KEYEXCHANGE” 
• “2” or “XCN_AT_SIGNATURE”

With ADFS, the certificate needs to support key exchange so the required KeySpec is "1" or "XCN_AT_KEYEXCHANGE". If the other option is chosen, you may end up with failures when trying to log in to CRM and will see errors in the event viewer on the ADFS server similar to below.

Exception type: NotSupportedException 

    Exception message: The private key does not support the exchange KeySpec.

   at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.DecryptKey(String algorithm, Byte[] keyData)

   at System.IdentityModel.Selectors.SecurityTokenResolver.SimpleTokenResolver.TryResolveSecurityKeyCore(SecurityKeyIdentifierClause keyIdentifierClause, SecurityKey& key)

While the certificate will still appear valid using "XCN_AT_SIGNATURE" and will secure the webpages, authentication will not succeed with ADFS so the certificate will need to be regenerated. 

CRM Reporting Issue - The system cannot contact a domain controller to service the authentication request

Had a case come into my queue indicating a client’s reports in their production CRM were not working so I logged in and began taking a look. At first glance, I was seeing the generic “Reporting Error – The report cannot be displayed. (rsProcessingAborted)” that we see for 90% of reporting issues so I proceeded to the SSRS logs where I was greeted with the following error:

System.Runtime.InteropServices.COMException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350) ---> Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350)

Interesting to say the least and something I had never seen before. My inclination was that a domain controller was down or somehow the connection to AD was severed (but then again, how was I able to log on the server). I ran a set command to check the logon sever and was able to ping it fine, along with all of the other DCs. Next stop was at the DNS/Gateway settings on the NIC – compared everything to another SQL/SSRS server that was working fine. Again, nothing seemed out of place.
Off to the handy dandy event viewer! Immediately I noticed a flood of errors indicating that the automatic backups and log shipping were failing. Amidst those errors I found a familiar message:

Whatever was causing the reporting issue was also causing the backups to fail. I made my way through the events until I reached the beginning of the flood and there it was, in all its glory:

BackupDiskFile::CreateMedia: Backup device '\\goxsaXXXX\D$\MSSQL11.XXXX\MSSQL\Backup\Log_Shipping\XXXX_20160524043001.trn' failed to create. Operating system error 1331(This user can't sign in because this account is currently disabled.).

Easy enough – the SQLAdmin account running SSRS (and the other SQL services) cannot authenticate because it’s disabled.

 

Had the customer check AD and sure enough, the account was disabled. Once enabled, reports began working again. 

In all, this was a very strange error message for the actual issue. It could have really led us on a wild goose chase into the networking side of things. Probably saved 2-3 hours by reading through the error logs and being thorough in investigation before going back to the customer with the wrong answer. Moral of the story: do your homework.

Use SSL for Outgoing Connection / Incoming Connection Radio Buttons Greyed Out for CRM Server Side Sync POP3-SMTP Profile

If you need to create a Server Side Sync profile for your POP3 and/or SMTP servers without using SSL for your on-premise CRM 2013, 2015, or 2016 environments you may notice that you are unable to change the option to not use SSL. The radio buttons are simply greyed out and stuck set to “Yes”. This can be problematic if you want to run over basic HTTP.

Luckily, these buttons can be enabled via the following PowerShell commands run on the CRM server:

Add-PSSnapin Microsoft.Crm.Powershell
$setting=Get-CrmSetting ServerSideSyncEmailSettings
$setting.AllowNonSSLEmail=$True
Set-CrmSetting $setting
Get-CrmSetting -SettingType ServerSideSyncEmailsettings
Exit

After the commands are complete, refresh your browser and the buttons should now be active.

A Server Certificate Could Not Be Validated for URL - SQL Server Reporting Services 2012

If you ever get the following error during the system checks of the CRM installer, check to see if you can even access the Report Manager URL from the SSRS server:

A Server Certificate Could Not Be Validated for URL: http://reportservername/ReportServer 

When checking the Report Manager URL (http://reportservername/Reports) using HTTP - not HTTPS - you may see the following error:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

You will probably also get a certificate error accessing the Report Service URL (http://reportservername/ReportServer) which will end up resulting in a 404 error screen if you proceed. So why is SSRS looking for SSL/TLS when using HTTP?

The answer is rather simple – it’s configured to do so. Here is how to change that:

1. Go to the Report Server directory. Typically it is {Drive}:\MSRS11\Reporting Services\ReportServer

2. Open the rsreportserver.xml file in notepad or other editing software.

3. In this configuration file, look for or run a find for SecureConnectionLevel. It should be about 12 lines from the top. You will likely see this key set to a value of 2.

4. Change the value to 0 so that the line reads:
<Add Key=”SecureConnectionLevel” Value=”0” />

5. Save and close the file. Restart the SQL Reporting Services service and try accessing the Report Manager URL once more. You should now get the site as expected over HTTP.

6. Go back to your CRM server and re-run the installation wizard.

Of course, the alternative to going this route is to actually provide SSRS with an SSL certificate and access it over HTTPS only but some may not want the added hassle or cost.

CRM and IIS Machine Keys

When you have multiple CRM web (front-end) servers splitting the load, requests can bounce between servers or they can be set to remain on the same server through the NLB/Firewall settings. The latter option is commonly referred to as “Sticky Sessions” or “Persistence”.

If requests bounce between servers (whether by design or by fault), you may notice issues when accessing certain functionality within CRM. Often it will be accompanied by the following error message in the event viewer:

Event code: 3012 
Event message: An error occurred processing a web or script resource request. The resource identifier failed to decrypt. 

Exception information: 
    Exception type: HttpException 
    Exception message: Unable to validate data.
   at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType, Boolean signData)
   at System.Web.UI.Page.DecryptString(String s, Purpose purpose)
   at System.Web.Handlers.ScriptResourceHandler.ProcessRequest(HttpContextBase context, VirtualFileReader fileReader, Action`2 logAction, Boolean validatePath)

Basically what is happening is one server encrypted (validated) some of the traffic with an automatically generated key and another server received it but cannot decrypt it because it has different keys. These keys are known as “machine keys” and are found within IIS. By default, these keys are all set to generate automatically and no two are the same. When you are using multiple web servers, the machine keys need to be set statically and shared amongst them. Here is how to do this, starting on the first web server:

1. Open IIS, select the Microsoft Dynamics CRM Website, and double-click “Machine Key”
   
   
2. Uncheck the “Automatically generate at runtime” and “Generate a unique key for each application” boxes under both “Validation Key” and “Decryption Key”.
   
   
3. Click “Generate Keys” on the right side of the screen.
   
   
4. This will create new, random keys.
   
   
5. Click apply.
   
   
6. From here, repeat steps 1 and 2 on the remaining web servers. Instead of generating new keys as in step 3, copy and paste the keys generated from earlier into the remaining servers so that all servers have the same set of keys.
7. Once keys on all servers are set and applied, reset IIS on all the boxes. That should do it.

CRM 2016 Update 0.1 Bug with ADFS for Server 2012 R2

Following a successful upgrade to CRM 2016 and installation of the 0.1 Update, users could no longer authenticate against ADFS using the “internal” URL.

On the ADFS server I was seeing Event ID 364 in the Event Viewer:

Exception details:
Microsoft.IdentityServer.Web.InvalidRequestException: MSIS7042: The same client browser session has made '6' requests in the last '0' seconds. Contact your administrator for details.

In efforts to get this working, I tried completely re-configuring CRM for claims and IFD and even recreated the Relying Party Trusts. All with no luck but apparently, I was not alone on this issue and Microsoft has acknowledged it as a bug with Update 0.1. The following forum thread has more info:

https://social.msdn.microsoft.com/Forums/windowsserver/en-US/9d4040eb-81fa-4144-ae4b-85ca4610aa1d/crm-2016-problem-with-claimsbased-authentication?forum=crm

These are the only real details on this issue currently available:

There were major code changes in Ara UR1 for authentication. The affected code is in Microsoft.Crm.Core.Security.Identity.IdentityExtensions.GetUserPrincipalName(). We are unable to cast from type ClaimsIdentity to a new type CrmIdentity. Therefore, the variable is null, and we cannot retrieve the information.

Please note that only the ‘internal’ URL is affected when IFD is setup. The ‘external’ URL (e.g. https://orgname.domain.com) that uses forms authentication works fine still. I will update this blog post as more information becomes available.

UPDATE: Service Pack 1 for CRM 2016 resolves this issue.

Host Header Configured on Selected Site Does Not Resolve to an IP Address that is Assigned to the Local System.

I was recently performing an in-place upgrade from CRM 2015 to CRM 2016 when during the system checks I received the following warning:

Host Header "notCRM.domain.com" configured on selected site does not resolve to an IP address that is assigned to the local system. This conflict must be reconciled for CRM Server to function properly. 

The fact that it was giving me a host header that did not pertain to the CRM website was odd so I took a look at IIS. What I discovered was that there was another website setup in IIS on the CRM Server. And that site had the site ID of 1.

As I was the original installer of CRM for this deployment, I knew that this site wasn’t here when CRM was installed. CRM originally had the site ID of 1, not this other site. Somebody had been in here messing around and changed the site IDs manually. After asking enough questions, I learned who did this and why. To make a long story short, their site needed to have the ID of 1. CRM, on the other hand, does not care so I rolled with the punches and decided I would work around this rather than cause a fuss.

From what I could deduce about the error message, the CRM installer was being told to look at the IIS website that had an ID of 1 to perform the upgrade on. I decided to switch the CRM site back to the ID of 1. To do this, you must first change the other site’s ID to a number not in use (e.g. 3, in my case). To do this, go into the site’s advanced settings.

After the other site was changed, I reverted the CRM site back to 1 and reset IIS.

The upgrade wizard will need to be restarted for it to pick up the change but once you make it back to the system checks, the warning should be gone.

The upgrade succeeded without issue but we weren’t done just yet. Since the other site needed to have the ID of 1, I needed to revert changes to the site IDs which in itself isn’t complicated. What was not taken into account was a registry key in HKLM\Software\Microsoft\MSCRM called “website”.

This registry key indicates which website directory in IIS CRM is using. Since CRM is using the ID of 2, this key needed to be switched to “LM/W3SVC/2”. In case you have been wondering what the ID controls, it is just an identifier used in the directory name for trace files and log files. It’s nothing critical that will not let CRM operate as normal.

By now you are probably thinking “Why didn’t you change this key rather than switching the site IDs to resolve the warning message during the upgrade?!?!” Well... it didn’t occur to me until afterward I completed the upgrade. Hindsight is 20/20!

The Authentication Endpoint Username Was Not Found On The Configured Secure Token Service and Port 808

While not recommended, circumstances sometime present the need to install CRM and ADFS on the same server. If this is done on Windows Server 2012 R2, you will likely run into issues connecting anything through the discovery service such as the Outlook Add-In, Email Router, and Plugin Registration Tool. The error will read:

The authentication endpoint Username was not found on the configured Secure Token Service.

If taken literally, one would be tempted to go to the ADFS Configuration and enable the /adfs/services/trust/13/username endpoint as shown below:

DO NOT DO THIS! On the outside chance that it does actually fix the immediate issue it can cause further authentication problems and is NOT recommended – despite what some other blogs/forums may tell you.

Further investigation should eventually lead you to this error in the ADFS event viewer logs:

System.ServiceModel.AddressAlreadyInUseException: There is already a listener on IP endpoint 0.0.0.0:808. This could happen if there is another application already listening on this endpoint or if you have multiple service endpoints in your service host with the same IP endpoint but with incompatible binding configurations. —> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted

With ADFS for Server 2012 R2, net.tcp port 808 is utilized for a function of the federation service. You may recall that port 808 is also used by CRM for its sandbox service so it’s easy to see the problem here. Port conflict! Luckily, the solution is rather simple. Just open a PowerShell prompt on the server and enter the following:

Set-ADFSProperties -nettcpport 809

(Note: In the above the command, 809 is just an example of a port that can be used and is not required. If something else is running over port 809 on your server, substitute it for some other number.)

After the command has been run, restart the ADFS service, IIS, and the email router service (if the email router was having the issue connecting) – to be on the safe side, you could just reboot the whole server. Give it another go and you should be in business! If you are wondering why this hasn’t been an issue with past versions of ADFS, it's because the federation services used to run on ports 1500 and 1501.

Next Button Grayed Out in Internet-Facing Deployment Configuration Wizard

While recently assisting a colleague with setting up IFD, we stumbled upon an interesting bug with CRM 2016 in which we could not advance in the IFD wizard because the “Next” button was grayed out and disabled. No matter what we typed in the field, it would not enable the button. Tried an IISReset, restarting CRM services, and even a reboot but they all failed to help.

Realizing this had to be an issue within MSCRM_CONFIG database, we took a look at the tables that hold the IFD properties that we input in these wizards. What we found was that somehow the NVarCharColumn value of ExternalRelyingPartyPassiveIdentifier in the FederationProviderProperties table was set to “uri:ifdMicrosoftCRM”. Typically we would see the Auth URL in this field so we updated it manually to “https://auth.domain.com” in SQL. Make sure to include the “https://” - even though we do not enter this in the field during the configuration wizard, it gets appended automatically and set in the database. Issue an IISRESET on the CRM server and then re-launch the CRM Deployment Manager.

Please note that editing the database manually via SQL is not supported by Microsoft and you are doing so at your own risk. Always take backups and use caution.

CertificateDuration of ADFS Token Certificates

At the end of a past blog post on my page (http://blog.gagepennisi.com/2016/01/understanding-adfs-token-signing-and.html), I touched on the subject of extending the certificate validity period from the default of one year. Recall that when these certificates expire or rollover automatically, CRM becomes inaccessible so reducing the frequency of how soon these expire will reduce the downtime associated. Let’s dig in:

1. Open an administrative PowerShell prompt on the ADFS server
2. If using Server 2008 R2 (ADFS 2.0), add the ADFS Snap-in:
             Add-PSSnapin Microsoft.ADFS.PowerShell
3. Set the CertificateDuration property (Note: ‘1095’ below is an example and represents the number of days desired. 1,095 days = 3 years):
             Set-AdfsProperties -CertificateDuration 1095
4. Create new token certificates with the new duration:
             Update-AdfsCertificate –Urgent
5. Update the ADFS metadata stored in the CRM databases by running through the Claims-based Authentication and IFD Wizards in the CRM Deployment Manager.
6. Issue an IISRESET on CRM Server(s).

CRM and ADFS should both now be working with your extended-duration token certificates!

Note: If you have just installed ADFS, and not yet set CRM up for Claims-based Authentication/IFD, you do not need to perform steps 5 and 6.

CRM Database Log Growth Issue

A customer of ours had come to us facing a rather interesting issue. Every night around 1:00AM their CRM database log file would grow to 31GB and cause the log drive to fill up. When users would log into CRM in the morning, they would receive SQL errors stating that their transactions could not be completed because of this.

Given that this issue occurred on a regular schedule, I determined that the issue had to be attributed to some sort of automated job. Figuring the first place to check would be the out-of-the-box CRM Maintenance Jobs, I downloaded the Job Editor from Codeplex (https://crmjobeditor.codeplex.com/ - Very useful tool that everyone should be using) and got to investigating.

Right off the bat, I saw the error on the Deletion Service:

Deletion Service encountered an internal error: System.Data.SqlClient.SqlException (0x80131904): Invalid object name 'SubscriptionTrackingDeletedObject'

From here I moved to SQL and queried the SubscriptionTrackingDeletedObject table of the CRM database. What I found was astounding – the table contained 137 MILLION records. Basically after seeing this I knew that the job had to just be timing out – we tested my theory by running the job manually and immediately saw the log file grow to the expected 31GB.

It was decided to clean this table up manually via truncating it. Before you say “Oh no! Don’t delete records via SQL directly!” let’s explain what this table actually is. When records are deleted from CRM, there are also records that get inserted into this SubscriptionTrackingDeletedObject table.  This table gives the Deletion Service Job ObjectIDs that have been removed so that further cleanup can be performed asynchronously. So essentially, it is just a table of deleted records which gives the Deletion Service knowledge to clean up some other areas of CRM (e.g. POA records, duplicate detection records, etc...) if necessary. Once cleaned up, the records from the table are removed. We understood this and decided the need to clean this table outweighed having the other areas of CRM cleaned up (as you will learn later, this wasn’t a concern for us because of how the records got in there). Please note I cannot condone the practice of editing SQL manually without full knowledge of the possible repercussions. Always consult with Microsoft support if in doubt and remember what works in one scenario, may not work in all.

After the table was truncated, CRM was tested and the deletion service job was run manually – this time not failing with the error above and the log did not grow to 31GB. Before calling this case closed, I still needed to understand what caused this problem in the first place. What could have possibly created so many records in such a short period of time? Luckily, I had the right people involved on the customer’s end and we were able to determine that there had been a malfunctioning Scribe job that was running unnoticed for some time. The job was bulk creating and deleting 50,000 records at a time within CRM but it had since been fixed. Case closed.

Why You Should Have TCP Port 80 Open Outbound On Your ADFS Servers

Active Directory Federation Services (ADFS) performs a lot of tasks when it comes to authenticating users into CRM securely. One of those tasks in particular is a certification revocation check to validate that the certificates being used are still valid. ADFS completes this process by reaching out to certification revocation lists (CRLs) over TCP port 80 – basic HTTP communication.

What we’ve seen is that businesses will want to lock down their ADFS servers just to be on the “safe side” and that includes closing TCP Port 80 outbound (e.g. no internet access). If left in its default state, ADFS will break and cause authentication to fail as it knows that it is supposed to check the CLRs to validate the certificate before issuing a token to allow a user into CRM. If it cannot do this, it will not issue a token. You may see an error similar to the following in the ADFS event viewer logs after a failed authentication attempt:

Event ID: 364
Microsoft.IdentityServer.AuthenticationFailedException: MSIS3014: The encryption certificate of the relying party trust 'https://crm.domain.com/' identified by thumbprint '01DEDF6E6F532BF7357457EBEC31DA82SFDA1234' is not valid. It might indicate that the certificate has been revoked, has expired, or that the certificate chain is not trusted. ---> Microsoft.IdentityServer.Service.SecurityTokenService.RevocationValidationException: MSIS3014: The encryption certificate of the relying party trust 'https://crm.domain.com/' identified by thumbprint '01DEDF6E6F532BF7357457EBEC31DA82SFDA1234' is not valid. It might indicate that the certificate has been revoked, has expired, or that the certificate chain is not trusted.

So what are your options?

1. Have your networking team open TCP 80 outbound on the ADFS server(s). This would also apply to all ADFS Proxies or WAP servers. While opening a port might seem less secure at face value it would actually be the opposite as ADFS is able to validate the certificates being used. 
2. The less preferred, but still acceptable, method would be turning off the Certificate Revocation Check of ADFS. The check is controlled individually for each relying party in ADFS so it would need to be turned off for all one by one. To do this open an admin PowerShell prompt and issue the following command:

                             Set-ADFSRelyingPartyTrust  -TargetName <relyingpartytrustName>  

                             -EncryptionCertificateRevocationCheck None

CRM Organization Import Issue and SSRS MaxRequestLength

While importing an organization to CRM 2011 I came across an error during the import wizard process which was causing the import to fail:

14:54:33|   Info| PublishReportsFromDatabase: Creating report in Reporting Services.  ReportId: 9f973403-bc84-e111-88bc-0050569e0001, Name: INVOICE PAYMENTS
14:54:34|  Error| Error while updating organization information: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Crm.Reporting.RuntimeReportServer.UploadReport(String path, Byte[] reportDefinition, String name, String description, Boolean overwriteExistingReport)
   at Microsoft.Crm.Reporting.RuntimeReportServer.UploadReport(SRSReport report, String reportNameOnSrs, String name, String description, Boolean isSharedReport, Boolean overwriteExistingReport)
   at Microsoft.Crm.Setup.Server.Utility.ReportsUtility.OrganizationPublishReportsScaleGroup(IDbCommand command, Uri reportingUrl, String orgUniqueName, Boolean ignoreCustomReportsFailure, Boolean publishOnlyCustomReports)
   at Microsoft.Crm.Tools.Admin.DBImportHelper.RePublishReports(IDbCommand command, Guid organizationId, String organizationUniqueName, Uri reportUrl)
   at Microsoft.Crm.Tools.Admin.ImportOrganizationInstaller.UpdateOrganizationInfo(Guid organizationId, OrganizationGroupsInfo organizationInfo, String organizationFriendlyName, String organizationUniqueName, Uri reportServerUrl, Int32 PercentUpdateOrganization, ICollection`1 users)

This was not an upgrade of any sort - just simply a CRM migration to a new environment so there were no versioning differences from a CRM perspective. While the error above is lacking much detail, it did give us enough to begin troubleshooting. I could clearly see that this issue was occurring with a report titled “Invoice Payments” but knew nothing else. At this point, I enlisted the help of a colleague (The amazing Lauren Hudgins) who is more familiar with report writing to see if there was anything peculiar about this report. After obtaining a copy of the report’s RDL file, we weren't able to notice anything wrong in particular with the way the report was written but did think it was rather large for an RDL file – nearly 5MB – there were quite a few embedded images.

We decided to attempt uploading the report directly into SSRS and were met with a much more helpful error message - “Maximum request length exceeded”. What this was telling us rang a bell with what we noticed earlier regarding the RDL file size. By default, SSRS has a limit on the report file size that it will allow to be imported. This limit is 4MB but can be increased by doing the following:

Open the web.config of the Report Manager (%\Program Files\Microsoft SQL Server\MSSQL.X\Reporting Services\ReportManager) and find the “executionTimeout” setting. It should look something like this:
  <executionTimeout = “9000” /> 

On this line, add the maxRequestLength attribute with the value (in KB) needed to upload the report. This value is not in here by default. It should now look like this (this shows a 10MB limit):
       <executionTimeout = “9000” maxRequestLength=”10240″ />

Save the file and then repeat these steps in the web.config of the Report Server (%\Program Files\Microsoft SQL Server\MSSQL.X\Reporting Services\ReportServer). Once both files have been modified and saved, restart the SSRS service.

Following this change, we were able to upload the RDL to SSRS directly to verify that the change worked and then subsequently were able to complete the organization import for CRM without issue. Big thanks to Lauren for her assistance on this one.

Understanding the ADFS Token Signing and Decrypting Certificates Rollover Process

Active Directory Federation Services (ADFS) creates and manages the two certificates used for the tokens issued. These are the Token-signing and Token-decrypting certificates. By default, these certificates are valid for one year from their creation and around the one-year mark, they will renew themselves automatically via the Auto Certificate Rollover feature in ADFS. Once this happens, CRM can no longer properly authenticate users as it still holds the old certificates’ metadata in the database. This is easily resolved by rerunning the “Configure Claims Based Authentication” and “Configure Internet Facing Deployment” wizards from the CRM Deployment Manager and then issuing an IISRESET on the CRM server(s). More details and resolution can be found in this KB: http://support.microsoft.com/kb/2686840

While most CRM administrators understand that aspect, there are a number of settings and configurations that lead up to this issue that are less well-known to most. One of the biggest complexities is understanding EXACTLY when CRM will be going down because of the Auto Certificate Rollover and how to avoid it. We will be going through that today.

We will start with the process that ADFS goes through for certificate renewal:

1. ADFS determines that its certificates will be expiring soon.
2. ADFS creates new certificates and sets them as secondary certificates.
3. ADFS updates the new certificates to primary certificates.

There are a number of settings for ADFS only accessible via PowerShell that control the Auto Certificate Rollover options and properties for the process above. To access these, open an administrative PowerShell prompt and execute the following (Note that if you are using ADFS 2.0, you will need to add the ADFS PowerShell Snap-In first by executing Add-PSSnapin Microsoft.ADFS.PowerShell):

Get-ADFSProperties

This will display a listing of the deployment properties for ADFS, including the properties around the certificates and rollover. For our purposes, we will keep our focus on just a handful of these properties:

• AutoCertificateRollover
• CertificateDuration
• CertificateGenerationThreshold
• CertificatePromotionThreshold
• CertificateRolloverInterval

So what do all of these values mean? Below are the same steps provided earlier but now account for the values in the table above.

1. ADFS determines that its certificates will be expiring within 20 days.
2. ADFS creates new certificates valid for 365 days and sets them as secondary certificates.
3. After 5 days’ time the Certificate Management Cycle kicks off and ADFS updates the new certificates to primary certificates.

As you can see, knowing these values can greatly help in planning for certificate rollover. Here is an example:

In the screenshot below, we can see our primary certificates expire on 2/12/2015 and ADFS has already created new certificates to rollover. The new (secondary) certificates were created 20 days prior to the expiration of the primary certificates (1/23/2015). On 1/28/2015, 5 days after the creation of the new certificates, ADFS will change them to primaries.

Great. So what now? BEAT ADFS TO THE PUNCH!

In the above example, you know your deadline is 1/28/2015. Rather than sitting back and waiting until CRM goes down, plan a short outage afterhours and roll the certificates over manually! You can force ADFS to generate new certificates and promote them to primaries immediately using the following command in PowerShell:

Update-ADFSCertificate –Urgent

Once the new certificates are in place in ADFS, re-run the Claims and IFD Wizards in the CRM Deployment Manager to update the metadata and issue an IISReset on the CRM server(s). Voila! Happy CRM users!

Of course, given the properties we have at our disposal to modify there is much more you can do to create a better life for yourself. For example, set the CertificateDuration to 1095 days (three years) rather than just 365 (one year) so this is not as frequent of an issue. Another idea would be to set the CertificateGenerationThreshold lower so the actual rollover date is closer to the true expiration of the certificate. Or just turn off AutoCertificateRollover altogether, set a reminder, and take care of it all manually before expiration!

CRM 2016 New Form Rendering Engine

With the release of CRM 2016 (2015 Update 1 for CRM Online), comes a new form rendering engine built to provide better performance of form loads. The two main changes are focused around loading process of the form and the handling of the cache.

However, while the new rendering engine was built to help with performance you may actually notice the opposite taking place. In heavily customized environment, you might experience long stalls during form loads with messages reading “requesting data from CRM” and/or “loading business logic”.

If you are currently being plagued by these messages, it may be a good idea to turn the new rendering engine off. To do this, simply go to: Administration -> System Setting, scroll all the way down and you will see the “Use legacy form rendering” option. Turning it to “yes” will disable the new engine.

While this will help in the short term, it is advised to figure out what on your form is causing conflict with the new engine so that it can be fixed and the new engine can be turned back on for even better performance.

ADFS Logon Page Loop Issue

I recently setup CRM 2015 and ADFS 3.0 (the version that comes on Server 2012 R2, aka ADFS 2.2) on new servers for a customer. Everything configured fine and initial tests proved successful when logging in with the domain admin account used to set everything up. However, I quickly found an issue when other users tried to access CRM via the external URL. At the ADFS login page, a user would enter his or her credentials as usual and try to login but rather than giving a 302 redirect back to CRM for access, it redirected back to the ADFS login page. This presented no errors on screen or in the CRM event viewer – it was as if I never tried logging in. If I purposefully entered invalid credentials, it provided the error regarding incorrect user ID or password so I knew authentication against AD was taking place successfully and that something was wrong with the passing of the token.  Also of note was the fact that despite the external URL not working, all users were able to access CRM just fine using the internal (pass-through auth) URL.

Nothing appeared in the ADFS Admin event viewer logs but upon closer inspection, the Security log in the event viewer on the ADFS server was loading up with Audit Failure notifications – Event ID 4625. The failure reason indicated “Unknown user name or bad password” for the ADFS service account. As any logical person would assume, I figured the account was locked out, the password expired, or I entered invalid credentials during setup. Unfortunately, upon resetting the password in Active Directory, the audit failures persisted.

After some digging around, I uncovered that adding the ADFS service account to the Windows Authorization Access Group in Active Directory was the resolution. The members of this group have access to the computed tokenGroupsGlobalAndUniversal attribute on User objects. I added the account to the group, restarted the ADFS service, and all users were then able to access CRM via the external URL as expected!

UPDATE (1/27/16) - While adding the ADFS service account to the Windows Authorization Access Group fixed the issue for most users accessing CRM, it managed to rear its ugly head for a handful of users again. The fix this time was similar - I added the account to the Pre-Windows 2000 Compatible Access Group. Users in this group have read access over all users in the domain. Typically, the 'Everyone' group is present in here by default so this is not usually an issue but this customer in particular removed it. So to conclude, make sure group membership is set on both of these groups if you are experiencing the loop issue.