Yet Another ADFS Looping Issue

I recently applied the Dynamics 365 Update to a CRM 2016 Service Pack 1 environment that was setup for IFD and ran into some unexpected behavior. This environment had about 15 organization setup in it. When trying to log on using the external URLs (e.g. orgname.domain.com) access worked without issue but using the internal URL for any organization (e.g. crm.domain.com/orgname) CRM and ADFS would run in a continuous loop. Sometimes it would error out, sometimes it would just keep looping forever.

Started off by checking event viewers on both the CRM and ADFS server. I noticed that on the times it did error out after a few loops instead of looping nonstop, it resulted in an event log similar to one I saw before caused by a bug with the 0.1 Update for CRM 2016 (http://blog.gagepennisi.com/2016/04/crm-2016-update-01-bug-with-adfs-for.html):

MSIS7042: The same client browser session has made '6' requests in the last '17' seconds. Contact your administrator for details.

So immediately I thought “Great, here we go again.” There was also nothing in the CRM event viewer.

Hours of trying different different troubleshooting tactics for similar situations (including this one with the same looping behavior but with the external URL - http://blog.gagepennisi.com/2016/01/adfs-logon-page-loop-issue.html) yielded nothing. Rebuilt the Relying Party Trusts, reconfigured IFD, created a self-signed certificate to verify it wasn’t a certificate issue, etc…

Instead of keeping on with the guess and check, I realized I needed to get more information around the problem so I started with a Fiddler trace – great web debugging tool, if you haven’t heard of it (http://www.telerik.com/fiddler). Unfortunately, in this case all it did was confirm the behavior I was seeing in the browser which was a constant redirect between D365 and ADFS.

After that, I decided to run a platform trace on the CRM server to see if CRM would give me any insight but nothing really stood out to me. Almost at my wit’s end, I decided to rope in my colleague Dan Francis to get another set of eyes on it. After some review of the platform trace he noticed one little line that seemed a bit odd:

>Multi-org sharable cache loading system and non-system metadata with build number 7.0.0.3543 and language 1033

CRM 2016 is version 8.0 and D365 is 8.2 so the fact that we were seeing any refernce to 7.0.X.XXXX (CRM 2015) was not inline with what was to be expected. We checked the Deployment Manger and realized there was an old, disabled organization from CRM 2015 that was not upgraded (because it was disabled at the time of the original CRM 2016 upgrade). After some deliberation, we decided to remove that organization from the deployment and boom! The internal URL began working as expected!

Basically what we were able to surmise from this is that when the internal URL is being used, all organizations – whether enabled or disabled – are checked for versioning and the lowest build number is used to load this “multi-org sharable cache” and apparently D365 doesn’t play nicely with older versions hanging around. Moral of the story – update/upgrade all of your orgs or just remove them.

Big thanks to Dan the Man!

CRM and TLS 1.0

While performing a routine installation of CRM 2016, I stumbled upon a new error during the system checks at the end of CRM installation wizard. The error was for the SQL Server check and read:

Could not connect to the following SQL Server: 'XXXXXX-SQL02'. Verify that the server is up and running and that you have SQL Server administrative credentials. [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Having never seen this particular error, I resorted to my friend Google who led me down a wormhole or possible issues and resolutions related to certificates and protocols. Turns out that this was indeed related to a security protocol – TLS 1.0 to be exact. It was disabled on the SQL Server as part of our new security template for new servers. So with the help of one of Cloud Application Engineers, Spencer Ashworth, we enabled TLS 1.0. This is done through a simple registry change.

Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server\ and locate the DWORD named Enabled. Set the value to 1, close the registry and then reboot the server.

Following the reboot and re-run of the CRM installation wizard, the system checks all passed and CRM was able to be successfully installed. Big thanks to Spencer who was the one that recalled CRM’s requirement of TLS 1.0 and knew the resolution!

CRM Diagnostics Page

When troubleshooting CRM performance issues everyone typically wants to point fingers at the servers – whether it be SQL, CRM, or even SSRS – but a commonly overlooked factor is the network. Not known to everyone is a tool built into CRM called the CRM Diagnostics Page. It is basically a very simple way to check network performance. To access the CRM Diagnostics Page simply go to:

http(s)://<YourCRMServerURL>/tools/diagnostics/diag.aspx

This will land you at a page that looks like the following:

On this page, you can clearly see the series of tests available but what are they?

1. Latency Test

This calculates the average time taken to download a small text file 20 times. The file downloaded from the CRM server is /_static/Tools/Diagnostics/smallfile.txt. CRM is designed to work best with latency under 150 milliseconds. Latency can be a huge factor in performance for offices far away the CRM server’s location.

2. Bandwidth Test

The bandwidth test downloads image files and records the speeds. The recorded speeds are averaged together and provided in the results summary. Bandwidth should ideally be higher than 50 KB/sec. This test, along with the Latency Test, are the most helpful on this page.

3. Browser Info

This is a JavaScript pull of the local browser details such as browser name, version, cookie status, operating system and the user-agent string.

4. IP Address

This is the IP address of the client computer as known to the server. The IP address is server-side dynamic and represents the IP address which was used to contact the server.

5. JavaScript tests

The series of JavaScript tests times loops and returns their execution time. They are essentially just memory/CPU stress tests on the client machine.

6. Organization Info

Provides general server information- organization name, time on server and client and URL.

Bonus! 

This is also a neat little secret: http(s)://<YourCRMServerURL>/home/home_debug.aspx.

It has some of the same information as the diagnostics page - Not overly helpful but can possibly come in handy troubleshooting. I find it just to be a nice, concise page for deployment information to help with documentation.

HTTP to HTTPS Redirect for CRM using Claims-Based Authentication and IFD

In some cases, our customers would like HTTP to HTTPS redirection setup for CRM when using Claims-Based Authentication and IFD. There are a number of articles and blogs out on the internet that use various methods to accomplish this (e.g. URL Rewriting) but I find that a lot are overly complicated if all you are looking to do is have the redirect functioning internally (you do not want port 80 open on your firewall anyways). Below is what I perceive to be the easiest method to achieving this – using the HTTP Redirection capabilities of IIS:

1. First we will need to install HTTP Redirection on the CRM Web Server if it is not installed already. In Server Manager, go to Add Roles and Features and choose to add the HTTP Redirection feature under the Web Server role.

2. Create two folders in a directory that you know will not be deleted. Name these folders something along the lines of “CRM Internal Redirect” and “CRM External Redirect”. Folder names and paths aren’t too important but in my example I just put them in the inetpub directory.

3. Next, go into IIS. Make sure to remove the HTTP (port 80) binding from the CRM site before proceeding with the next steps.

4. Since two URLs can be used to be accessed CRM when it is setup for Claims and IFD, we will need to create the redirects for both URLs. Begin by creating a new website in IIS for the Internal URL.

• Site Name: Something descriptive such as “CRM Internal Redirect”
• Physical Path: Browse to the CRM Internal Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your internal CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, open the CRM Deployment Manager and go to properties. Select the Web Addresses tab and you should see your internal URL.

5. Just like step 4, create another new site in IIS – this time for the external URL:

• Site Name: Something descriptive such as “CRM External Redirect”
• Physical Path: Browse to the CRM External Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your exteransl CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, this is the unique name of your CRM organization followed by the domain (e.g. orgname.domain.com).

6. You should now have IIS looking similar to this:

7. Click on your Internal Redirect site and open the HTTP Redirect module:

8. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your internal URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

9. Now do the same for the External redirect site. Click on your External Redirect site and open the HTTP Redirect module. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your external URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

That’s it. You should now be able to enter either of the CRM URLs with HTTP into a browser and have it redirected to HTTPS appropriately.

Insufficient Winsock Resources Available to Complete Socket Connection Initiation

Seemingly out of the blue, one of our customers could no longer log in to their CRM. As it appeared to be an authentication issue, I began troubleshooting on their ADFS server. Upon opening the ADFS Management Console, I was greeted with the following error message:

In the event viewer were a number of errors similar to this:

Microsoft.IdentityServer.Configuration.ReadServiceConfigFailedException: MSIS2001: Configuration service URL is not configured. ---> System.InsufficientMemoryException: Insufficient winsock resources available to complete socket connection initiation. ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 127.0.0.1:1500

Well, this was a new one for me but at least the error message actually gave a description instead of something generic. Logic dictates that if there isn’t enough of something, give it more. In this case, it appeared to me that ADFS was complaining about not having enough ephemeral ports so I figured I would increase the number available.

1. Start Registry Editor.
2. Locate the following subkey in the registry, and then click Parameters:  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. On the Edit menu, click New, and then add the following registry entry:
   Value Type: DWORD
   Value Name: MaxUserPort
   Value Data: 65534 (decimal)
4. Reboot server

After completing the above, ADFS was able to open the connection to its database and users were able to log back in to CRM. Now, it may be worthwhile to investigate exactly why ports suddenly became busy… perhaps there is a socket leak or new applications were installed… but this should at least get you up and running in the meantime.

Calling LoadLibraryEx on ISAPI filter DefaultAddonFilter.dll failed

I recently had an interesting issue with a client’s CRM 2011 to 2016 go-live. Following all of my upgrade work and initial testing, I typically like to reboot the CRM and SQL servers and then test again. In this particular instance, when the servers came back online I tested CRM again only to receive a lovely error message from Internet Explorer:

HTTP Error 500.0 – Internal Server Error

Calling LoadLibraryEx on ISAPI filter C:\Program Files\Microsoft Dynamics CRM\Server\bin\DefaultAddonFilter.dll failed.

After a fair amount of digging I noticed that the Visual C++ 2013 Redistributable (64-bit) was not present in the add/remove programs list on the CRM server. It seemingly uninstalled itself during the reboot because otherwise I would have had this issue pre-reboot. So, I went out and grabbed the installer package for Visual C++ 2013 Redistributable (64-bit) from MS (http://go.microsoft.com/fwlink/p/?LinkId=402059) and reinstalled it. After that and an IISReset, CRM began functioning as normal and everything retested everything successfully.

Unfortunately, I am at a loss as to why this happened – perhaps something to do with server patching. Also of note is that the redistributable did not remove itself from the SQL server.

Reports in CRM not working - The type initializer for 'Microsoft.Crm.LocatorService' threw an exception.

We had a customer running into an issue while trying to run reports from CRM. Of course, the error in the CRM UI was the usual rsProcessingAborted error - Cannot create a connection to Data Source. So as I always need to do, I dug into the SSRS logs. The errors I could extract from there were:

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The type initializer for 'Microsoft.Crm.LocatorService' threw an exception. ---> 

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: Cannot load Counter Name data because an invalid index 'W3SVC_W3WP' was read from the registry.

From what I could surmise, there was something wrong with the performance counters on the server(s). I did find that Microsoft does have a KB for this error but it is in reference to update rollups on the CRM Outlook Client. Luckily for me, the fix turned out to be similar!

1. On the CRM Server, open an administrative command prompt.
2. Type “lodctr /r” and press enter.
3. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
4. Restart CRM Services and run an IISReset
5. On the SSRS Server, open an administrative command prompt.
6. Type “lodctr /r” and press enter.
7. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
8. Restart SSRS Services.

After following the above process, reports in CRM began working again. The commands run rebuild the performance counters.