ADFS Cache Issues

From time-to-time, employees may leave a company but end up coming back. With all that comes user management, accounts, access, permissions, etc…. While that is all easy enough from an Active Directory Standpoint, this can cause some complications with access into CRM/Dynamics 365 when using ADFS for claims-based authentication. Particularly if the user’s original AD account was deleted and then recreated with the same username.

After the account is recreated, you may learn that there isn’t any issue re-enabling the user in CRM or Dynamics 365 but that the user cannot login via ADFS. There is also a similar scenario when companies migrate domains and keep the same user names, only changing the domain. Some behavior you may notice are 404 errors, wrong username showing in event viewer logs, and general ADFS errors. While I haven’t been able to pinpoint why this happens sometimes and not others, what I do know is the problem lies within ADFS.

Basically what seems to be going on is the ADFS server has a cache of information already built up about the users, including GUIDs, SIDs, UPNs, etc… So when an authentication request comes through it checks for the username in its cache along with any associated records and matches what it can rather than pulling from Active Directory. When it does this it can pull the information from the old, previously-deleted account and then error out.

Anyways, without further ado – here are two methods that I’ve used successfully to get through these situations. Sometimes one works and sometimes the other works – just some more mystery around this situation…

Method 1: Disable the LSA Cache
This method involves disabling the LSA cache by setting it’s maximum size to 0 – essentially making it nonexistent.

1. Log on to the ADFS Server.
2. Open Registry Editor.
3. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.
4. Create a new DWORD value in Lsa named LsaLookupCacheMaxSize and set the value to 0.
5. Close Registry Editor.
6. Have problem user try to log in again.

Whether or not this works, go back in after testing and delete the value you just created. ADFS may suffer from performance issues if it is left in place for a long period of time.

Method 2: Use PSTools to update SID
Going this route requires the download of PsTools – a command-line tool that helps with server administration. In our scenario, it is the PsGetSid command that will be useful as it will force update the SID of the account held in cache from AD.

1. Log on to the ADFS server.
2. Download PSTools from https://docs.microsoft.com/en-us/sysinternals/downloads/psgetsid.
3. Extract out the contents of the zip file into an accessible directory.
4. Run PSGetSid from command prompt using the following syntax: C:\{Directory}\PsGetsid.exe domain\crmuser
5. Have problem user try to log in again.

If neither of those methods works, there may be something else wrong. I would start by verifying that all other users can access CRM/D365 without issue and go from there. My apologies for not having exact error messages or screenshots on this post – it was just an issue I’ve run into in the past a few times and thought it may help someone out regardless as I’ve realized there isn’t much information out there about it.

Connecting Dynamics 365 On-Premise to Power BI

If you're looking to setup the integration between Dynamics 365 On-Premise (using IFD) and Power BI, you may stumble upon this lovely TechNet: https://technet.microsoft.com/en-us/library/dn708055.aspx#PBI_op

All seems great, right? Well, that would be the case if all the steps were accurate...

To start, the PowerShell commands in step 1 to enable OAuth on the Dynamics 365 Server just do not work. The first three will work fine but upon execution of the 4th to actually set the change, it will throw an error. Instead, use this syntax to accomplish the same thing:

Add-PSSnapin Microsoft.Crm.PowerShell
$ClaimsSettings = Get-CrmSetting -SettingType OAuthClaimsSettings
$ClaimsSettings.Enabled = $true
Set-CrmSetting -Setting $ClaimsSettings

Afterwards, run Get-CrmSetting -SettingType OAuthClaimsSettings and you should see enabled set to "True" as shown below.

Moving onward to step 3 in the process where we register the Power BI Desktop OAuth 2.0 client with ADFS. You will notice in the instructions that it states "open a Windows PowerShell window and run the following PowerShell command on the PC where you are running Power BI Desktop". This is simply wrong. The "Add-AdfsClient" command is for... ADFS. Shocking, I know. So ignore the statement about the PC running Power BI and run the command provided on your ADFS server instead. Luckily, this one is correct and works. If you run Get-AdfsClient afterwards, you will see the addition of Power BI.

One last helpful tip - When copying commands from websites (even here), I always recommend pasting into Notepad before bringing them where they need to be executed (e.g. SSMS, PowerShell, etc...). This will get rid of any problematic formatting.

That's really it from a server perspective. Best of luck!

Issue Installing CRM 2016 SRS Data Connector with SQL 2016

In my last post, I mentioned that I had received some Windows Server 2016 boxes to test setup of CRM 2016/Dynamics 365. In addition to them being the 2016 OS, they also came with SQL Server 2016. The installation of CRM went off without a hitch (after working through the Search service problem) and the databases created in SQL as expected.

However, when it came to installation of the SRS Data Connector, things got a little hairy. The installation began but quickly threw this error message:

Error| System.Exception: Action Microsoft.Crm.Setup.SrsDataConnector.AddBindingRedirectForRdlHelper failed. ---> System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Program Files\Microsoft SQL Server\MSRS13.MSSQLSERVER\Reporting Services\ReportManager\web.config'.

To make a long story short, SQL 2016 no longer creates/uses the path specified in the error message and the RTM Data Connector installer does not check the version of SSRS being used so it will just keep trying to use that directory.

Now, I had some previous experience installing the Data Connector with SQL 2016 involved. From that, I recalled that the installer, when asked to check for updates on the first screen, actually went out and got some new install bits. This time around I was not so lucky – it just told me there were no updates available (I even verified that the Windows Firewall was off and that the box was allowing Windows Updates)… it had internet access but if you’re reading this and your SQL server does not have access to the internet, you should keep reading. Regardless, I needed a solution and the internet was not full of suggestions with this being pretty new technology.

I decided to check the Microsoft Update Catalog to see if I could download and manually patch the update in. To my delight, I came across a listing for “Setup Update for Microsoft Dynamics CRM 2016 Reporting Extensions” and this ultimately helped solve my issue. Here is a step-by-step process of what to do if you are facing the same situation:

1. If your initial install failed, go into the server’s control panel and uninstall the data connector.

2. Go to http://www.catalog.update.microsoft.com/

3. Search for “3129794” and you should get back one listing. Click the download button.

4. A new screen will pop up with a ton of .cab files. Find the one named “srs_kb3129794_amd64_1033_b585b78e247dc7611e1e406a6c132cdee112690c.cab” (just look for 1033) and download that. I will admit that I am not sure of the purpose of all these files but the 1033 one did the trick for me. Later on, you will notice that another file we work with is also “1033” so that probably has something to do with it.

5. Once you have the .cab file downloaded to the server, open it up to reveal the patch file. Right-click on it and choose “Extract”.

6. After the extraction, you will have the .msp file available for use. Go to the directory of the SRS Data Connector’s installation files and locate a folder named “Update”. In here, you will see another .msp file named “Srs_KB3121695_Amd64_1033”.

7. Cut and paste that file out of the “Update” folder – just put it anywhere else on the server.

8. Copy and paste the .msp you just downloaded into the now blank “Update” folder - essentially replacing it.

You should go from this:

To this:

9. Re-run the installer as normal and it should complete without issue.


Just to note a few things:

• I initially played a bunch of trial and error with getting the update patched in before I discovered the ability to replace the .msp in the Update folder. I first went down the path of editing the config.xml to include the patch and then called the installer from a command line specifying it to use the config file. This is how I have always patched in updates for the server installer but it does not seem to work the same way for the Data Connector. I also tried loading the patch into the temp files that the installer looks to… also didn’t work.

• If your base installer’s update file has a different number than “1033” at the tail end of it, you may need to match it up with the correct .cab file from the Update Catalog. I did not encounter this scenario so I cannot test or prove out this theory but I think logic dictates… 

• There is another blog out there about applying SP1/2 to the failed installation and then manually editing file structure. I will not discredit that being a possible workaround but personally, I do not like the idea of relying on an update to fix a broken install.

Good luck!

Issue Installing CRM 2016 (Dynamics 365) on Windows Server 2016

I recently got my hands on some Windows Server 2016 boxes and wanted to test out the CRM 2016/Dynamics 365 installation process on them. Preparation began as usual and I kicked off the wizard. Some time after that I was presented with the following error message:

Action.Microsoft.Crm.Setup.Common.InstallWindowsSearchAction failed. The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)

What I quickly learned was that the Windows Search service, while coming pre-installed on Windows Server 2016, is set to disabled by default.

I enabled the service by setting it to automatic startup, started it, and then hit retry on the error diagloue from the CRM installation wizard. The installation continued and finished with no further issues.

I love the easy fixes!

High Availability and Disaster Recovery Options for Dynamics 365 (CRM)

Planning the infrastructure for your Dynamics 365 environment is a critical step in the deployment process, especially when the software is mission critical to business needs and day-to-day operations. In scenarios like these, it is imperative to have high availability and even disaster recovery plans in place. So the question is: What options do you have when it comes to Dynamics 365 on premise? In this article, we will break it down from both an application (Dynamics 365) and database (SQL Server) perspective.

High Availability – Dynamics 365
For most companies, having high availability for their Dynamics 365 deployment is a must and the setup for this is similar to that of other web-based applications. Keep in mind that through this section we are strictly referring to Dynamcis 365 servers with the “Front End” or “Full” role – in other words, the servers that host the website in IIS. Having multiple “Back End”-only servers (those with the Async and Sandbox services) is also a good idea but those will natively load balance themselves if they are in the same deployment.

• Network Load Balancing – Probably the most obvious method for high availability is running multiple Dynamics 365 servers in a load-balancing configuration. This will not only help with carrying the day-to-day load of traffic but also serve as the foundation for a good high availability option. If a server were to fail, just pop it out of the NLB pool. Load balancing comes in a few different flavors because of all the applications out there but mainly it comes down to two options – hardware and software.

• Hardware Load Balancing involves the use of a third-party appliance such as F5 or NetScaler that sits in front of the Dynamics 365 servers to handle which server receives the user requests. While this is the preferred option as it does not create any additional processes for the servers to handle and usually provides a wider array of configuration options (e.g. monitoring, security, filtering, etc…), it can be more difficult and much more costly to implement.

• Software Load Balancing on the other hand is relatively easy to setup and in the case of Windows NLB, is free, as you have already paid for your Windows Server licenses. Critics of software load balancing will contend that it is not “true” high-availability due to the lack of many key features found in hardware load balancing. The fact stands that Windows NLB is just Round Robin DNS so if a server were to go down, requests would still be sent to the faulted server until it is told otherwise.

• As one can probably infer, this is simply a cost vs. ROI decision – which is the case with most IT situations. If the hardware option is being considered, make sure to reach out to your appliance vendor to ensure there are no known issues with it and Dynamics 365. It is also a good idea to inquire about documentation for setup. For more information about load balancing Dynamics 365 in general and the required configurations, check out this TechNet: https://technet.microsoft.com/en-us/library/hh699803.aspx. 

• Backup Server(s) – While this method may be a little more rigid, it is still effective and again, has a couple options. The meat and potatoes of this route is having a server to fall back to in the event that your primary Dynamics 365 server(s) fails and NLB has not been implemented.

• VM Snapshots – If you are reading this, you most likely understand the concept of VM snapshots so we will not dive too heavily into this. The idea is to have automatic snapshots taken automatically on a set interval that the Dynamics 365 server can be reverted to or new server built from in the event of failure. Reverting the existing server is the easier option but also the more risky of the two. Spinning up a new server is safer but more involved – and requires the shutdown of the existing server as you will need to use the same host name and IP address.

• Laying in wait – We have had some customers opt for this route as the return to uptime is a little quicker and/or VMs were not being used. Dynamics 365 deployments can obviously be comprised of multiple servers – but not all need to be active or enabled. The concept here is to build a secondary Dynamics 365 server as part of the deployment but leave it disabled until it is needed. If the primary server were to fail, all that would be required is enabling of the server via the Deployment Manager and changes to DNS (and firewall rules, if applicable).

High Availability – SQL Server

Now that we have covered high availability from an application perspective, what about the database? Don’t worry! Microsoft has that covered too!

• SQL Server Failover Cluster – There is nothing new about this concept. Two SQL Servers are setup to use a shared disk, one server goes down and the other picks up the load. When installing Dynamics 365 just make sure to point connection strings to the cluster name and not a single node and you should not have any issues but be sure to test failover prior to going live. Just to note - although you can install Dynamics 365 to a SQL Server cluster configured for either active-active or active-passive clustering, the cluster will function in an active-passive manner. While clustering is tried and true, Microsoft decided to go a step further with...

• SQL Server Always On – New to SQL Server 2012 (Enterprise Edition), Always On introduced the concept of Availability Groups. An Availability Group is simply a container for a set of databases that fail over together. For purposes of Dynamics 365 databases, this is important considering there are always at least two databases per deployment. With Always On, the databases in the Availability Groups are synchronized amongst all the configured replicas. We won’t get into the details of setting up Always On (there are tons of articles out there already) but be sure to reference this TechNet for how to configure Dynamics 365 with SQL Always On: https://technet.microsoft.com/en-us/library/jj822357.aspx 

Disaster Recovery – Dynamics 365 and SQL Server

While having a single server fail is far more likely than seeing your entire data center swallowed into a pit only to be burnt up in the Earth’s core, it is a situation that should also be planned for! Unlike the sections regarding high availability, we need to think of a DR scenario in terms of the entire infrastructure – both Dynamics 365 and SQL Server – because if there is a disaster, chances are it is taking both so these plans go hand-in-hand. 

Over the years, we have implemented CRM/Dynamics 365 disaster recovery plans for numerous customers and as a result, have learned (what we consider) the best option – which is what we will be covering. This is not to say there are no other viable methods (such as VM snapshot migrations or SQL Log Shipping), but we will keep focus on our preferred option. In a nutshell, we are going to have a separate, self-contained Dynamics 365 environment in the DR data center that has data synced from the production SQL servers in the primary site.

SQL Server

Let us start from a database perspective since it is a little more complex and setting up Dynamics 365 for DR will require this information. As previously touched on, SQL Always On will come in handy yet again and recall the concept of Availability Groups as this will play heavy into this strategy.

For DR scenarios with Dynamics 365, we will need two Availability Groups in SQL AlwaysOn – one for the MSCRM_CONFIG database to be synchronized between the two primary data center nodes and another to synchronize the Organization_MSCRM database between not only the two primary data center SQL Servers but also a third SQL Server in your DR data center. The reason for this will become clear as you keep reading.

Dynamics 365

Going into the Dynamics 365 server installation(s), the third SQL node should already be built and ready with SQL Always On. Unlike the installation of the Dynamics 365 servers in the primary data center, you do NOT want to set the SQL connection to use the Availability Group but rather just the SQL Server located in the DR data center. The reasoning behind this is the MSCRM_CONFIG database – this database is specific per deployment of Dynamics 365 and two cannot exist in the same SQL server/instance (nor can the name be changed). Remember that this Dynamics 365 DR deployment is to be isolated from the primary.

At this point, you may be asking yourself how this all comes together. When the primary data center goes down (mainly both primary SQL nodes), the organization database will be failed over to the third node in DR. Once that happens, launch the deployment manager on the Dynamics 365 DR Server and import the organization database into the deployment. That should take about 5 minutes and then Dynamics 365 will be back up and running.

Great! We have the servers all setup in DR and are ready for that giant pit to open up beneath the data center! Now what? RUN THROUGH A TEST DISASTER RECOVERY SCENARIO! This cannot be stressed enough – you do not want to wait for a real disaster to find some small configuration issue messed up the entire plan.

URL Redirect Based on IP Address

As you are most likely aware, CRM/D365 has two URLs to access it when setup for IFD. The first URL, referred to as the “internal URL”, will allow authentication to be passed through and not display the ADFS login page for credentials. As the name suggests, this URL should only be available from within the network and not exposed out over the internet. The second URL, (not surprisingly) referred to as the “external URL”, will not pass through authentication and will display the ADFS login form for users to enter credentials. This URL is the one exposed out over the internet but is also available from within the network.

Customers often like the convenience of not having to enter credentials using the internal URL but do not like having two URLs to use. As a result, we are tasked with providing a solution to this dilemma. Back in the days of ADFS 2.0, we could easily handle this with a much simpler tweak in IIS of the ADFS server. Unfortunately, with the newer versions of ADFS no longer using IIS as its backbone this is no longer an option and a URL Rewrite rule now must be implemented on the CRM/D365 server. Here’s how:

1. Open IIS on the CRM/D365 Front End (web) Server and go to the URL Rewrite module of the CRM Website.

2. Click the link to “Add Rule(s)” and just select “Blank Rule” under Inbound rules.

3. Give the rule an applicable name and in the Match URL section, leave both the ”Requested URL” and “Using” fields as their defaults (shown in screen capture). In the “Pattern” field type in (.*).

4. This is where it gets fun. In the Conditions section, you will add three rules in the order shown below.

     a. For the first rule, change “crmorgname” to the actual name of your organization.
     b. For the second rule, change the pattern to match that of your internal IP range.
     c. Just enter the third rule as shown – no changes needed – ([^\.]*)\.(.*)

5. In the Action section, change the “Action type” to “Redirect” and for the Redirect URL enter your internal CRM/D365 URL followed by the syntax shown in the screenshot. Leave “Append query string” checked and set the Redirect type to “Permanent (301)”.

6. Apply the URL Rewrite rule and test it by going to the external URL from a machine that has an IP address matching that of the ranges specified on your network. If everything was setup correctly, the external URL will redirect to the internal URL and pass-through authentication will occur.

Tracing ADFS Logon Failures - Enabling ADFS Auditing

If you are ever faced with a situation where you are seeing a ton of logon failures in your ADFS logs and you’re not sure where they are coming from, you will soon learn that the basic logs do not provide any insight into their origins. But fear not! Some additional auditing can be enabled to help track down your problem child.

You will likely start with “Event 342 – The user name or password is incorrect” in the ADFS Admin logs – this is about as useless as it gets.

In my scenario, I was seeing about 1,500 of those events per minute. Needless to say, something was wrong but we had no idea what could possibly be trying to authenticate that much. Luckily, ADFS has some built-in auditing that can be of more use in situations like this. Start out by opening the ADFS Management Console and choose the option “Edit Federation Service Properties…” (it’s in the column on the right). Once in the properties screen, click on the “Events” tab. Here you should see 5 checkboxes – 2 of which are unchecked. Check those boxes (Success audits and Failure audits) and click OK.

Now that ADFS is setup for auditing, you need to tell the server to allow it. Head on over to the Local Security Policy of the ADFS server – this can be found from the Server Manager. Once here, drop down “Advanced Audit Policy Configuration” and then “System Audit Policies – Local Group Policy Object”. Locate the “Object Access” policy and then open the “Audit Application Generated” subcategory. Check the three boxes on this screen and then click OK.

You should now be all set to revisit your Event Viewer. Look into the Security events under the Windows Logs and you should now see events with ID 411 for “Classic Audit Failure” with the source as “AD FS Auditing”.

Go ahead and open one of those bad boys up…. Ahhhh finally some useful information! A quick look through will reveal the Client IP, which should be the machine sending the invalid authentication requests to ADFS.

The process from here is self-explanatory but head on over to the machine with the offending client IP address and find out what the problem is. In my case, there was a long-forgotten email router still setup which kept trying to access CRM. After figuring out the problem, it is recommended to disable auditing.

To Run The Email Router Configuration Manager, You Must Be A Local Administrator

Just had an unusual error message pop-up when trying to open the Email Router Configuration Manager following the update to Dynamics 365 (on-prem). It was something I had never seen before with previous versions of CRM – 2016 included. The message read:

To run the Email Router Configuration Manager, you must be a local administrator. Make sure that you are a member of the Administrator group before you start this program.

Well… this was an internal Dynamics 365 environment in which my account was a Domain Admin so I was a bit confused but I played along. Went ahead and added my account in as a local admin on the server and rebooted for good measure. That didn’t help. No matter how much I restarted the router service, rebooted, tried adding permissions, etc… it all fell on deaf ears. Googling the error message resulted in no meaningful results – which usually means 1 of 2 things – my router is completely hosed or I just uncovered a nice new bug with Dynamics 365!

After thinking about it for a few more minutes, I figured why not try running it AS AN ADMINISTRATOR!?!?! Sure enough, 3 clicks later and I was in… Easy workaround and I hope it saves some head-scratching!

CRM and TLS 1.0

While performing a routine installation of CRM 2016, I stumbled upon a new error during the system checks at the end of CRM installation wizard. The error was for the SQL Server check and read:

Could not connect to the following SQL Server: 'XXXXXX-SQL02'. Verify that the server is up and running and that you have SQL Server administrative credentials. [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Having never seen this particular error, I resorted to my friend Google who led me down a wormhole or possible issues and resolutions related to certificates and protocols. Turns out that this was indeed related to a security protocol – TLS 1.0 to be exact. It was disabled on the SQL Server as part of our new security template for new servers. So with the help of one of Cloud Application Engineers, Spencer Ashworth, we enabled TLS 1.0. This is done through a simple registry change.

Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server\ and locate the DWORD named Enabled. Set the value to 1, close the registry and then reboot the server.

Following the reboot and re-run of the CRM installation wizard, the system checks all passed and CRM was able to be successfully installed. Big thanks to Spencer who was the one that recalled CRM’s requirement of TLS 1.0 and knew the resolution!

HTTP to HTTPS Redirect for CRM using Claims-Based Authentication and IFD

In some cases, our customers would like HTTP to HTTPS redirection setup for CRM when using Claims-Based Authentication and IFD. There are a number of articles and blogs out on the internet that use various methods to accomplish this (e.g. URL Rewriting) but I find that a lot are overly complicated if all you are looking to do is have the redirect functioning internally (you do not want port 80 open on your firewall anyways). Below is what I perceive to be the easiest method to achieving this – using the HTTP Redirection capabilities of IIS:

1. First we will need to install HTTP Redirection on the CRM Web Server if it is not installed already. In Server Manager, go to Add Roles and Features and choose to add the HTTP Redirection feature under the Web Server role.

2. Create two folders in a directory that you know will not be deleted. Name these folders something along the lines of “CRM Internal Redirect” and “CRM External Redirect”. Folder names and paths aren’t too important but in my example I just put them in the inetpub directory.

3. Next, go into IIS. Make sure to remove the HTTP (port 80) binding from the CRM site before proceeding with the next steps.

4. Since two URLs can be used to be accessed CRM when it is setup for Claims and IFD, we will need to create the redirects for both URLs. Begin by creating a new website in IIS for the Internal URL.

• Site Name: Something descriptive such as “CRM Internal Redirect”
• Physical Path: Browse to the CRM Internal Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your internal CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, open the CRM Deployment Manager and go to properties. Select the Web Addresses tab and you should see your internal URL.

5. Just like step 4, create another new site in IIS – this time for the external URL:

• Site Name: Something descriptive such as “CRM External Redirect”
• Physical Path: Browse to the CRM External Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your exteransl CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, this is the unique name of your CRM organization followed by the domain (e.g. orgname.domain.com).

6. You should now have IIS looking similar to this:

7. Click on your Internal Redirect site and open the HTTP Redirect module:

8. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your internal URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

9. Now do the same for the External redirect site. Click on your External Redirect site and open the HTTP Redirect module. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your external URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

That’s it. You should now be able to enter either of the CRM URLs with HTTP into a browser and have it redirected to HTTPS appropriately.

Insufficient Winsock Resources Available to Complete Socket Connection Initiation

Seemingly out of the blue, one of our customers could no longer log in to their CRM. As it appeared to be an authentication issue, I began troubleshooting on their ADFS server. Upon opening the ADFS Management Console, I was greeted with the following error message:

In the event viewer were a number of errors similar to this:

Microsoft.IdentityServer.Configuration.ReadServiceConfigFailedException: MSIS2001: Configuration service URL is not configured. ---> System.InsufficientMemoryException: Insufficient winsock resources available to complete socket connection initiation. ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 127.0.0.1:1500

Well, this was a new one for me but at least the error message actually gave a description instead of something generic. Logic dictates that if there isn’t enough of something, give it more. In this case, it appeared to me that ADFS was complaining about not having enough ephemeral ports so I figured I would increase the number available.

1. Start Registry Editor.
2. Locate the following subkey in the registry, and then click Parameters:  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. On the Edit menu, click New, and then add the following registry entry:
   Value Type: DWORD
   Value Name: MaxUserPort
   Value Data: 65534 (decimal)
4. Reboot server

After completing the above, ADFS was able to open the connection to its database and users were able to log back in to CRM. Now, it may be worthwhile to investigate exactly why ports suddenly became busy… perhaps there is a socket leak or new applications were installed… but this should at least get you up and running in the meantime.

Calling LoadLibraryEx on ISAPI filter DefaultAddonFilter.dll failed

I recently had an interesting issue with a client’s CRM 2011 to 2016 go-live. Following all of my upgrade work and initial testing, I typically like to reboot the CRM and SQL servers and then test again. In this particular instance, when the servers came back online I tested CRM again only to receive a lovely error message from Internet Explorer:

HTTP Error 500.0 – Internal Server Error

Calling LoadLibraryEx on ISAPI filter C:\Program Files\Microsoft Dynamics CRM\Server\bin\DefaultAddonFilter.dll failed.

After a fair amount of digging I noticed that the Visual C++ 2013 Redistributable (64-bit) was not present in the add/remove programs list on the CRM server. It seemingly uninstalled itself during the reboot because otherwise I would have had this issue pre-reboot. So, I went out and grabbed the installer package for Visual C++ 2013 Redistributable (64-bit) from MS (http://go.microsoft.com/fwlink/p/?LinkId=402059) and reinstalled it. After that and an IISReset, CRM began functioning as normal and everything retested everything successfully.

Unfortunately, I am at a loss as to why this happened – perhaps something to do with server patching. Also of note is that the redistributable did not remove itself from the SQL server.

Reports in CRM not working - The type initializer for 'Microsoft.Crm.LocatorService' threw an exception.

We had a customer running into an issue while trying to run reports from CRM. Of course, the error in the CRM UI was the usual rsProcessingAborted error - Cannot create a connection to Data Source. So as I always need to do, I dug into the SSRS logs. The errors I could extract from there were:

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The type initializer for 'Microsoft.Crm.LocatorService' threw an exception. ---> 

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: Cannot load Counter Name data because an invalid index 'W3SVC_W3WP' was read from the registry.

From what I could surmise, there was something wrong with the performance counters on the server(s). I did find that Microsoft does have a KB for this error but it is in reference to update rollups on the CRM Outlook Client. Luckily for me, the fix turned out to be similar!

1. On the CRM Server, open an administrative command prompt.
2. Type “lodctr /r” and press enter.
3. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
4. Restart CRM Services and run an IISReset
5. On the SSRS Server, open an administrative command prompt.
6. Type “lodctr /r” and press enter.
7. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
8. Restart SSRS Services.

After following the above process, reports in CRM began working again. The commands run rebuild the performance counters.

An attempt was made to access a socket in a way forbidden by its access permissions

When setting up the Microsoft Dynamics Email Router, there can be a lot of obstacles in getting emails to send and receive properly. One of which turns out to be McAfee VirusScan. If you get the following error when testing the connection on the Outgoing profile, you may want to see if McAfee (or any other virus protection software) is running on the server:

Outgoing Status: Failure – An error occurred while checking the connection to the email server {smtp server name}. An attempt was made to access a socket in a way forbidden by its access permissions.

In order to fix this, right click on the McAfee icon in the system tray and open up the VirusScan Console.

In the VirusScan Console, double-click “Access Protection”.

Under Categories, click on “Anti-virus Standard Protection” and then check to see if there are any checkmarks next to “Prevent mass mailing worms from sending mail”. If there are, simply uncheck them.

Click OK and close out of the VirusScan Console. Open the email router and try testing again. You should now be seeing green!

Report Server cannot load the SQLPDW extension

While troubleshooting SSRS issues, you may run into the following error messages in logs and traces:

Report Server (MSSQLSERVER) cannot load the SQLPDW extension.

Report Server (MSSQLSERVER) cannot load the TERADATA extension.

On numerous occasions I have seen both customers and colleagues believe that these errors were the source of their problems. After which, the wild goose chase to resolve these errors ensues. I am here to tell you that it is highly unlikely that they are related to your problem at hand (especially if you are only using SSRS for CRM reporting).

These errors occur because the Teradata and SQLPDW extensions are registered in the Reporting Services configuration file by default but the assemblies for these extensions do not get installed in the same manner. Typically these extensions are used with big data and business analytics.

If the errors do not bother you, simply ignore them and track down other error messages. However, if you are the type that prefers clean logs you can take the following steps to alleviate the error messages from being logged:

• Open the rsreportserver.config file. This can usually be found in \Program Files\Microsoft SQL Server\MSRSXX_X.MSSQLSERVER\Reporting Services\ReportServer.
• In this config file, simply search for and comment out the entries related to SQLPDW and TERADATA. There should be 2 entries for SQLPDW and 3 for TERADATA. Below is an example:

<!--Extension Name="SQLPDW" Type="Microsoft.ReportingServices.DataExtensions.SqlDwConnectionWrapper,Microsoft.ReportingServices.DataExtensions"/-->

<!--Extension Name="SQLPDW" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.MSSQLADW.MSSqlAdwSQCommand,Microsoft.ReportingServices.SemanticQueryEngine">

    <Configuration>

        <EnableMathOpCasting>False</EnableMathOpCasting>

    </Configuration>

</Extension-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.DataExtensions.TeradataConnectionWrapper,Microsoft.ReportingServices.DataExtensions"/-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.Teradata.TdSqlSQCommand,Microsoft.ReportingServices.SemanticQueryEngine">

    <Configuration>

        <EnableMathOpCasting>True</EnableMathOpCasting>

        <ReplaceFunctionName>oREPLACE</ReplaceFunctionName>

    </Configuration>

</Extension-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.Teradata.TdSqlModelGenerator,Microsoft.ReportingServices.SemanticQueryEngine"/-->

• Save and close the config file.
• Restart the SSRS service.

CRM Reporting Issue - The system cannot contact a domain controller to service the authentication request

Had a case come into my queue indicating a client’s reports in their production CRM were not working so I logged in and began taking a look. At first glance, I was seeing the generic “Reporting Error – The report cannot be displayed. (rsProcessingAborted)” that we see for 90% of reporting issues so I proceeded to the SSRS logs where I was greeted with the following error:

System.Runtime.InteropServices.COMException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350) ---> Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350)

Interesting to say the least and something I had never seen before. My inclination was that a domain controller was down or somehow the connection to AD was severed (but then again, how was I able to log on the server). I ran a set command to check the logon sever and was able to ping it fine, along with all of the other DCs. Next stop was at the DNS/Gateway settings on the NIC – compared everything to another SQL/SSRS server that was working fine. Again, nothing seemed out of place.
Off to the handy dandy event viewer! Immediately I noticed a flood of errors indicating that the automatic backups and log shipping were failing. Amidst those errors I found a familiar message:

Whatever was causing the reporting issue was also causing the backups to fail. I made my way through the events until I reached the beginning of the flood and there it was, in all its glory:

BackupDiskFile::CreateMedia: Backup device '\\goxsaXXXX\D$\MSSQL11.XXXX\MSSQL\Backup\Log_Shipping\XXXX_20160524043001.trn' failed to create. Operating system error 1331(This user can't sign in because this account is currently disabled.).

Easy enough – the SQLAdmin account running SSRS (and the other SQL services) cannot authenticate because it’s disabled.

 

Had the customer check AD and sure enough, the account was disabled. Once enabled, reports began working again. 

In all, this was a very strange error message for the actual issue. It could have really led us on a wild goose chase into the networking side of things. Probably saved 2-3 hours by reading through the error logs and being thorough in investigation before going back to the customer with the wrong answer. Moral of the story: do your homework.

CRM and IIS Machine Keys

When you have multiple CRM web (front-end) servers splitting the load, requests can bounce between servers or they can be set to remain on the same server through the NLB/Firewall settings. The latter option is commonly referred to as “Sticky Sessions” or “Persistence”.

If requests bounce between servers (whether by design or by fault), you may notice issues when accessing certain functionality within CRM. Often it will be accompanied by the following error message in the event viewer:

Event code: 3012 
Event message: An error occurred processing a web or script resource request. The resource identifier failed to decrypt. 

Exception information: 
    Exception type: HttpException 
    Exception message: Unable to validate data.
   at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType, Boolean signData)
   at System.Web.UI.Page.DecryptString(String s, Purpose purpose)
   at System.Web.Handlers.ScriptResourceHandler.ProcessRequest(HttpContextBase context, VirtualFileReader fileReader, Action`2 logAction, Boolean validatePath)

Basically what is happening is one server encrypted (validated) some of the traffic with an automatically generated key and another server received it but cannot decrypt it because it has different keys. These keys are known as “machine keys” and are found within IIS. By default, these keys are all set to generate automatically and no two are the same. When you are using multiple web servers, the machine keys need to be set statically and shared amongst them. Here is how to do this, starting on the first web server:

1. Open IIS, select the Microsoft Dynamics CRM Website, and double-click “Machine Key”
   
   
2. Uncheck the “Automatically generate at runtime” and “Generate a unique key for each application” boxes under both “Validation Key” and “Decryption Key”.
   
   
3. Click “Generate Keys” on the right side of the screen.
   
   
4. This will create new, random keys.
   
   
5. Click apply.
   
   
6. From here, repeat steps 1 and 2 on the remaining web servers. Instead of generating new keys as in step 3, copy and paste the keys generated from earlier into the remaining servers so that all servers have the same set of keys.
7. Once keys on all servers are set and applied, reset IIS on all the boxes. That should do it.

Host Header Configured on Selected Site Does Not Resolve to an IP Address that is Assigned to the Local System.

I was recently performing an in-place upgrade from CRM 2015 to CRM 2016 when during the system checks I received the following warning:

Host Header "notCRM.domain.com" configured on selected site does not resolve to an IP address that is assigned to the local system. This conflict must be reconciled for CRM Server to function properly. 

The fact that it was giving me a host header that did not pertain to the CRM website was odd so I took a look at IIS. What I discovered was that there was another website setup in IIS on the CRM Server. And that site had the site ID of 1.

As I was the original installer of CRM for this deployment, I knew that this site wasn’t here when CRM was installed. CRM originally had the site ID of 1, not this other site. Somebody had been in here messing around and changed the site IDs manually. After asking enough questions, I learned who did this and why. To make a long story short, their site needed to have the ID of 1. CRM, on the other hand, does not care so I rolled with the punches and decided I would work around this rather than cause a fuss.

From what I could deduce about the error message, the CRM installer was being told to look at the IIS website that had an ID of 1 to perform the upgrade on. I decided to switch the CRM site back to the ID of 1. To do this, you must first change the other site’s ID to a number not in use (e.g. 3, in my case). To do this, go into the site’s advanced settings.

After the other site was changed, I reverted the CRM site back to 1 and reset IIS.

The upgrade wizard will need to be restarted for it to pick up the change but once you make it back to the system checks, the warning should be gone.

The upgrade succeeded without issue but we weren’t done just yet. Since the other site needed to have the ID of 1, I needed to revert changes to the site IDs which in itself isn’t complicated. What was not taken into account was a registry key in HKLM\Software\Microsoft\MSCRM called “website”.

This registry key indicates which website directory in IIS CRM is using. Since CRM is using the ID of 2, this key needed to be switched to “LM/W3SVC/2”. In case you have been wondering what the ID controls, it is just an identifier used in the directory name for trace files and log files. It’s nothing critical that will not let CRM operate as normal.

By now you are probably thinking “Why didn’t you change this key rather than switching the site IDs to resolve the warning message during the upgrade?!?!” Well... it didn’t occur to me until afterward I completed the upgrade. Hindsight is 20/20!