ADFS Cache Issues

From time-to-time, employees may leave a company but end up coming back. With all that comes user management, accounts, access, permissions, etc…. While that is all easy enough from an Active Directory Standpoint, this can cause some complications with access into CRM/Dynamics 365 when using ADFS for claims-based authentication. Particularly if the user’s original AD account was deleted and then recreated with the same username.

After the account is recreated, you may learn that there isn’t any issue re-enabling the user in CRM or Dynamics 365 but that the user cannot login via ADFS. There is also a similar scenario when companies migrate domains and keep the same user names, only changing the domain. Some behavior you may notice are 404 errors, wrong username showing in event viewer logs, and general ADFS errors. While I haven’t been able to pinpoint why this happens sometimes and not others, what I do know is the problem lies within ADFS.

Basically what seems to be going on is the ADFS server has a cache of information already built up about the users, including GUIDs, SIDs, UPNs, etc… So when an authentication request comes through it checks for the username in its cache along with any associated records and matches what it can rather than pulling from Active Directory. When it does this it can pull the information from the old, previously-deleted account and then error out.

Anyways, without further ado – here are two methods that I’ve used successfully to get through these situations. Sometimes one works and sometimes the other works – just some more mystery around this situation…

Method 1: Disable the LSA Cache
This method involves disabling the LSA cache by setting it’s maximum size to 0 – essentially making it nonexistent.

1. Log on to the ADFS Server.
2. Open Registry Editor.
3. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.
4. Create a new DWORD value in Lsa named LsaLookupCacheMaxSize and set the value to 0.
5. Close Registry Editor.
6. Have problem user try to log in again.

Whether or not this works, go back in after testing and delete the value you just created. ADFS may suffer from performance issues if it is left in place for a long period of time.

Method 2: Use PSTools to update SID
Going this route requires the download of PsTools – a command-line tool that helps with server administration. In our scenario, it is the PsGetSid command that will be useful as it will force update the SID of the account held in cache from AD.

1. Log on to the ADFS server.
2. Download PSTools from https://docs.microsoft.com/en-us/sysinternals/downloads/psgetsid.
3. Extract out the contents of the zip file into an accessible directory.
4. Run PSGetSid from command prompt using the following syntax: C:\{Directory}\PsGetsid.exe domain\crmuser
5. Have problem user try to log in again.

If neither of those methods works, there may be something else wrong. I would start by verifying that all other users can access CRM/D365 without issue and go from there. My apologies for not having exact error messages or screenshots on this post – it was just an issue I’ve run into in the past a few times and thought it may help someone out regardless as I’ve realized there isn’t much information out there about it.

Issue Installing CRM 2016 SRS Data Connector with SQL 2016

In my last post, I mentioned that I had received some Windows Server 2016 boxes to test setup of CRM 2016/Dynamics 365. In addition to them being the 2016 OS, they also came with SQL Server 2016. The installation of CRM went off without a hitch (after working through the Search service problem) and the databases created in SQL as expected.

However, when it came to installation of the SRS Data Connector, things got a little hairy. The installation began but quickly threw this error message:

Error| System.Exception: Action Microsoft.Crm.Setup.SrsDataConnector.AddBindingRedirectForRdlHelper failed. ---> System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Program Files\Microsoft SQL Server\MSRS13.MSSQLSERVER\Reporting Services\ReportManager\web.config'.

To make a long story short, SQL 2016 no longer creates/uses the path specified in the error message and the RTM Data Connector installer does not check the version of SSRS being used so it will just keep trying to use that directory.

Now, I had some previous experience installing the Data Connector with SQL 2016 involved. From that, I recalled that the installer, when asked to check for updates on the first screen, actually went out and got some new install bits. This time around I was not so lucky – it just told me there were no updates available (I even verified that the Windows Firewall was off and that the box was allowing Windows Updates)… it had internet access but if you’re reading this and your SQL server does not have access to the internet, you should keep reading. Regardless, I needed a solution and the internet was not full of suggestions with this being pretty new technology.

I decided to check the Microsoft Update Catalog to see if I could download and manually patch the update in. To my delight, I came across a listing for “Setup Update for Microsoft Dynamics CRM 2016 Reporting Extensions” and this ultimately helped solve my issue. Here is a step-by-step process of what to do if you are facing the same situation:

1. If your initial install failed, go into the server’s control panel and uninstall the data connector.

2. Go to http://www.catalog.update.microsoft.com/

3. Search for “3129794” and you should get back one listing. Click the download button.

4. A new screen will pop up with a ton of .cab files. Find the one named “srs_kb3129794_amd64_1033_b585b78e247dc7611e1e406a6c132cdee112690c.cab” (just look for 1033) and download that. I will admit that I am not sure of the purpose of all these files but the 1033 one did the trick for me. Later on, you will notice that another file we work with is also “1033” so that probably has something to do with it.

5. Once you have the .cab file downloaded to the server, open it up to reveal the patch file. Right-click on it and choose “Extract”.

6. After the extraction, you will have the .msp file available for use. Go to the directory of the SRS Data Connector’s installation files and locate a folder named “Update”. In here, you will see another .msp file named “Srs_KB3121695_Amd64_1033”.

7. Cut and paste that file out of the “Update” folder – just put it anywhere else on the server.

8. Copy and paste the .msp you just downloaded into the now blank “Update” folder - essentially replacing it.

You should go from this:

To this:

9. Re-run the installer as normal and it should complete without issue.


Just to note a few things:

• I initially played a bunch of trial and error with getting the update patched in before I discovered the ability to replace the .msp in the Update folder. I first went down the path of editing the config.xml to include the patch and then called the installer from a command line specifying it to use the config file. This is how I have always patched in updates for the server installer but it does not seem to work the same way for the Data Connector. I also tried loading the patch into the temp files that the installer looks to… also didn’t work.

• If your base installer’s update file has a different number than “1033” at the tail end of it, you may need to match it up with the correct .cab file from the Update Catalog. I did not encounter this scenario so I cannot test or prove out this theory but I think logic dictates… 

• There is another blog out there about applying SP1/2 to the failed installation and then manually editing file structure. I will not discredit that being a possible workaround but personally, I do not like the idea of relying on an update to fix a broken install.

Good luck!

Issue Installing CRM 2016 (Dynamics 365) on Windows Server 2016

I recently got my hands on some Windows Server 2016 boxes and wanted to test out the CRM 2016/Dynamics 365 installation process on them. Preparation began as usual and I kicked off the wizard. Some time after that I was presented with the following error message:

Action.Microsoft.Crm.Setup.Common.InstallWindowsSearchAction failed. The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)

What I quickly learned was that the Windows Search service, while coming pre-installed on Windows Server 2016, is set to disabled by default.

I enabled the service by setting it to automatic startup, started it, and then hit retry on the error diagloue from the CRM installation wizard. The installation continued and finished with no further issues.

I love the easy fixes!

High Availability and Disaster Recovery Options for Dynamics 365 (CRM)

Planning the infrastructure for your Dynamics 365 environment is a critical step in the deployment process, especially when the software is mission critical to business needs and day-to-day operations. In scenarios like these, it is imperative to have high availability and even disaster recovery plans in place. So the question is: What options do you have when it comes to Dynamics 365 on premise? In this article, we will break it down from both an application (Dynamics 365) and database (SQL Server) perspective.

High Availability – Dynamics 365
For most companies, having high availability for their Dynamics 365 deployment is a must and the setup for this is similar to that of other web-based applications. Keep in mind that through this section we are strictly referring to Dynamcis 365 servers with the “Front End” or “Full” role – in other words, the servers that host the website in IIS. Having multiple “Back End”-only servers (those with the Async and Sandbox services) is also a good idea but those will natively load balance themselves if they are in the same deployment.

• Network Load Balancing – Probably the most obvious method for high availability is running multiple Dynamics 365 servers in a load-balancing configuration. This will not only help with carrying the day-to-day load of traffic but also serve as the foundation for a good high availability option. If a server were to fail, just pop it out of the NLB pool. Load balancing comes in a few different flavors because of all the applications out there but mainly it comes down to two options – hardware and software.

• Hardware Load Balancing involves the use of a third-party appliance such as F5 or NetScaler that sits in front of the Dynamics 365 servers to handle which server receives the user requests. While this is the preferred option as it does not create any additional processes for the servers to handle and usually provides a wider array of configuration options (e.g. monitoring, security, filtering, etc…), it can be more difficult and much more costly to implement.

• Software Load Balancing on the other hand is relatively easy to setup and in the case of Windows NLB, is free, as you have already paid for your Windows Server licenses. Critics of software load balancing will contend that it is not “true” high-availability due to the lack of many key features found in hardware load balancing. The fact stands that Windows NLB is just Round Robin DNS so if a server were to go down, requests would still be sent to the faulted server until it is told otherwise.

• As one can probably infer, this is simply a cost vs. ROI decision – which is the case with most IT situations. If the hardware option is being considered, make sure to reach out to your appliance vendor to ensure there are no known issues with it and Dynamics 365. It is also a good idea to inquire about documentation for setup. For more information about load balancing Dynamics 365 in general and the required configurations, check out this TechNet: https://technet.microsoft.com/en-us/library/hh699803.aspx. 

• Backup Server(s) – While this method may be a little more rigid, it is still effective and again, has a couple options. The meat and potatoes of this route is having a server to fall back to in the event that your primary Dynamics 365 server(s) fails and NLB has not been implemented.

• VM Snapshots – If you are reading this, you most likely understand the concept of VM snapshots so we will not dive too heavily into this. The idea is to have automatic snapshots taken automatically on a set interval that the Dynamics 365 server can be reverted to or new server built from in the event of failure. Reverting the existing server is the easier option but also the more risky of the two. Spinning up a new server is safer but more involved – and requires the shutdown of the existing server as you will need to use the same host name and IP address.

• Laying in wait – We have had some customers opt for this route as the return to uptime is a little quicker and/or VMs were not being used. Dynamics 365 deployments can obviously be comprised of multiple servers – but not all need to be active or enabled. The concept here is to build a secondary Dynamics 365 server as part of the deployment but leave it disabled until it is needed. If the primary server were to fail, all that would be required is enabling of the server via the Deployment Manager and changes to DNS (and firewall rules, if applicable).

High Availability – SQL Server

Now that we have covered high availability from an application perspective, what about the database? Don’t worry! Microsoft has that covered too!

• SQL Server Failover Cluster – There is nothing new about this concept. Two SQL Servers are setup to use a shared disk, one server goes down and the other picks up the load. When installing Dynamics 365 just make sure to point connection strings to the cluster name and not a single node and you should not have any issues but be sure to test failover prior to going live. Just to note - although you can install Dynamics 365 to a SQL Server cluster configured for either active-active or active-passive clustering, the cluster will function in an active-passive manner. While clustering is tried and true, Microsoft decided to go a step further with...

• SQL Server Always On – New to SQL Server 2012 (Enterprise Edition), Always On introduced the concept of Availability Groups. An Availability Group is simply a container for a set of databases that fail over together. For purposes of Dynamics 365 databases, this is important considering there are always at least two databases per deployment. With Always On, the databases in the Availability Groups are synchronized amongst all the configured replicas. We won’t get into the details of setting up Always On (there are tons of articles out there already) but be sure to reference this TechNet for how to configure Dynamics 365 with SQL Always On: https://technet.microsoft.com/en-us/library/jj822357.aspx 

Disaster Recovery – Dynamics 365 and SQL Server

While having a single server fail is far more likely than seeing your entire data center swallowed into a pit only to be burnt up in the Earth’s core, it is a situation that should also be planned for! Unlike the sections regarding high availability, we need to think of a DR scenario in terms of the entire infrastructure – both Dynamics 365 and SQL Server – because if there is a disaster, chances are it is taking both so these plans go hand-in-hand. 

Over the years, we have implemented CRM/Dynamics 365 disaster recovery plans for numerous customers and as a result, have learned (what we consider) the best option – which is what we will be covering. This is not to say there are no other viable methods (such as VM snapshot migrations or SQL Log Shipping), but we will keep focus on our preferred option. In a nutshell, we are going to have a separate, self-contained Dynamics 365 environment in the DR data center that has data synced from the production SQL servers in the primary site.

SQL Server

Let us start from a database perspective since it is a little more complex and setting up Dynamics 365 for DR will require this information. As previously touched on, SQL Always On will come in handy yet again and recall the concept of Availability Groups as this will play heavy into this strategy.

For DR scenarios with Dynamics 365, we will need two Availability Groups in SQL AlwaysOn – one for the MSCRM_CONFIG database to be synchronized between the two primary data center nodes and another to synchronize the Organization_MSCRM database between not only the two primary data center SQL Servers but also a third SQL Server in your DR data center. The reason for this will become clear as you keep reading.

Dynamics 365

Going into the Dynamics 365 server installation(s), the third SQL node should already be built and ready with SQL Always On. Unlike the installation of the Dynamics 365 servers in the primary data center, you do NOT want to set the SQL connection to use the Availability Group but rather just the SQL Server located in the DR data center. The reasoning behind this is the MSCRM_CONFIG database – this database is specific per deployment of Dynamics 365 and two cannot exist in the same SQL server/instance (nor can the name be changed). Remember that this Dynamics 365 DR deployment is to be isolated from the primary.

At this point, you may be asking yourself how this all comes together. When the primary data center goes down (mainly both primary SQL nodes), the organization database will be failed over to the third node in DR. Once that happens, launch the deployment manager on the Dynamics 365 DR Server and import the organization database into the deployment. That should take about 5 minutes and then Dynamics 365 will be back up and running.

Great! We have the servers all setup in DR and are ready for that giant pit to open up beneath the data center! Now what? RUN THROUGH A TEST DISASTER RECOVERY SCENARIO! This cannot be stressed enough – you do not want to wait for a real disaster to find some small configuration issue messed up the entire plan.

URL Redirect Based on IP Address

As you are most likely aware, CRM/D365 has two URLs to access it when setup for IFD. The first URL, referred to as the “internal URL”, will allow authentication to be passed through and not display the ADFS login page for credentials. As the name suggests, this URL should only be available from within the network and not exposed out over the internet. The second URL, (not surprisingly) referred to as the “external URL”, will not pass through authentication and will display the ADFS login form for users to enter credentials. This URL is the one exposed out over the internet but is also available from within the network.

Customers often like the convenience of not having to enter credentials using the internal URL but do not like having two URLs to use. As a result, we are tasked with providing a solution to this dilemma. Back in the days of ADFS 2.0, we could easily handle this with a much simpler tweak in IIS of the ADFS server. Unfortunately, with the newer versions of ADFS no longer using IIS as its backbone this is no longer an option and a URL Rewrite rule now must be implemented on the CRM/D365 server. Here’s how:

1. Open IIS on the CRM/D365 Front End (web) Server and go to the URL Rewrite module of the CRM Website.

2. Click the link to “Add Rule(s)” and just select “Blank Rule” under Inbound rules.

3. Give the rule an applicable name and in the Match URL section, leave both the ”Requested URL” and “Using” fields as their defaults (shown in screen capture). In the “Pattern” field type in (.*).

4. This is where it gets fun. In the Conditions section, you will add three rules in the order shown below.

     a. For the first rule, change “crmorgname” to the actual name of your organization.
     b. For the second rule, change the pattern to match that of your internal IP range.
     c. Just enter the third rule as shown – no changes needed – ([^\.]*)\.(.*)

5. In the Action section, change the “Action type” to “Redirect” and for the Redirect URL enter your internal CRM/D365 URL followed by the syntax shown in the screenshot. Leave “Append query string” checked and set the Redirect type to “Permanent (301)”.

6. Apply the URL Rewrite rule and test it by going to the external URL from a machine that has an IP address matching that of the ranges specified on your network. If everything was setup correctly, the external URL will redirect to the internal URL and pass-through authentication will occur.

Tracing ADFS Logon Failures - Enabling ADFS Auditing

If you are ever faced with a situation where you are seeing a ton of logon failures in your ADFS logs and you’re not sure where they are coming from, you will soon learn that the basic logs do not provide any insight into their origins. But fear not! Some additional auditing can be enabled to help track down your problem child.

You will likely start with “Event 342 – The user name or password is incorrect” in the ADFS Admin logs – this is about as useless as it gets.

In my scenario, I was seeing about 1,500 of those events per minute. Needless to say, something was wrong but we had no idea what could possibly be trying to authenticate that much. Luckily, ADFS has some built-in auditing that can be of more use in situations like this. Start out by opening the ADFS Management Console and choose the option “Edit Federation Service Properties…” (it’s in the column on the right). Once in the properties screen, click on the “Events” tab. Here you should see 5 checkboxes – 2 of which are unchecked. Check those boxes (Success audits and Failure audits) and click OK.

Now that ADFS is setup for auditing, you need to tell the server to allow it. Head on over to the Local Security Policy of the ADFS server – this can be found from the Server Manager. Once here, drop down “Advanced Audit Policy Configuration” and then “System Audit Policies – Local Group Policy Object”. Locate the “Object Access” policy and then open the “Audit Application Generated” subcategory. Check the three boxes on this screen and then click OK.

You should now be all set to revisit your Event Viewer. Look into the Security events under the Windows Logs and you should now see events with ID 411 for “Classic Audit Failure” with the source as “AD FS Auditing”.

Go ahead and open one of those bad boys up…. Ahhhh finally some useful information! A quick look through will reveal the Client IP, which should be the machine sending the invalid authentication requests to ADFS.

The process from here is self-explanatory but head on over to the machine with the offending client IP address and find out what the problem is. In my case, there was a long-forgotten email router still setup which kept trying to access CRM. After figuring out the problem, it is recommended to disable auditing.

To Run The Email Router Configuration Manager, You Must Be A Local Administrator

Just had an unusual error message pop-up when trying to open the Email Router Configuration Manager following the update to Dynamics 365 (on-prem). It was something I had never seen before with previous versions of CRM – 2016 included. The message read:

To run the Email Router Configuration Manager, you must be a local administrator. Make sure that you are a member of the Administrator group before you start this program.

Well… this was an internal Dynamics 365 environment in which my account was a Domain Admin so I was a bit confused but I played along. Went ahead and added my account in as a local admin on the server and rebooted for good measure. That didn’t help. No matter how much I restarted the router service, rebooted, tried adding permissions, etc… it all fell on deaf ears. Googling the error message resulted in no meaningful results – which usually means 1 of 2 things – my router is completely hosed or I just uncovered a nice new bug with Dynamics 365!

After thinking about it for a few more minutes, I figured why not try running it AS AN ADMINISTRATOR!?!?! Sure enough, 3 clicks later and I was in… Easy workaround and I hope it saves some head-scratching!

Yet Another ADFS Looping Issue

I recently applied the Dynamics 365 Update to a CRM 2016 Service Pack 1 environment that was setup for IFD and ran into some unexpected behavior. This environment had about 15 organization setup in it. When trying to log on using the external URLs (e.g. orgname.domain.com) access worked without issue but using the internal URL for any organization (e.g. crm.domain.com/orgname) CRM and ADFS would run in a continuous loop. Sometimes it would error out, sometimes it would just keep looping forever.

Started off by checking event viewers on both the CRM and ADFS server. I noticed that on the times it did error out after a few loops instead of looping nonstop, it resulted in an event log similar to one I saw before caused by a bug with the 0.1 Update for CRM 2016 (http://blog.gagepennisi.com/2016/04/crm-2016-update-01-bug-with-adfs-for.html):

MSIS7042: The same client browser session has made '6' requests in the last '17' seconds. Contact your administrator for details.

So immediately I thought “Great, here we go again.” There was also nothing in the CRM event viewer.

Hours of trying different different troubleshooting tactics for similar situations (including this one with the same looping behavior but with the external URL - http://blog.gagepennisi.com/2016/01/adfs-logon-page-loop-issue.html) yielded nothing. Rebuilt the Relying Party Trusts, reconfigured IFD, created a self-signed certificate to verify it wasn’t a certificate issue, etc…

Instead of keeping on with the guess and check, I realized I needed to get more information around the problem so I started with a Fiddler trace – great web debugging tool, if you haven’t heard of it (http://www.telerik.com/fiddler). Unfortunately, in this case all it did was confirm the behavior I was seeing in the browser which was a constant redirect between D365 and ADFS.

After that, I decided to run a platform trace on the CRM server to see if CRM would give me any insight but nothing really stood out to me. Almost at my wit’s end, I decided to rope in my colleague Dan Francis to get another set of eyes on it. After some review of the platform trace he noticed one little line that seemed a bit odd:

>Multi-org sharable cache loading system and non-system metadata with build number 7.0.0.3543 and language 1033

CRM 2016 is version 8.0 and D365 is 8.2 so the fact that we were seeing any refernce to 7.0.X.XXXX (CRM 2015) was not inline with what was to be expected. We checked the Deployment Manger and realized there was an old, disabled organization from CRM 2015 that was not upgraded (because it was disabled at the time of the original CRM 2016 upgrade). After some deliberation, we decided to remove that organization from the deployment and boom! The internal URL began working as expected!

Basically what we were able to surmise from this is that when the internal URL is being used, all organizations – whether enabled or disabled – are checked for versioning and the lowest build number is used to load this “multi-org sharable cache” and apparently D365 doesn’t play nicely with older versions hanging around. Moral of the story – update/upgrade all of your orgs or just remove them.

Big thanks to Dan the Man!

CRM and TLS 1.0

While performing a routine installation of CRM 2016, I stumbled upon a new error during the system checks at the end of CRM installation wizard. The error was for the SQL Server check and read:

Could not connect to the following SQL Server: 'XXXXXX-SQL02'. Verify that the server is up and running and that you have SQL Server administrative credentials. [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Having never seen this particular error, I resorted to my friend Google who led me down a wormhole or possible issues and resolutions related to certificates and protocols. Turns out that this was indeed related to a security protocol – TLS 1.0 to be exact. It was disabled on the SQL Server as part of our new security template for new servers. So with the help of one of Cloud Application Engineers, Spencer Ashworth, we enabled TLS 1.0. This is done through a simple registry change.

Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server\ and locate the DWORD named Enabled. Set the value to 1, close the registry and then reboot the server.

Following the reboot and re-run of the CRM installation wizard, the system checks all passed and CRM was able to be successfully installed. Big thanks to Spencer who was the one that recalled CRM’s requirement of TLS 1.0 and knew the resolution!

CRM Diagnostics Page

When troubleshooting CRM performance issues everyone typically wants to point fingers at the servers – whether it be SQL, CRM, or even SSRS – but a commonly overlooked factor is the network. Not known to everyone is a tool built into CRM called the CRM Diagnostics Page. It is basically a very simple way to check network performance. To access the CRM Diagnostics Page simply go to:

http(s)://<YourCRMServerURL>/tools/diagnostics/diag.aspx

This will land you at a page that looks like the following:

On this page, you can clearly see the series of tests available but what are they?

1. Latency Test

This calculates the average time taken to download a small text file 20 times. The file downloaded from the CRM server is /_static/Tools/Diagnostics/smallfile.txt. CRM is designed to work best with latency under 150 milliseconds. Latency can be a huge factor in performance for offices far away the CRM server’s location.

2. Bandwidth Test

The bandwidth test downloads image files and records the speeds. The recorded speeds are averaged together and provided in the results summary. Bandwidth should ideally be higher than 50 KB/sec. This test, along with the Latency Test, are the most helpful on this page.

3. Browser Info

This is a JavaScript pull of the local browser details such as browser name, version, cookie status, operating system and the user-agent string.

4. IP Address

This is the IP address of the client computer as known to the server. The IP address is server-side dynamic and represents the IP address which was used to contact the server.

5. JavaScript tests

The series of JavaScript tests times loops and returns their execution time. They are essentially just memory/CPU stress tests on the client machine.

6. Organization Info

Provides general server information- organization name, time on server and client and URL.

Bonus! 

This is also a neat little secret: http(s)://<YourCRMServerURL>/home/home_debug.aspx.

It has some of the same information as the diagnostics page - Not overly helpful but can possibly come in handy troubleshooting. I find it just to be a nice, concise page for deployment information to help with documentation.

HTTP to HTTPS Redirect for CRM using Claims-Based Authentication and IFD

In some cases, our customers would like HTTP to HTTPS redirection setup for CRM when using Claims-Based Authentication and IFD. There are a number of articles and blogs out on the internet that use various methods to accomplish this (e.g. URL Rewriting) but I find that a lot are overly complicated if all you are looking to do is have the redirect functioning internally (you do not want port 80 open on your firewall anyways). Below is what I perceive to be the easiest method to achieving this – using the HTTP Redirection capabilities of IIS:

1. First we will need to install HTTP Redirection on the CRM Web Server if it is not installed already. In Server Manager, go to Add Roles and Features and choose to add the HTTP Redirection feature under the Web Server role.

2. Create two folders in a directory that you know will not be deleted. Name these folders something along the lines of “CRM Internal Redirect” and “CRM External Redirect”. Folder names and paths aren’t too important but in my example I just put them in the inetpub directory.

3. Next, go into IIS. Make sure to remove the HTTP (port 80) binding from the CRM site before proceeding with the next steps.

4. Since two URLs can be used to be accessed CRM when it is setup for Claims and IFD, we will need to create the redirects for both URLs. Begin by creating a new website in IIS for the Internal URL.

• Site Name: Something descriptive such as “CRM Internal Redirect”
• Physical Path: Browse to the CRM Internal Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your internal CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, open the CRM Deployment Manager and go to properties. Select the Web Addresses tab and you should see your internal URL.

5. Just like step 4, create another new site in IIS – this time for the external URL:

• Site Name: Something descriptive such as “CRM External Redirect”
• Physical Path: Browse to the CRM External Redirect folder created in step 2.
• Binding Type: HTTP
• IP Address: Unless you have multiple IP addresses on your server, you can leave this on All Unassigned. Otherwise, select the IP address used by CRM.
• Port: 80
• Host Name: This will your exteransl CRM URL which was configured when setting up Claims/IFD. If you are unsure of it, this is the unique name of your CRM organization followed by the domain (e.g. orgname.domain.com).

6. You should now have IIS looking similar to this:

7. Click on your Internal Redirect site and open the HTTP Redirect module:

8. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your internal URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

9. Now do the same for the External redirect site. Click on your External Redirect site and open the HTTP Redirect module. In the HTTP Redirect settings, check the box for “Redirect requests to this destination:” and then enter your external URL with HTTPS:// prefixed. Leave the other boxes unchecked and status code at 302. Apply the configuration.

That’s it. You should now be able to enter either of the CRM URLs with HTTP into a browser and have it redirected to HTTPS appropriately.

Insufficient Winsock Resources Available to Complete Socket Connection Initiation

Seemingly out of the blue, one of our customers could no longer log in to their CRM. As it appeared to be an authentication issue, I began troubleshooting on their ADFS server. Upon opening the ADFS Management Console, I was greeted with the following error message:

In the event viewer were a number of errors similar to this:

Microsoft.IdentityServer.Configuration.ReadServiceConfigFailedException: MSIS2001: Configuration service URL is not configured. ---> System.InsufficientMemoryException: Insufficient winsock resources available to complete socket connection initiation. ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 127.0.0.1:1500

Well, this was a new one for me but at least the error message actually gave a description instead of something generic. Logic dictates that if there isn’t enough of something, give it more. In this case, it appeared to me that ADFS was complaining about not having enough ephemeral ports so I figured I would increase the number available.

1. Start Registry Editor.
2. Locate the following subkey in the registry, and then click Parameters:  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. On the Edit menu, click New, and then add the following registry entry:
   Value Type: DWORD
   Value Name: MaxUserPort
   Value Data: 65534 (decimal)
4. Reboot server

After completing the above, ADFS was able to open the connection to its database and users were able to log back in to CRM. Now, it may be worthwhile to investigate exactly why ports suddenly became busy… perhaps there is a socket leak or new applications were installed… but this should at least get you up and running in the meantime.

Calling LoadLibraryEx on ISAPI filter DefaultAddonFilter.dll failed

I recently had an interesting issue with a client’s CRM 2011 to 2016 go-live. Following all of my upgrade work and initial testing, I typically like to reboot the CRM and SQL servers and then test again. In this particular instance, when the servers came back online I tested CRM again only to receive a lovely error message from Internet Explorer:

HTTP Error 500.0 – Internal Server Error

Calling LoadLibraryEx on ISAPI filter C:\Program Files\Microsoft Dynamics CRM\Server\bin\DefaultAddonFilter.dll failed.

After a fair amount of digging I noticed that the Visual C++ 2013 Redistributable (64-bit) was not present in the add/remove programs list on the CRM server. It seemingly uninstalled itself during the reboot because otherwise I would have had this issue pre-reboot. So, I went out and grabbed the installer package for Visual C++ 2013 Redistributable (64-bit) from MS (http://go.microsoft.com/fwlink/p/?LinkId=402059) and reinstalled it. After that and an IISReset, CRM began functioning as normal and everything retested everything successfully.

Unfortunately, I am at a loss as to why this happened – perhaps something to do with server patching. Also of note is that the redistributable did not remove itself from the SQL server.

Reports in CRM not working - The type initializer for 'Microsoft.Crm.LocatorService' threw an exception.

We had a customer running into an issue while trying to run reports from CRM. Of course, the error in the CRM UI was the usual rsProcessingAborted error - Cannot create a connection to Data Source. So as I always need to do, I dug into the SSRS logs. The errors I could extract from there were:

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The type initializer for 'Microsoft.Crm.LocatorService' threw an exception. ---> 

Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: Cannot load Counter Name data because an invalid index 'W3SVC_W3WP' was read from the registry.

From what I could surmise, there was something wrong with the performance counters on the server(s). I did find that Microsoft does have a KB for this error but it is in reference to update rollups on the CRM Outlook Client. Luckily for me, the fix turned out to be similar!

1. On the CRM Server, open an administrative command prompt.
2. Type “lodctr /r” and press enter.
3. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
4. Restart CRM Services and run an IISReset
5. On the SSRS Server, open an administrative command prompt.
6. Type “lodctr /r” and press enter.
7. You will then get the message ‘Info: Successfully rebuilt performance counter setting from system backup store’.
8. Restart SSRS Services.

After following the above process, reports in CRM began working again. The commands run rebuild the performance counters.

An attempt was made to access a socket in a way forbidden by its access permissions

When setting up the Microsoft Dynamics Email Router, there can be a lot of obstacles in getting emails to send and receive properly. One of which turns out to be McAfee VirusScan. If you get the following error when testing the connection on the Outgoing profile, you may want to see if McAfee (or any other virus protection software) is running on the server:

Outgoing Status: Failure – An error occurred while checking the connection to the email server {smtp server name}. An attempt was made to access a socket in a way forbidden by its access permissions.

In order to fix this, right click on the McAfee icon in the system tray and open up the VirusScan Console.

In the VirusScan Console, double-click “Access Protection”.

Under Categories, click on “Anti-virus Standard Protection” and then check to see if there are any checkmarks next to “Prevent mass mailing worms from sending mail”. If there are, simply uncheck them.

Click OK and close out of the VirusScan Console. Open the email router and try testing again. You should now be seeing green!

Report Server cannot load the SQLPDW extension

While troubleshooting SSRS issues, you may run into the following error messages in logs and traces:

Report Server (MSSQLSERVER) cannot load the SQLPDW extension.

Report Server (MSSQLSERVER) cannot load the TERADATA extension.

On numerous occasions I have seen both customers and colleagues believe that these errors were the source of their problems. After which, the wild goose chase to resolve these errors ensues. I am here to tell you that it is highly unlikely that they are related to your problem at hand (especially if you are only using SSRS for CRM reporting).

These errors occur because the Teradata and SQLPDW extensions are registered in the Reporting Services configuration file by default but the assemblies for these extensions do not get installed in the same manner. Typically these extensions are used with big data and business analytics.

If the errors do not bother you, simply ignore them and track down other error messages. However, if you are the type that prefers clean logs you can take the following steps to alleviate the error messages from being logged:

• Open the rsreportserver.config file. This can usually be found in \Program Files\Microsoft SQL Server\MSRSXX_X.MSSQLSERVER\Reporting Services\ReportServer.
• In this config file, simply search for and comment out the entries related to SQLPDW and TERADATA. There should be 2 entries for SQLPDW and 3 for TERADATA. Below is an example:

<!--Extension Name="SQLPDW" Type="Microsoft.ReportingServices.DataExtensions.SqlDwConnectionWrapper,Microsoft.ReportingServices.DataExtensions"/-->

<!--Extension Name="SQLPDW" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.MSSQLADW.MSSqlAdwSQCommand,Microsoft.ReportingServices.SemanticQueryEngine">

    <Configuration>

        <EnableMathOpCasting>False</EnableMathOpCasting>

    </Configuration>

</Extension-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.DataExtensions.TeradataConnectionWrapper,Microsoft.ReportingServices.DataExtensions"/-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.Teradata.TdSqlSQCommand,Microsoft.ReportingServices.SemanticQueryEngine">

    <Configuration>

        <EnableMathOpCasting>True</EnableMathOpCasting>

        <ReplaceFunctionName>oREPLACE</ReplaceFunctionName>

    </Configuration>

</Extension-->

<!--Extension Name="TERADATA" Type="Microsoft.ReportingServices.SemanticQueryEngine.Sql.Teradata.TdSqlModelGenerator,Microsoft.ReportingServices.SemanticQueryEngine"/-->

• Save and close the config file.
• Restart the SSRS service.

The private key does not support the exchange KeySpec.

When using Active Directory Federation Services (ADFS) for claims-based authentication with Dynamics CRM, one of the requirements is a SSL certificate. If a new certificate has to be procured, it is imperative to make sure the certificate request (CSR) is being generated with the correct KeySpec, if required.

There are two options for KeySpec:

• “1” or “XCN_AT_KEYEXCHANGE” 
• “2” or “XCN_AT_SIGNATURE”

With ADFS, the certificate needs to support key exchange so the required KeySpec is "1" or "XCN_AT_KEYEXCHANGE". If the other option is chosen, you may end up with failures when trying to log in to CRM and will see errors in the event viewer on the ADFS server similar to below.

Exception type: NotSupportedException 

    Exception message: The private key does not support the exchange KeySpec.

   at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.DecryptKey(String algorithm, Byte[] keyData)

   at System.IdentityModel.Selectors.SecurityTokenResolver.SimpleTokenResolver.TryResolveSecurityKeyCore(SecurityKeyIdentifierClause keyIdentifierClause, SecurityKey& key)

While the certificate will still appear valid using "XCN_AT_SIGNATURE" and will secure the webpages, authentication will not succeed with ADFS so the certificate will need to be regenerated. 

CRM Reporting Issue - The system cannot contact a domain controller to service the authentication request

Had a case come into my queue indicating a client’s reports in their production CRM were not working so I logged in and began taking a look. At first glance, I was seeing the generic “Reporting Error – The report cannot be displayed. (rsProcessingAborted)” that we see for 90% of reporting issues so I proceeded to the SSRS logs where I was greeted with the following error:

System.Runtime.InteropServices.COMException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350) ---> Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350)

Interesting to say the least and something I had never seen before. My inclination was that a domain controller was down or somehow the connection to AD was severed (but then again, how was I able to log on the server). I ran a set command to check the logon sever and was able to ping it fine, along with all of the other DCs. Next stop was at the DNS/Gateway settings on the NIC – compared everything to another SQL/SSRS server that was working fine. Again, nothing seemed out of place.
Off to the handy dandy event viewer! Immediately I noticed a flood of errors indicating that the automatic backups and log shipping were failing. Amidst those errors I found a familiar message:

Whatever was causing the reporting issue was also causing the backups to fail. I made my way through the events until I reached the beginning of the flood and there it was, in all its glory:

BackupDiskFile::CreateMedia: Backup device '\\goxsaXXXX\D$\MSSQL11.XXXX\MSSQL\Backup\Log_Shipping\XXXX_20160524043001.trn' failed to create. Operating system error 1331(This user can't sign in because this account is currently disabled.).

Easy enough – the SQLAdmin account running SSRS (and the other SQL services) cannot authenticate because it’s disabled.

 

Had the customer check AD and sure enough, the account was disabled. Once enabled, reports began working again. 

In all, this was a very strange error message for the actual issue. It could have really led us on a wild goose chase into the networking side of things. Probably saved 2-3 hours by reading through the error logs and being thorough in investigation before going back to the customer with the wrong answer. Moral of the story: do your homework.

Use SSL for Outgoing Connection / Incoming Connection Radio Buttons Greyed Out for CRM Server Side Sync POP3-SMTP Profile

If you need to create a Server Side Sync profile for your POP3 and/or SMTP servers without using SSL for your on-premise CRM 2013, 2015, or 2016 environments you may notice that you are unable to change the option to not use SSL. The radio buttons are simply greyed out and stuck set to “Yes”. This can be problematic if you want to run over basic HTTP.

Luckily, these buttons can be enabled via the following PowerShell commands run on the CRM server:

Add-PSSnapin Microsoft.Crm.Powershell
$setting=Get-CrmSetting ServerSideSyncEmailSettings
$setting.AllowNonSSLEmail=$True
Set-CrmSetting $setting
Get-CrmSetting -SettingType ServerSideSyncEmailsettings
Exit

After the commands are complete, refresh your browser and the buttons should now be active.

A Server Certificate Could Not Be Validated for URL - SQL Server Reporting Services 2012

If you ever get the following error during the system checks of the CRM installer, check to see if you can even access the Report Manager URL from the SSRS server:

A Server Certificate Could Not Be Validated for URL: http://reportservername/ReportServer 

When checking the Report Manager URL (http://reportservername/Reports) using HTTP - not HTTPS - you may see the following error:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

You will probably also get a certificate error accessing the Report Service URL (http://reportservername/ReportServer) which will end up resulting in a 404 error screen if you proceed. So why is SSRS looking for SSL/TLS when using HTTP?

The answer is rather simple – it’s configured to do so. Here is how to change that:

1. Go to the Report Server directory. Typically it is {Drive}:\MSRS11\Reporting Services\ReportServer

2. Open the rsreportserver.xml file in notepad or other editing software.

3. In this configuration file, look for or run a find for SecureConnectionLevel. It should be about 12 lines from the top. You will likely see this key set to a value of 2.

4. Change the value to 0 so that the line reads:
<Add Key=”SecureConnectionLevel” Value=”0” />

5. Save and close the file. Restart the SQL Reporting Services service and try accessing the Report Manager URL once more. You should now get the site as expected over HTTP.

6. Go back to your CRM server and re-run the installation wizard.

Of course, the alternative to going this route is to actually provide SSRS with an SSL certificate and access it over HTTPS only but some may not want the added hassle or cost.