CRM Reporting Issue - The system cannot contact a domain controller to service the authentication request

Had a case come into my queue indicating a client’s reports in their production CRM were not working so I logged in and began taking a look. At first glance, I was seeing the generic “Reporting Error – The report cannot be displayed. (rsProcessingAborted)” that we see for 90% of reporting issues so I proceeded to the SSRS logs where I was greeted with the following error:

System.Runtime.InteropServices.COMException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350) ---> Microsoft.Crm.Reporting.DataExtensionShim.Common.ReportExecutionException: The system cannot contact a domain controller to service the authentication request. Please try again later. (Exception from HRESULT: 0x80090350)

Interesting to say the least and something I had never seen before. My inclination was that a domain controller was down or somehow the connection to AD was severed (but then again, how was I able to log on the server). I ran a set command to check the logon sever and was able to ping it fine, along with all of the other DCs. Next stop was at the DNS/Gateway settings on the NIC – compared everything to another SQL/SSRS server that was working fine. Again, nothing seemed out of place.
Off to the handy dandy event viewer! Immediately I noticed a flood of errors indicating that the automatic backups and log shipping were failing. Amidst those errors I found a familiar message:

Whatever was causing the reporting issue was also causing the backups to fail. I made my way through the events until I reached the beginning of the flood and there it was, in all its glory:

BackupDiskFile::CreateMedia: Backup device '\\goxsaXXXX\D$\MSSQL11.XXXX\MSSQL\Backup\Log_Shipping\XXXX_20160524043001.trn' failed to create. Operating system error 1331(This user can't sign in because this account is currently disabled.).

Easy enough – the SQLAdmin account running SSRS (and the other SQL services) cannot authenticate because it’s disabled.

 

Had the customer check AD and sure enough, the account was disabled. Once enabled, reports began working again. 

In all, this was a very strange error message for the actual issue. It could have really led us on a wild goose chase into the networking side of things. Probably saved 2-3 hours by reading through the error logs and being thorough in investigation before going back to the customer with the wrong answer. Moral of the story: do your homework.

ADFS Logon Page Loop Issue

I recently setup CRM 2015 and ADFS 3.0 (the version that comes on Server 2012 R2, aka ADFS 2.2) on new servers for a customer. Everything configured fine and initial tests proved successful when logging in with the domain admin account used to set everything up. However, I quickly found an issue when other users tried to access CRM via the external URL. At the ADFS login page, a user would enter his or her credentials as usual and try to login but rather than giving a 302 redirect back to CRM for access, it redirected back to the ADFS login page. This presented no errors on screen or in the CRM event viewer – it was as if I never tried logging in. If I purposefully entered invalid credentials, it provided the error regarding incorrect user ID or password so I knew authentication against AD was taking place successfully and that something was wrong with the passing of the token.  Also of note was the fact that despite the external URL not working, all users were able to access CRM just fine using the internal (pass-through auth) URL.

Nothing appeared in the ADFS Admin event viewer logs but upon closer inspection, the Security log in the event viewer on the ADFS server was loading up with Audit Failure notifications – Event ID 4625. The failure reason indicated “Unknown user name or bad password” for the ADFS service account. As any logical person would assume, I figured the account was locked out, the password expired, or I entered invalid credentials during setup. Unfortunately, upon resetting the password in Active Directory, the audit failures persisted.

After some digging around, I uncovered that adding the ADFS service account to the Windows Authorization Access Group in Active Directory was the resolution. The members of this group have access to the computed tokenGroupsGlobalAndUniversal attribute on User objects. I added the account to the group, restarted the ADFS service, and all users were then able to access CRM via the external URL as expected!

UPDATE (1/27/16) - While adding the ADFS service account to the Windows Authorization Access Group fixed the issue for most users accessing CRM, it managed to rear its ugly head for a handful of users again. The fix this time was similar - I added the account to the Pre-Windows 2000 Compatible Access Group. Users in this group have read access over all users in the domain. Typically, the 'Everyone' group is present in here by default so this is not usually an issue but this customer in particular removed it. So to conclude, make sure group membership is set on both of these groups if you are experiencing the loop issue.