Discovering that the User Profile Synchronization Service will not start is not an uncommon problem for anyone who has spent time working with the User Profile Service application in SharePoint 2010. I have a client that relies on the serivce application to be available, and for the synch process to work too. This was not the first time we had run into issues with the user profile synchronization service not starting. Usually a cursory review of Spence Harbar’s article “Stuck on Starting”: Common Issues with SharePoint Server 2010 User Profile Synchronization would sort the problem right out, but not this time.
Thankfully this was on a test environment, but we had recently performed some changes and there was a strong chance that performing the same changes could break production when it was scheduled in a few weeks time. I had some time on my side to work this out, but did not think it was going to take as long as it did.
Symptoms
First, a bit of background on this client’s environment. In addition to SQL backups, there are SharePoint farm backups performed on a nightly basis (yes, even on a test system). When a SharePoint farm back up is performed (using the PowerShell cmdlet Backup-SPFarm) it has to unprovision the user profile synchronization service and re-provision it once it has completed. Every 2-3 weeks the re-provisioning does not work. The solution is usually simple, start it up under Services on Server and everything is fine. Unable to come up with a reliable explanation as to why it was happening, I wrote a short PowerShell script to check on its status once a day so at least it could be acted on. This script was working like a charm, but this time was different.
Usually when I hit start on the user profile synchronization service and enter my farm account password, I wait a few moments and everything is fine. This time, the service was back in a stopped state almost instaneously. Not to be fazed, I pulled out the ULS logs but to my surprise there were no log entries! Nothing, nada, zip, nil – even on verbose!. Not any indication that I had even asked for it to start up. During my debugging steps I got it to tell me that it had successfully un-provisioned the user profile synchronization service, but that was about it. A very perculiar event, I had really expected to see something there. What made this more mysterious was that I couldn’t see anything in the Windows event logs either.
Diagnosis
At this stage the most obvious thing to do was delete the user profile service application and recreate it, keeping the Profile and Social databases. In hindsight I wish I had made a backup of the Sync database as ultimately I don’t think it was to blame, and too many days had passed while I was trying to work this out between other work. Unfortunately still, the user profile synchronization service will not start. As a summary, here is what I had tried:
- Provisioning the user profile syncronization service with PowerShell (taken from the AutoSPInstaller scripts)
- Rebooting the server
- Checking the farm account was a local administrator
- Checking for failed timer jobs
- Re-enabling NetBIOS over TCP/IP (this was one of the changes I had made to the system) then removing the user profile service, rebooting and re-creating the service application.
- Checking everything on Spence’s article “Stuck on Starting”: Common Issues with SharePoint Server 2010 User Profile Synchronization, then checking it twice again.
- Monitoring the server with Process Monitor in some hope to see an issue somewhere
I was running out of options, and with a bit of dispair I ran the SharePoint Products Configuration Wizard on all servers in the farm. To my astonishment, within a couple of minutes of the SharePoint Products Configuration Wizard finishing, the user profile synchronization service had started all by itself!
A review of the configration wizard logs shows this:
A NotifyEventArgs event received in TaskNotifyEventHandler. Task: secureresources NotifyEventArgs.EventCriticalityType: Successfully secured resource 'C:Program FilesMicrosoft Office Servers14.0\Synchronization Service' of type 'Directory' by granting security group 'NetworkService' 'Read, Write, Execute' permissions . Message: info
The only takeaway from this is that there were incorrect permissions on the Synchronization Service folder. As part of its steps, the SharePoint Products Configuration Wizard “Performs resource security enforcement on the server. For example, security is enforced on files, folders, and registry keys”. More information can be found on this TechNet article.
For reference, the permissions that were set on the C:Program FilesMicrosoft Office Servers14.0Synchronization Service folder were:
- SYSTEM – Full control
- WSS_ADMIN_WPG – Full control
- Administrators – Full control
- WSS_WPG – Read & execute, List folder contents, Read
- Network Service – “Special” – All except Full control, Change permissions, Take ownership
However, not all was quite right. When I went to manage the user profile service application, I was missing some information:
As well as getting ULS log entries like:
UserProfileServiceUserStatisticsWebPart:LoadControl failed, Exception: System.IO.FileLoadException: The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
UserProfileServiceAudienceStatisticsWebPart:LoadControl failed, Exception: System.IO.FileLoadException: The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
UserProfileServiceImportStatisticsWebPart:LoadControl failed, Exception: System.IO.FileLoadException: The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
I tried deleting and then recreating the user profile service to no avail. I was still getting these error messages. Process Monitor was confirming that the files could not be found.
Thinking that perhaps there was some strange cached copy of a file that IIS was hanging onto, I performed an iisreset and then everything was working again.
Post Moretom
The problem that the user profile synchronization service will not start is a most interesting and perculiar one. The only major change that I had recently performed was the installation of Project Server. I certainly would not have expected that to cause any issues, particularly around permissions on the Syncronization Service folder. Nonetheless the service is working again. I was a little surprised that I hadn’t come across anyone else with this particular issue. The absense of logs really put me in the dark as I spent a bit of time on this over the course of a week trying to nut it out. Hopefully this will help someone else out too.