UPS problems post SharePoint 2010 SP1 and June 2011 CU Refresh

Today on a client’s test environment I installed SharePoint 2010 Service Pack 1, and the June 2011 Cumulative Update Refresh.  The process went fairly well, and completed our test plan with success.  The test plan failed to include any of the service applications, and when I went to try and do a user profile sync, it failed.

I checked the service applications, and noticed that the User Profile Synchronization Service had stopped.  I tried starting it up, but after awhile it just stopped again.  I peeked at the event viewer which revealed this error:

Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.

This seemed quite odd, there were no errors or warnings generated from the updates, it appeared that one of the UPS databases didn’t update properly.  I had a look in the upgrade log and found this:

[OWSTIMER] [SPUpgradeSession] [DEBUG] [10/12/2011 12:31:22 PM]: NeedsUpgrade [SynchronizationDatabase Name=SP_SyncDB] returned: False.

I imagined that this meant that the SyncDB didn’t need to be upgraded, which seemed peculiar.  I did some research online and most recommendations were to remove and re-provision the user profile service.  I wasn’t really keen on that idea.

I then came across this forum entry and tuccitown’s response gave me the idea – that the account I was using for the UPS, the spfarm account, didn’t have permissions.  I then added the spfarm account to the local administrators group, restarted the server (for good measure, a restart of all services probably would have also worked) and I was able to start the user profile synchronization service.

Interestingly, I had a look at the ULS logs after the service was provisioned, and saw this:

UpgradeMetaverseAndManagementAgent(): Indexing MV distionguishedName. It may take a long time.

So it appears that this service post-update tries to upgrade its components, and because it didn’t have permissions it failed.  And yes, it really said distionguisedName.

A second problem arose shortly after.  I attempted an incremental sync, and was instantly greeted by an error, and in the ULS it showed:

System.IO.FileLoadException: The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

I suspected that as the user profile sync service had just upgraded, the binaries must be out of whack.  Another reboot and I was able to kick off an incremental sync.

To wrap things up, I removed the spfarm account from the local administrators group, gave it one more reboot and everything continued to work!