UPS problems post SharePoint 2010 SP1 and June 2011 CU Refresh

Today on a client’s test environment I installed SharePoint 2010 Service Pack 1, and the June 2011 Cumulative Update Refresh.  The process went fairly well, and completed our test plan with success.  The test plan failed to include any of the service applications, and when I went to try and do a user profile sync, it failed.

I checked the service applications, and noticed that the User Profile Synchronization Service had stopped.  I tried starting it up, but after awhile it just stopped again.  I peeked at the event viewer which revealed this error:

Cannot open the FIM Synchronization Service database because the database schema version in existing database does not match the required version.

This seemed quite odd, there were no errors or warnings generated from the updates, it appeared that one of the UPS databases didn’t update properly.  I had a look in the upgrade log and found this:

[OWSTIMER] [SPUpgradeSession] [DEBUG] [10/12/2011 12:31:22 PM]: NeedsUpgrade [SynchronizationDatabase Name=SP_SyncDB] returned: False.

I imagined that this meant that the SyncDB didn’t need to be upgraded, which seemed peculiar.  I did some research online and most recommendations were to remove and re-provision the user profile service.  I wasn’t really keen on that idea.

I then came across this forum entry and tuccitown’s response gave me the idea – that the account I was using for the UPS, the spfarm account, didn’t have permissions.  I then added the spfarm account to the local administrators group, restarted the server (for good measure, a restart of all services probably would have also worked) and I was able to start the user profile synchronization service.

Interestingly, I had a look at the ULS logs after the service was provisioned, and saw this:

UpgradeMetaverseAndManagementAgent(): Indexing MV distionguishedName. It may take a long time.

So it appears that this service post-update tries to upgrade its components, and because it didn’t have permissions it failed.  And yes, it really said distionguisedName.

A second problem arose shortly after.  I attempted an incremental sync, and was instantly greeted by an error, and in the ULS it showed:

System.IO.FileLoadException: The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

I suspected that as the user profile sync service had just upgraded, the binaries must be out of whack.  Another reboot and I was able to kick off an incremental sync.

To wrap things up, I removed the spfarm account from the local administrators group, gave it one more reboot and everything continued to work!

SSRS Connection Failure

I had setup a test environment with SQL Server Reporting Services in integrated mode with SharePoint 2010.  My initial tests had confirmed everything had been setup correctly, including Kerberos.  I had even setup a few test reports to a SQL database and everything was fine.  I then rolled in a copy of production data and the reports just didn’t want to work.

I attempted to modify the data connection which was looking at a SharePoint list, but wouldn’t connect successfully, instead coming up with the error The report server has encountered a configuration error.

clip_image002

I checked the SSRS logs (default location: C:\Program Files\Microsoft SQL Server\MSRS10_50.MSSQLSERVER\Reporting Services\LogFiles) and saw this in the text:

Call to TestConnectForDataSourceDefinitionAction().

ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.LogonFailedException: Logon attempt for user ‘tst.sprsunattended.svc‘ failed., Microsoft.ReportingServices.Diagnostics.Utilities.LogonFailedException: Log on failed. Ensure the user name and password are correct. —> System.ComponentModel.Win32Exception: Logon failure: unknown user name or bad password

I initially thought this was permissions related, so in an effort to give this account full access to the web application, I discovered something peculiar:

 image

The short account name had been truncated to the first 20 characters!  I had seen this before in other scenarios, and this is a short explanation on TechNet:

“The SAM-Account-Name attribute (also known as the pre–Windows 2000 user logon name) is limited to 256 characters in the schema. However, for the purpose of backward compatibility the limit is 20 characters.”

While I had checked the SharePoint service accounts, I didn’t pay too much attention to those for SSRS.  I’m just thinking to Buy Windows 7 Ultimate.
Anyways tho, i updated the Execution Account details in the Reporting Services Configuration Manager to the 20 character account name and pressed Apply.

image

Without a service restart or an IIS reset, I then proceeded back to the data source and retested the connection, this time with success!

clip_image002[5]

Multiple controls with the same ID ‘xxxx’ were found.

A client rang up last week with a problem with their SharePoint environment.  On their publishing site, every time they went to view the Pages library within Manage Content & Structure they got an error.  Within the ULS logs, this was found:

Unexpected     System.Web.HttpException: Multiple controls with the same ID ‘xxxx’ were found. FindControl requires that controls have unique IDs.

To fix this, I went to SharePoint designer and edited the client’s site.  On the Properties of the Pages library, I noticed there were two views with the same name of xxxx!

I simply renamed one of the views, and I could then see the pages under Manage Content & Structure.  I’m not sure how the client did it, and why SharePoint didn’t prevent this from happening.

Fixing missing server side dependencies: [MissingFeature]

Have you ever seen this while looking through the SharePoint Health Analyzer on your farm?

[MissingFeature] Database [Z_PRODRESTORE] has reference(s) to a missing feature: Id = [20e9ecab-2d22-41eb-892d-933afc32793e]. The feature with Id 20e9ecab-2d22-41eb-892d-933afc32793e is referenced in the database [Z_PRODRESTORE], but is not installed on the current farm. The missing feature may cause upgrade to fail. Please install any solution which contains the feature and restart upgrade if necessary.

It is certainly a bit cryptic.  SharePoint has a reference to a feature that doesn’t exist anymore, and its not too keen to give you any clues about it.  This tends to happen when a feature has been removed from the farm, but not deactivated from the sites that may have been using it.  I encountered this recently with five of these such errors, and here is my approach:

Try and work out what the feature is

This doesn’t work in every case, but you can give it a shot, run this PowerShell command:

(Get-SPSite http://site ).Features

You may get some information out of the Properties property.  This helped me identify two InfoPath forms that I had removed from Central Admin, but hadn’t deactivated on a site collection.  Simply re-adding the forms, then deactivating the forms in the offending site collection made two of these errors go away.

The problem with this approach is that there may not be information available in the Properties property, and you may need to check every site collection to find the missing feature. 

Solution 1 – the easy way

Thankfully there is a nice, elegant solution put together by Phil Childs:  Removing features from a content database in SharePoint 2010 using PowerShell

Phil’s script will check all of the sites in a content database and programmatically remove the feature references – I’ve run this and it works as advertised.  If you can, run this on a non-production environment first!

Solution 2 – the hard way

I also found another way of achieving the same result and inspired by a comment left on a forum somewhere (and I cannot find the link!).  Essentially their suggestion was to create a feature with the same feature ID as per the error, install it onto the farm, deactivate the feature, and remove the dummy feature from the farm.

I tested this approach and I must say this did work!  Chapter 13 of Professional SharePoint 2010 Administration gave an excellent example on how to create a dummy feature and solution package.  Same note as above about running this on a test system before trying it on production.

Overall

If you can’t work out the missing feature and fix it easily, then go with Phil’s method per Solution 1.  If anyone is crazy enough to want to know more information about how to achieve Solution 2 and more info on the purest forskolin , drop me a comment below.

The Perfect SharePoint Upgrade

Upgrading WSS or MOSS to SharePoint 2010 at times can be an interesting challenge.  Microsoft gives plenty of guidance on how to perform an upgrade, either in-place or with a database attach method.  Having done a few upgrades now for different clients, I’ve recently completed one that could be described as the perfect upgrade, and thought I’d my experience.

The client already had a SharePoint 2010 environment, and a number of other legacy WSS and MOSS installations.  One of their WSS sites had previously been attempted by someone else but had failed when they tried to go live.  This meant that there was already some negative notions from the business that trying this again could be a problem. 

The only way forward was to start afresh.  Not knowing all the details from the previous attempt meant that I would need to get back to basics.  I decided that the best way to cover off all the issues would be to complete a trial upgrade, and have it thoroughly tested before attempting it for real.

We also managed to get great business buy-in to help us ensure they were happy that everything was working as expected.  The business representative even devised a very comprehensive test plan.

From performing the trial upgrade, we found a whole heap of problems, including:

  • Network communication problems
  • Authentication issues
  • Problems around search
  • Excel service issues
  • Absolute URLs
  • User understanding

Methodically, I worked through all of the issues and one-by-one solved these such that the business was happy to sign-off of the trial upgrade.  Surprisingly, there were not any problems with the actual SharePoint upgrade process itself!

I was very confident that the upgrade would be fairly smooth sailing.  I had made notes with every single step I performed to get the site ready for acceptance.  I had even made some timing notes so I could expect when things were happening. 

When it came time to perform the upgrade for real and go-live, everything worked perfectly.  There were only a handful of minor issues raised after the upgrade, all cosmetic and easily fixed.

There are some takeaways from this particular project:

  1. Have a good relationship with your client, and set their expectations that this is going to be a potentially complex process.
  2. Perform a dry run, and get business buy in for testing – including test plans.
  3. Generate a procedure document as part of the dry run, and document everything you do to make the dry run successful.  Having a re-executable plan should make the upgrade process a no-brainer.
  4. Communicate to everyone involved and keep them in the loop when appropriate.
  5. Be available after the upgrade to assist with any problems that may arise..

Reflecting on this, it is clear that process over technology is what helped make this a perfect SharePoint upgrade.