Migrating from MOSS 2007 to SharePoint 2013

Technology needs upgrade and SharePoint is no exception infact newer versions of SharePoint are usually released in 3 year cycles. It is essential for businesses to upgrade to newer versions if they want to continuously leverage the new technological advancements in the IT industry.

When it comes to migrating SharePoint farms between different versions it has always been both interesting and challenging. The level of complexity in a SharePoint migration is proportional to the level of customiszations applied to the environment. All most all the SharePoint environments that I have worked on are customized to different levels of complexity, some are heavily customized with many complex custom workflows (SharePoint/.Net workflows, K2 and Nintex) and integrated with external systems via custom connectors or Business Connectivity Services (BCS), etc… These type of complex environments are often challenging to migrate and require proper planning and a phased approach.

In this article I will briefly talk about the some key aspects that are worth considering when migrating from MOSS 2007 to SharePoint 2013. I like to cover the migration from MOSS 2007 to SharePoint 2013 because this will include migration from SharePoint 2010 to SharePoint 2013 as well.

1. Migrating Content has to be a 2 step process, if no 3^rd party tools are used. I.e. Upgrade MOSS to a temporary 2010 environment via database attach and then move the content to SP2013 using the same approach. 3^rd party tools such as AvePoint can also be utilized. Eventhough most of the content can be migrated ‘as is’, it’s important to use this opportunity to clean up the content before moving to SP2013.
2. Migrating code, this has to be done manually and it’s a key component in the process of migration. The existing code needs to be analyzed and evaluated in detail before deciding on whether to build the solution from scratch or to do a code migration. You must look at code-based customizations, whether custom-built, in-house code, or third party tools, add-ins, or web parts, and determine what will and won’t work in the target version of SharePoint.
3. The odds are pretty good that some of your customizations won’t work, and you will have to decide whether the functionality those customizations provide is still necessary, or whether the customization can be left behind. If the customization is necessary, but incompatible, you will have to determine the cost and effort required to remediate the customization so that it works with your target version. Most likely the core source code might be reusable so that you can avoid reinventing the wheel and thus reduce cost and effort.
4. The effort that I have just mentioned is the same whether you are going to 2010 or 2013. And, in fact, it’s extremely likely that the remediation you undertake will result in the customization being ready for either version.
5. It’s not the target version of SharePoint that is the issue rather it is the mess you have in your 2007 environment that is the real problem. Once you solve that, you are ready to migrate to either 2010 or 2013.
6.  Permission and Authentication model, since SP2013 uses claims authentication by default and classic authentication is soon to be deprecated, special attention needs to be given to the new security model when architecting the whole solution. In my view point it’s better to do the permission migration separate to content and code migration, probably after the code and content migration which will help you to isolate and identify any specific permission related issues that may occur.

Some of the other general things that you might have to consider in successful migration and adaption of the new SP2013 are

1. End user training and adoption
2. Administrator training and familiarity of the system
3. IT gets familiar with 2013 before making the decision.

If you move users from 2007 to 2010, that’s going to require training—a significant amount of training, in fact. SharePoint 2010 has rough edges in discoverability, and gaps in usability, that were solved in SharePoint 2013. That means that your users will require more training going from 2007 to 2010, than from 2007 to 2013. You’re also going to have to train users twice. You’ll have to train them to use 2010, and then when you do finally move to SharePoint 2013, you will have to train them again because the user interface and user experience is quite different in each version of SharePoint.

Taxonomy Field Values are not visible for Users other than Site Administrators

Background

Recently in one of the SharePoint 2010 implementations we faced the interesting issue of Taxonomy field values in Lists and Pages were not visible for users except Site Administrators.

 After spending sometime investigating I found out that in each site collection there is a hidden taxonomy list (TaxonomyHiddenList) which has all the taxonomy field values from the Manage Metadata Terms Database. This list will be synchronized hourly with the Manage Metadata database using the Taxonomy Update Scheduler  timer job.

 Reason

Users don’t have permission to access TaxonomyHiddenList

Solution

Give  ‘NT AUTHORITY\authenticated users’  read permission to the hidden list located at <sitecollection>/lists/TaxonomyHiddenList

Crawl issue with SharePoint 2010/FAST Search Server 2010

Backgroud:

During one of the recent FAST implementations, I faced another interesting issue with crawling internet sites and it is confirmed by Microsoft to be a product (SharePoint 2010) issue.  The details of my findings are given below.

Problem:

In SharePoint server 2010, you received following error when you try to crawl an anonymous + NTLM enabled web site (non-SharePoint site)

 “Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has “Full Read” permissions on the SharePoint Web Application being crawled.”

Cause:

It is confirmed to be  a product issue. 

Such issue can occur when you ever set a default content access account. In such scenario, Search will incorrectly pick up the default account and assume the site to crawl is NTLM and it fails to crawl the site using NTLM, it does not even fall back to anonymous.

Resolution:

MS Product Group has confirmed this to be product issue. However we were told, since changes required to fix the issue is big, they will more consider fixing it in future release of the product. 

Workarounds:

The below 2 workarounds are available to this issue, either of them will solve the issue

I.    Remove and re-provision a new search service application ( I know its a pain spcially if you have already set up hunderds of crawl rules/etc…). After the new search service application  is provisioned, DO NOT set or change the default content access account. (unfortunately, there is no public interface that can be used to take away the default content access account once it is there). Even retyping the password for the same default content access account will set the authentication type to NTLM as explained and will trigger the issue.

OR

II.    Create a crawl rule for the start address of the site and configure it to use cookie. 

1. Create a dummy text file, i.e. dummy_cooki.txt, to contain some dummy text:   E.g. Test: testing
2. Create a crawl rule to use cookie to crawl, and locate the cookie created in step 1 
3. Specify whatever Error.aspx (it is more a dummy or filler, and is not really used)
4.  Crawl again

 The idea is to configure SharePoint to crawl the site using cookie (from some degree, this can also be interpreted as anonymous). As long as the target site does not have the extra dummy cookie (many web sites don’t specifically look for cookies other than those sent by themselves), crawl can succeed.

Have a nice Day !!!

FAST Post Setup Configuration Failed

In one of my recent FAST implementations to a large Government Organization in Western Australia, I faced numerous interesting issues and would like to share them with the SharePoint communitiy in my next few articles.

In my view point, even though FAST is a very powerful and useful component when it comes to integration with SharePoint it need some major improvements, fixes and specially better error handling.

 

Issue 1: “Post Configuration of FAST failed”

Error/Exception in FAST install Logs:

Utility.WriteException – Exception -  : Exception – Microsoft.SharePoint.Search.Extended.Installer.Mahasen.Common.Utility.ExecutorException: Command returned non-zero         at Microsoft.SharePoint.Search.Extended.Installer.Mahasen.Common.Utility.Executor.CheckReturnCode(Int32 returnCode)
at Microsoft.SharePoint.Search.Extended.Installer.Mahasen.Common.Utility.Executor.ExecuteCommand(String command, String arguments, String workingDirectory, Int32 timeout, ILogWriter logWriter)

FAST farm Post Configuration failed with the above error message which is very generic and makes no sense, which forced us to reflect some dlls to find out the root cause of the issue.

The reason for the above error was the failure of  the command ”d:\fastsearch\bin\msdtcsetup -admin”.

Again we dont have any clue of what prevents msdtcsetup from running therefore next we used NetMon (MS Network monitor tool) to analyse and further trouble shoot the network traffic. Netmon was very useful in finding out the exact root cause which was MSDC service permission was limited to Local Admin, LOCAL and INTERACTIVE by a group policy.

Since we found out the extract root cause the resolution was pretty simple, which is to add “Network Services” to the MSDC service permission group along with the other accounts and set the service to start up Automicatically.

As I mentioned earlier FAST needs better error handling…………..:(

 

Have a nice day !!!!

Read Issue 2:

Re-link Managed Metadata – Invalid Terms in Taxonomy Field

 In  one of my recent SharePoint 2010 implementations, I faced an interesting issue during the content migration from one of the Staging environments to UAT with regard to Managed Metadata fields. The problem was with taxonomy fields /Managed Metadata fields (custom fields) in publishing pages, where they became invalid after the migration. Even re-linking the fields with the terms sets both manually and programatically (using a feature receiver)  didn’t help and the fields remained invalid. But it was possible to manually link the fields with the terms in each page after a page edit. This is not a practicall approach in an environment where thousands of pages are already created.

The cause of the problem was related with IDs. IDs for terms in a term store is unique to each environment therefore IDs in the Staging environment are different to the ones in the new Prod environment, but the taxonomy fields remain to be linked with the old IDs even after the migration. Therefore the fields needs to be re-linked with the new IDs. Thanks to Chris for the intial solution using powershell.

Chris built a powershell script to update the fields in lists with new IDs, I have customized it to work with publishing pages.

Below is the code for the script and I recommend to run it in test mode first to verify that there are no errors/exceptions.

# Note: Before executing the script change the following parameters values at the end of the file
#     -webAddress: Root Level Site collection on which the Taxonomy Fields needs to be relinked
#     -test: If true, the script will display the output of Fields and metadatas that will be affected, the script wont do any changes to the evironment.
#      Its recommened to run the script in test mode for the first time and verify the changes that will be made.
#      Values are either $true OR $false
#  -recurse: Will determine whether the Taxonomy Fields in subsites needs to be linked as well. Recommended to set it to True
#      Values are either $true OR $false
#
#  E.g. @ UpdateTaxonomyFields -webAddress http://xxxxxx -test:$false -recurse:$true
#  

 [Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint") 

 [Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Taxonomy") 

 function UpdateTaxonomyFields 

 { 

     param 

     ( 

         [string]$webAddress = "", 

         [switch]$recurse, 

         [switch]$test

     ) 

     $web = Get-SPWeb $webAddress

     $taxonomySession = Get-SPTaxonomySession -Site $web.Site 

     $taxonomyType = [Type]::GetType("Microsoft.SharePoint.Taxonomy.TaxonomyField, Microsoft.SharePoint.Taxonomy, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c", $true) 

      
  $web.AllowUnsafeUpdates = $true;
     foreach ($list in $web.Lists) 

     { 
# Iterate only through pages library 
   if ($list.Title -eq "Pages")
  {

         try 

         { 

             foreach ($field in $list.Fields) 

             { 
                 $query = New-Object Microsoft.SharePoint.SPQuery 

                 $listItems = $list.GetItems($query) 

                 if ($field.GetType() -eq $taxonomyType) 

                 { 

                     Write-Host "Entering List '", $list.Title, "' Located At:", $list.DefaultViewUrl, "On Field:", $field.Title 

                     if (-not $field.IsTermSetValid) 

                     { 

                         Write-Host "Current Term Set Invalid:", $field.TermSetId, " Field Name:", $field.Title 

                     } 

                     foreach ($item in $listItems) 

                     { 

                         $currentTerms = $item[$field] 

                         if (-not [string]::IsNullOrEmpty($currentTerms)) 

                         { 

                             $newTerms = New-Object Microsoft.SharePoint.Taxonomy.TaxonomyFieldValueCollection $field

                             if ($test) 

                             { 

                                 Write-Host "Term Replacement Starting"

                             } 

                             foreach ($term in $currentTerms) 

                             { 

                                 Write-Host $term

                                 $fullTermLabel = $term.Label 

                                 $lastIndexOfTermSeparator = $fullTermLabel.LastIndexOf(":") 

                                 $termLabel = $fullTermLabel.Substring($lastIndexOfTermSeparator + 1) 

                                 $existingTerms = $taxonomySession.GetTerms($termLabel, $false) 

                                 if ($existingTerms.Count -gt 1) 

                                 { 

                                     if ($test) 

                                     { 

                                         Write-Host "Found Terms"

                                     } 

                                     $termIsFound = $false

                                     $matchedTerm = ""

                                     foreach ($foundTerm in $existingTerms) 

                                     { 

                                         $foundTermPath = $foundTerm.GetPath().Replace(";", ":") 

                                         if ($foundTermPath -eq $fullTermLabel) 

                                         { 

            Write-Host ""
                                             Write-Host "Matching Term Found Found Term:", $foundTermPath, "Full Term:", $fullTermLabel

                                             $termIsFound = $true

                                             $matchedTerm = $foundTerm

                                         } 

                                     } 

                                     if (-not $termIsFound -and $test) 

                                     { 

                                         Write-Host "No exact term found"

                                     } 

                                     elseif ($termIsFound) 

                                     { 

                                         Write-Host "Replacing Term:", $fullTermLabel, "New Term:", $matchedTerm.GetPath() 

                                         $termValueAsString = "-1" + ";#" + $matchedTerm.GetPath() + [Microsoft.SharePoint.Taxonomy.TaxonomyField]::TaxonomyGuidLabelDelimiter + $matchedTerm.Id.ToString() 

                                         Write-Host $termValueAsString

                                         $termValue = New-Object Microsoft.SharePoint.Taxonomy.TaxonomyFieldValue $termValueAsString

                                         $newTerms.Add($termValue) 

                                     } 

                                     if ($test) 

                                     { 

                                         Write-Host "End of Found Terms"

                                     } 

                                 } 

                                 elseif ($existingTerms.Count -le 0) 

                                 { 

                                     Write-Host "Term For Item '", $item.Id, "' At List '", $list.Title, "' Located At", $list.DefaultViewUrl, "Has No Matching Term"

                                 } 

                                 else 

                                 { 

                                     if ($test) 

                                     { 

                                         try 

                                         { 

                                             Write-Host "    Old Term:", $term.Label, "New Term:", $existingTerms[0].GetPath() 

                                         } 

                                         catch 

                                         { 

                                             Write-Host "Error Occurred Full Term Label:", $fullTermLabel, "| Term Label:", $termLabel, "| Found Terms Count:", $existingTerms.Count 

                                         } 

                                     } 

                                     else 

                                     { 

                                         Write-Host "Replacing Term:", $fullTermLabel, "New Term:", $existingTerms[0].GetPath() 

                                         $termValueAsString = "-1" + ";#" + $existingTerms[0].GetPath() + [Microsoft.SharePoint.Taxonomy.TaxonomyField]::TaxonomyGuidLabelDelimiter + $existingTerms[0].Id.ToString() 

                                         Write-Host $termValueAsString

                                         $termValue = New-Object Microsoft.SharePoint.Taxonomy.TaxonomyFieldValue $termValueAsString

                                         $newTerms.Add($termValue) 

                                     } 

                                 } 

                             } 

                             try 

                             { 

                                 if ($test) 

                                 { 

                                     Write-Host "Term Replacement Finished"

                                 } 

                                 else 

                                 { 

                                     if ($field.AllowMultipleValues) 

                                     { 

                                         $field.SetFieldValue($item, $newTerms) 

                                         $web.AllowUnsafeUpdates = $true;
# Since we are updating publishing pages need to check them out first.
           $item.File.CheckOut();
           $item.Update();
           $item.File.CheckIn("Keywords field Metadata relinked");
           $item.File.Publish("Keywords field Metadata relinked");
           $item.File.Approve("Keywords field Metadata relinked");
          

                                     } 

                                     elseif ($newTerms.Count -gt 0) 

                                     { 

                                         $field.SetFieldValue($item, $newTerms[0]) 

                                         $web.AllowUnsafeUpdates = $true;
          
           $item.File.CheckOut();
           $item.Update();
           $item.File.CheckIn("Keywords field Metadata relinked");
           $item.File.Publish("Keywords field Metadata relinked");
           $item.File.Approve("Keywords field Metadata relinked");

                                     } 

                                 } 

                             } 

                             catch [System.Management.Automation.ExtendedTypeSystemException] 

                             { 

                                 Write-Host $_.Exception.ToString() 

                             } 

                         } 

                         else 

                         { 

                             Write-Host "Item Is Empty:", $item.Id, $item.Title 

                         } 

                     } 

                 } 

             } 

         } 

         catch [System.Management.Automation.ExtendedTypeSystemException] 

         { 

             Write-Host $_.Exception.ToString() 

         }
  
   }

     } 

      
  $web.AllowUnsafeUpdates = $false;
 
     if ($recurse) 

     { 

         foreach ($childWeb in $web.Webs) 

         { 

             if ($test) 

             { 

                 Write-Host "Child Web:", $childWeb.Url 

             } 

             UpdateTaxonomyFields -webAddress $childWeb.Url -test:$test -recurse:$recurse

         } 

     } 

 }     

Disclaimer: Use the above script at your own risk! The author accept no liability for any loss or corruption of data that may occur from the use of the above script.

Programmatically work with External Content Types (BDC models) using BCS Object Model

BCS object model becomes handy in situations where we want to build our own web part or web control to display external data using external content types that are already deployed. This article briefly explains and gives you an overview of the BCS Object Model, so that you can quickly jump-start into building custom solutions using BCS Object Model.

As a initial step we need add the references to the relevant assemblies and namespaces in order to make use of the BCS Object model.  Microsoft.BusinessData assembly and the namespaces mentioned below (circled in red) will be used for this purpose.

Having added the relevant assembly and namespaces, I will outline a step by step approach using the Object Model in order to connect and get information from the external content types.

Initially we need to establish a connection and get a reference to BDC Service in SPFarm services, as follows

BdcService service = SPFarm.Local.Services.GetValue<BdcService>();

Next step is to get the Metadata catalog which contains the external content types, and this can be achieved by using GetDatabaseBackedMetadataCatalog method of BdcService. This method requires the service context as a parameter, since the code is going to be executed within a visual web part, we can use current service context as below.

 IMetadataCatalog catalog = service.GetDatabaseBackedMetadataCatalog(SPServiceContext.Current);

Once the metadata catalog reference is in place, we can retrieve the particular entity that we are interested in using the GetEntity method of ImetaDataCatalog using the Site URL and the entity name as parameters.

Catalog.GetEntity(site.Url, “MSICNewInformation”);

Note: If we want to get references to all the entities of the deployed external content types, then we can use GetEntities method of ImetaDataCatalog which takes wild card (‘*’) as input parameter.

 catalog.GetEntities(“*”);

Having covered the key parts, next we will move into a practical scenario (code given below) where the status of an application is retrieved by passing in the submission date and the number of the application as parameters. Obviously this information is kept in an external database.

As mentioned above we get the reference to the Entity Object, therefore its properties, methods and method instances can be accessed and queries can be executed. In below example we make use of the SpecificFinder method instance of the external content type. 

methodInstance.MethodInstanceType == MethodInstanceType.SpecificFinder

Since the  SpecificFinder method of the external content type returns a single entity instance we make use of the FindSpecific method of Entity object in the BCS Object Model. This method returns single entity instance based on the parameters provided. Parameters are of type Object (which will have the value of the identifier) and LobSystemInstance.

A new object of type Identity is created based on the input parameters (txtdate and txtMsicNumber) and passed to the FindSpecific method along with the LobSystemInstance.

The returned entity instance will have the result filtered based on the input parameters and the status of the application can be retrieved from it.

Reference:         Get Started with Using the Runtime Object Model

Thanks for reading my article and in my next article I will cover some areas in Managed Metadata Services and Taxonomy of SharePoint 2010.

Making BCS work for anonymous users – Part 2

We had a look at how the security/permissions needs to be configured using SharePoint Designer in the first article ‘Making BCS work for anonymous users – Part 1’.  In this article I will outline what changes needs to be done via Central Admin to complete the task and make the external content type accessible for anonymous users in internet facing sites.

Once the relevant changes (mentioned in Part1) are completed and external content types are available in the Central Admin (under the Business Connectivity Services) as shown below, object permissions needs to be set to external content types.

Select the external content type that you need to expose to external users and click the ‘Set Object Permission’ in the ribbon, the dialog shown below will popup where you can add users/groups and set specific permissions. Add the built-in user group ‘NT AUTHORITY\ANONYMOUS LOGON’ and set ‘execute’ permission.

 Note: In some cases SharePoint wouldn’t find the built in group ‘NT AUTHORITY\ANONYMOUS LOGON’ and therefore you cant set the permission using the UI,  although you can’t set these permissions in the UI you can set them by editing the XML of the underlying BDC model directly.  The XML of the BDC model for the external content type includes <AccessControlEntry> elements that specify what rights an individual user or group has to the BCS external content type.  Adding users to the BCS permissions in the UI creates additional entries in the XML of the model.  To give anonymous users access to the BCS model we need to add the following entry to several <AccessControlList> elements in the BDC model’s XML.

<AccessControlEntry Principal=”NT Authority\Anonymous Logon”>
  <Right BdcRight=”Execute” />
</AccessControlEntry>

The easiest way to do this is, in Central Admin select the BCS service application you created and add the Execute right for a specific user.  This will give you an entry that you can do a search and replace on in the XML file later. Then export the BDC Model (with permissions) and edit it. Do a search and replace for the user you added and replace them with ‘NT AUTHORITY\ANONYMOUS LOGON’. Go back to Central Admin and use the Import button on the ribbon to import the edited BDC model into your BCS service app. (make sure you import with permissions).

That’s it, now the external data (may be in a external list or in Business data webparts) that is based on your external content type can now be accessed by anonymous users.

Thanks for reading the article and in my next article I will cover some areas on how to programmatically work with external content types.