SharePoint 2013 Incoming Email Issue : Emails disappear


 

Introduction

The other day we had a SharePoint 2013 development environment which had been configured up to receive email into libraries.

The configuration had been working for a while with no problems, there are issues with the incoming email on older versions of SharePoint 2013 but this farm was patched to May 2015.

We used the following Technet article to configure incoming email: https://technet.microsoft.com/en-us/library/cc262947.aspx

Unfortunately, email was never getting into SharePoint. The service had been working without any problems for a few weeks.

 

Problem Analysis

 

On investigation of the problem showed that email was successfully routed via Exchange to the SharePoint server via the locally installed SMTP service. The email was arriving successfully into the drop folder and after a few minutes the email disappeared. However, it would never appear in the document library! The email just disappeared.

Another symptom of the problem was seen when accessing the document library settings page for Incoming Email. Not all the settings would be loaded, the page was missing a large set of settings.

Normally the page looks like this :-

incoming-email-settings-library-ok

 

However, it was only showing this information :-

incoming-email-settings-library-broken

 

When someone attempted to make a change to the email settings for the list then the following error would be thrown:-


Server Error in '/' Application.

Error in the application.Description: An unhandled exception occurred during the execution of the current web request. 

Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.SharePoint.SPException: Error in the application.

Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace: [SPException: Error in the application.]   
Microsoft.SharePoint.SPList.UpdateDirectoryManagementService(String oldAlias, String newAlias) +1239   
Microsoft.SharePoint.SPList.Update(Boolean bFromMigration) +311   
Microsoft.SharePoint.ApplicationPages.EmailSettingsPage.SubmitButton_Click(Object sender, EventArgs args) +1347   
System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +146   
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3586

Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.34248

 

Strange behaviour indeed!

I fired up ULSViewer (http://www.microsoft.com/en-us/download/details.aspx?id=44020) on the offending server. Whilst watching the ULS log, another email was sent to the document library. After a minute or so a message popped up.

Cannot load type EWSSharePointEventReceiver from Assembly: [Assembly Name].

So for some reason the assembly could not be loaded, hmm so why is that?
I had a look through the GAC and could find the assembly name. However, using a .Net Reflector such as Just Decompile (http://www.telerik.com/products/decompiler.aspx) or .NET Reflector (http://www.red-gate.com/products/dotnet-development/reflector/). I could see that the class was not in the assembly. Speaking with the developers, I found out the class names had been changed. So the next thing I checked was the event receivers.

With a bit of PowerShell, I found a set of event receivers which were being fired by the item email received event.

I removed the event receiver using the following PowerShell and running it in the SharePoint 2013 PowerShell Management Shell:-

 


$eventReceiverName="EWSSharePointEventReceiver*";
$siteUrl="https://sharepoint";
$site = Get-SPSite $siteUrl;
$rootWeb = $site.RootWeb;
$allWebs = $rootWeb.AllWebs();

foreach($checkList in $allListsInWeb)        
{   
	Write-Host "Checking List: " $checkList.Title " for event receiver class: " $eventReceiverName;
	$eventReceivers = $checkList.EventReceivers | ?{$_.Class -like $eventReceiverName};            
 
        If($eventReceivers -ne $null -and $eventReceivers.Count -gt 0)
        {                   
             foreach($removeReceiver in $eventReceivers)                        
             {                       
	           Write-Host "Removing Event Receiver: $removeReceiver";
		   $removeReceiver.Delete();
             }
 
             $checkList.Update();
        }
}

 

After the corrupt event receivers were removed from the list then resending the email to SharePoint worked! Now the email appeared in the document library.

SharePoint 2013/Online Attachment Viewer Enhancement


Introduction

Update: 22nd January 2016 – Further updates to fix an issue with Minimal Download Strategy (MDS) and works correctly when used in multiple list view app parts on the same page.

Recently I had a client that wanted a better way to get to the attachments that were linked to a SharePoint list item from a list view web part. This request had me thinking and I didn’t think it would be too difficult to achieve using SharePoint 2013’s Client Side Rendering (CSR) user interface components. Anyway one night I decided on an approach and put the following together. attachment-viewer-example

Solution

The solution uses client side rendering to override the output that the field displays in a list view. This is achieved using a list template override to change the Attachment’s field rendering. The solution makes use of the SharePoint Callout Manager which is used to display links to the attachment. I found a set of articles which have been posted in the Resources section below really good for getting to grips with using the Callout Manager. The implementation works on both SharePoint 2013 and SharePoint Online. The one issue with the approach is that you have to reference the JavaScript file for each view, so if you create a new view then that view needs to be setup. This isn’t a problem if you are creating a custom list with a schema.xml as you can reference the JavaScript in the list’s schema. The solution makes use of the SharePoint 2013 JSOM API and the list of attachments is only created one the user has clicked on the paperclip icon to see the list of attachments. This is to try and keep the load down on the server when there are lots of attachments for a particular list item.

Resources

The following websites provided a great resource in getting the Callout Manager and SharePoint JSOM working. http://blog.alexboev.com/2012/07/custom-callouts-in-sharepoint-2013.html

http://www.learningsharepoint.com/2013/05/09/sharepoint-2013-callout-popup-tutorial/

SharePoint 2013 – Custom CallOut with File Preview – Obilogic

http://msdn.microsoft.com/en-us/library/office/dn268594(v=office.15).aspx

Setup

Watch the video showing you how to setup the attachment viewer.

To take advantage of the list attachment viewer and set it up for a particular list view, do the following:-

  • Download the zipfile, found at the bottom of the post
  • Unzip the zip file to extract the ITSP.SP.AttachmentField.js file.
  • Upload the ITSP.SP.AttachmentField.js into a suitable location, I would suggest the Site Assets library.
    • Create a folder called Scripts within the Site Assets library
    • Upload the ITSP.SP.AttachmentField.js into this folder, this keeps your content organised.
    • Alternatively you could package up the JS file to deploy the files by using a Module into a site collection.
  • Browse to the SharePoint list that you wish to enable the viewer on.
    • Make sure that the list has the Attachment field showing as this is the column that is going to be overridden
  • Click “Edit Page”

attachment-viewer-stepone-editpage

  • Click on the List Viewer Web Part, choose Edit Web Part

attachment-viewer-steptwo-editwebpart

  • The web part properties will appear on the right hand side
  • Expand the miscellaneous section
  • In the JSLink field type in the location of the ITSP.SP.AttachmentField.js file
  • E.g ~site/SiteAssets/Scripts/ITSP.SP.AttachmentField.js
  • Use the ~site token to substitute for the local site url

attachment-viewer-stepthree-jslink

  • Click Apply
  • Click Ok
  • Save the Page

Now create a list item and add some attachments to it. When you click on the paperclip icon you will get a callout which will display all the attachments for that list item. attachment-viewer-paperclip attachment-viewer-example Please let me know if you find it useful, I have tested the script with 5/6 attachments, it should scale with 20 or so documents as the list of documents is built each time you click on a particular document.

Update 21st May 2014

A new version of the script has been released to add support for the Minimal Download Strategy feature. This helps to reduce the amount of bytes being downloaded to the client. For more information please read this MSDN Blog Post.

The behaviour that you would start seeing is that the attachment callout would appear once and then never again. More information can be found here:-

Update 17th November 2015

A new version of the script has been released to add support for being used in multiple list view app parts on the same page.
Please use this version instead.

Thanks to Dar for reporting this issue.

Update 4th January 2016

It seems that I did not do enough testing and a further issue was discovered by Ken when having multiple list view web parts on the page.

Update 22nd January 2016

There still seemed to be an issue with MDS and also the control was polluting the global JavaScript pool, this update fixed those has been tested on SharePoint Online.

Download:
ITSP.SP.AttachmentField-20160122.zip

 

How to: SharePoint Search PDF Document Preview using Acrobat


Introduction

This is what we are looking to build, a SharePoint 2013 search experience where PDFs can be previewed within Search.

pdf_preview_demo

 

On a recent project I built a solution which made extensive use of the SharePoint 2013 search features. Microsoft have been pushing building solutions with SharePoint 2013 and I have to say it does give a great experience in terms of performance and functionality.

One of the main reasons for selecting search based solution is that it scales and this solution needed to provide quick access to documents. The complexity was that due to the nature of the documents they each had unique security permissions. The solution stored a few hundred thousand documents, this would have been a killer for performance if using traditional data querying methods.

95% of the documents were PDFs and I thought it would be cool if we could provide a preview of the PDF in the search results. Unfortunately we didn’t have Office Web Apps and so I started to think about how I could build something similar. The preview would work in a similar way to the search visualisation for Office when Office Web Apps is available.

At the time Office Web Apps wasn’t available and so we could use its ability to view PDFs.

One of the important goals was to ensure that the solution would show the PDF preview window in the hover panel but would also fail gracefully when a user didn’t have the necessary components installed on their machine.

 

The solution relies on the following:-

  • SharePoint 2013
  • SharePoint Search Centre
  • Custom Display Template
  • Acrobat Reader / Acrobat
  • Acrobat Reader / Acrobat’s Web Browser Browser Plugin
  • the PDF Object JavaScript library

 

Adobe Acrobat Reader Pre-requisite Configuration

As mentioned we need a few pre-requisites for this solution to work.

The user must have Acrobat or Acrobat Reader installed and configured to use the PDF Web Browser plugin. You can see my previous post, which explains how to enable the PDF Web Browser plugin.

https://blog.ithinksharepoint.com/2011/02/21/open-pdf-files-from-a-document-library-in-a-new-window/

Adobe Acrobat Reader can be downloaded from: https://get.adobe.com/uk/reader/

 

SharePoint 2013 Setup

So the SharePoint 2013 Search configuration requires the following:-

  • custom Search Item Display Template
  • custom Search Hover Panel Display Template
  • configured Search Result Type

The Search Result type is required to actually configure search to use our custom display templates to render PDF results with our preview hover panel.

We are expecting that you are using a search centre for the sake of this blog post but the solution could be deployed into any site collection.

 

Setting up display templates

Lets set this thing up. We will need to have a SharePoint 2013 Visual Studio project created so that we can put the display templates into source control, plus we need to be able to deploy them too!

  • Start Visual Studio 2012 / Visual Studio 2013
  • Create a new project using the SharePoint 2013 Empty Project
  • Give it a suitable name and location

pdf_vs_newproject

  • Provide the URL to the SharePoint search centre site collection that will be used for development.
  • Choose a Sandbox Solution
  • Click Create

Once we have a Visual Studio project setup we need to create our display templates. The best way to learn is to take an existing display template and hover panel and make a new set. Display Templates are found in the master page document library. This can be found in a site collection’s root web, for example. http://sharepoint/search/_catalogs/masterpages.

Within the Master Pages document library is a sub folder called DisplayTemplates, this is where the Display Templates for the site collection are stored. I suggest that you create your own folder within this so that you can differentiate your changes from the out of the box display templates.

The deployment of display templates is a little tricky because they are made up of two parts. A display template is made up of an html file which SharePoint parses and creates a JavaScript representation of automatically. This occurs each time the .html file is saved.

Unfortunately if you deploy a Display Template via a feature the JavaScript representation is not created automatically. To get around this, I used the following process:-

As we mentioned earlier, we will create our display templates by copying an existing display template, renaming and modifying. This is achieved by the following:-

  • Browse to your SharePoint site.
  • Click on the cog icon in the top right hand section of the page. Click on Site Settings.
  • Click on display page layouts and master pages Pages link under “Look and Feel”.
  • Browse to the Display Templates folder
  • Download the following html display templates
  • Item_PDF.html and Item_PDF_HoverPanel.html

Now that we have a copy of the display templates, we need to add them to our Visual Studio project Create a new feature in Visual Studio by clicking on the Features node in the Solution Explorer window and choosing new feature.

  • Rename the feature folder to an appropriate name such as:-
    • PDFPreviewTemplateWithAcrobatimport
  • Open the new feature and fill in the feature’s information.

pdf_vs_featuredef

Next we will add a SharePoint Project Item to deploy the Display Templates.

  • Right-Click on the project and choose new item
  • From within the SharePoint project items choose the Module project item and call it something appropriate such as “PDFDisplayTemplates”
  • Drag the files into the newly created Project Item

If you cannot drag them then you are likely running Visual Studio as Administrator and you will need to copy the files into the folder created for PDFDisplayTemplates project item and then use the View All Files icon (found at the top of solution explorer) and add them by right clicking on each item and choosing “Include in Project” from the menu.

  • Next, rename the files to something appropriate, like this
    • item_pdf.html –> item_pdf_acrobat.html
    • item_pdf_hoverpanel.html –> item_pdf_acrobat_hoverpanel.html

Once you have uploaded your files into Visual Studio then we need to edit the files and configure Visual Studio to deploy the files as part of the solution by changing the file properties in Visual Studio solution explorer to change your deployment type from “No Deployment” to “Element File”.

We also need to update the display templates so that the name and description allows a user to identify the template. These descritions are modified using the metadata for the display template which isactually provided by a set of headers in the .html file.

Please see the screenshots below, we have modified the name and description to:-

  • Name: PDF Item with Acrobat
  • Description: Displays a result tailored for a Portable Document Format (PDF) document and displays a preview using Adobe Acrobat.

 

Before:-

pdfitem_properties_before

After:-

pdfitem_properties_after

The last step is to add the PDFObject.js file to Visual Studio so that its deployed with the solution. We could do this in a number of ways, I am going to include it within the PDF Display Template feature so that we only have to activate the one feature.

 

Finally I am going to clean up the the feature’s element.xml file so that it is deployed into the correct location within SharePoint, if you remember it should be deployed into the masterpage document library.

So here is the element.xml before:-

pdffeature_elementsbefore

and here is the element.xml after:-

pdffeature_elements_after

Once you have updated your display template’s name then they are ready to be deployed to SharePoint via the visual studio solution. Install your solution and activate the feature that was created.

 

Modifying the display templates to show PDFs

Hopefully now you have the display templates uploaded into SharePoint, though they are just copies of an existing display templates, so we’ll need to update the display template with the code to show PDFs.

Next we need to modify the contents of the display template, I have already create updated versions which you can download but to lets give you an overview of what I have changed.

 

Item_Pdf_Acrobat.html

This has been modified so that the line reading

var hoverUrl = "~sitecollection/_catalogs/masterpage/Display Templates/Search/Item_PDF_HoverPanel.js";

is updated to the use are version of the hover panel.

var hoverUrl = "~sitecollection/_catalogs/masterpage/Display Templates/Search/item_pdf_acrobat_hoverpanel.js";

Item_Pdf_Acrobat_HoverPanel.html

This has also been modified to update the name and description metadata elements. The entire html has been replaced and rather than explain all the changes, download the solution zip file and take a look at the file within Visual Studio.

The standout piece is the following section, which is called after the display template is rendered using the AddPostRenderCallback() function. This checks to see if the file is a pdf, and if it is will try and create the Acrobat Web Viewer window, if it fails for whatever reason the preview div is hidden.

<!--#_
AddPostRenderCallback(ctx, function()
{
var csrId = ctx.CurrentItem.csr_id;

if(ctx.CurrentItem.SecondaryFileExtension=="pdf")
{
try
{
var employeePDFDocument = new PDFObject({
url: $urlHtmlEncode(ctx.CurrentItem.OriginalPath),
pdfOpenParams: {
navpanes: 0,
toolbar: 0,
statusbar: 0,
view: "FitV"
}

}).embed($htmlEncode(id + HP.ids.viewer));
}
catch(e)
{
var viewDivId = $htmlEncode(id + HP.ids.viewer);
var viewDiv = document.getElementById(viewDivId);
if(viewDiv!=null)
{
viewDiv.style.display = 'none';
}

}
}

});

_#-->

Next the display templates need to be updated, I will show you the following approach which is the way that Microsoft recommend when developing display templates.

  • browse to the master page catalogue using http://sharepoint/search/_layouts/15/masterpages
  • click on the library tab in the ribbon toolbar
  • click on open in windows explorer
    • if you get an error make sure that IE is setup to include the web site in the Local Intranet zone and also if you are developing on Windows Server 2008/2012 make sure you have the Desktop Experience feature installed.
  • Once you have the windows explorer folder open you can now copy and paste the files from Visual Studio into Display Templates folder. SharePoint will take care of updating the JavaScript representation of your html file

Now we have our updated display templates we need to configure them to be used!

 

Configure search result: Tell SharePoint about the display templates

The approach to configuring search to use our display templates is by setting up a result type. Search Result types are the new Search Scopes but with SharePoint 2013 they also allow you to specify how each search result is displayed. This is really powerful and one of my favourite features of SharePoint 2013!

This feature allows you to have a list of search results and if a search result for example had a custom content type with custom metadata that result could be displayed differently to show that additional information.

Anyway, on to setting up the custom search result type:-

  • browse to your SharePoint search centre site collection
  • click on the cog icon and then choose site settings
  • Click Search Result Types (under Site Collection Administrator heading)
  • Click new search result type
    • Name: PDF using Acrobat
    • Sources: All Sources
    • Type of Content: PDF
    • Display Template: PDF using Acrobat
    • Tick optimize for frequent use

Lets try it out!

Deploy the solution to your SharePoint environment and add some PDF content to your SharePoint sites. Perform an incremental crawl or wait for your continuous crawl to pick up the content.

If you perform a search for  pdf, you should get some search results with the content uploaded, as you hover over the PDF search result you should get a preview of the PDF!

pdf_preview_demo

 

Troubleshooting the solution

Then the solution was being built then there were a couple of problems that I had. To be honest most of the issues were down to the Adobe Acrobat configuration.

Please take a look at my post Opening PDFs in a New Window and the section

 

Solution Files and resources

The Visual Studio Project, PowerShell script and Display Templates can be found here:

ITSP.SP.PDFSearchPreviewWithAcrobat.zip

 

 

Issue applying Workflow Associations to Content Types


 

Introduction

 

First, let’s set the scene. We have built a solution for a customer with a site collection and within that site collection a sub-web which has a large number of document libraries in it.

For the next release we were looking to setup records management and use the out of the box disposition workflow to allow the customer to decide whether to delete the document or not. The workflow is triggered by an information policy.

The upgrade went well all the feature upgrades worked nicely however then we hit a snag setting up the workflows.

The process of applying a workflow to a content type is pretty straight forward and we did not have any problems with the development, staging or UAT environments. Of course its only when we go to setup in production did we hit a problem.

Applying the workflow to a content type was achieved by doing the following:-

  • Browse to the root of your site collection
  • Click on Site Content –> Settings
  • Click on Site Content Types
  • Click on your content type that you wish to configure
  • Click “Workflow Settings”
  • Click Add a Workflow which takes you into the following screen

addingworkflowtocontenttype

If you look at the image at the bottom of the page you will see the option “Update all content types”, so we set this and clicked Ok.

The page just hung there processing, we left it for a while but still it was hung there. Knowing SharePoint we thought that is fine we’ll leave it. Two hours later still nothing so we started to check the SharePoint Unified Logs and unfortunately we found an error

 

SharePoint Foundation   Database        fa46    High    at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock)     at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean retryfordeadlock)     at Microsoft.SharePoint.Library.SPRequestInternalClass.GetListsWithCallback(String bstrUrl, Guid foreignWebId, String bstrListInternalName, Int32 dwBaseType, Int32 dwBaseTypeAlt, Int32 dwServerTemplate, UInt32 dwGetListFlags, UInt32 dwListFilterFlags, Boolean bPrefetchMetaData, Boolean bSecurityTrimmed, Boolean bGetSecurityData, Boolean bPrefetchRelatedFields, ISP2DSafeArrayWriter p2DWriter, Int32& plRecycleBinCount)     at Microsoft.SharePoint.Library.SPRequestInternalClass.GetListsWithCallback(String bstrUrl, Guid foreignWebId, String bstrListInternalName, Int32 dwBaseType, Int32 dwBaseTypeAlt, Int32 dwServerTemplate, UInt32 dwGetListFlags, UInt32 dwListFilterFlags, Boolean bPrefetchMetaData, Boolean bSecurityTrimmed, Boolean bGetSecurityData, Boolean bPrefetchRelatedFields, ISP2DSafeArrayWriter p2DWriter, Int32& plRecycleBinCount)     at Microsoft.SharePoint.Library.SPRequest.GetListsWithCallback(String bstrUrl, Guid foreignWebId, String bstrListInternalName, Int32 dwBaseType, Int32 dwBaseTypeAlt, Int32 dwServerTemplate, UInt32 dwGetListFlags, UInt32 dwListFilterFlags, Boolean bPrefetchMetaData, Boolean bSecurityTrimmed, Boolean bGetSecurityData, Boolean bPrefetchRelatedFields, ISP2DSafeArrayWriter p2DWriter, Int32& plRecycleBinCount)     at Microsoft.SharePoint.SPListCollection.EnsureListsData(Guid webId, String strListName)     at Microsoft.SharePoint.SPListCollection.GetListByName(String strListName, Boolean bThrowException)     at Microsoft.SharePoint.Workflow.SPWorkflowAssociation.EnsureUtilityList(SPWeb web, String workflowName, String listTitle, Guid Id, SPListTemplateType templateType, Boolean forceListCreation, String alternateNameRes, String descriptionRes)     at Microsoft.SharePoint.Workflow.SPWorkflowAssociation.EnsureTaskList(SPWeb web, String workflowName, String createTaskListTitle, Guid createTaskListGuid, Boolean forceListCreation)     at Microsoft.SharePoint.Workflow.SPWorkflowAssociationCollection.SetUtilityLists(SPWorkflowAssociation wa, Boolean forceUtilityListCreation)     at Microsoft.SharePoint.Workflow.SPWorkflowAssociationCollection.AddCore(SPWorkflowAssociation wa, Guid id, SPList list, Boolean forceUtilityListCreation)     at Microsoft.SharePoint.Workflow.SPContentTypeWorkflowAssociationCollection.AddCoreCT(SPWorkflowAssociation wa)     at Microsoft.SharePoint.Workflow.SPContentTypeWorkflowAssociationCollection.PushDownAssociation(SPWorkflowAssociation associationTemplate, Boolean bUpdateIfExisting, MethodBase mbChangeEntry)     at Microsoft.SharePoint.Workflow.SPContentTypeWorkflowAssociationCollection.UpdateOrAdd(SPWorkflowAssociation associationTemplate)     at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)     at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)     at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)     at Microsoft.SharePoint.SPChangeMonitor.ApplyChangesCore(Object ct, Boolean applyAll, Type typeFilter, Boolean bFilterInclude)     at Microsoft.SharePoint.SPContentType.PushDownWorkflowChangesToDerivedCTCore(SPContentType ct, Boolean fCloseWebAsNecessary)     at Microsoft.SharePoint.SPContentType.PushDownWorkflowChangesToListCTs(SPContentTypeCollection cts, IEnumerable`1 cids, Boolean throwOnSealedOrReadOnly)     at Microsoft.SharePoint.SPContentType.PushDownChanges(CodeToPushDownChangesToDerivedCT derivedCTPushdownCode, CodeToPushDownChangesToListCTs listDerivedCTsPushdownCode, Boolean throwOnSealedOrReadOnly, IList`1 exceptions)     at Microsoft.SharePoint.SPContentType.UpdateWorkflowAssociationsOnChildren(Boolean bGenerateFullChangeList, Boolean bPushdownDerivedCT, Boolean bPushdownListCTs, Boolean bThrowOnSealedOrReadOnly)     at Microsoft.SharePoint.WorkflowServices.ApplicationPages.WrkSetngPage.OnClick_Update(Object sender, EventArgs e)     at System.Web.UI.WebControls.LinkButton.RaisePostBackEvent(String eventArgument)     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     at System.Web.UI.Page.ProcessRequest()     at System.Web.UI.Page.ProcessRequest(HttpContext context)     at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)     at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)     at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)     at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)     at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)     at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)     at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

 

The reason for this error was explained in the next log entry which was a little more concise:-

Getting Error Message for Exception System.Web.HttpException (0x80004005): Request timed out

 

So a bit of head scratching and we thought well we should really make this change via PowerShell but the problem is how to do that?

 

Solution

 

Well its funny some of the method calls you see on SharePoint objects. Obviously the guys at Microsoft have seen this issue before. When I was updating the SPContentType I remember seeing the function UpdateWorkflowAssociation() but then there was the much more useful UpdateWorkflowAssociationsOnChildren()!

Thanks to the team at Microsoft – that is just what we were looking for!

The fix was the following piece of PowerShell:-

 

$web = Get-SPWeb [your site collection root web url]; $contentTypeToUpdate = $web.ContentTypes | ?{$_.Name –eq “[Your Content Name Here]”}; $contentTypeToUpdate.UpdateWorkflowAssociationsOnChildren($true, $true, $true, $false);

I used ULSViewer to watch the function running SPSqlClient sessions to see that the process was working correctly and more importantly it wasn’t timing out!

 

To talk through the PowerShell we are doing the following:-

  • Getting the SharePoint web for the root of our site collection which contains the content type to update
  • Get the content type that we need to update
  • Call the UpdateWorkflowAssociationsOnChildrenUpdate()
    • mark all the content as changed = true
    • push down to all derived content types = true
    • push down to all content types that are associated to lists = true
    • throw an error if you encounter a read-only or sealed content type = false

Lessons Learned

 

When you are doing upgrades always try and use a similar sized data set as production!

When it does not work through the UI – bring on the PowerShell!

 

Anyway I hope that helps someone else get themselves out of trouble!

Experiences with SharePoint Disposition Approval Workflow


 

Introduction

 

Recently I have been working on document lifecycle management project. One of the targets for the project was to try and use out of the box (OOB) SharePoint features rather than develop additional components. I am all for this, it reduces support and maintenance overhead and the technical debt of the solution.

The solution ended up implementing a multiple content types with information management policies for auditing and retention policies.

The retention policy had a number of stages which were:-

  • Created date + 1 day make the document into a record
  • Created date + X years execute the “Disposition Approval Workflow”

The number of years would vary depending on the type of document.

Before I go any further I should talk about the Disposition Approval Workflow, this is a workflow which has been in SharePoint since SharePoint 2007 and is made available using the Disposition Workflow feature.

The workflow when run against a document has one step. The step creates a task which presents the user with the option to either delete the document or retain it. The user can also provide comments.

What should happen is that if the user chooses to delete the document then the workflow will delete the document.

However, this fails if the document is declared as a record. Unfortunately the out of the box disposition approval workflow which has been designed to provide a mechanism to delete a document cannot delete a document if its a record.

The workflow displays the following error or words to the effect of “Cannot delete the document”

This was a little frustrating so what do we do?

 

Options

 

After some thought about the problem I came up with two approaches:-

  • redesign the Disposition Approval Workflow to work how we want it to
  • add a step to undeclare the document as a record

Thinking back to one of the key targets, keep the amount of custom development to a minimum, I decided to go for the second option.

Unfortunately, there is no expiration action which allows you to undeclare a record. So I looked at the options to create one.

Fortunately there are a couple of reasonable examples, they might be a little lacking in detail but there is enough to understand the process.

 

Solution

 

So the following solution was built, the solution is made up of the following parts:-

  • A class which implements the IExpirationAction interface and perform the action to undeclare a document as a record
  • A feature which has a feature receiver which adds the IExpirationAction to the PolicyTemplates
    A class UndeclareRecordExpirationAction is created which implements the IExpirationAction interface as follows:-
    The IExpirationAction interface has a function OnExpiration which passes in the SPListItem to delete and the date that the document has expired on.
    The function implementation would be to check that the document is not already on hold and also that the document isn’t a record already.
    if the document is on hold then an exception is raised so that the process goes no further.

Two static functions were created which register / unregister the ExpirationPolicyAction with SharePoint.

 

A feature was created which implemented FeatureActivatation and FeatureDeactivation methods. These call into the appropriate static functions in the UndeclareRecordExpirationAction class.

 

Once the solution is installed and the feature activated then a new policy action appears here:

image

 

To configure the policy do the following:-

  • Browse to the site collection
  • Browse to the site settings page
  • Click Site Content Types
  • Click on the content type that you wish to apply the policy to
  • Click on “Information Management Policy settings”
  • Click “Define a policy..”
    • Click Ok
    • Click Enable Retention
    • Add a retention stage…

    The information policy was modified so that it has the following steps:-

    Created + 1 day = Declare record (reoccurrence 1 month)

 

    Created + 6 Years = Undeclare Record (reccurrence 1 year)

 

    Created + 6 Years = Start Workflow (reoccurence 1 year)

The option to specify how to manage retention on records was set that the same policy was used for both records and non-records.

I set the reoccurrence for the Declare record action to one month so that if a document was undeclared and then the record managers decided not to delete the document through the workflow it would then picked up again.

 

Testing the solution

The process of testing these expiration policies was a little tricky so I will cover the method that used. Information policies are applied by timer jobs. The following timer jobs are used:-

  • Information Management Policy Timer Job
  • Expiration Timer job

The expiration timer job does the actual work of applying/running the various steps of a retention policy. By default it runs once a week in SharePoint 2013.

To speed things up I did the following to test/debug the solution.

  • Compile the assembly hosting our UndeclareRecordExpirationAction class as a debug build.
  • Deploy the solution
  • Restart SharePoint Timer Service
  • Attach the Visual Studio debugger to the “owstimer.exe” process
  • From Central Admin, start the timer job through the monitoring->review job definintions
  • Browse to the Expiration Policy Job Definitions
  • Click on the “Expiration policy” link for the one associated to the Web Application where your application is hosted and click “Run now”
  • Wait for the visual studio debugger to hit the breakpoint

If you are having problems getting the debugger to hit the breakpoint then:-

  • Start the SharePoint Management Shell
  • From within the PowerShell session
  • restart the owstimer service using Restart-Service sptimerv4
  • restart IIS using iisreset

Improvements

 

The only thing that I may look to do is rather than have two stages for “Undeclare record” and “Starting a workflow” is merge them into two and have a “undeclare record and start workflow action” so that there is less delay with the processing.

 

Conclusion

 

I hope people find this useful, please let me know if you do.

 

Here are links to the code samples:-

ITSP.SP.Policy.UndeclareRecordAction.zip

PowerShell Script to Restart SharePoint 2013 Farm services


 

Introduction

 

A while back I wrote a quick script to ‘Restart services on a SharePoint 2010 Farm’, this blog post can be found here.

 

Anyway, I have slightly updated it for SharePoint 2013. Its pretty much the same though I am also restarting the Search process too and also checking that the service exists on the server before restarting.

How you find it useful

 

The Script

 

Here is the script:-

 <pre>
param
(
    [Parameter(Mandatory=$false, HelpMessage='-ServiceNames Optional, provide a set of service names to restart.')]
    [Array]$ServiceNames=@("SharePoint Timer Service","SharePoint Administration", "SharePoint Server Search 15", "World Wide Web Publishing Service","IIS Admin Service")
);

Write-Host "Attempting to get SharePoint Servers in Farm" -ForegroundColor White;
$farm = Get-SPFarm;
$servers = $farm.Servers;
Write-Host "Found" $servers.Count "Servers in Farm (including database servers)" -ForegroundColor White;
foreach($server in $servers)
{
    if($server.Role -ne [Microsoft.SharePoint.Administration.SPServerRole]::Invalid)
    {
        Write-Host "Attempting to restart services on" $server.Name -ForegroundColor White;
        foreach($serviceName in $ServiceNames)
        {
            $serviceInstance = Get-Service -ComputerName $server.Name -Name $serviceName -ErrorAction SilentlyContinue;
            if($serviceInstance -ne $null)
            {
                Write-Host "Attempting to restart service" $serviceName ".." -ForegroundColor White -NoNewline;
                try
                {
                    $restartServiceOutput="";
                    Restart-Service -InputObject $serviceInstance;
                    Write-Host " Done!" -ForegroundColor Green;
                }
                catch
                {
                    Write-Host "Error Occured: " $_.Message;
                }
            }
        }
    }
}
</pre>

Download the script here: Restart-SPServices2013.zip

Building a SharePoint 2013 Search Refiner with Custom Intervals


Introduction

SharePoint 2013 Search has really improved over its SharePoint 2010 predecessor. These improvements are down to the integration of the FAST search feature set into SharePoint 2013.

One of the nice features is the slider graph refiner which gives a visual representation of the results that can be filtered. However, unfortunately it doesn’t work well for custom managed properties.

Here is the refiner using the out of the box Last Modified Managed Property

clip_image002

Here is the refiner using a custom Date Time Managed Property called Timeline (this is the date of the document).

clip_image004

As you can see the date range is only for the last year, the numbers have commas in them and as the following image shows the descriptions for refining the data between the different intervals does not really make sense.

clip_image006

In a recent project this led to the decision to use a normal text refiner over the slider bar graph. However, the user base asked for the refinement options to be changed to a custom set of values. The out of the box date time refinement has the following options:-

· Last Year

· Last Six Months

· Last Month

· Last Week

· Last day

However the requirement was to have a custom set of refinements as shown below:-

timeline_datetime_refiner

Investigation

So to achieve the requirement above we need be able to do two things:-

· Configure how the refiner web part calculates the additional refiners

· Configure the text being used to display the refinement options to the user

How to configure the refinement calculations

The first thing that I did was have a look and see if there was anyone else who had tackled this issue and unfortunately I didn’t come up with much on MSDN or out there on the web.

The next step was to start looking at the customisation options for the web part.

By editing the web part properties and clicking on the “Choose Refiners” button you can decide which managed properties are going to be refinement options. Microsoft have given quite a lot of flexibility to custom the refiners but these options are not available for date/time data types . There are more options for the other data types such as text or number.

For example a managed property which is based on a number gives you the opportunity to create manual intervals to refine by.

clip_image007

However, a refiner based on a date/time data type doesn’t have much flexibility. This is probably easier to see via a screenshot.

clip_image009

There is an option to define the dates in the search schema but when that option is select then you get the following error message.

clip_image011

So this leads to the question what do I do now?

Quite often with web parts I start looking at the actual web part xml definition to try and workout if there are any appropriately named properties or values. So I exported the web part and opened up the web part in notepad.

After a lot of looking about I found the following web part property, “SelectedRefinementControlsJson”:-

clip_image013

This property contains a JSON string which defines the refinement control configuration. For each refiner that is selected a set of properties are stored within this string, these include properties such as

· sortBy

· sortOrder

· maxNumberOfRefinementItems

· DisplayTemplate

· etc

However, there were two properties which has interesting names and they were:-

clip_image015

clip_image017

I presumed that I could change the useDefaultDateIntervals to false and then provide some sort of JSON object for the intervals property. However, the difficult was trying to work out what the intervals structure would look like!

I started to debug the refiners using the Internet Explorer Developer Toolbar but didn’t really find much to help reverse engineer the process.

The approach that I quite often take is to examine the .NET code using a tool like Reflector. So it was time to reflect the appropriate web part. The refinement web part is of type:-

· Microsoft.Office.Server.Search.WebControls.RefinementScriptWebPart

After some searching I found the appropriate code.

The ResultScriptWebPart has a private method called GetRefinementControls() this instatiates another class called RefinementControl. A RefinementControl object is created for each Refinement Control JSON object and part of that object looks at the useDefaultDateIntervals property and if this has been set then the GetDefaultDateIntervalSpecString() is loaded.

clip_image019

The function GetDefaultDateIntervalSpecString() gives the clue of how the intervals parameter should be formatted.

clip_image021

So the intervals are basically an array of integers so the representation of this in JSON is the following:-

clip_image023

And of course to ensure that the custom intervals are loaded we need to change the useDefaultDateIntervals property to false.

clip_image025

So just to go back the intervals that have been set for the refiners are the number of days that should be removed from todays date to create the interval. The intervals -1065,-730,-365,-183,-30,-7, 0 translates to the less than 3+ years, 2-3 years, 1-2 years, 6 months to 1 year, 1-6 months, 1 week – 1 month and today.

So now we are able to control the refinement calculations, the next step is how to configure the labels displayed to the user.

If we don’t configure the labels we get the following being displayed to the user, which is obviously not good enough.

How to configure the refinement labels

Now I haven’t found a simpler solution for this but if anyone finds an alternative solution which for example overrides a particular web part property then please let me know.

The approach that I took in the end was to create my own filter display template. I haven’t covered display templates in my blog as yet. I am planning a post to discuss my findings with SharePoint 2013 Search development soon. However, the following articles provide some good information.

Anyway, display templates are found in the master page catalog document library which can be found at http://sharepoint/_catalogs/masterpage/displaytemplates. An alternate method to browse to this list is by going to the SharePoint site collection, click Site settings cog icon -> Site Settings->Master Pages and Page Layouts (under the Web Designer Galleries section).

All the out of box refinement filters can be found under the DisplayTemplates/Filters folder. I downloaded the Filter_Default.html file which is used as the standard out of the box text refinement filter. The file was renamed the file to Filter_CustomDateTime.html and edited in Visual Studio.

Here is an example of the contents of the Display Template

clip_image027

The <head> tag was updated to give a new title and then the following script was added which will go and set the labels presented to the end user. The JavaScript was added after all the default label data has been loaded, this was to ensure the custom configuration was not overwritten.

clip_image029

The display template was loaded back into SharePoint, I suggest that you put it in a custom folder and use a Module to deploy the file. In my example I am putting the filter in the following directory, Display Templates/iThinkSharePoint/.

Now we have the custom display template and we have configured our web part to use custom intervals there is one last step.

Can you guess what it is?

So we need to tell the web part to use our custom filter display template, this is another modification of the web part property, SelectedRefinementControlsJson. With some care look through the JSON string and find the refinement configuration for our custom Managed Property.

clip_image031

Now the web part configuration can be saved and uploaded into SharePoint as a new web part in the web part gallery.

Solution

So to recap the final solution has the following components:-

· Custom Web Part Configuration which includes the custom intervals, references the custom display template

· Custom Display Template for the Refinement control

This creates the following refiner:-

timeline_datetime_refiner

You can download the example Display Template and Web Part Configuration XML file from here:-

ITSPSearchRefineCustomIntervals.zip

If you find this useful please let me know and please ask if you need some more clarification.