Friday, June 24, 2011

Delay Loading of Data in SharePoint 2010 Web Parts

Sometimes your web-parts may take a long time to load their data, e.g. when connecting to external data through BCS, doing SPSiteDataQuery across a large number of sites, or when iterating over a user's site memberships to read some items from lists in different site-collections. Put a few of such web-parts on a dashboard page and wait for the combined load time of all those web-parts to complete before the page is shown. Not a nice user experience. These days users expect something as shown in this short screencast:


If you've used ASP.NET Ajax UpdatePanels, you might wish to utilize the asynchronous partial page update experience seen on postbacks also during page load. The simple thing seems to be calling __doPostBack for each UpdatePanel from the page load JavaScript event to trigger the Ajax async partial postback. That won't work, as only one concurrent postback is allowed by ASP.NET Ajax, so only one of your web-parts will work as expected, the other __doPostBack calls will get canceled by the ScriptManager.

A simple solution to this problem, is to put an asp:timer control inside the UpdatePanel and let it trigger a postback to your web-part code. Then load the data and update the content of the UpdatePanel during this async Ajax postback.

Here are the code to two base classes that implements this delayed load approach:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web.UI;
using System.Web.UI.WebControls;
 
namespace Puzzlepart.SharePoint.WebParts
{
    public class AjaxPanelWebPart : System.Web.UI.WebControls.WebParts.WebPart
    {
        protected UpdatePanel AjaxPanel;
 
        protected override void OnLoad(EventArgs e)
        {
            base.OnLoad(e);
            this.EnsureChildControls();
        }
 
        protected override void CreateChildControls()
        {
            AjaxPanel = new UpdatePanel()
            {
                ID = this.ID + "UpdatePanel1",
                UpdateMode = UpdatePanelUpdateMode.Conditional
            };
            Controls.Add(AjaxPanel);
            UpdatePanelConfigurator.AddUpdatePanelProgress(AjaxPanel);
        }
 
        protected virtual void ApplyUserActions()
        {
            //to be overridden in derived classes
        }
 
        protected void RebindControlsWhenNoViewState()
        {
            if (Page.IsPostBack == true &&
            System.Web.UI.ScriptManager.GetCurrent(Page).IsInAsyncPostBack == false)
            {
                ApplyUserActions();
            }
        }
    }
 
 
    public class AjaxPanelDelayedLoadWebPart : AjaxPanelWebPart
    {
        protected Timer LoadTimer;
 
        protected override void OnLoad(EventArgs e)
        {
            base.OnLoad(e);
            this.EnsureChildControls();
        }
 
        protected override void CreateChildControls()
        {
            base.CreateChildControls();
            CreateLoadTimer();
        }
 
        private void CreateLoadTimer()
        {
            LoadTimer = new Timer()
            {
                ID = this.ID + "LoadTimer1",
                Interval = 1 //millisecond
            };
            LoadTimer.Tick += new EventHandler<EventArgs>(LoadTimer_Tick);
            AjaxPanel.ContentTemplateContainer.Controls.Add(LoadTimer);
        }
 
        protected void LoadTimer_Tick(object sender, EventArgs e)
        {
            LoadTimer.Enabled = false;
            try
            {
                ApplyUserActions();
            }
            catch (Exception ex)
            {
                Label msg = new Label()
                {
                    Text = "An error occurred in delayed load: " + ex.Message,
                    ToolTip = ex.ToString()
                };
                Controls.Add(msg);
            }
        }
    }
}

The ApplyUserActions method is where you should fetch your data and update the content of the AjaxPanel member control. All the controls of your web-part should be created as usual, remember to call base.CreateChildControls in your derived web-parts to ensure that the Ajax controls get created.

Note that no slow data must be fetched and bound to your web-part controls during page load, e.g. in the CreateChildControls or OnPreRender methods, as this defeats the purpose of delay loading the data in the ApplyUserActions async postback method. A typical scenario is creating and configuring an SPGridView control in CreateChildControls and then fetch the data and set the grid's DataSource and call the grid's DataBind method in the overridden ApplyUserActions method in your derived web-part.

Note that all page event code get executed on partial postbacks for all web-parts on the page. This can cause problems that are unrelated to the web-part that triggers the postback, manifested as ScriptResource.axd JavaScript errors. Some problems are related to viewstate handling, such as the "Error=Value cannot be null. Parameter name: container" SPGridView exception. The simple solution is to turn off viewstate, and then call the RebindControlsWhenNoViewState method to load and bind the data when the postback is not an async Ajax postback. This must also be done for all controls that do not use viewstate, otherwise they will end up empty after e.g. modal dialogs that reload the page on close.

This ASP.NET Timer approach allows the page to load quickly, then each web-part will in turn get the timer tick postback and update itself using ASP.NET Ajax partial page updates. Note that this code won’t work as a sandboxed web-part. The UpdatePanel control requires the ScriptManager, which isn’t accessible from the sandboxed worker process.

The more professional way of getting real asynchronous loading for web-part content is to use PageAsyncTask as shown in Chapter 9 in Wictor Wilen's excellent SharePoint 2010 Web Parts in Action book. It does require a bit more code, but will allow parallell data fetching and thus faster page load time. It also works without using any UpdatePanels as all is done server-side.

Thursday, May 19, 2011

New Sites and the SharePoint 2010 Content Type Hub

The SharePoint 2010 content type hub does quite a good job of managing and publishing a centrally controlled set of content types. There are a few quirks and limitations, some of them documented in Content Type Hub FAQ and Limitations by Chaks' SharePoint Corner; not to forget the content type publishing timer jobs that actually push the content types to the subscribers.

One of the less documented areas of using a content type hub (HUB) is what happens to new site-collections that are provisioned? What if I have list definitions in my features, how can I be sure that their referenced content types have been provisioned at feature activation time? Can I deploy my enterprise content types feature at both the hub site-collection and also at new site-collections that users create?


First, when a new site-collection is created, it will immediately have all the published content types from the parent SharePoint web-application's connected Managed Metadata Service (MMS) application's defined HUB, automatically provisioned into its local content type gallery. Note that this applies only to the published content types as configured in the source hub. Content types that are not published, will not exist in your new site-collection. Note that hub content types are not by default published; this must be configured for every single content type in the source hub.

So if your list definitions depend on global content types that have not been published in the HUB, your feature activation will fail. You can of course solve this by publishing the applicable global content types in the source hub and run the timer jobs first, as this will ensure that new site-collections will have the enterprise content types auto-provisioned from the MMS HUB.

However, you can also deploy your enterprise content types feature to both the content type hub and to any other site-collection that you create. This works fine as the site content type definitions are identical, including the content type ID structure - after all it is the same content type CAML feature. This won't affect subscribing to the content type hub, and publishing new, updated or derived content types from the hub works just as expected.

Activate your site content type feature before activating your list definition feature, or any other feature that depends on the site content types being provisioned, to ensure that they exists locally in the new site-collection even if not yet published in the HUB.

As your taxonomy is subject to change, so are your enterprise content types. Thus, your deployment strategy for enterprise content types needs to handle change. I strongly recommend using the Open-Closed Principle for modifying and extending the enterprise content types. The Open-Closed Principle is based on using a set of immutable base content types that you derive from to make new specialized content types, inheriting fields from the base. The immutable base of the Open-Closed Principle coincides nicely with provisioning global content types through both a feature and the content type hub, as by policy any changes are made by extending the former through the latter.

Even trivial stuff such as providing standardized company templates for Word and other Office applications, is best done by publishing new derived content types. Use the content type hub to inherit your base PzlDocument into PzlDocumentMemo and attach a template, go to "Manage publishing for this content type" to publish the content type. Wait for, or run, the two HUB timer jobs, and then add the Word template to the applicable document libraries.

Now you're in for a surprise later on. The next time you try to create a new site-collection after publishing modifications in the HUB, you might get this "content type is read only" error:


The ULS log typically contains an exception like this:

SPContentTypeReadOnlyException
Error code: -2146232832
The content type is read only or updateChildren is true and one of the child objects of the content type is read only.


The root cause for this is that published content types are by default read-only in the subscribers. What typically leads to this error is the need to use code when provisioning content types, e.g. when renaming and reordering fields, or when adding the enterprise keywords field to your content type. Another typical scenario where code is required is managed metadata fields; see How to provision SharePoint 2010 Managed Metadata columns by Wictor Wilén.

Making changes to the site content type definition in the FeatureActivated code and then calling SPContentType Update with updateChildren=true will work fine, until someone creates a new derived content type in the source hub and publish it. Your carefully tested code will suddenly crash, as the published child content type is read-only! Alas, what better proof that the deployed and the published global content types are the same?

Luckily, the change is isolated to the new inherited content type, thus it can safely be ignored when deploying the base content types. Use this overloaded Update method when modifying the global content types:

public void Update(
         bool updateChildren := true,
         bool throwOnSealedOrReadOnly := false
)

The HUB change did not affect your global content type due to using the Open-Closed governance policy for enterprise content types. See my SharePoint 2010 Open-Closed Taxonomy post to learn more about this recommended policy.

The content type hub and the Managed Metadata Service are perhaps the best new features in SharePoint 2010, still there are some uncharted areas that make developers reluctant at using the MMS HUB. There are a lot of articles at Technet and MSDN on the architecture, but way too little about deployment scenarios and issues such as those in this post.

Saturday, May 07, 2011

Site Lifecycle Management using Retention Policies

The ootb governance tools for site lifecycle management (SLM) in SharePoint 2010 have not improved from the previous version. You're still stuck with the Site Use Confirmation and Deletion policies that will just periodically e-mail site owners and ask them to confirm that their site is still in use. There is no check for the site or its content actually being used, it is just a dumb timer job. If the site is not confirmed as still being active, the site will then be deleted - even if it is still in use. As deleting a site is not covered by any SharePoint recycle bin mechanism (coming in SP1), Microsoft also provides the site deletion capture tool on CodePlex.

Wouldn't it be nice if we could apply the information management policies for retention and disposition of content also for SharePoint 2010 sites? Yes we can :) By using a content type to identify and keep metadata for a site, the standard information management policies for content expiration can be configured to implement a recurring multistage retention policy for site disposition.

Create a site information content type and bind it to a list or library in your site definition, and ensure that this list contains one SiteInfo item with the metadata of the site. Typical metadata are site created date, site contact, site type, cost center, unit and department, is restricted site flag, last review date, next review date, and last update timestamp. Restrict edit permissions for this list to just site owners or admins.

Enable retention for the SiteInfo content type to configure your site lifecycle management policy as defined in your governance plan.


Add one or more retention stages for the SiteInfo content type as needed by your SLM policy. You will typically have a first stage that will start a workflow to notify the site owner of site expiration and ask for disposition confirmation. Make sure that the site owner knows about and enacts on your defined governance policies for manual information management, such as sending valuable documents to records management. Then there will be a second stage for performing the site disposition steps triggered by the confirmation.

You can also implement custom information management policy expiration formula or expiration action for use when configuring your retention policy. You typically do this when your policy requires retention events that are not based on date fields only. See Sahil Malik's Authoring custom expiration policies and actions in SharePoint 2007 which is still valid for SharePoint 2010.


Use a custom workflow or custom expiration action to implement the site disposition steps: user removal, automated content clean-up and archiving, and finally trigger deletion of the site. If the site is automatically deleted by a custom workflow, or marked for deletion to be processed by a custom timer job, or a custom action just sends an e-mail to the site-admin, is up to your SLM policy.

If you need to keep the site in a passive state for e.g. 6 months before deleting it, you can use a delegate control in your site master pages to prevent access to passive sites or you can move the site to an archive web-app that use a "deny write" / "deny all" access policy to prevent access. Note that the former is not real security, just content targeting for the site. The latter is real security, as "deny" web-app policies overrides site specific access rights granted to SharePoint groups and users. This allows for keeping the site users and groups "as-is" in case the site can be reactivated again according to your SLM policies. If site owners can do housekeeping on a site while passive, then grant them access by creating extra "steward" accounts that are not subject to being denied access.

I recommend removing all users from the default site members group before deleting the site, otherwise the site will not be deleted from the site memberships list in the user's my site.

The astute reader may wonder how the content type retention policy knows if the site is actually in use. The answer is quite simple; each SPWeb object provides a LastItemModifiedDate property. This timestamp is also stored in the SharePoint property bag. Use a delegate control in your site's master page to check and push the timestamp to a date-time field the SiteInfo item, so that the rentention policy can trigger on it. Remember to use SystemUpdate when updating the SiteInfo, otherwise you will change the site's LastItemModifiedDate to now. You can also use a custom expiration formula that inspects the last modified timestamp for the site when the information management policy timer job runs.

We also use the site information content type in our Puzzlepart projects to provide a search-driven site directory. It is quite simple to make a nicely categorized and searchable site catalog by simply using one or more customized the search results web-parts. This search-driven catalog can of course be sorted by the search result 'write' managed property, which must be mapped to the crawled property field that contains the LastItemModifiedDate of a site.


Using a search-driven approach makes it unnecessary to have a classic site directory list. The site metadata is simply stored directly in a list within each site, managed by the respective site owners. This is more likely to keep the site metadata up-to-date rather than going stale in a central site directory list that no one maintains.

I hope this post have given you some new ideas on how to store, manage and use site metadata both for site lifecycle management and for providing a relevant search-driven site directory.

Friday, April 29, 2011

Using SP2010 BCS Resource Files for BDC Model Settings

You've probably seen way too many Business Connectivity Services (BCS) demos using SharePoint Designer (SPD), showing how simple it is to connect to a SQL Server 2008 database and automagically create external content types and operations, with external lists that can be used both to display and update external data.

Have you ever wondered how to manage those data source connection settings across multiple SharePoint 2010 farms? How do you change the SQL Server name and other login information when deploying to your test and staging farm, and then to your production farm without using SPD again? Even good BCS books such as Professional Business Connectivity Services in SharePoint 2010 have very little coverage of BCS external system settings in Central Admin and of how to actually use BDC resource files. There is a nice end-to-end overview in the Migrating Business Connectivity Services External Content Types in SharePoint 2010 article on MSDN, also lacking some of these details.

Data source connection settings are stored as External System properties that can be configured from Central Admin by managing the BDC service application. This can be a bit confusing, as you might run into the "there are no configurable properties" message when trying to manage Settings for an External System. The trick is to remember that these settings are not for the external system per se, but for a specific external system instance aka connection. Chose the External Systems view in the ribbon, click the link of the applicable external system to see the instances, then select the instance and finally click Settings in the ribbon. Change the connection properties and click OK.


In the example data connection I've used the Secure Store Service (SSS) application for the login information because the target database requires SQL Server authentication instead of passthrough integrated Windows authentication.

Instead of manually changing these External System settings whenever deploying a new version across your development, testing, staging and production farms, you can use BDC resource files to apply the settings. This is done by exporting and importing resource files, either using Central Admin, code or Powershell.


Prototype and test your BCS solution on your development farm first, then package your solution into a feature as explained in How to: Deploy a Declarative BDC Model with a Feature on MSDN. Remember to change the model name and external system name, including the entity namespace into durable names, or at least change the entity version, from your SPD prototype when packing your BDC model.

The declarative BDC model is really just some XML stored in a BDCM file. In addition to the BDC model file, SharePoint 2010 also supports using BDC resource files for specific metadata elements that commonly change, such as SQL Server connection configuration. A resource file is really just some XML stored in a BDCR file, that can be merged with the stored model without deleting the existing model and its configuration from the BCS metadata store.


The simplest way to get a BDCR file to start with, is to export the BDC resources using Central Admin. Chose the BDC Models view in the ribbon, then select the applicable model and finally click Export in the ribbon. Set the file type to Resource and select which resources to include in the exported BDCR file, typically properties, and click Export to save the selected set of resources.


Edit the the resource XML to change e.g. the name of the SQL Server (RdbConnection Data Source) and save your changes, using one file per BDC model and target farm. Only the XML for settings that should be updated when applying the resource file need be in the file.


The edited BDC resource file can now be applied to the applicable BDC model in one of your farms, typically when deploying a new version of a model to the staging or production farm. Chose the BDC Models view in the ribbon, then select the applicable model and finally click Import in the ribbon. Click Browse to select the applicable resource file and set the file type to Resource. Then select which resources to import from BDCR file, typically properties, and click Import to load the selected set of resources.


Validate that the correct settings were imported by reviewing the import log warnings, and by reviewing e.g. the settings for your external system instance or the permissions for the BDC model and its external content types.


See How to: Use a Resource File to Specify Localized Names, Properties, and Permissions on MSDN to include an edited BDC resource file in a feature in your Visual Studio 2010 package. Note that BDC models created with SPD cannot be exported from CA as they are not complete. Such declarative BDC models must be exported from SPD.