Showing posts with label SearchDriven. Show all posts
Showing posts with label SearchDriven. Show all posts

Wednesday, October 23, 2013

Roadmap for Responsive Design and Dynamic Content in SharePoint 2013

Responsive design combined with dynamic user-driven content and mobile first seems to be the main focus everywhere these days. The approach outlined in Optimizing SharePoint 2013 websites for mobile devices by Waldek Mastykarz and his How we did it series show how it can be achieved using SharePoint 2013.

But what if you have thousands of pages with good content that you want make resposive? What approach will you use to adapt all those articles after migrating the content over from SharePoint 2010? This post suggests a roadmap for gradually transforming your static content to become dynamic content.

First of all, the metadata of your content types must be classified so that you know the ranking of the metadata within a content type, so that it can be used in combination with device channel panels and resposive web design (RWD) techniques to prioritize what to show on what devices. This will most likely introduce more specialized content types with more granular metadata fields. All your content will need to be edited to fit the new content classification, at least the most important content you have. By edited, I mean that the content type of articles must be changed and the content adapted to the new metadata; in addition, selecting a new content type will also cause a new RWD page layout to be used for the content. These new RWD page layouts and master pages are also something you need to design and implement, as part of your new user experience (UX) concept.

While editing all the (most important) existing content, it is also a good time to ensure that the content gets high quality tagging according to your information architecture (IA), as tagging is the cornerstone of a good, dynamic user experience provided by term-driven navigation and search-driven content. Editing the content is the most important job here, tagging the content is not required to enable RWD for your pages.

Doing all of this at once as part of migrating to SP2013 is by experience too much to be realistic, so this is my recommended roadmap:

Phase 1
- Focus on RWD based on the new content types and their new prioritized metadata and new responsive master pages and page layouts
- Quick win: revise the search center to exploit the new search features, even if tagging is postponed to a later phase (IA: findability and search experience)
- Keep the existing information architecture structure, and thus the navigation as-is
- Keep the page content as-is, do not add search-driven content to the pages yet, focus on making the articles responsive
- Most time-consuming effort: adapting the content type of all articles, cutting and pasting content within each article to fit the new prioritized metadata structure

Phase 2
- Focus on your new concept for structure and navigation in SP2013 (IA: content classification and structure, browse and navigate UX)
- Tagging of the articles according to the new IA-concept for dynamic structuring of the content (IA: term sets for taxonomy)
- Keep the page content as-is, no new search-driven UX in this phase, just term-driven navigation
- Most time-consuming effort: tagging all of your articles, try scripting some auto-tagging based on the existing structure of the content

Phase 3
- Focus on search-driven content in the pages according to the new concept  (IA: discover and explore UX)
- New routines and processes for authors, approvers and publishers based on new SP2013 capabilities (IA: content contributor experience)
- Most time-consuming effort: tune and tag the content of all your articles to drive the ranking of the search-driven content according to the new concept

Phase 4
- Content targeting in the pages based on visitor profile segmentation, this kind of user-driven content is also search-driven content realized using query rules (and some code)

The IA aspects in the roadmap are taken from my SharePoint Information Architecture from the Field article.

Monday, May 07, 2012

Exploring Search Results Step-by-Step in SharePoint 2010

Using search to provide a news archive in SharePoint 2010 is a wellknown solution. Just add the core results web-part to a page and configure it to query for your news article content type and sort it in descending order. Then customize the result XSLT to tune the content and layout of the news excerpts to look like a new archive. Add also the search box, the search refiners and the results paging web-parts and you have a functional news archive in no time.

This post is about providing contextual navigation by adding "<< previous", "next >>" and "result" links to the article pages, to allow users to explore the result set in a step-by-step manner. Norwegians will reckognize this way of exploring results from finn.no.


For a user or visitor to be able to navigate the results, the result set must be cached per user. The search results are in XML format, and it contains a sequential id and the URL for each hit. This allows the navigation control to use XPath to locate the current result by id, and get the URLs for the previous and next results. The user query must also be cached so that clicking the "result" link will show the expected search results.

Override the CoreResultsWebPart as shown in my Getting Elevated Search Results in SharePoint 2010 post to add per-user caching of the search results. If your site allows for anonymous visitors, you need to decide on how to keep tab on them. In the code I've used the requestor IP address, which is not 100% foolproof, but this allows me to avoid using cookies for now.

namespace Puzzlepart.SharePoint.Presentation
{
    [ToolboxItemAttribute(false)]
    public class NewsArchiveCoreResultsWebPart : CoreResultsWebPart
    {
        public static readonly string ScopeNewsArticles 
            = "Scope=\"News Archive\"";
 
        private static readonly string CacheKeyResultsXmlDocument 
            = "Puzzlepart_CoreResults_XmlDocument_User:";
        private static readonly string CacheKeyUserQueryString 
            = "Puzzlepart_CoreResults_UserQuery_User:";
        private int _cacheUserQueryTimeMinutes = 720;
        private int _cacheUserResultsTimeMinutes = 30;
 
        protected override void CreateChildControls()
        {
            try
            {
                base.CreateChildControls();
            }
            catch (Exception ex)
            {
                var error = SharePointUtilities.CreateErrorLabel(ex);
                Controls.Add(error);
            }
        }
 
        protected override XPathNavigator GetXPathNavigator(string viewPath)
        {
            //return base.GetXPathNavigator(viewPath);
 
            SetCachedUserQuery();
            XmlDocument xmlDocument = GetXmlDocumentResults();
            SetCachedResults(xmlDocument);
 
            XPathNavigator xPathNavigator = xmlDocument.CreateNavigator();
            return xPathNavigator;
        }
 
 
        private XmlDocument GetXmlDocumentResults()
        {
            XmlDocument xmlDocument = null;
 
            QueryManager queryManager = 
            SharedQueryManager.GetInstance(Page, QueryNumber).QueryManager;
 
            Location location = queryManager[0][0];
            string query = location.SupplementaryQueries;
            if (query.IndexOf(ScopeNewsArticles, 
                StringComparison.CurrentCultureIgnoreCase) < 0)
            {
                string userQuery = 
                    queryManager.UserQuery + " " + ScopeNewsArticles;
                queryManager.UserQuery = userQuery.Trim();
            }
 
            xmlDocument = queryManager.GetResults(queryManager[0]);
            return xmlDocument;
        }
 
        private void SetCachedUserQuery()
        {
            var qs = HttpUtility.ParseQueryString
                    (Page.Request.QueryString.ToString());
            if (qs["resultid"] != null)
            {
                qs.Remove("resultid");
            }
            HttpRuntime.Cache.Insert(UserQueryCacheKey(this.Page), 
               qs.ToString(), null
               Cache.NoAbsoluteExpiration, 
               new TimeSpan(0, 0, _cacheUserQueryTimeMinutes, 0));
        }
 
        private void SetCachedResults(XmlDocument xmlDocument)
        {
            HttpRuntime.Cache.Insert(ResultsCacheKey(this.Page), 
               xmlDocument, null
               Cache.NoAbsoluteExpiration, 
               new TimeSpan(0, 0, _cacheUserResultsTimeMinutes, 0));
        }
 
        private static string UserQueryCacheKey(Page page)
        {
            string visitorId = GetVisitorId(page);
            string queryCacheKey = String.Format("{0}{1}"
                CacheKeyUserQueryString, visitorId);
            return queryCacheKey;
        }
 
        private static string ResultsCacheKey(Page page)
        {
            string visitorId = GetVisitorId(page);
            string resultsCacheKey = String.Format("{0}{1}"
                CacheKeyResultsXmlDocument, visitorId);
            return resultsCacheKey;
        }
 
        public static string GetCachedUserQuery(Page page)
        {
            string userQuery = 
                (string)HttpRuntime.Cache[UserQueryCacheKey(page)];
            return userQuery;
        }
 
        public static XmlDocument GetCachedResults(Page page)
        {
            XmlDocument results = 
                (XmlDocument)HttpRuntime.Cache[ResultsCacheKey(page)];
            return results;
        }
 
        private static string GetVisitorId(Page page)
        {
            //TODO: use cookie for anonymous visitors
            string id = page.Request.ServerVariables["HTTP_X_FORWARDED_FOR"
                ?? page.Request.ServerVariables["REMOTE_ADDR"];
            if(SPContext.Current.Web.CurrentUser != null)
            {
                id = SPContext.Current.Web.CurrentUser.LoginName;
            }
            return id;
        }
    }
}

I've used sliding expiration on the cache to allow for the user to spend some time exploring the results. The result set is cached for a short time by default, as this can be quite large. The user query text is, however, small and cached for a long time, allowing the users to at least get their results back after a period of inactivity.

As suggested by Mikael Svenson, an alternative to caching would be running the query again using the static QueryManager page object to get the result set. This would require using another result key element than the dynamic <id> number to ensure that the current result lookup is not scewed by new results being returned by the search. An example would be using a content type field such as "NewsArticlePermaId" if it exists.

Overriding the GetXPathNavigator method gets you the cached results that the navigation control needs. In addition, the navigator code needs to know which is the result set id of the current page. This is done by customizing the result XSLT and adding a "resultid" parameter to the $siteUrl variable for each hit.

. . . 
 <xsl:template match="Result">
    <xsl:variable name="id" select="id"/>
    <xsl:variable name="currentId" select="concat($IdPrefix,$id)"/>
    <xsl:variable name="url" select="url"/>
    <xsl:variable name="resultid" select="concat('?resultid=', $id)" />
    <xsl:variable name="siteUrl" select="concat($url, $resultid)" />
. . . 

The result set navigation control is quite simple, looking up the current result by id and getting the URLs for the previous and next results (if any) and adding the "resultid" to keep the navigation logic going forever.

namespace Puzzlepart.SharePoint.Presentation
{
    public class NewsArchiveResultsNavigator : Control
    {
        public string NewsArchivePageUrl { get; set; }
 
        private string _resultId = null;
        private XmlDocument _results = null;
 
        protected override void CreateChildControls()
        {
            base.CreateChildControls();
 
            _resultId = Page.Request.QueryString["resultid"];
            _results = NewsArchiveCoreResultsWebPart.GetCachedResults(this.Page);
 
            if(_results == null || _resultId == null)
            {
                //render nothing
                return;
            }
 
            AddResultsNavigationLinks();
        }
 
        private void AddResultsNavigationLinks()
        {
            string prevUrl = GetPreviousResultPageUrl();
            var linkPrev = new HyperLink()
            {
                Text = "<< Previous",
                NavigateUrl = prevUrl
            };
            linkPrev.Enabled = (prevUrl.Length > 0);
            Controls.Add(linkPrev);
 
            string resultsUrl = GetSearchResultsPageUrl();
            var linkResults = new HyperLink()
            {
                Text = "Result",
                NavigateUrl = resultsUrl
            };
            Controls.Add(linkResults);
 
            string nextUrl = GetNextResultPageUrl();
            var linkNext = new HyperLink()
            {
                Text = "Next >>",
                NavigateUrl = nextUrl
            };
            linkNext.Enabled = (nextUrl.Length > 0);
            Controls.Add(linkNext);
        }
 
        private string GetPreviousResultPageUrl()
        {
            return GetSpecificResultUrl(false);
        }
 
        private string GetNextResultPageUrl()
        {
            return GetSpecificResultUrl(true);
        }
 
        private string GetSpecificResultUrl(bool useNextResult)
        {
            string url = "";
 
            if (_results != null)
            {
                string xpath = 
                    String.Format("/All_Results/Result[id='{0}']", _resultId);
                XPathNavigator xNavigator = _results.CreateNavigator();
                XPathNavigator xCurrentNode = xNavigator.SelectSingleNode(xpath);
                if (xCurrentNode != null)
                {
                    bool hasNode = false;
                    if (useNextResult)
                        hasNode = xCurrentNode.MoveToNext();
                    else
                        hasNode = xCurrentNode.MoveToPrevious();
 
                    if (hasNode && 
                        xCurrentNode.LocalName.Equals("Result"))
                    {
                        string resultId = 
                        xCurrentNode.SelectSingleNode("id").Value;
                        string fileUrl = 
                        xCurrentNode.SelectSingleNode("url").Value;
                        url = String.Format("{0}?resultid={1}"
                           fileUrl, resultId);
                    }
                }
            }
 
            return url;
        }
 
        private string GetSearchResultsPageUrl()
        {
            string url = NewsArchivePageUrl;
 
            string userQuery = 
                NewsArchiveCoreResultsWebPart.GetCachedUserQuery(this.Page);
            if (String.IsNullOrEmpty(userQuery))
            {
                url = String.Format("{0}?resultid={1}", url, _resultId);
            }
            else
            {
                url = String.Format("{0}?{1}&resultid={2}"
                    url, userQuery, _resultId);
            }
 
            return url;
        }
 
    }
}

Note how I use the "resultid" URL parameter to discern between normal navigation to a page and result set navigation between pages. If the resultid parameter is not there, then the navigation controls are hidden. The same goes for when there are no cached results. The "result" link could always be visible for as long as the user's query text is cached.

You can also provide this result set exploration capability for all kinds of pages, not just for a specific page layout, by adding the result set navigation control to your master page(s). The result set <id> and <url> elements are there for all kind of pages stored in your SharePoint solution.

Monday, April 30, 2012

Almost Excluding Specific Search Results in SharePoint 2010

Sometimes you want to hide certain content from being exposed through search in certain SharePoint web-applications, even if the user really has access to the information in the actual content source. A scenario is intranet search that is openly used, but in which you want to prevent accidental information exposure. Think of a group working together on reqruiting, where the HR manager use the search center looking for information - you wouldn't want even excerpts of confidential information to be exposed in the search results.

So you carefully plan your content sources and crawl rules to only index the least possible amount of information. Still, even with crawl rules you will often need to tweak the query scope rules to exclude content at a more fine-grained level, or even add new scopes for providing search-driven content to users. Such configuration typically involves using exclude rules on content types or content sources. This is a story of how SharePoint can throw you a search results curveball, leading to accidental information disclosure.

In this scenario, I had created a new content source JobVault for crawling the HR site-collection in another SharePoint web-application, to be exposed only through a custom shared scope. So I tweaked the rules of the existing scopes such as "All Sites" to exclude the Puzzlepart JobVault content source, and added a new JobReqruiting scope that required the JobVault content source and included the content type JobHired and excluded the content type JobFired.

So no shared scopes defined in the Search Service Application (SSA) included JobFired information, as all scopes either excluded the HR content source or excluded the confidential content type. To my surprise our SharePoint search center would find and expose such pages and documents when searching for "you're fired!!!".

Knowing that the search center by default uses the "All Sites" scope when no specific scope is configured or defined in the keyword query, it was back to the SSA to verify the scope. It was all in order, and doing a property search on Scope:"All Sites" got me the expected results with no confidential data in it. The same result for Scope:"JobReqruiting", no information exposure there either. It looked very much like a best bet, but there where no best bet keywords defined for the site-collection.


The search center culprit was the Top Federated Results web-part in our basic search site, by default showing results from the local search index very much like best bets. That was the same location as defined in the core results web-part, so why this difference?

Looking into the details of the "Local Search Results" federated location, the reason became clear: "This location provides unscoped results from the Local Search index". The keyword here is "unscoped".


The solution is to add the "All Sites" scope to the federated location to ensure that results that you want to hide are also excluded from the federated results web-part. Add it to the "Query Template" and optionally also to the "More Results Link Template" under the "Location Information" section in "Edit Federated Location".


Now the content is hidden when searching. Not through query security trimming, but through query filtering. Forgetting to add the filter somewhere can expose the information, but then only to users that have permission to see the content anyway. The results are still security trimmed, so this no actual information disclosure risk.

Note that this approach is no replacement for real information security; if that is what you need, don't crawl confidential information from an SSA that is exposed through openly available SharePoint search, even on your intranet.

Monday, April 16, 2012

Getting Elevated Search Results in SharePoint 2010

I often use the SharePoint 2010 search CoreResultsWebPart in combination with scopes, content types and managed properties defined in the Search Service Application (SSA) for having dynamic search-driven content in pages. Sometimes the users might need to see some excerpt of content that they really do not have access to, and that you don't want to grant them access to either; e.g. to show a summary to anonymous visitors on your public web-site from selected content that is really stored in the extranet web-application in the SharePoint farm.

What is needed then is to execute the search query with elevated privileges using a custom core results web-part. As my colleague Mikael Svenson shows in Doing blended search results in SharePoint–Part 2: The Custom CoreResultsWebPart Way, it is quite easy to get at the search results code and use the SharedQueryManager object that actually runs the query. Create a web-part that inherits the ootb web-part and override the GetXPathNavigator method like this:

namespace Puzzlepart.SharePoint.Presentation
{
    [ToolboxItemAttribute(false)]
    public class JobPostingCoreResultsWebPart : CoreResultsWebPart
    {
        protected override void CreateChildControls()
        {
            base.CreateChildControls();
        }
 
        protected override XPathNavigator GetXPathNavigator(string viewPath)
        {
            XmlDocument xmlDocument = null;
            QueryManager queryManager = 
              SharedQueryManager.GetInstance(Page, QueryNumber)
                .QueryManager;
            SPSecurity.RunWithElevatedPrivileges(delegate()
            {
                xmlDocument = queryManager.GetResults(queryManager[0]);
            });
            XPathNavigator xPathNavigator = xmlDocument.CreateNavigator();
            return xPathNavigator;
        }
    }
}

Running the query with elevated privileges means that it can return any content that the app-pool identity has access to. Thus, it is important that you grant that account read permissions only on content that you would want just any user to see. Remember that the security trimming is done at query time, not at crawl time, with standard SP2010 server search. It is the credentials passed to the location's SSA proxy that is used for the security trimming. Use WindowsIdentity.GetCurrent() from the System.Security.Principal namespace if you need to get at the app-pool account from your code.

You would want to add a scope and/or some fixed keywords to the query in the code before getting the results, in order to prevent malicious or accidental misuse of the elevated web-part to search for just anything in the crawled content of the associated SSA that the app-pool identity has access to. Another alternative is to run the query under another identity than the app-pool account by using real Windows impersonation in combination with the Secure Store Service (see this post for all the needed code) as this allows for using a specific content query account.

The nice thing about using the built-in query manager this way, rather than running your own KeywordQuery and providing your own result XML local to the custom web-part instance, is that the shared QueryManager's Location object will get its Result XML document populated. This is important for the correct behavior for the other search web-parts on the page using the same QueryNumber / UserQuery, such as the paging and refiners web-parts.

The result XmlDocument will also be in the correct format with lower case column names, correct hit highlighting data, correct date formatting, duplicate trimming, getting <path> to be <url> and <urlEncoded>, have the correct additional managed and crawled properties in the result such as <FileExtension> and <ows_MetadataFacetInfo>, etc, in addition to having the row <id> element and <imageUrl> added to each result. If you override by using a replacement KeywordQuery you must also implement code to apply appended query, fixed query, scope, result properties, sorting and paging yourself to gain full fidelity for your custom query web-part configuration.

If you don't get the expected elevated result set in your farm (I've only tested this on STS claims based web-apps; also see ForceClaimACLs for the SSA by my colleague Ole Kristian Mørch-Storstein), then the sure thing is to create a new QueryManager instance within the RWEP block as shown in How to: Use the QueryManager class to query SharePoint 2010 Enterprise Search by Corey Roth. This will give you correctly formatted XML results, but note that the search web-parts might set the $ShowMessage xsl:param to true, tricking the XSLT rendering into show the "no results" message and advice texts. Just change the XSLT to call either dvt_1.body or dvt_1.empty templates based on the TotalResults count in the XML rather than the parameter. Use the <xmp> trick to validate that there are results in the XML that all the search web-parts consumes, including core results and refinement panel.

The formatting and layout of the search results is as usual controlled by overriding the result XSLT. This includes the data such as any links in the results, as you don't want the users to click on links that just will give them access denied errors.

When using the search box web-part, use the contextual scope option for the scopes dropdown with care. The ContextualScopeUrl (u=) parameter will default to the current web-application, causing an empty result set when using the custom core results web-part against a content source from another SharePoint web-application.

Thursday, February 16, 2012

Reusable SPGridView with Multiple Filter and Sort Columns

The venerable SPGridView still has its use in SharePoint 2010 when your data is not stored in a list or accessible as an external content type through BCS. A typical example is when using a KeywordQuery to build a search-driven web-part feeding on results as DataTable as show in How to: Use the SharePoint 2010 Enterprise Search KeywordQuery Class by Corey Roth. Another example is cross-site queries using SPSiteDataQuery.

The SPGridView can use a DataTable as its data source, but several things from sorting arrows to filtering don't work as expected when not using an ObjectDataSource. As Shawn Kirby shows in SPGridView WebPart with Multiple Filter and Sort Columns it is quite easy to implement support for such features.

In this post, I show how to generalize Shawn's web-part code into a SPGridView derived class and a data source class wrapping a DataTable, isolating this functionality from the web-part code itself, for both better reusability and separation of concerns.

First the simple abstract data source class that you must implement to populate your data set:

namespace Puzzlepart.SharePoint.Core
{
    public abstract class SPGridViewDataSource
    {
        public abstract DataTable SelectData(string sortExpression);
 
        protected void Sort(DataTable dataSource, string sortExpression)
        {
            //clean up the sort expression if needed - the sort descending 
            //menu item causes the double in some cases 
            if (sortExpression.ToLowerInvariant().EndsWith("desc desc"))
                sortExpression = sortExpression.Substring(0, sortExpression.Length - 5);
 
            //need to handle the actual sorting of the data
            if (!string.IsNullOrEmpty(sortExpression))
            {
                DataView view = new DataView(dataSource);
                view.Sort = sortExpression;
                DataTable newTable = view.ToTable();
                dataSource.Clear();
                dataSource.Merge(newTable);
            }
        }
    }
}

The SPGridViewDataSource class provides the SelectData method that you must override, and a completed Sort method that allows the SPGridView to sort your DataTable. Note that this class must be stateless as required by any class used as an ObjectDataSource. Its logic cannot be combined with the grid view class, as it will get instantiated new every time the ObjectDataSource calls SelectData.

Then the derived grid view with support for filtering and sorting, including the arrows and filter images:

namespace Puzzlepart.SharePoint.Core
{
    public class SPGridViewMultiSortFilter : SPGridView
    {
        public SPGridViewMultiSortFilter()
        {
            this.FilteredDataSourcePropertyName = "FilterExpression";
            this.FilteredDataSourcePropertyFormat = "{1} = '{0}'";            
        }
 
        private ObjectDataSource _gridDS;
        private char[] _sortingSeparator = { ',' };
        private string[] _filterSeparator = { "AND" };
 
        public ObjectDataSource GridDataSource
        {
            get { return _gridDS; }
            private set
            {
                _gridDS = value;
                this.DataSourceID = _gridDS.ID;
            }
        }
 
        public bool AllowMultiSorting { get; set; }
        public bool AllowMultiFiltering { get; set; }
 
        string FilterExpression
        {
. . .
        }
 
        string SortExpression
        {
. . .
        }
 
        protected override void CreateChildControls()
        {
            base.CreateChildControls();
 
            this.Sorting += new GridViewSortEventHandler(GridView_Sorting);
            this.RowDataBound += new GridViewRowEventHandler(GridView_RowDataBound);
        }
 
        protected void GridView_Sorting(object sender, GridViewSortEventArgs e)
        {
            EnsureChildControls();
            string direction = e.SortDirection.ToString();
            direction = (direction == "Descending") ? " DESC" : "";
 
            SortExpression = e.SortExpression + direction;
            e.SortExpression = SortExpression;
 
            //keep the object dataset filter
            if (!string.IsNullOrEmpty(FilterExpression))
            {
                _gridDS.FilterExpression = FilterExpression;
            }
        }
 
        protected void GridView_RowDataBound(object sender, GridViewRowEventArgs e)
        {
            EnsureChildControls();
            if (sender == null || e.Row.RowType != DataControlRowType.Header)
            {
                return;
            }
 
            BuildFilterView(_gridDS.FilterExpression);
            SPGridView grid = sender as SPGridView;
 
            // Show icon on filtered and sorted columns 
            for (int i = 0; i < grid.Columns.Count; i++)
            {
. . .
            }
        }
 
        void BuildFilterView(string filterExp)
        {
. . .
 
            //update the filter
            if (!string.IsNullOrEmpty(lastExp))
            {
                FilterExpression = lastExp;
            }
 
            //reset object dataset filter
            if (!string.IsNullOrEmpty(FilterExpression))
            {
                _gridDS.FilterExpression = FilterExpression;
            }
        }
 
        public ObjectDataSource SetObjectDataSource(string dataSourceId, SPGridViewDataSource dataSource)
        {
            ObjectDataSource gridDS = new ObjectDataSource();
            gridDS.ID = dataSourceId;
            gridDS.SelectMethod = "SelectData";
            gridDS.TypeName = dataSource.GetType().AssemblyQualifiedName;
            gridDS.EnableViewState = false;
            gridDS.SortParameterName = "SortExpression";
            gridDS.FilterExpression = FilterExpression;
            this.GridDataSource = gridDS;
 
            return gridDS;
        }
    }
}

Only parts of the SPGridViewMultiSortFilter code is shown here, see download link below for the complete code. Note that I have added two properties that controls whether multi-column sorting and multi-column filtering are allowed or not.

This is an excerpt from a web-part that shows search results using the grid:

namespace Puzzlepart.SharePoint.Presentation
{
    [ToolboxItemAttribute(false)]
    public class JobPostingRollupWebPart : WebPart
    {
        protected SPGridViewMultiSortFilter GridView = null;
 
        protected override void CreateChildControls()
        {
            try
            {
                CreateJobPostingGrid();
            }
            catch (Exception ex)
            {
                Label error = new Label();
                error.Text = String.Format("An unexpected error occurred: {0}", ex.Message);
                error.ToolTip = ex.StackTrace;
                Controls.Add(error);
            }
        }
 
        private void CreateJobPostingGrid()
        {
            //add to control tree first is important for view state handling  
            Panel panel = new Panel();
            Controls.Add(panel);

            GridView = new SPGridViewMultiSortFilter();
 
  . . .
 
            GridView.AllowSorting = true;
            GridView.AllowMultiSorting = false;
            GridView.AllowFiltering = true;
            GridView.FilterDataFields = "Title,Author,Write,";
 
  . . .
 
            panel.Controls.Add(GridView) 

            //set PagerTemplate after adding grid to control tree

 
            PopulateGridDataSource();
 
            //must bind in OnPreRender
            //GridView.DataBind();  
        }
 
        protected override void OnPreRender(EventArgs e)
        {
            GridView.DataBind();
        }
 
        private void PopulateGridDataSource()
        {
            var dataSource = new ApprovedJobPostingDataSource();
            var gridDS = GridView.SetObjectDataSource("gridDS", dataSource);
            //add the data source
            Controls.Add(gridDS);
        }
    }
}

Note how the data source is created and assigned to the grid view, but also added to the control set of the web-part itself. This is required for the grid's DataSourceId binding to find the ObjectDataSource at run-time. Also note that data binding cannot be triggered from CreateChildControls as it is too early in the control's life cycle. The DataBind method must be called from OnPreRender to allow for view state and child controls to load before the sorting and filtering postback events

Finally, this is an example of how to implement a search-driven SPGridViewDataSource:

namespace Puzzlepart.SharePoint.Presentation
{
    public class ApprovedJobPostingDataSource : SPGridViewDataSource
    {
        private string _cacheKey = "Puzzlepart_Godkjente_Jobbannonser";
 
        public override DataTable SelectData(string sortExpression)
        {
            DataTable dataTable = (DataTable)HttpRuntime.Cache[_cacheKey];
            if (dataTable == null)
            {
                dataTable = GetJobPostingData();
                HttpRuntime.Cache.Insert(_cacheKey, dataTable, null
DateTime.Now.AddMinutes(1), Cache.NoSlidingExpiration);
            }
 
            this.Sort(dataTable, sortExpression);
 
            return dataTable;
        }
 
        private DataTable GetJobPostingData()
        {
            DataTable results = new DataTable();
            string jobPostingManager = "Puzzlepart Jobbannonse arbeidsleder";
            string jobPostingAssistant = "Puzzlepart Jobbannonse assistent";
            string approvedStatus = "0";
 
            SPSite site = SPContext.Current.Site;
            KeywordQuery query = new KeywordQuery(site);
            query.QueryText = String.Format(
"ContentType:\"{0}\" ContentType:\"{1}\" ModerationStatus:\"{2}\""
jobPostingManager, jobPostingAssistant, approvedStatus);
            query.ResultsProvider = SearchProvider.Default;
            query.ResultTypes = ResultType.RelevantResults;
 
            ResultTableCollection resultTables = query.Execute();
            if (resultTables.Count > 0)
            {
                ResultTable searchResults = resultTables[ResultType.RelevantResults];
                results.Load(searchResults, LoadOption.OverwriteChanges);
            }
 
            return results;
        }
    }
}

That was not too hard, was it? Note that SearchProvider Default work for both FAST (FS4SP) and a standard SharePoint 2010 Search Service Application (SSA).

All the code can be downloaded from here.

Tuesday, November 01, 2011

SharePoint Information Architecture from the Field

Over the years, I've written quite a few articles on how to technically structure your SharePoint solutions into web-apps, sites, subsites, lists and document libraries. All based on having a defined Information Architecture (IA) as a basis for the solution design, or at least being able to reason about your content management using my "chest of drawers" and "news paper" analogies. The latter is based on my experiences from the field as most companies don't have a well-defined IA in place.

Common questions from customers are "what is Information Architecture?" and "what is the value of having an IA for SharePoint, can't we just create sites and doc-libs on the fly as needed?". Not to forget "how do we go about creating an Information Architecture for SharePoint?". So here are some IA advice from Puzzlepart projects:

In short, Information Architecture defines how to classify and structure your content so that it is easy for content consumers to find and explore relevant content, while making it simple for workers to contribute and manage content in an efficient manner.

The business goal of having an Information Architecture for your SharePoint solution is enabling workers to contribute, store and mange content in a manner that is simple and efficient, enabling more content sharing; and at the same time making it easy for workers to browse and find content they need, while also making it easy for workers to discover and explore relevant content they didn't know of. The outcome is more knowledgeable workers that are better informed about what’s going on in the company and about the range of intellectual property possessed by other employees, while also saving time wasted on finding information, and time wasted on incorrect or outdated information.

An outcome is a metric that customers use to define the successful realization of the objectives and goals. Outcomes happens after the project has delivered on its objectives and goals, and the customers must themselves work against securing the outcomes to achieve the desired business value.

The business value of having a working IA is capturing company knowledge from employees with better quality of shared content, which combined with good findability drive more knowledgeable workers that make better decisions and better faster processes. In addition, more and better content sharing helps user not only discover and explore content, but also people such as subject matter experts, allowing employees to build and expand their network throughout the company, helping the company to retain talented employees through social ties and communities. Access to discover more and better content and people expertise is central to enabling innovation and process improvement, as new knowledge is a trigger for new ideas and for identifying new opportunities.

The process of defining your IA for your SharePoint solution should focus on these objectives and goals:
  • Analyze and define the content classification and structure for the solution 
    • goal: identify what content to manage and plan how to store it in SP, leading to sites, subsites and doc-lib structure organized into SP web-apps
  • Analyze and define how to browse and navigate the content
    • goal: make it simple and efficient for users to find and use known content that they need in their daily work to drive better faster processes
  • Analyze and define how to discover and explore the content
    • goal: make it easy for users to stumble upon novel shared knowledge based on "common focus" to trigger innovation and build social ties
  • Provide simple and efficient content contributor experience with liberal appliance of default metadata values, storing content close to the authors 
    • goal: make workers contributors, not knowledge management grunts, and help them store content correctly with better metadata and tagging, driving findability and "common focus" content discovery; drive better sharing and collaboration
  • Analyze and define the starter content types with metadata and term set taxonomy based on the defined site and doc-lib architecture, with a strong focus on needed search experience capabilities
    • goal: enable content management and support both search-driven and "common focus" content; drive findability, sharing and innovation
  • Analyze and define the policies for social tagging and rating in the solution, also in relation to user profile interests, skills and responsibility tagging
    • goal: drive "common focus" content discovery, drive findability, drive social communities, drive innovation
  • Analyze and define the search experience, focusing on both search-driven content and on search center scopes and refiners
    • goal: drive findability and provide both search-driven and "common focus" content
  • Enable disposition of redundant and irrelevant content 
    • goal: provide users with better, correct and up-to-date information, drive findability, save storage cost, save process cost
Including the search experience helps you avoid an initial version of your IA with a too narrow scope, which is easy to do for a starter IA when e.g. analyzing only project collaboration needs or just the document types of an attachment-based intranet. It makes you focus on what you get out, not only what you put in.

Note that navigation is not IA, its just one way to explore the content. Using navigation to structure your content is just reapplying the fileshare folder approach, which we all know doesn't work too good for findability and discovery. Navigation should not define the statical IA structure for the content, do a LATCH analysis to model the possible IA structures, and choose one of them to define the statical IA structure. The site map is closer to define statical IA structure than navigation, still it is only good for logical IA structure and cannot be expected to be used directly as the physical IA structure in SharePoint.

In an upcoming article I will give some practical advice from the field on how to define and realize the Information Architecture for your SharePoint solution in an agile fashion.

Saturday, May 07, 2011

Site Lifecycle Management using Retention Policies

The ootb governance tools for site lifecycle management (SLM) in SharePoint 2010 have not improved from the previous version. You're still stuck with the Site Use Confirmation and Deletion policies that will just periodically e-mail site owners and ask them to confirm that their site is still in use. There is no check for the site or its content actually being used, it is just a dumb timer job. If the site is not confirmed as still being active, the site will then be deleted - even if it is still in use. As deleting a site is not covered by any SharePoint recycle bin mechanism (coming in SP1), Microsoft also provides the site deletion capture tool on CodePlex.

Wouldn't it be nice if we could apply the information management policies for retention and disposition of content also for SharePoint 2010 sites? Yes we can :) By using a content type to identify and keep metadata for a site, the standard information management policies for content expiration can be configured to implement a recurring multistage retention policy for site disposition.

Create a site information content type and bind it to a list or library in your site definition, and ensure that this list contains one SiteInfo item with the metadata of the site. Typical metadata are site created date, site contact, site type, cost center, unit and department, is restricted site flag, last review date, next review date, and last update timestamp. Restrict edit permissions for this list to just site owners or admins.

Enable retention for the SiteInfo content type to configure your site lifecycle management policy as defined in your governance plan.


Add one or more retention stages for the SiteInfo content type as needed by your SLM policy. You will typically have a first stage that will start a workflow to notify the site owner of site expiration and ask for disposition confirmation. Make sure that the site owner knows about and enacts on your defined governance policies for manual information management, such as sending valuable documents to records management. Then there will be a second stage for performing the site disposition steps triggered by the confirmation.

You can also implement custom information management policy expiration formula or expiration action for use when configuring your retention policy. You typically do this when your policy requires retention events that are not based on date fields only. See Sahil Malik's Authoring custom expiration policies and actions in SharePoint 2007 which is still valid for SharePoint 2010.


Use a custom workflow or custom expiration action to implement the site disposition steps: user removal, automated content clean-up and archiving, and finally trigger deletion of the site. If the site is automatically deleted by a custom workflow, or marked for deletion to be processed by a custom timer job, or a custom action just sends an e-mail to the site-admin, is up to your SLM policy.

If you need to keep the site in a passive state for e.g. 6 months before deleting it, you can use a delegate control in your site master pages to prevent access to passive sites or you can move the site to an archive web-app that use a "deny write" / "deny all" access policy to prevent access. Note that the former is not real security, just content targeting for the site. The latter is real security, as "deny" web-app policies overrides site specific access rights granted to SharePoint groups and users. This allows for keeping the site users and groups "as-is" in case the site can be reactivated again according to your SLM policies. If site owners can do housekeeping on a site while passive, then grant them access by creating extra "steward" accounts that are not subject to being denied access.

I recommend removing all users from the default site members group before deleting the site, otherwise the site will not be deleted from the site memberships list in the user's my site.

The astute reader may wonder how the content type retention policy knows if the site is actually in use. The answer is quite simple; each SPWeb object provides a LastItemModifiedDate property. This timestamp is also stored in the SharePoint property bag. Use a delegate control in your site's master page to check and push the timestamp to a date-time field the SiteInfo item, so that the rentention policy can trigger on it. Remember to use SystemUpdate when updating the SiteInfo, otherwise you will change the site's LastItemModifiedDate to now. You can also use a custom expiration formula that inspects the last modified timestamp for the site when the information management policy timer job runs.

We also use the site information content type in our Puzzlepart projects to provide a search-driven site directory. It is quite simple to make a nicely categorized and searchable site catalog by simply using one or more customized the search results web-parts. This search-driven catalog can of course be sorted by the search result 'write' managed property, which must be mapped to the crawled property field that contains the LastItemModifiedDate of a site.


Using a search-driven approach makes it unnecessary to have a classic site directory list. The site metadata is simply stored directly in a list within each site, managed by the respective site owners. This is more likely to keep the site metadata up-to-date rather than going stale in a central site directory list that no one maintains.

I hope this post have given you some new ideas on how to store, manage and use site metadata both for site lifecycle management and for providing a relevant search-driven site directory.

Thursday, April 28, 2011

Using Dynamic Stored Procedures in BCS Finders

We use quite a lot of Business Connectivity Services (BCS) at my current SharePoint 2010 project both for traditional integration of data from external systems into web-parts and lists, and also for crawling external systems for integrating those system using search and search-driven web-parts.

One of our integration partners prefers to provide their integration points as SQL Server 2008 stored procedures, which is very well supported by BCS. BCS supports both stored procedures and table valued functions, called "routines" in SharePoint Designer (SPD). SharePoint Designer is dependent on being able to extract metadata about the returned table data set when adding External Content Type (ECT) operations or when using the Operations Design View.

Alas, the provided integration sprocs used dynamic SQL statements, and for technical reasons this could not be rewritten to inline SQL select statements. This is as always a problem with tooling such as SPD, as no result set metadata can be discovered. When connecting SPD to the external system, I got no fields in the Data Source Elements panel in the Read List operation's Return Parameter Configuration. Rather I got three errors and a warning.


The workaround is quite simple and requires the use of a SQL Server table variable, which defines the result set and allows SPD to discover the table metadata. Rewrite the stored procedure by declaring a table variable, insert the result of the dynamic SQL statement into the variable, and finally return the result set by reading the table variable. The changes to the sproc is shown in blue in this example:

CREATE PROCEDURE [dbo].[GetFavorites]
AS
BEGIN
SET NOCOUNT ON;
 
DECLARE @DbName AS NVARCHAR(max) = 'ARISModellering1'
DECLARE @ObjDef AS NVARCHAR(max) = dbo.GetArisTableName(@DbName, 'ObjDef')
DECLARE @Model AS NVARCHAR(max) = dbo.GetArisTableName(@DbName, 'Model')
 
DECLARE @FavoriteSQL AS NVARCHAR(max) = N'
SELECT
ARISBPDATA.BOOKMARKS.ID AS FavoriteId
, ARISBPDATA.BOOKMARKS.DESCRIPTION AS FavoriteName
, ModelType.ModelTypeName AS FavoriteType
, LOWER(ARISBPDATA.BOOKMARKS.USERNAME) AS UserName
, ''http://puzzlepart/index.jsp?ExportName=ARISModellering&modelGUID='
+ ARISBPDATA.BOOKMARKS.DATAKEY AS FavoriteUrl
FROM
ARISBPDATA.BOOKMARKS
INNER JOIN ' + @Model + ' m ON ARISBPDATA.BOOKMARKS.OBJID = m.Id
INNER JOIN ModelType ON m.TypeNum = ModelType.ModelTypeId
'
 
 
DECLARE @userFavs TABLE(FavoriteId int not null, FavoriteName nvarchar(max), 
FavoriteType nvarchar(max), UserName nvarchar(max), FavoriteUrl nvarchar(max))
 
insert @userFavs EXEC sp_executeSQL @FavoriteSQL
 
select * from @userFavs
 
END

Refreshing the external system data connection and then creating the ECT read list operation now works fine, and all the return type errors and warnings are gone.


Note that the classic #temp table workaround won't work with SPD, you have to use a table variable in your stored procedure. The sproc will now use more memory, so the BCS best practice for keeping finder result sets small applies.

The table_variable declaration is also a good place to make sure that the identifier column is "not null" and that it is a supported BCS data type such as "int32". External lists cannot be created from an ECT whose identifier field is unsupported, such as SQL Server "bigint", and I strongly recommend using a supported identifier data type right from the start. Getting the ECT identifier wrong in the BCS model will give you problems later on when using SharePoint Designer.

Friday, August 27, 2010

SharePoint 2010 My Tasks Web Part using Search Driven Cross-Site Query with Muenchian Grouping

There always seems to be a requirement for rolling up data from all sites in one or more SharePoint solutions, such as getting a list of my tasks, a list of new documents this week, or creating a searchable news archive for publishing sites; or such as creating a site map or dynamic site directory based on metadata collected in your site provisioning workflow, that are later maintained by site owners.

SharePoint has several web-parts that can do cross-list and cross-subsite queries, such as the Content Query web-part, but all restricted to a single site-collection. In addition, there are the classic Data View web-part and the new XSLT List View web-parts that can be configured using SharePoint Designer. These web-parts can connect to a diverse set of data sources, from internal SharePoint lists to external REST and OData services.

Still, the simplest solution for cross-site/cross-solution rollups is to customize the ootb search web-parts against custom search scopes in the Search Service application. In most cases, no coding will be required, pure configuration of SharePoint will go a long way. This post will show how to configure a search driven "My Tasks" web-part that will show all tasks assigned to the user across all SharePoint sites across all indexed SharePoint solutions. The unstyled cross-site task rollup web-part looks like this, included some debug info:


First you need to configure the results scope behind the search driven web-part in Central Admin. Start by adding a new scope in 'Search Service Application>Scopes' called TaskRollup using the rules as shown here:


If you can't see ContentType when adding a rule, then go to 'Search Service Application>Metadata Properties' and edit the managed property to set Allow this property to be used in scopes.

As the TaskStatus site column is not mapped to any managed property by default, you must map the crawled property ows_Status to one before it can be used. Go to 'Search Service Application>Metadata Properties' and create a managed property called TaskStatus using the mapping as shown here:


Do not go creative with the naming, stay away from spaces and special characters such as ÆØÅ - a SharePoint best practice for any artifact name used as an identifier or an URL fragment. For example, a name like "Contoso Web Ingress" first gets encoded as "Contoso_x0020_Web_x0020_Ingress" when stored, and then once more encoded as "Contoso_x005F_x0020_Web_x005F_x0020_Ingress" in a search result XML.

A full crawl is required after adding or changing crawled or managed properties. Do a full crawl of the content source you used in the TaskRollup scope. Note that there must be some matching content stored in SharePoint for these properties to be added to the property database in the first place. Thus after provisioning new site content types or site columns, you must add some sample content and then do a full recrawl of the applicable content source.

Verifying that the full crawl of the SharePoint sites content source finished without errors completes the Central Admin configuration. Now it's time to configure the ootb Search Core Results web-part to become the customized My Tasks web-part.

Open a team-site and add the Search Core Results web-part to a page. Switch to page edit mode and select 'Edit Web Part' to open the Search Core Results settings panel. Rename the web-part 'Title' to Task Rollup (cross-site) and set the 'Cross Web-Part Query ID' to User query and 'Fixed Keyword Query' to scope: "TaskRollup" as shown here:


The Search Core Results web-part requires a user query, or a configured fixed or appended query, to actually perform a search. No configured or no user query will just show a message asking for query input. The cross-page query ID setting User query is chosen here for reasons explained later.

If you want to further limit what tasks are shown in the My Tasks web-part, just add more query keywords to the 'Append Text to Query' setting as shown here:


The My Tasks web-part will show the two task fields 'Status' and 'Assigned to' in the task list. Any managed crawled property can be added to the search results by configuring the 'Fetched Properties' setting. Add the following XML <Column Name="AssignedTo"/> <Column Name="TaskStatus"/> as shown here:


You need to uncheck the 'Use Location Visualization' setting to enable the controls for customizing the result set and XSL formatting. See A quick guide to CoreResultsWebPart configuration changes in SharePoint 2010 by Corey Roth to learn more about the new search location concept in SharePoint 2010. Read all his Enterprise Search posts for an excellent introduction to the improved SharePoint 2010 search services and web-parts.

After adding 'TaskStatus' and 'AssignedTo' to the fetched properties, you will also need to customize the XSL used to format and show the search results to also include your extra task fields. Click the 'XSL Editor' button in the 'Display Properties' section of the web-part settings panel, and add the fields to the match="Result" xsl:template according to your design. Note that the property names must be entered in lower case in the XSL.

The astute reader will have noticed the nice grouping of the search results. This is done using the Muenchian method as SharePoint 2010 still uses XLST 1.0, thus no simple XSLT 2.0 xsl:for-each-group. The customized "My Tasks" results XSL creates a key called 'tasks-by-status' that selects 'Result' elements and groups them on the 'taskstatus' field as shown here:


Again, note the requirement for lower case names for the fetched properties when used in the XSL. Use the <xmp> trick to see the actual result XML.

The final part of the puzzle is how to turn the cross-site task list into a personal task list. Unfortunately, the [Me] and [Today] filter tokens cannot be used in the enterprise search query syntax, so some coding is required to add such dynamic filter tokens. Export the customized Search Core Results web-part to disk to start packaging into a WSP solution.

Create a new TaskRollupWebPart web-part SPI in your web-parts feature in Visual Studio 2010. Make the new web-part class inherit from CoreResultsWebPart in the Microsoft.Office.Server.Search assembly. Override the methods shown here to add dynamic filtering of the query through the SharedQueryManager for the web-part page:

namespace PuzzlepartTaskRollup.WebParts
{
[ToolboxItemAttribute(false)]
public class TaskRollupWebPart : 
Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart
{
QueryManager _queryManager;
protected override void OnInit(EventArgs e) {
  base.OnInit(e);
  _queryManager = SharedQueryManager.GetInstance(this.Page).QueryManager;
}
 
protected override System.Xml.XPath.XPathNavigator GetXPathNavigator(string viewPath)
{
  SPUser user = SPContext.Current.Web.CurrentUser;
  _queryManager.UserQuery = string.Format("scope:\"TaskRollup\" AssignedTo:\"{0}\""
user.Name);
  return base.GetXPathNavigator(viewPath);
}
 
protected override void CreateChildControls()
{
  base.CreateChildControls();
  //debug info
  //Controls.Add(new Label { Text = string.Format("FixedQuery: {0}<br/>
AppendedQuery: {1}<br/>UserQuery: {2}", 
FixedQuery, AppendedQuery, _queryManager.UserQuery) });
}
}
}

The code in GetXPathNavigator is what adds the current user to the QueryManager.UserQuery to filter tasks based on the assigned user by [me]. There are five query objects available on a search web-part page, where QueryId.Query1 is the default. This is also what is exposed in the web-part settings as the 'User Query' option. Use the GetInstance(Page, QueryId) overload in SharedQueryManager to get at a specific cross-page query object.

Replace the content of the TaskRollupWebPart.webpart file with the exported Search Core Results configuration. This will ensure that all the configuration done to customize the ootb web-part into the My Tasks web-part is applied to the new TaskRollupWebPart. A small change is needed in the metadata type element to load the new TaskRollupWebPart code rather than the CoreResultsWebPart code:

<webParts>
<webPart xmlns="http://schemas.microsoft.com/WebPart/v3">
<metaData>
<type name="PuzzlepartTaskRollup.WebParts.TaskRollupWebPart, 
$SharePoint.Project.AssemblyFullName$" />
<importErrorMessage>$Resources:core,ImportErrorMessage;</importErrorMessage>
</metaData>

Build the feature and deploy the package to your test site from Visual Studio 2010. Add the web-part to a page and verify that you get only your tasks as expected.

I know that this seems like a lot of work, but a search-driven web-part is easily created and tested before lunch. The inevitable styling & layout using XSL and CSS is what will burn hours, as usual.
 
A drawback of search driven web-parts or code is the delay before new/updated content is shown due to the periodical crawling schedule, typically five or ten minutes. On the positive side, the results will be automatically security trimmed for you based on the logged on user - no authentication hassles or stored username password required as with the XSL List View.

Note that most enterprise search classes are still sealed in SharePoint 2010 as in SharePoint 2007, except the CoreResultsWebPart and some new classes, so you're limited to what customizations can be achieved with configuration or the SharedQueryManager. Search driven web-parts works equally well in SharePoint 2007, except that there is no SharedQueryManager, but rather the infamous search results hidden object (SRHO) which is unsupported.

Recommended: SharePoint Search XSL Samples and the Search Community Toolkit at CodePlex.