Controlling Content Database Size in SharePoint

A SharePoint content database can be up to 4TB with data (max 200GB is recommended). However, storage size is not the problem; it is the recovery time to restore all that data that is the availability problem. The recovery time decides for how long your business critical solution will be down. As SharePoint can spread its content across multiple databases, it is recommended that your architecture segments different content across different databases based on IA and other user experience aspects, plus business requirements for availability and recovery time. Plan for structuring your solutions with a strong focus on your information architecture (IA).

Here are some options for how to control the size of the content databases, without disposing and deleting content:

A) Use an ootb Record Center as an archive for old content: The users must manually send each document to the RC using e.g. move and leave a link; note that only the latest major version with metadata is kept – all version history is lost. The information management policies supported by SharePoint for retention and disposition can be used to automate the cleanup.
As the RC has its own content databases, the live collaboration databases will grow slower or even shrink as outdated information is moved to the archive. Keeping the live databases small ensures shorter recovery time; while the recovery time for the archived content can be considerable, but not business critical.
Search must be configured appropriately to cover both live and archived content.

B) Use a third-party archiving solution for SharePoint from e.g. MetaLogix or AvePoint. This has the same pros & cons as in option A, but the functionality is probably better in relation to keeping version history and batch management of outdated content.
Search must be configured appropriately to cover both live and archived content.

C) Use a third-party remote blob storage (RBS) solution for SharePoint, such as MetaLogix StoragePoint, so that documents are registered in the database, but not stored there. This gives smaller content databases, but more complicated backup and recovery as the content now resides both in databases and on disk. Provided that you don’t lose both at the same time, the recovery time should be shorter.
Search will work as before, as all content is still logically in the “database”.

D) Use powershell scripts or other code to implement the disposition of outdated content. The script can e.g. copy old documents to disk and delete old versions from the content database; the drawback being that all metadata will be lost and there is no link left in SharePoint.
The databases size will shrink as data is actually deleted, and backup and recovery is more complicated as content is now both in the database and on disk (same as for option C).
Search can be configured to also crawl and index the files on disk, but content ranking will suffer as the valuable metadata is lost.

My recommendation is to consider option A first, especially if you are able to define automated rules and exploit the built-in information management policies in SharePoint. The keyword is *able* - in my experience, everyone is positive to having automated retention and disposition, but noone even at large banks and law firms are able to come up with the policies.

Always consider using RBS for databases larger than 200GB, and note that RBS also helps you meet the disk IOPS requirements of SharePoint.

New Sites and the SharePoint 2010 Content Type Hub

The SharePoint 2010 content type hub does quite a good job of managing and publishing a centrally controlled set of content types. There are a few quirks and limitations, some of them documented in Content Type Hub FAQ and Limitations by Chaks' SharePoint Corner; not to forget the content type publishing timer jobs that actually push the content types to the subscribers.

One of the less documented areas of using a content type hub (HUB) is what happens to new site-collections that are provisioned? What if I have list definitions in my features, how can I be sure that their referenced content types have been provisioned at feature activation time? Can I deploy my enterprise content types feature at both the hub site-collection and also at new site-collections that users create?

First, when a new site-collection is created, it will immediately have all the published content types from the parent SharePoint web-application's connected Managed Metadata Service (MMS) application's defined HUB, automatically provisioned into its local content type gallery. Note that this applies only to the published content types as configured in the source hub. Content types that are not published, will not exist in your new site-collection. Note that hub content types are not by default published; this must be configured for every single content type in the source hub.

So if your list definitions depend on global content types that have not been published in the HUB, your feature activation will fail. You can of course solve this by publishing the applicable global content types in the source hub and run the timer jobs first, as this will ensure that new site-collections will have the enterprise content types auto-provisioned from the MMS HUB.

However, you can also deploy your enterprise content types feature to both the content type hub and to any other site-collection that you create. This works fine as the site content type definitions are identical, including the content type ID structure - after all it is the same content type CAML feature. This won't affect subscribing to the content type hub, and publishing new, updated or derived content types from the hub works just as expected.

Activate your site content type feature before activating your list definition feature, or any other feature that depends on the site content types being provisioned, to ensure that they exists locally in the new site-collection even if not yet published in the HUB.

As your taxonomy is subject to change, so are your enterprise content types. Thus, your deployment strategy for enterprise content types needs to handle change. I strongly recommend using the Open-Closed Principle for modifying and extending the enterprise content types. The Open-Closed Principle is based on using a set of immutable base content types that you derive from to make new specialized content types, inheriting fields from the base. The immutable base of the Open-Closed Principle coincides nicely with provisioning global content types through both a feature and the content type hub, as by policy any changes are made by extending the former through the latter.

Even trivial stuff such as providing standardized company templates for Word and other Office applications, is best done by publishing new derived content types. Use the content type hub to inherit your base PzlDocument into PzlDocumentMemo and attach a template, go to "Manage publishing for this content type" to publish the content type. Wait for, or run, the two HUB timer jobs, and then add the Word template to the applicable document libraries.

Now you're in for a surprise later on. The next time you try to create a new site-collection after publishing modifications in the HUB, you might get this "content type is read only" error:

The ULS log typically contains an exception like this:

SPContentTypeReadOnlyException
Error code: -2146232832
The content type is read only or updateChildren is true and one of the child objects of the content type is read only.

The root cause for this is that published content types are by default read-only in the subscribers. What typically leads to this error is the need to use code when provisioning content types, e.g. when renaming and reordering fields, or when adding the enterprise keywords field to your content type. Another typical scenario where code is required is managed metadata fields; see How to provision SharePoint 2010 Managed Metadata columns by Wictor Wilén.

Making changes to the site content type definition in the FeatureActivated code and then calling SPContentType Update with updateChildren=true will work fine, until someone creates a new derived content type in the source hub and publish it. Your carefully tested code will suddenly crash, as the published child content type is read-only! Alas, what better proof that the deployed and the published global content types are the same?

Luckily, the change is isolated to the new inherited content type, thus it can safely be ignored when deploying the base content types. Use this overloaded Update method when modifying the global content types:

public void Update(

         bool updateChildren := true,

         bool throwOnSealedOrReadOnly := false

)

The HUB change did not affect your global content type due to using the Open-Closed governance policy for enterprise content types. See my SharePoint 2010 Open-Closed Taxonomy post to learn more about this recommended policy.

The content type hub and the Managed Metadata Service are perhaps the best new features in SharePoint 2010, still there are some uncharted areas that make developers reluctant at using the MMS HUB. There are a lot of articles at Technet and MSDN on the architecture, but way too little about deployment scenarios and issues such as those in this post.

Site Lifecycle Management using Retention Policies

The ootb governance tools for site lifecycle management (SLM) in SharePoint 2010 have not improved from the previous version. You're still stuck with the Site Use Confirmation and Deletion policies that will just periodically e-mail site owners and ask them to confirm that their site is still in use. There is no check for the site or its content actually being used, it is just a dumb timer job. If the site is not confirmed as still being active, the site will then be deleted - even if it is still in use. As deleting a site is not covered by any SharePoint recycle bin mechanism (coming in SP1), Microsoft also provides the site deletion capture tool on CodePlex.

Wouldn't it be nice if we could apply the information management policies for retention and disposition of content also for SharePoint 2010 sites? Yes we can :) By using a content type to identify and keep metadata for a site, the standard information management policies for content expiration can be configured to implement a recurring multistage retention policy for site disposition.

Create a site information content type and bind it to a list or library in your site definition, and ensure that this list contains one SiteInfo item with the metadata of the site. Typical metadata are site created date, site contact, site type, cost center, unit and department, is restricted site flag, last review date, next review date, and last update timestamp. Restrict edit permissions for this list to just site owners or admins.

Enable retention for the SiteInfo content type to configure your site lifecycle management policy as defined in your governance plan.

Add one or more retention stages for the SiteInfo content type as needed by your SLM policy. You will typically have a first stage that will start a workflow to notify the site owner of site expiration and ask for disposition confirmation. Make sure that the site owner knows about and enacts on your defined governance policies for manual information management, such as sending valuable documents to records management. Then there will be a second stage for performing the site disposition steps triggered by the confirmation.

You can also implement custom information management policy expiration formula or expiration action for use when configuring your retention policy. You typically do this when your policy requires retention events that are not based on date fields only. See Sahil Malik's Authoring custom expiration policies and actions in SharePoint 2007 which is still valid for SharePoint 2010.

Use a custom workflow or custom expiration action to implement the site disposition steps: user removal, automated content clean-up and archiving, and finally trigger deletion of the site. If the site is automatically deleted by a custom workflow, or marked for deletion to be processed by a custom timer job, or a custom action just sends an e-mail to the site-admin, is up to your SLM policy.

If you need to keep the site in a passive state for e.g. 6 months before deleting it, you can use a delegate control in your site master pages to prevent access to passive sites or you can move the site to an archive web-app that use a "deny write" / "deny all" access policy to prevent access. Note that the former is not real security, just content targeting for the site. The latter is real security, as "deny" web-app policies overrides site specific access rights granted to SharePoint groups and users. This allows for keeping the site users and groups "as-is" in case the site can be reactivated again according to your SLM policies. If site owners can do housekeeping on a site while passive, then grant them access by creating extra "steward" accounts that are not subject to being denied access.

I recommend removing all users from the default site members group before deleting the site, otherwise the site will not be deleted from the site memberships list in the user's my site.

The astute reader may wonder how the content type retention policy knows if the site is actually in use. The answer is quite simple; each SPWeb object provides a LastItemModifiedDate property. This timestamp is also stored in the SharePoint property bag. Use a delegate control in your site's master page to check and push the timestamp to a date-time field the SiteInfo item, so that the rentention policy can trigger on it. Remember to use SystemUpdate when updating the SiteInfo, otherwise you will change the site's LastItemModifiedDate to now. You can also use a custom expiration formula that inspects the last modified timestamp for the site when the information management policy timer job runs.

We also use the site information content type in our Puzzlepart projects to provide a search-driven site directory. It is quite simple to make a nicely categorized and searchable site catalog by simply using one or more customized the search results web-parts. This search-driven catalog can of course be sorted by the search result 'write' managed property, which must be mapped to the crawled property field that contains the LastItemModifiedDate of a site.

Using a search-driven approach makes it unnecessary to have a classic site directory list. The site metadata is simply stored directly in a list within each site, managed by the respective site owners. This is more likely to keep the site metadata up-to-date rather than going stale in a central site directory list that no one maintains.

I hope this post have given you some new ideas on how to store, manage and use site metadata both for site lifecycle management and for providing a relevant search-driven site directory.

Five Steps to Structure Your SharePoint Sites (Part III)

At every SharePoint customer and in every SharePoint project, the issue of how to realize a future-proof, working site structure arise. I've written a few blog posts on classification and structuring of SharePoint sites before (part 1 and part 2), focusing on scalable site structure design within the SharePoint system boundaries. This post is about getting from a classic site map based approach to a methodology driven by Information Architecture and the site classification pyramid, combined with the SharePoint containment hierarchy and SharePoint governance best practices. The latter is important as the management and usage of a site, and the service level agreement for the site, strongly influence how the overall SharePoint solution should be partitioned to adhere to architecture best practices.

Most information architects come up with a content organization model like a site map with hierarchical URL scheme. "This is the draft model, and we want the URL structure to be like this in our SharePoint solution" they say. "That's a very useful mental model of the site" I respond, "but what you really want is a good, intuitive navigation concept, helping users find what they need - like an invisible guiding hand; not that specific URL structure". Having a simple and easy mental model for the site structure is important, as it makes it easy to both contribute and find content, driving findability when the amount of information stored and managed in SharePoint grows.

Alas, that is not the primary issue with the site map approach for designing SharePoint site structures. The site map is closer to define statical IA structure than navigation, still it is only good for logical IA structure and cannot be expected to be used directly as the physical IA structure in SharePoint.

Using a site map approach to design a SharePoint site structure will only work in simple scenarios with homogenous solution domains, such as a publishing portal or a collaboration solution with limited document storage and versioning. It will fail when the SharePoint solution domains and their usage, governance and service level agreements differ, not to forget when the overall amount of data stored in SharePoint surpasses the 100GB limit per content database. The wanted URL structure might not be possible to realize in most cases anyhow when adhering to SharePoint architecture best practices.

A more structured approach is needed to design a SharePoint site structure according to architectural and governance best practices. My methodology involves a process with these five steps:

1. Review your site map draft model against the SharePoint site classification pyramid and your governance plan for each of your identified site types.
2. Partition your sites into solution domains based on site type and governance policies aspects such as business purpose, usage patterns, storage requirements, management boundaries, information security and isolation, operations and service level agreements.
3. Adapt the site structure to the SharePoint containment hierarchy, adhering to limits and best practices based on the partitioned solution domains.
4. Use the main SharePoint architectural components explicit and wildcard site-collections to ensure a scalable design that allows for future growth and expansion into new business domains. Use wildcard site-collections by default as these are super scalable. Adapt or customize SharePoint global menu and current menu to your navigation concept, to hide the solution partitioning.
5. Define your URL scheme using host headers, manage paths and site provisioning policies.

Following these five steps is a top-down approach that is more understandable for non-techies. However, from step 3 onwards, good technical knowledge of SharePoint architecture is required.

To successfully design a working and scalable SharePoint site structure, you need a site classification scheme. I recommend the classic site classification pyramid as it outlines typical solution domains with governance and site types, helping you classify sites and draw domain boundaries in your site model.

Then, after identifying the different solution domains that make up your overall SharePoint solution, you must map the identified solution domains to the SharePoint containment hierarchy. It is imperative that you understand the containment hierarchy and its boundaries, otherwise your solution will slowly stop working as more and more users create more and more sites and fill them with more and more information.

Site-collections is the main architectural component of SharePoint, in combination with the content database that stores the site-collection's content and documents. A web-application always have a single root (1st tier) site-collection, which is the home site of the web-application. A 2nd tier of site-collections can be added to the web-application as needed to scale the application, using SharePoint managed paths. A typical scenario is to add a new solution domain. Another typical scenario is to add room for more content and documents; because a site-collection cannot span content databases and adding more space thus requires a new site-collection. Use subsites for anything that is just a section of an existing solution domain or site. See the golf and car communities in the example, which functionally are "sites" in their own right. Read my post Classification and Structuring of SharePoint Sites (Part I) to learn more.

It's easy to choose what solution goes into the root site-collection of a web-application, typically this is the portal to the overall solution. As there is a limited set of explicit site-collections per SharePoint web-application, these should be reserved for a planned, wellknown small number of predefined sites. Use the limited set of available explicit site-collections for the top two types in the site classification pyramid. Note that root and explicit site-collection storage cannot be scaled as you cannot add another content database to a site collection. Use as few explicit site-collection as possible, stay well below the limit of 20 managed paths. Expect everybody to require vanity URLs such as ://puzzlepart/HR/ for their site, so a strict governance policy for 'explicit' is required.

Use wildcard site-collections when the predictable number of sites is too large for explicit, typically for the bottom three types in the site classification pyramid. Also use wildcard site-collections even for central portals and division areas when there is no absolute requirement for an 'explicit' 2nd tier top-level site. See Puzzlepart TV in the example; even if the channel sites are classified as 'division area' (yellow color code) they do not need to be 'explicit'. In fact, due to their massive storage requirements for rich media content, best practices dictate the use of 'wildcard'. Always use wildcard site-collections for solutions have a limited life span such as projects.

Finally, after composing your scalable site structure design from root, explicit and wildcard site-collections, define the URL scheme that underlies the site structure. Use short descriptive URL elements for web-applications, managed paths and site-collections. The combination of these elements become the URL of the top-level sites that are what the users actually interacts with when using SharePoint. Note how the URL structure is affected by the SharePoint containment hierarchy. Compare ://puzzlepart/pzl-tv/ and ://puzzlepart/pzltv/ in the example, the latter contains child sites of the former in the overall navigation concept. In addition, I recommend defining and enforcing policies for all URL elements that users can provision, such as subsites, lists and libraries, to avoid non-friendly URLs with anything from spaces to special characters.

It is very likely that you will need to customize the ootb SharePoint navigation provider and menu rendering to have a consistent navigation experience across sites, sections and pages. Users get very confused and annoyed when the content of the top menu changes as they navigate from site to site, which is very understandable. There are several ways to implement common menu and navigation across site-collections, but that's left as an exercise for my readers.

The capacity and planning boundaries are mostly the same in SharePoint 2010 as in SharePoint 2007 for web-applications, site-collections and content databases. The most significant change is that SharePoint 2010 now supports max 250 000 site-collections per web-application, five times the limit for 2007. The max number of content databases per web-application has increased from 100 to 300, on the other hand the max number of site-collections per content database have been reduced from 50 000 to max 5000 with a recommendation of 2000 site-collections per content database. None of these numbers are hard limits, these are numbers for acceptable SharePoint performance.

Minimal SharePoint Governance Plan - Part III

This is part III in a mini-series about the Minimal SharePoint Governance Plan needed to get you started with your SharePoint governance efforts.This part gives a more detailed overview of the mininal governance plan. The overview comprises operational and functional areas from SharePoint architecture, via site, user and information lifecycle management, to realization of SharePoint solutions. As you will see, there is a multitude of governance aspects, even if just focusing on the technical aspects. Executing on all governance aspects from day one is not viable, that is why I recommend to start with simple governance.

The is no one governance plan to rule them all. Governance is too multi-faceted for a single set of policies to fit all the different site types across diverse business areas. Governance for controlled publishing sites will be very different from Enterprise 2.0 pull-style situational solutions, as will it differ for management of project sites and team sites, and as for community, social and personal sites. Adapt your governance plan according to the targeted solution.

The site classification scheme shown here is from the Technet SharePoint Governance Checklist Guide, refer to page 20 for more details.

Architectural Governance

The governance plan for the SharePoint solution must define a logical architecture model based on your Information Architecture analysis, adhering to architectural components, farm deployment, capacity and planning recommendations on Technet.

• Farm design policies for a robust and flexible platform
• Sharing and isolation policies for applications and information
• Site-collection structure policies to drive overall solution architecture & governance
• Information asset structure policies to ensure classification, management and findability

The objective of having architectural policies is to create a workable farm and solution design considering hard and soft SharePoint limits.

Site Lifecycle Management

The governance plan for site lifecycle management (SLM) must specify policies for managing sites from creation to disposition. Define a classification scheme for site types and adapt your governance plan to each site type. It is recommended to develop timer jobs to automate and enforce SLM policies.

You need a site sweeper job that disposes expired, abandoned and useless sites from your solution to ensure that the overall effect produces useful business results. Make sure that knowledge captured in obsolete sites are retained through information management tasks before permanently disposing the sites. Alas, don't be afraid of self-service provisioning, after all more is different.

• SLM policies must be defined and enforced
• Site Provisioning
    o Implement custom provisioning if ootb functionality is not sufficient
    o Provisioning policy per site type defines level of automation and self-service
    o Use provisioning wizard to collection data related to SLM
    o Store SLM data in site properties or a site inventory list
• Site Retention
    o Do not rely on database backup for retention, backup retention might be shorter than site retention
    o Prepare to restore sites deleted by users
• Site Disposition
    o Implement custom site sweeper if ootb functionality is not sufficient
    o Standard site sweeper is only for site-collections (site use confirmation)
    o Define a procedure for information management when disposing a site

Note that there is no ootb site directory in SharePoint 2010. Still, just create a shared custom list and use it for inventory management of sites as part of your SLM implementation.

DocAve Backup & Recovery a recommended 3rd-party tool, it provides capabilities beyond ootb SharePoint 2010, such as item-level recovery. For site retention, the CodePlex MSIT Site Delete Capture tool is an option.

User Lifecycle Management / Identity & Access Management

The governance plan for user lifecycle management (ULM) must specify policies for managing users from onboarding to termination. ULM is directly related to information security (access and auditing), information management, and Identity & Access Management (IAM). Employees come and go, resulting in SharePoint data that nobody manages, or in worst case, lost knowledge.

Implementing good site disposition policies and good information management policies will reduce the efforts required for user lifecycle management, as obsolete sites and information then will be disposed of in a timely manner - keeping the quantity of orphaned data down.

• ULM policies must be defined and enforced
• Site memberships and permissions must be assigned for new users
• Site and information asset permissions & ownership must be handled when
    o Account is terminated
    o User transfers to another business role or department
• A policy for reassignment of ownership must be defined

Having tools for management of user permissions, ownership and lifespan is nice, but no prerequisite. LigthningTools DeliverPoint or Axceler ControlPoint are recommended partner solutions for user management.

Content Type Governance / Information Management

The governance plan must specify policies for content management according to your Information Architecture analysis and taxonomy. A taxonomy is realized in SharePoint using Site Content Types for information asset types and Term Sets for coherent tagging of information. Content types combined with metadata tagging is essential for information classification and for driving findability.

Content types defines the static classification hierarchy of the information managed in SharePoint. A content type is built from a set of fields defining the metadata of the content type, further detailing the classification of the information. Some metadata fields require the use of a controlled vocabulary for content tagging.

Information management policies can be assigned to content types. The most important are for retention and disposition of content, helping you manage e.g. outdated content to ensure the relevance and timeliness of your information.

• Reuse the Open-Closed Enterprise Taxonomy across web-applications and site-collections
• Always use a core Content Type Hub store for the enterprise taxonomy
• The core content type store defines company specific immutable base content types
• Ensure that all additional content types derives from the core content types, by extending the immutable base
• Use few required metadata fields, max 3-5 per content type
• Use sensible default values where possible
• Define polices for
    o Reusable content (document repository)
    o Retention of outdated content / historical archive (document repository)
    o Retention of expired content (records repository or disposition)
    o Regulatory compliance records (records management)
    o Disposition of content
• Retention policies are important for driving findability, use them to prevent irrelevant search results
• Define and enforce behavior using
    o Information Management policies
    o Workflows / Event receivers
• Use Information Management policies for
    o Retention, disposition, auditing, labeling / barcodes
• Ensure that content types are evolved according to best practices

The new SharePoint 2010 multi-stage retention support, combined with workflow and document /records repositories allows for implementing and enforcing sophisticated content management policies. Note that the new SharePoint 2010 Content Organizer only supports Document-based content types, in addition to e-mail messages.

Managed Metadata Governance

The governance plan must specify policies for management of the managed metadata used when tagging content in SharePoint. Managed metadata is a controlled vocabulary defined in the corporate taxonomy, realized in SharePoint 2010 as term sets defined in the Managed Metadata Service.

• Reuse the Open-Closed Enterprise Taxonomy across web-applications and site-collections
• Always use a core Managed Metadata Service term store for the enterprise taxonomy
• Allow local Managed Metadata Services for isolated, locally managed term stores
• Always use synonyms when defining terms, consistent content tagging is essential for content management and for driving findability
• Use term translation to support other languages for the term
• Avoid random or haphazard tagging due to unintelligible terms
• Enable managed keywords for user-driven freeform tagging of content
• Ensure that term sets are evolved according to best practices
• Define and enforce a policy for reviewing open term sets for improper usage

Note that search do not comprise term synonyms or translations when searching, it only finds the stored key term. The same applies to faceted search – or 'refinement panels' as they are called.

You can have multiple Term Set stores and Content Type Hub inventories in SharePoint 2010. This allows for combining both enterprise definitions and local definitions to support both shared and isolated taxonomy configurations. See Plan to share terminology and content types on Technet.

Social Tagging Governance

The governance plan must specify policies for management of the social tagging features

• Use managed keywords to enable folksonomy for content (list items)
• Use social tagging to enable folksonomy for "anything with an URL"
• Allow for managed metadata and managed keywords to be included in social tags
• Define and enforce a policy for reviewing the folksonomy tags for improper usage

Note that the social tagging of "anything with an URL" is provided by the SharePoint2010 User Profile Service application, not the Managed Metadata Service application. Thus, social tags have no explicit relation to the term store at all. The same applies to other SharePoint2010 social features such as ranking and social bookmarking.

Document Template Governance

The governance plan must specify policies for using Office templates in content types.

• Use a shared set of enterprise Office templates
• Manage and store templates in a SharePoint document library at a central location
• Do not store templates directly in content types, always reference the central shared templates
• Make use of the Office 2010 Backstage or the document information panel for managing metadata directly in Office

Office 2010 now has support for storing templates in a SharePoint repository. Use AD group policies to populate 'File > Save As' and to lock down storage locations such as file shares and local disk.

List & Library Definition Governance

The governance plan must specify policies for managing content in lists and libraries. It is strongly recommended to use only lists based on site content types, rather than directly customizing list definitions. Enforcement of consistent classification and information management policies depends on using site content types.

• List content
    o Use only a few content types per list
    o Content types in a list must be cohesive
    o Prefer list views over "dumb" folders
    o Use SharePoint 2010 folders when appropriate
• List permissions
    o Prefer using inherited permissions
    o Avoid user item level permissions
• Enforce content management policies using
    o Versioning, check-in/out, workflows / event receivers
• Information Rights Management (IRM)
    o Policies for document access and usage restrictions
    o Applies IRM policies when document is downloaded from library
    o Enable by installing Active Directory Rights Management Services (AD RMS)
• Information management policies
    o Prefer implementing IM policies on content types rather than on lists or libraries

Note that some of the new SharePoint 2010 features work only for document libraries, such as the Unique Document ID, Document Set and Content Organizer features.

Permissions Governance

The governance plan must specify policies for how to manage access to sites and information assets, including which permissions users and groups have. All experience shows that simple permission policies are more secure. The more intricate and fine-grained permissions assignments you have, the harder it is to know who has access to what – and the more likely it is that there will be information security breaches exposing confidential information.

• Use SP groups to manage user group memberships
• Build your SP groups from AD security groups
    o Management of AD group members is typically a bottleneck, thus avoid it
• Do not assign permissions to single users, always assign to SP groups
• Prefer inherited groups (role assignments)
• Prefer inherited permission levels (role definitions)
• Use unique permissions at site level (favored) or list/library level only when absolutely required
• Avoid assigning item level permissions
• Site-collections are preferred security management boundaries

The visibility into what a user has access to has improved a bit in SharePoint 2010, so has the usage reporting capabilities. Still, 3rd-party tools such as LigthningTools DeliverPoint or Axceler ControlPoint might be required for professional permissions management beyond the built-in SharePoint 2010 Permissions Tool.

Search Governance

The governance plan must specify policies for driving findability through indexing and search. The Information Architecture analysis defines the information taxonomy and organization blueprint realized in a SharePoint site structure capable of storing and managing your content. The site structure combined with content types enables findability through consistent classification and tagging of content.

• Ensure ease of adding information assets to correct location
    o Users should not have to enter a lot of required metadata
    o Users should not have to browse/navigate extensively to store content
    o Task context should deduce location, e.g. CRM client document store
• Metadata tagging through content types for all findable assets
• Use content type retention policies to prevent irrelevant, outdated search results
• Use search scopes to provide search context: people, tasks, articles, project, archive, etc.
• Use faceted search (refinement panel)
• Ensure and enforce information isolation
    o Farm design must prevent configuration mistakes from exposing confidential information by accident
    o Use separate service application groups or even separate shared services farms

The most valuable search is the one that connects a user to other people, as people are often the best sources of information and knowledge, especially tacit knowledge – know-how relating to new better business performance or to novel information that generate flow of new knowledge – in short, that ignites innovation. A former CEO of Hewlett Packard famously observed: "If HP knew what HP knows, we would be three times as profitable".

Findability is more than just search capabilities, it also includes the SharePoint 2010 social computing features such as “Tags & Notes” for tagging and social bookmarking. Tag clouds, metadata-based navigation and filtering, and even the My Site activity feed are all enablers of driving findability.

Note how I say "driving findability"; findability is not something you just enable, you have to actively manage and adapt the Search Service application settings according to your business needs. Just enabling search is just as bad as not managing your user's expectations for what to expect from enterprise search.

All parts of this mini-series:
Part I - SharePoint Governance - Eating an Elephant
Part II - Start with Simple Governance
Part III - Minimal Governance Plan (this post)

Start with Simple Governance - Part II

This is part II in a mini-series about the Minimal SharePoint Governance Plan needed to get you started with your SharePoint governance efforts. The objective is to create a simple and viable plan that covers the core governance aspects for operations and solution management. The plan comprises the functional areas of textbook enterprise governance plans, skipping organizational aspects to focus on technical aspects. The focus is on lifecycle management of SharePoint sites and users, classification and management of content, and on driving findability, all hosted on a robust SharePoint farm.

Required Operational Governance

The governance plan for the SharePoint environments must specify policies for both farm infrastructure and operations. The architecture of the farm must be adequate for hosting the planned solutions within defined service level agreements (SLA), and at the same time be positioned for future expansion into other solution areas.

Operational aspects that must be covered by the governance plan:

• Availability
    o Farm with redundancy
    o Monitoring
• Backup and Recovery
    o Retention and disposition policies must be defined and enforced
    o Restore specific information assets
    o Tested disaster recovery plan
    o Complete solution can be restored within allowed time limit

Usage of SCOM SharePoint management packs is recommended. For recovery is DocAve Backup & Recovery a recommended 3rd-party tool, it provides capabilities beyond ootb SharePoint 2010, such as item-level recovery.

Required Solution Governance

The governance plan for the SharePoint solution must specify policies for governing the sites and information assets that make up the solution. This especially applies to how to manage solutions over time as they evolve in response to changed business requirements.

Solution aspects that must be covered by the governance plan:

• Site Lifecycle Management (SLM)
    o Policies for provisioning, retention and disposition must be defined
    o Automation of SLM through site provisioning forms and timer jobs
    o Site delete capture for retention beyond database backups
• Content Type and metadata definitions (taxonomy)
    o Policies for retention and disposition, including archival and records, must be defined
    o Classification of all information assets, from sites to documents and items
    o Cover the core content types for your company (the immutable base content types)
    o Enable findability through consistent classification and tagging of content

SharePoint has little built-in support for SLM, but most SharePoint workflow tools provides support for user input forms and site creation and management. For site retention, the CodePlex MSIT Site Delete Capture tool is an option.

There is good taxonomy support in SharePoint 2010 provided by the Managed Metadata Service and the Content Type Hub mechanism. Note that neither managed metadata nor social tagging is included in SharePoint Foundation 2010, these are SharePoint Server 2010 service applications.

Optional Solution Governance

In addition to the above aspects, governance policies for how to manage users, their permissions and their ownership of sites and data should be defined. Employees come and go, resulting in SharePoint data that nobody manages, or in worst case, lost knowledge.

Solution aspects that should be covered by the governance plan:

• User Lifecycle Management (ULM)
    o Manage the lifecycle of accounts due to onboarding, transfering, and termination of users
    o Policies for permissions and ownership of sites and information assets
    o Automation of ULM though partner/open solutions
• Visibility into usage
• Visibility into permissions

The visibility into what a user has access to has improved a bit in SharePoint 2010, so has the usage reporting capabilities. Still, 3rd-party tools such as LigthningTools DeliverPoint or Axceler ControlPoint might be required dependent on needs.

SharePoint Governance Checklist

Microsoft provides a SharePoint governance checklist guide whitepaper that my customers find very useful. The checklist covers important governance and lifecycle management aspects that must be included in a governance plan. It is strongly recommended to review all areas of your governance plan against this checklist.

Aspects covered includes:
• Design-time and run-time governance
• Roles and ownership
• Information architecture, navigation and findability
• Branding
• Infrastructure and operations
• Testing and development

Each checklist has a related tips & information section, effectively making this a pocket governance guide.

All parts of this mini-series:
Part I - SharePoint Governance - Eating an Elephant
Part II - Start with Simple Governance (this post)
Part III - Minimal Governance Plan

SharePoint Governance - Part I - Eating an Elephant

As part of my work doing SharePoint architecture & design at different Puzzlepart customers, the governance aspect always comes up. Everybody seems to know the issues with uncontrolled site sprawl in SharePoint and want a way to manage their SharePoint solution in a controlled manner. I usually refer them to the SharePoint Products and Technologies Governance Checklist Guide and the authoratitive Sample SharePoint Governance Plan by Joel Oleson and Mark Wagner at Technet. The latter plan also available in a revised, shorter SharePoint Collaboration Service Governance Plan variant.

I have to tell you, my customers are always baffled by the size of the plan (33 pages) and the diverse and detailed set of governance areas covered by the plan:

• Resources: people, teams, roles, technical
• Governance hierarchy
• Operational policies
• Application usage policies
• Communication plan
• Training plan
• Support plan

They understand that all the areas have their importance, but feel that the plan is too comprehensive for their planned initial usage of SharePoint. I agree, it is better to start with a simple viable governance plan covering core governace aspects, rather than taking the big bang enterprise governance plan approach and fail on its execution.

In my consulting knowledge arsenal, I have a leaner governance plan adapated to getting started with SharePoint, focusing on the minimal level of governance needed. The next two posts in this mini-series covers this minimal governance plan:

Part I - Eating an Elephant (this post)
Part II - Start with Simple Governance
Part III - Minimal Governance Plan

The Minimal SharePoint Governance Plan has a functional focus, and do not cover organizational aspects at all. You will need to cover more and more areas in your governance plan as SharePoint grows enterprise on you. Just eat the elephant one bite at a time.

Most importantly, make sure that the governance plan is understood pragmatically, adopted consistently, applied productively, and managed sustainably. Realizing the plan is the daunting task of the organization, and it is your task to make them realize that.

SharePoint 2010 Open-Closed Taxonomy

A taxonomy is realized in SharePoint using content types for information asset types and term sets for coherent tagging of information. Content types defines the static classification hierarchy of the information managed in SharePoint. A content type is built from a set of fields defining the metadata of the content type, further detailing the classification of the information. Some metadata fields require the use of a controlled vocabulary, realized in SharePoint 2010 as term sets defined in the managed metadata service.

A central design recommendation for both content types and term sets is the Open-Closed principle: open for extension, closed for modification. The Open-Closed principle states "software entities (content types, term sets) should be open for extension, but closed for modification"; that is, such an entity can allow its configuration to be modified through extension without altering the core base entity definition. This is especially valuable in a production environment, where changes to base definitions will necessitate code reviews, unit tests, and other such procedures to qualify those changes. The principle isolates the changes to the extension entity definitions, because the core base entities are immutable. This makes it simpler to evolve the entities in a controlled manner while keeping a common, consistent core base definition.

SharePoint is restrictive on what changes can be made when evolving or specializing content types, with very limited support for modifying existing content type definitions. The recommended practice is to use a base set of core content types (the closed part) and then perform all specialization and variations on content types inheriting from the base content types (the open part). Evolving term sets is less restrictive than content types and allows for much more flexibility in changes to their definition, including renaming and merging terms.

New in SharePoint 2010 is the capability to share content types and term sets across multiple site-collections and even across farms. These enterprise content types and term sets can be combined with local site content types and local term sets, the giving a combined syndication effect. The Open-Closed principle can be applied also to this syndicated taxonomy: use the enterprise content type inventory and the enterprise term set inventory as the core taxonomy base (the closed part), and use the site inventories to extend the taxonomy with local specializations (the open part).

In technical SharePoint 2010 service application terms: create a primary term store and a primary content type hub, then extend those locally with additional local term sets and local content type inventories subscribing to the primary core taxonomy.

Applying the Open-Closed principle both for content type and term set definitions, and for content type and term set syndication, allows for defining and enforcing governance policies that makes it possible to have centrally controlled lifecycle management of the core base definitions of your SharePoint taxonomy, while still allowing local extension and adaptation of the taxonomy.

See SharePoint 2010 Intra-Farm Term Store Syndication for details on realizing taxonomy syndication, including configuration of permissions per web-app, user or group for adding or modifying term sets.

Classification and Structuring of SharePoint Sites (Part I)

As part of my work as a SharePoint architect, I almost always run into poorly designed and structured SharePoint solutions. Mostly because consultants hardly ever seems to read and really understand the Technet SharePoint Site Structure Planning and the SharePoint Logical Design and Architecture guidelines or do any information architecture (IA) analysis. What is needed is a site classification scheme based on solution domains, usage, management and governance as shown in this article with structuring guidelines for each site category.

This post and the next is targeted at a technical audience; if back-end architecture details is not for you, jump straight to part III for a simple five step process aimed at business users, analysts and information architects.

IA defines the structural design of your information space to facilitate content organization, realized in a SharePoint site structure to store and manage your content. Skipping the IA and site structure planning will work for some time, perhaps forever, but eventually most SharePoint solutions grows into the software boundaries of the platform. The effect will be poor performance and frustrating user experience due to the unmanageable number of sites popping up everywhere.

SharePoint supports a flexible IA structure by providing several types of content containers, and you really need to understand how to use them:

• Root site collection: container for portal, home sites
• Explicit site collection: container for department, services, regional sites
• Wildcard site collection: container for team, community, project sites
• Top-level site (site home): container for content and subsites and channels
• Subsite (site section): container for content and even more subsites and channels

Note that it is a best practice to not use the terms "site" or "web" when classifying and structuring sites. Experience shows that these terms are so overloaded that is just causes confusion. Even the latest SharePoint guidance makes a classical MSDN mess using the term "root site" in some parts when referring to top-level sites. Also stay away from using site-definition names such as team-sites and workspaces.

In addition, I use "managed site-collection" instead of the term "managed path" to get consistent naming in my site taxonomy. Managed paths are central when constructing your URL space, but thats a secondary goal and only important as part of your IA structural design to facilitate content organization and driving findability. It is the site-collections held at managed path explicit and wildcard inclusion URLs that is the primary goal, plus of course that held at the root path.

It is important that you look at your diverse set of information and develop a classification scheme for the content that will go into your SharePoint solution. Develop a classification of sites and follow the planning guidelines on how to structure your sites according to SharePoint recommended practices. I like to classify sites into four major categories as shown here (click to enlarge):

Sites are classified into the four major categories based on the management policies (corporate to community, controlled to uncontrolled) and the life-cycle policies (predefined and planned, predictable and ad-hoc) of the site content. The four categories are numbered 1 to 4 ranging from most specific to very generic classification of sites.

Sites always belong to one site-collection. A site-collection is just that, a collection of sites. A site-collection contains one top-level site, that again can contain a hierarchy of subsites. Hence site-collection. As there is a limited set of explicit site-collections per SharePoint web-application, these should be reserved for a planned, wellknown small number of predefined sites. Use wildcard site-collections when the predictable number of sites is too large for explicit, and for ad-hoc sites that come and go in unknown numbers. Finally, there can be only one root site-collection per SharePoint web-application.

The following figure applies the SharePoint site structure design limits to my site classification scheme:

The major categories provide a recommendation of the type of site-collection to use; and for each category and site-collection type, the site classification scheme provides a recommendation for how to use top-level sites (TLS) and subsites to provide the structural IA design. There are two major ways to structure content within a site-collection: wide and deep. A wide site structure uses multiple TLS sites (and hence multiple site-collections), while a deep site structure rather uses a single TLS site with multiple subsites with even more subsites. Both approaches can be combined to best structure the content according to your IA.

Using a wide site structure is applicable for sites like projects and team-sites, as this gives the best control over site configuration, administration and management in SharePoint. Each TLS is a separate site-collection, and each site-collection is a management boundary for users and groups, access control, content taxonomy, navigation, branding, SLA, all in all sites are separated from each other and each have their own life-cycle and governance policies.

Using a deep site structure is applicable for sites like portals, departmental or regional content, and shared communities. All subsites with the TLS site-collection share the same navigation and content taxonomy, the same members and groups, the same branding, all in all the same configuration and administration and the same governing policies and life-cycles.

You will of course also need to consider the site-definition type that you plan to apply to each site, as this will influence what should be structured together to create a well planned user experience. E.g. within a internet "Publishing portal" TLS you would most likely not add a "Team-site" subsite.

Note that root or explicit site-collection storage cannot be scaled as you cannot add another content database to a site collection. If scalable storage is a requirement, using wildcard site-collections is best practice.

I recommend making a SharePoint site diagram like this to vizualize how your site structure is built by combining root, explicit and wildcard site-collections (click to enlarge):

I hope this site classification scheme will help you understand how to plan and design your next SharePoint site structure. At least it should make planning workshops more focused and less confusing, both for you and your customers.

Read part II for more details on wide vs deep site structures, including SharePoint system boundaries.

Read part III Five Steps to Structure Your SharePoint Sites if you're unsure of where to start your site classification and structuring efforts.

SharePoint Governance Guidelines

The SharePoint Products and Technologies governance checklist guide on the Technet SharePoint Governance Resource Center is very popular among our customers. It covers aspects from architectural planning, roles and policies, to operational practices.

To complement this official checklist, I've collected some more concrete guidelines that my customers often request when getting started with MOSS. It covers additional aspects such as driving findability and user lifecycle management. The guidelines are available here: MOSS governance guidelines.pdf

If you have any suggestions for more recommended governance practices, please leave a comment.

[UPDATE] See the updated governance guidelines here, including new SharePoint 2010 aspects.