Thursday, November 27, 2008

Service+Schema Versioning: The Ripple Effect

In my SOA versioning, compatibility & composability session at NNUG this week, I stressed the importance of recognizing the ripple effect that incompatible or semantic changes to service contract artifacts will have. This illustration captures how versioning an artifact will affect upstream artifacts:


The ripple effect is a bottom-up effect, and an incompatible change on e.g. a customer address schema will cause an explosion of new versions of all affected contract artifacts. In the end, the change will propagate even to the service consumers.

Having a versioning strategy let you control the effects of the inevitable changes. Using a compatibility policy will help you alleviate some of the negative effects of versioning.

Sunday, November 23, 2008

Never buy a DELL XPS

Now it has happened again :(

The motherboard of my two year old DELL XPS 700 has broke again - only six months after it was replaced last April. DELL's support in Norway won't help me as XPS is just a subcontractor PC branded as DELL. I had to call international support in Ireland (or was it India), where they tried to cheat me of my Norwegian consumer rights. Then I had to stay home from work for three days because InfoCare didn't come as promised the first two times, without even notifying me that they wouldn't come anyway.

No more.

The XPS is now for sale at finn.no. In fact, I will never buy DELL again.

Wednesday, November 19, 2008

SharePoint: Reference Data Lookup

In my SharePoint Common Reference Data Management post, I outlined several options for how to maintain and use common master data as reference data across site-collections:

  • Replicated Lookup List
  • Remote Lookup List
  • External Database Lookup
  • External Hybrid Lookup List
I've made some figures showing the four reference data designs:
Lookup metadata and content.pdf

Note that if you plan to use custom lookup columns in your site content types to get a controlled vocabulary for your taxonomy, you must test and verify that the lookup columns can:
  • be used as site columns
  • be filtered to show a subset of lookup values
  • work in the DIP panel - fully, downgraded or not at all
  • reference data in other site-collections
  • be replicated across site-collections
A lookup column that can be filtered is useful as it is better positioned to support an evolving set of lookup values, e.g. hide values that become expired. Such a lookup column can be configured to work exactly as a choice column and still contain a centrally controlled vocabulary value-set.

Tuesday, November 18, 2008

NNUG Oslo 25. November

I'll be giving a session on service+schema compatibility & composability at NNUG Oslo on 25. November: details here.


Among the topics I will cover is the "Flexible/Strict" strategy.

[UPDATE 12-DEC-2008] Download slidedeck here.

Wednesday, November 12, 2008

Service+Schema Versioning: Flexible/Strict Strategy

In a few weeks time I will be giving a session at NNUG on SOA service and schema versioning strategies and practices. A central topic will be schema compatibility rules, where I will recommend creating a policy based on the "Strict", "Flexible" and "Loose" versioning strategies described in chapter 20.4 in the latest Thomas Erl series book Web Service Contract Design and Versioning for SOA. I guess David Orchard is the author/editor of part III in the book.

I recommend using a “Flexible/Strict” compatibility policy:

  • Flexible: Safe changes to schemas are backwards compatible and cause just a point version

  • Strict: All unsafe schema changes must cause a new schema version and thus a new service version

  • Do not require forwards compatible schemas (Loose, wildcard schemas) - schemas should be designed for extensibility, not to avoid versioning

  • Service interfaces should also have a Flexible/Strict policy
Safe changes is typically adding to schemas, while unsafe changes are typically modifying or removing schema components.

Note that forwards compatible schemas is not required to have forwards compatible services, as service compatibility is defined by the ability to validate messages. WCF uses a variant of 'validation by projection' (ignore unknown) for forwards compatibility, but also supports schema wildcards.


As Nicolai M. Josuttis shows in the book SOA in Practice (chapter 12.2.1 Trivial Domain-Driven Versioning), even simple backwards compatible changes might cause unpredicted side effects such as response times breaking SLAs and causing problems for consumers. It is much safer to provide a new service version with the new schema version, as if there is a problem, only the upgraded consumers that required the change will be affected.

Note that even adding backwards compatible schema components can be risky, but adding is typically safe. Josuttis recommends using "Strict" as it is a very simple and explicit policy, but I prefer "Flexible/Strict" as this gives more flexibility and less service versions to govern.

Avoid trying to implement some smart automagical mechanism for handling schema version issues in the service logic. Rather use backwards compatibility, explicit schema versions and support multiple active service versions. In addition, consider applying a service virtualization mechanism.

Friday, November 07, 2008

SharePoint: Content Type Guidelines

I've collected some recommended practices for SharePoint content types for my current customer. The guidelines are available here: ContentTypeGuidelines.pdf

The main advice is that content types must be based on your information architecture (IA) and goverened, to make it easy for your users to both contribute content and to drive findability across that content.

The general guideline for evolving the content type IA is to never change or rename content types or their aspects, make new ones and hide the old ones.

Wednesday, November 05, 2008

SharePoint: Common Reference Data Management

There are two kind of common reference data typically used in SharePoint:

  1. Data that is native to the portal
  2. Data that provided by external systems such as CRM, ERP, SCM
The usage and maintenance of the common reference data differs by their information type classification as described in later sections. Doing a thorough information architecture analysis is important for this aspect of SharePoint also.

The sharing of common data across multiple site-collections will also affect how native data can be stored, accessed and maintained.

External Common Data

External data used in SharePoint should not be imported into SharePoint lists; but must rather be used as pure reference data, and be maintained in their native systems. This is to avoid replication as much as possible due to the extra implementation and operational effort each added replication requires.

Use the web-parts provided by the external system to view the external data. Usage of the external data as lookup columns in lists is restricted to either:

The “Business Data Catalog” (BDC) is the standard MOSS mechanism for utilizing external data in MOSS, but this will require MOSS enterprise edition. Note that the BDC "Business Data" column cannot be used as a site column and hence not be part of a site content type.

Suggested third-party components:

Always test and verify that custom columns can be used as site columns if you plan to use them in site content types. Also test and verify that custom columns will work in DIP when used in a document content type, even if only added to list content types.

Native Common Data

Native reference data should be stored and managed in SharePoint lists. The advantage of using SharePoint lists is that you get a data management UI for free: just use standard SharePoint to maintain the data. The downside is the need for replication of shared data across site-collections that use the reference data, as OOTB lookup columns are restricted to a single site-collection.

The common data lists will be used as reference data sources for site columns of type “lookup”. This will ensure that common data is based on native SharePoint aspects, possibly enhanced with third-party SharePoint components.

It is recommended to create the reference data list and the corresponding lookup site column at the root site of the site collection, as the lookup column can then be used in lists in any sub-site to lookup the reference data (
cross-site lookup).

Externalized Native Common Data - External/Hybrid

Sharing common data in SharePoint lists across site-collections is not trivial. It must be considered if native common reference data should be externalized to a custom database and be treated as external data. The downside is that an application must then be implemented to allow for maintaining the externalized data. Data-aware tools such as InfoPath, Excel or Access can be utilized as the data maintenance front-end.

A variant of the external database is to use SharePoint itself as the external database, then you get the best from both alternatives (hybrid): a real master for the common data and the SharePoint UI for free. Use either a BDC lookup column or a third-party lookup column that works across site-collections.

Externalizing the native common data to a separate database will require more work on backup/restore. Using the SharePoint database avoids this.

Suggested third-party components:

Always test and verify that custom columns work across site-collections.

Replicating Common Data across Site-Collections

Native common data will be managed centrally in a master site collection, replicated to multiple target site collections. There will be one master list per target list type. Lists in sites in the target site collections can then reference the target common data lists using columns of type “lookup”.

Note: always prefer using externalized native common data instead of replication.

There are two options for how to maintain and replicate the shared data:

A) Overwrite data in targets on replication, i.e. data must only be entered in master as any updates to data in target lists will be lost on next replication. Updating target lists must be prohibited.

B) Two-way data replication to allow for updating data in targets in addition to master. Conflict resolution rules must be configured so that the master wins any auto-resolvable conflicts.

Which option to use for a list depends on the classification of the information asset that the list contains. E.g. country and product lists need not be updated except from in the master; while it is critical for the business that project information assets like contacts can be updated and used immediately.

Note that there will be latency in the replication of changes for both option A) and B). This latency can be configured. The impact on this replication latency must be analyzed based on the information types identified as common portal data. Different information types will have different tolerance for data staleness.

For option A) it must be considered to use a remote list viewer/editor in relevant sites to allow for editing master lists “remotely” from target sites.

Suggested third-party components:

Always test and verify that custom components such as web-parts and column types can be replicated.

SharePoint Lookup Column Types

Note that the standard “lookup” column is not a many-to-many relationship; it is just a choice list with a multi-select option. There are several third-party lookup types available to alleviate this limitation.

Suggested third-party components:

Always test and verify that custom columns can be replicated if used in lists and content types.

Codename "Bulldog" - Master Data Management

Microsoft last year acquired the Stratature +EDM master data management (MDM) platform. It will be incorporated as part of the next version of SharePoint, however only a few details are public. Read the roadmap here.

Monday, November 03, 2008

I&AM: Understanding Geneva

At PDC last week, the "Geneva" platform for federated claims-based identity and access management (I&AM) was announced. Here are some useful resources to get started:
Don't let the federation part of this stop you from looking at Geneva, it is utilizing claims for access management and access control you should focus on initially, externalizing it to the identity metasystem. Let the support for distributed I&AM across the extended enterprise be a nice feature that you get for free.