Modeling Business Processes for SOA, don't forget the Information Model

Recently I have done a few architecture reviews at customers struggling to get their SOA efforts going. The common problem of these attempts on service-enabling legacy systems to support agile business processes have been - except from system designers not grasping that SOA is in fact a paradigm shift from invoking-operations composition thinking, and those using RPC over web-services - that they focus solely on business process management (BPM) modeling. From this model they try to identify services without having a taxonomy for classification of services (activity services, information services, composite services), struggling with service granularity and how to identify what goes into the message and data contracts.

The problem is that modeling your business processes typically misses out on the central building block of service-oriented solutions: the data and the semantics of the data as related to the business processes. This is typical for workflow models like the 'UML Activity Diagram'.

The data should represented by a Common Information Model (CIM). The CIM should be the starting point of your modeling efforts, in combination with Business Process Modeling Notation (BPMN) diagrams, as BPMN encompass process, events, messages and business documents. Based on the CIM, you must then design a business process information model (BPIM) for each business process domain. Then you could follow the SOA mantra coined by David Linthicum: data-service-process (much like the TDD red-green-refactor) and create mashups based on composite services. 

Last week Jean-Jacques Dubray posted an article about 'The Seven Fallacies of Business Process Execution'. The article contains a lot of good points about the 'case-tool grail' dreams about business people doing agile modeling in BPMN and then just generating systems that automates the business processes.

The really interesting part of the article is in 'Fallacy #5: Business Process Execution' where Jean-Jacques differentiates business objects (resources) and their lifecycle from the business processes that drives the lifecycle of the resources. The article explains how the data aspects are missing from BPMN even if it is the most important part of the business - just the problem I see so often.

Another point made in the article is that it is the resource lifecycle events that are important when trying to identify the services that your business process will depend on. The business events advance the business data through a state machine until the end of the business process. This is what Jack van Hoof, Nick Malik, myself and others have blogged a lot about this year: it is the business events and their message and data (documents) that are the key artefacts in your modeling efforts when creating a service-oriented architecture.

The article also shows how process service composition (orchestration of service execution and resource lifecycle management; BPEL) is different from user-centric business processes (human workflows and tasks; BPMN, WS-HumanTask, BPEL4People).

The article goes on to discuss how development is the glue that joins the different artifacts that constitutes a SOA. Developers are needed to actually implement the BPMN using the provided services. I strongly agree. Even if you could do BPMN to BPEL code generation, professional testing and QA in a development-staging-production environment is still a must. I will only go as far as to allow business people manage a set of goverened business rules, never directly implement any process (i.e. declaratively configure the diamond shapes in flow charts). Read more about business rules/decision service at James Taylor's blog.

Finally, the need for a durable 'composite service container' for the orchestrations is also something that is often disputed during my SOA reviews. "Command-and-control" developers rarely consider how the availability of autonomous services affects the fault-tolerance in the overall SOA solution. The question I always ask is: "What is the availability of a business process that composes five services that each have 99% availability? Is it 99%?" The answer is of course less: 95%. This is where a container for providing long running + reliable processes and 'resource' state machines comes in handy.

I recommend that you read and, most importantly, understand the concepts and the architecture blueprint presented in the article. I have to give Jean-Jacques a big credit for the good figures he has made to illustrate the taxonomy and ontology presented in the article.

Enterprise Architects, Top-Down SOA

I had hoped that the age of non-programming architects was over, but yesterday I was let down. I attended the monthly meeting at the "Practical SOA forum" at SINTEF where an experienced architect ("20+ years of IT experience") presentet his groups work at a large public service company in Norway. The architect explained the EA modelling work going on to build a service-oriented enterprise, showed the EA tool and portal, documented SOA guidelines, refinement of the model into BPMN (but not into BPEL), etc.

All systematic and professional, but then he went on to complain about having to spend too much time with the developers and hired consultants to help them actually understand and implement the model according to the guidelines and policies - which prevented him from playing (my word) with the model to refine it and expand it into new areas of the business...

No wonder many of the pure top-down approach SOA projects are regarded as failures.

WCF: Caching Claims using System.Web.Caching

We use a custom legacy STS (Security Token Service) that assigns a ticket to our consumer applications, that they again pass as to our WCF services for authentication. The passed SAML token contains only the ticket, so we must then use the validated ticket to generate the System.IdentityModel claims used for authorization.

As the services are stateless, each WCF operation must recreate the AuthorizationContext and thus generate the claim sets again. When you compose a set of services into a business process, the claim sets will get generated over and over again. To avoid this, and get better performance, I needed to cache the claim sets by STS ticket for a limited time.

Many might not know this, but the System.Web.Caching.Cache can be used even in systems that are not an ASP.NET application or even hosted by IIS/ASP.NET. You will still get all the goodies, such as cache expiration, dependencies, throwing stuff out of the cache in case of low memory, etc.

We now cache the claim sets in a singleton using a generic <T> getter and setter with ticket-based keys like this:

public static class EApprovalCache
{
private static System.Web.Caching.Cache _cache = HttpRuntime.Cache;

public static Cache Cache
{
get { return _cache; }
}

public static object Add<T>(string ticket, T value)
{
string key = typeof(T).Name +":" + ticket;
return _cache.Add(key, value, null, DateTime.Now.AddHours(1), Cache.NoSlidingExpiration, CacheItemPriority.Normal, null);
}

public static T Get<T>(string ticket)
{
string key = typeof(T).Name +":" + ticket;
return (T)_cache[key];
}
}

The claim set is retrieved and added to the cache like this:

_authorizationContext = EApprovalCache.Get<EApprovalAuthorizationContext> (context.Ticket);
if (_authorizationContext == null)
{
_authorizationContext = new EApprovalAuthorizationContext(session);
EApprovalCache.Add<EApprovalAuthorizationContext> (context.Ticket, _authorizationContext);
}

The claims are cached for max one hour; but if the user logs out and in again, then the user will have gotten another ticket and the claim set would be generated from scratch and not read from the cache.

Getting started with MSE, on SQL Server Express

As you can imagine, I just had to try to install and test the Managed Services Engine for service virtualization when it was made public on CodePlex this week. For a quick intro to MSE, its relation to "Oslo" and SOA governance, and MSE future directions; read this InfoQ interview with William Oellermann.

Not bothering to read the installation guide, I just ran the installer on my Vista machine with SQL Server Express installed - and got a "Failed to create SQL database: MSE6DB" error. The cause of the error is that the installer by default does not target a named SQL Server instance when trying to create the database used by MSE and the sample services provided with the MSE6 toolkit.

The solution is documented in the install guide (but I'm repeating it here for my friend Anders Norås):

• Ensure that SQL Server Express is running on your computer, but close SSE Management Studio
• Open a command windows (CMD.EXE) and navigate to the folder of the MS6.msi installer
• Run the installer (using /i):
  msiexec /i mse6.msi SQLSERVER=.\SQLEXPRESS
• Locate the Microsoft.MSE.Repository.Service.exe.config file in the “~\Program Files\Microsoft Managed Services Engine” folder and change the data source accordingly:
  <DBConnString>Initial Catalog=MSE6DB;Data Source=.\SQLEXPRESS; . . . ;

Note that you must be an administrator the local system if you are not in the SSE server role 'dbcreator' or equivalent.

This gives you the database MSE6DB that contains the MSE repository:
Try starting the Managed Services Engine MMC snap-in. If you get a service repository (net.pipe://localhost/ServiceCatalog/Pipe) connection error, then restart the “MSE Catalog Server” and “MSE Runtime Server” services in that order to reload the database config.

Finally, you must install some missing perf counters to make the service virtualization (hosted endpoints) work at run-time, otherwise you'll get an error when invoking the services. You must also restart the “MSE Catalog Server” and “MSE Runtime Server” services for the changes to take effect.

Now you're set to follow the 30min walkthrough to learn more about MSE and the wonderful world of service virtualization and see why it is important for life-cycle management of services in SOA systems.

PS! all of the walkthrough is correct, just pay attention in the POX step: you need to keep the action as-is from the previous step, only change what the text say you should change (binding XML, etc). Do not rediscover or change the selected operation in the grid.

BizTalk Services - Internet Service Bus

Channel 9 has just published some videos on BizTalk Services, a.k.a the Internet Service Bus (ISB), providing services (S+S/SaaS) for your mashups. This is an ARCast.TV episode with Ron Jacobs: part I and part II.

Absolutely worth watching, BizTalk Services is a central part of the "Oslo" initiative.

Tools and platform for providing SOA and mashups

This week at the Microsoft SOA and BPM conference, two announcements were made that support the predictions of my last post (SOA: Top Two Tech Topics for 2008):

• Managed Services Engine v6.2 (service virtualization)
• The "Oslo" SOA and Software+Services platform for composite applications

"Oslo" is based on the BizTalk concept (BizTalk Server + BizTalk Services) supported by the .NET framework (WCF, WF), model-driven design tools, and of course a repository for service life-cycle management.

Read more about "Oslo" in the Directions on Microsoft article provided on Christian Weyer's blog.

SOA: Top Two Tech Topics for 2008

As Gartner has published their 'Top 10 Strategic Technology Areas for 2008', of which several are related to SOA; this is my two cents about two more specific topics that are important for you as an architect and developer of service-oriented systems:

• Service Virtualization
• Aspect-Oriented Connectivity

Read the relevant articles to learn more about these indeed very technical topics, you really cannot manage a complex set of hundreds and thousands of services without such SOA operations and lifecycle governance technologies. Microsoft is on the ball with the BizTalk ESB Guidance and the partnership with AmberPoint (in use at Vital Forsikring in Norway) and SOA Software.

Also check out the Connected Services Framework being used at telecos around the world (watch video). You can play with aggregated services and managed network mashups at the Connected Services Sandbox.

BizTalk Services SDK, RelayBinding: Firewall Issue

I recommend that you download and try the samples of the BizTalk Services SDK (identity & access control, connectivity, web-style stuff) to get a deeper understanding of what the "internet service bus" concept provides to service-oriented solutions - both for the SaaS approach and for the Software+Service approach.

An error that you might run into when trying any of the connectivity samples is the "No DNS entries exist for host connect.biztalk.net" exception when calling host.Open() from e.g. the EchoSample server. You'll get a really lost feeling when googling for this error gives no hits:

System.ServiceModel.EndpointNotFoundException was unhandled
Message="No DNS entries exist for host connect.biztalk.net."
Source="System.ServiceBus"
StackTrace: at System.ServiceBus.RelayedOnewayClient.Connect()

The cause of this problem is that the RelayBinding uses an outbound TCP connection to connect to sb://connect.biztalk.net/ to set up your hosted endpoint; and if you're behind a firewall, this will fail unless permitted by the firewall. Refer to the ReadMe file of the SDK for more details.

Information Model Mediation: Esperanto vs Babel Fish

In my quite popular post SOA: Canonical "Data" Model (picked up by Steve Jones, David Linthicum, Jean-Jacques Dubray and others), I used two analogies for the semantic mediation needed when composing different services into business processes: Esperanto and the Babel fish. This post is about how the two approaches differ in their use of the business process information model (BPIM). But first a short recap of the two terms:

Esperanto is a common world language; when you learn it, you can speak with any other person in the world that also known Esperanto. Everyone does of course still have their native language. See how similar this is to services that interact using a common information model (CIM) to express themselves, while every system still use their own domain model inside.

A Babel fish is a small fish that you put in your ear, which is capable of simultaneously translating any known language in the universe to yours, while adapting semantics between galaxies. See how similar this is to services that interact using a common logical data model (LDM) while every system still uses their own domain model to express themselves.

Both these approaches to semantic mediation in service-oriented solutions do use a business process information model (BPIM). They differ only in how they implement the transformation between formats: inside the services or outside the services.

The CIM approach to mediation is to use the BPIM directly in the service contracts and apply the transformation within the provided services [Hohpe/Woolf: Messaging Mapper pattern (477)]. This is a simple and viable approach; the services are self-contained, but require a bit more to implement and test due to the extra mediation requirements. The services will also need to be changed and tested when the BPIM changes. Dan North suggests this approach in his 'A Low-Tech Approach to Understanding SOA' article. This is still the most common approach, especially for those without an ESB.

There is one minor challenge with the CIM approach: chances are that the Esperanto of an alien galaxy will differ from your Esperanto. Thus, when outsourcing an activity service such as 'credit check', you need to adapt to the service provider's business concepts. So even if you use a CIM for all your enterprise services, you will still need to implement some sort of mediation to do the context mapping between the two federated business domains.

The huge advantage of the CIM approach is that it doesn't require any ESB-style intermediary. The services speak "CIM" natively and thus no transformation to/from the common data model is needed. Note that the service implementation must still map between the CIM format and its internal data formats.

The LDM approach involves using an orchestration mechanism to compose business processes from the set of domain specific services. The services just provide their functions independent of who the requesters (consumers) are and how they try to interact with the services. Think of your SOA solution as having Event Driven Architecture (EDA) to free yourself from "invoking operations" and think of business events and messages instead. It is the task of the Babel fish to mediate and transform the messages sent between consumers and providers [Hohpe/Woolf: Message Translator pattern (87)]. The intermediary can be just the service composition mechanism, but an Service Bus is a more robust and flexible mediation and transformation mechanism.

The LDM approach makes the services themselves simpler and easier to implement and test as they now have just a single responsibility. The services need not change when the BPIM changes, that complexity has been shifted to the intermediary. Implementing semantic mediation outside the services is proposed by Jack van Hoof, David Chappell (Pope of ESB), Bobby Woolf and many others.

Enterprise-Level Business Concepts

Dan North has expanded on Arjen Poutsma's SOA 'holiday request form' analogy in his post 'A Low-Tech Approach to Understanding SOA'. Applying the paper form analogy to design the interactions and messages of your business processes is an approach that I really recommend.

What got my attention was the recommendation at the end of the post in the 'Avoid a universal domain model' section: Do not introduce an “enterprise information architecture” / “universal data dictionary” trying to force everyone to use the same domain model. Instead, introduce business concepts. This is effectively the higher-level, ubiquitous language that ties together all of the finer-grained domain models behind each service. The services use the enterprise-level business concepts when interacting, which decouples the service consumer from the service provider and allows them to evolve independently.

This is spot on with my advice for making a business process business process information model (BPIM) and avoid enforcing a "enterprise data model" or "canonical schemas" across your services. Not only does BPIM allow for loosely-coupled, evolvable services; it also allows for the services to be semantic covenants which is important for service composition.

Time to move on

As of Monday next week, I'll start working at Microsoft Consulting Services here in Oslo, Norway. I'll be working as a solution architect focusing on service orientation and integration, which is what interests me most - as long time readers of this blog might have noticed.

You may also have noticed that I've dropped CRM from the blog title, as I have not worked with MSCRM for more than a year now. I will not be involved with MSCRM even if I move to MCS. Alas, CRM cases these days involve a lot off business processes, workflow and integrations, so the random MSCRM related post might still appear at this blog.

To my colleagues in Objectware: So long and thanks for all the fish!

Managed File Transfer, RoboCopy, Sheduled Tasks

This spring I blogged about our research for a simple and inexpensive Managed File Transfer (MFT) solution. After reviewing the pros and cons of BITS for two-way robust high volume file transfers for a WAN scenario, RoboCopy was chosen due to its simplicity, configurability and logging capabilities.

Our MFT needs are covered by setting up a set of source-target pairs of UNC shares on the WAN and using the continuous monitoring mode of RoboCopy to move any files dropped in the source folders in a robust and reliable manner to the target folders. Use e.g. these RoboCopy options to do reliable transfers every five minutes when there is at least one new file:
ROBOCOPY.EXE . . . /Z /S /MOV /MON:1 /MOT:5
Refer to the documentation installed with RoboCopy to learn more about the different options.

I recommend making a command file (.CMD) that contains the RoboCopy jobs for each source-target folder pair. Note that when running in continuous monitoring mode RoboCopy will run "forever" and will not exit. Thus, you cannot just put multiple RoboCopy jobs into the command file, as the first job will block the command file, hindering all but the first process to start. Luckily, you can use the 'start' command to spawn/fork new processes:

start robocopy.exe \\mftserver001\uncshare1 \\dmserver001\uncshare1
start robocopy.exe \\dmserver001\uncshare2 \\mftserver001\uncshare2

We have used Scheduled Tasks to launch the RoboCopy processes instead of implementing a Windows Service. This makes configuring and running the jobs more accessible to the operations department as all they need to know is standard Scheduled Tasks (and saved me from implementing another Windows Service and installer). A few configuration tips:

• Use 'At system startup' to launch the command file only once and when the server (re)starts; i.e. like a service that has automatic startup
  
• Use 'Run As' to set the identity used to run the processes, i.e. the account that has the applicable NTFS and UNC share permissions
  
• Clear 'Run only if logged on' to ensure that the task is run like a service
  
• Clear the 'Stop the task if it runs for' option as the task should run "forever" in continuous monitoring mode
  

I strongly advice against running the command file using an account with admin rights. We use a limited account for the RoboCopy tasks, this identity has only the permissions needed to perform the managed file transfer.

Using a minimum permission set did of course cause an error when testing the RoboCopy task on a Windows Server 2003 machine: just running the command file worked fine, but running the scheduled task item got the status "Could not start'. To see the actual error of a task, use 'Advanced-View log' in the Scheduled Tasks main window and look for errors. You will typically find this error:

0x80070005: Access is denied.

I had granted the MFT identity read+execute rights on RoboCopy.exe and on the command file, so I was a bit puzzled. To make a long story short, Googling led me to "Access is denied" error message when you run a batch job on a Windows Server 2003-based computer. The problem was missing rights on the one part of the task that you do not think about: CMD.EXE itself.

Now everything was fine and dandy; and a restart of the server to simulate a server crash, followed by reviewing the processes using Task Manager showed the correct number of ROBOCOPY.EXE processes running in the background.

This MFT solution is truly based on the "Make everything as simple as possible, but not simpler" principle.

IIS6: App-Domain could not be created

Each time when I have to deploy a ASP.NET/WCF application to a new Windows Server 2003 IIS web-site, I run into a "Server Application Unavailable" error and this logged in the application event log:

Failed to execute request because the App-Domain could not be created. Error: 0x80070005 Access is denied.

Exception: System.IO.FileLoadException
Message: Could not load file or assembly 'System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Access is denied.

As always, I'm a bit puzzled by the access denied error as I have granted NTFS read+execute rights to the ASPNET and IUSR accounts as applicable to the application's disk folder and to the .NET 2.0 and 3.0 folders. Time for Microsoft System Internals Filemon to log file access failures to diagnose the problem.

Note to self: The cause of the problem is that the identity (e.g. NETWORK SERVICE) of the IIS application pool used to run the web application, needs read access to the application's web.config file to be able to launch the ASP.NET worker process.

Read the excellent MSDN article 'Extend Your WCF Services Beyond HTTP With WAS' to learn more about IIS/ASP.NET/WAS application pools, process models and app-domains.

Guerilla SOA, MEST

InfoQ has posted an interview with Jim Webber of ThoughtWorks about "Guerilla SOA", where he talks about how to design scalable SOA solutions that are evolvable and allow innovation in the different services that make up the solution.

The advice he offers covers topics that I have written about in my posts these last months:

• Share business messages owned by the business people (business process information model)
• SOA has no operations, only business messages delivered to letter boxes - services exchange messages to fulfill business processes
• Services cannot be invoked, they just receive event+payload messages and decide if and how to process it: "Could you possibly have a look at this message and maybe if it suits you do some processing on it" (semantic covenant: the service is always right)
• Business processes are workflows; you need to model long-lived conversations beyond request/response (SSDL, check out the Soya WCF SSDL toolkit at SourceForge)

This architectual style is coined MEST, as it resembles REST: it is based on message exchange through a uniform interface - the letterbox. As you will see, MEST is a decoupled scalable design pattern along the lines of EDA.

Listen to the interview and read the transcript: Jim Webber on "Guerilla SOA"

The Information Model for Composite Services is not about Solving Service Versioning

The business process information model (BPIM) is not intended for solving versioning issues for a specific service, that can be solved using message duck-typing or consumer-driven contracts. The BPIM is for mediating semantics between independent and autonomous services, allowing the services to be composed into biz processes and still be able to evolve each service separately. The autonomous SOA tenet should be used to achieve shared services and interchangeagility rather than focusing solely on reusability. A service is not autonomous before you can use the scissors on parts of a business process and replace the service by e.g. outsourcing it.

Indeed, service versioning is important for the evolvability of the service itself, but the BPIM is about the agility of the business processes in your service-oriented solution.

Do not confuse the BPIM with the EAI CDM, they are similar concepts at different architectual layers. The BPIM will contain a subset of the common information model (CIM), but is rather focused on the data+business documents and semantics of the business events and processes that the information model encompasses. Design the BPIM based on the CIM, ensuring that the model is canonical for each business process domain. The BPIM is targeted at the service composition layer, not at the data access services layer.

Related reading: SOA doesn’t need a Common Information Model

Lean Information Models for Composite Services

A business process information model (BPIM) is a model that is focused on the business events and the data+documents+messages needed as part of orchestrating the events into business processes by composing services. A central part of the creating the model is to keep the model as small as possible (like a "least common denominator" of the data), only passing the state of the process as part of the events. The data and business documents represent the state of an instance of a business process (what Jack van Hoof calls a dossier) and is all a business process step should need to know to be able to perform its actions.

It is important to recognize that the BPIM does not encompass all information that some part of a business process might ever want or need to know, it just needs to contain enough reference data. Neither does the BPIM have to contain data about business entites across all applications in your enterprise, i.e. the BPIM "customer" does not need to model the union of all the different customer fields from your CRM and ERP systems. This is why BPIM is different from a common information model (CIM) or the EAI Common Data Model [Hohpe/Woolf CDM pattern (355)].

The business process that need to know more about the referenced data of the dossier, will have to use services to look it up, as part of the service composition (orchestration). This is what is called the "pull model" in the Observer pattern [GoF] [Hohpe/Woolf Event Message (151)]. The business process can also augment the process state by calling services and adding data to the dossier. Thus, the mantra of making a slender model is "Ask For More".

Keeping the information model small is very important wrt to maintenance of the model, afterall the main purpose of the model is to mediate semantics in the service bus and its orchestrations of services. The number of mappings between the different elements (messages+dossiers and services) of a model will grow and change over time, thus having small model elements are crucial to keep the model maintainable. In addition, split your model into separate domains/bounded contexts and make context maps, as this makes the total number of mappings smaller.

The key maintenance advice is to keep the model elements small (smaller mappings) and the number of mappings between model elements to a minimum (less mappings).

Composite Services: Information Model, Federated Information Models

A SOA solution is typically implemented by composition of services provided by different systems. The solution might comprise a mix of of in-house services, 3rd party and outsourced services. In addition, the solution might also involve cross-enterprise service compositions.

To be able to efficiently compose the services, a model that ensures shared sematics is needed, and in my last post I described the business process information model (BPIM). David Chappell calls this approach semantic data integration. BPIM is a similar concept to the EAI canonical data model [Hohpe/Woolf CDM pattern (355)], but it is a model with a slightly different purpose at a different architectual layer and it is not just about the data. The BPIM is about the events, messages and data passed between services, not about having a unified superset of the entities within an enterprise.


The BPIM allows for composition of services without having to know and comply with the model of the underlying service logic, all you need to know is the message types. This is a big advantage, and allows for the consumers to be isolated from the details of the consumed services. E.g. all a service consumer need to know is the "AddressChange" data of the "CustomerHasMoved" event, not the complete data schema of the "Customer" service.

BPIM is closely related to the Common Information Model (CIM) concept. Note that the BPIM message types are projected compositions of the referenced CIM resources - not just simple compositions of resource objects. In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain.

The term 'domain' is borrowed from Domain Driven Design (DDD). Domain-driven design is not a technology or a methodology. It is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains. Read stories about how DDD can be applied to a diverse set of architectual problems: Practitioner Experience Reports.

Focusing on the bounded context for modeling the flow, events, messages, data and semantics involved in implementing the core business processes of the domain, should make it easier to come up with a working model. Note again that the information model is about more than just the data. Hence the name business process information model. Arvindra Sehmi and Beat Schwegler used the same term in Service-Oriented Modeling for Connected Systems, an article that also provides details about creating a service model.

Partner/ 3rd party/ outsourced services are not part of the core business domain according to DDD. If they were core processes in your business, how come they are so general that they can be outsourced or bought? Core processes are those that make your business unique and give you a competitive edge. DDD dictates using translators or an "anti-corruption layer" against services/systems that are not within the domain.

Having a canonical schema model at the service layer might be feasible within an enterprise, but should be avoided as this will cause very tight coupling to the One True Schema. Making every service depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. If agility at the service layer is less important for you, then such a service straight-jacket might initially feel good. Trying to make a enterprise data model (EDM) is not a good idea for the same reasons. Steve Jones has a good post about canonical form issues and how you cannot enforce your model upon the world: Single Canonical Form - not for SOA

Federated Business Process Information Models

DDD recommends splitting big, diverse and complex solutions into several bounded contexts. Set explicit boundaries based on e.g. orgranizational units and application usage. A natural boundary in SOA is partner/ 3rd party/ outsourced services. Each set of services that is not under your control and that you cannot enforce your information model upon, is a separate bounded context (domain model).

Note that also within your enterprise service model there will be several other bounded contexts for different domains, each with its own information model that is canonical per domain.

You should identify each model in play on the project and make a context map. A context map describes the points of contact between the domain models, in addition to outlining explicit translation for any communication between the models.

This figure show how a context map is used to show how two domains relate to each other:
The figure and the definitions of "bounded context" and "context map" is taken from the 'Strategic Design' chapter of the book "DOMAIN-DRIVEN DESIGN" by Eric Evans [Addison-Wesley, 2004].

Note how not all elements of a domain model needs to be mapped to other models. Only the interconnected parts needs to have a translation map. E.g. the credit check process is provided by a 3rd party, thus it exists in a separate external information model. Your BPIM needs to have a translation map to the other service to be able to invoke it. Note how similar this is to the purpose of the BPIM itself: translating between business process compositions and the underlying, composed services. Thus, the context map is the basis for modelling a set of Federated Business Process Information Models. A federated BPIM system map shows the integration of multiple enterprise service models, avoiding the pitfall of designing a single canonical data model across a set of different domains.

SOA: Canonical "Data" Model

An important topic when designing service oriented systems is how to enable different services to share semantics to be able to be composed into working solutions. Jack van Hoof has written a good article about this: How to mediate semantics in an EDA. A few weeks later, Nick Malik posted another good read about this topic: Canonical Model, Canonical Schema, and Event Driven SOA. Read Jack's post first.

They both talk about using a canonical data model (CDM) as the Esperanto / Babel fish to map between the format and semantics of the disparate systems taking part in a SOA solution. Note that CDM is not about having a common data model (EAI CDM) or a shared database across all systems in an enterprise, don't get fooled by the "data" in the term 'canonical data model'. CDM is about not making everybody have to speak English, but rather having CDM translators for each native system.

Btw, Gregor Hohpe sometimes use the term 'canonical domain model' on his blog, while using the term 'canonical data model' in the book "Enterprise Integration Patterns" [Hohpe/Woolf CDM pattern (355)]. I think it is better to talk about the business domain rather than about "data", as this help focusing on the business processes rather than databases and other technology. You'd be surprised how many biz people concern themselves with how the data model looks - maybe a leftover from the client-server days, to show that they know what an ER-diagram is? Focus on designing a business process information model (BPIM) for each business process domain.

Trying to enforce a One True Schema across your services (everyone has to speak English) is not a viable path, and it is also a recipe for future maintenance hell. Making every service contract depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. A simple change to e.g. the order entity will cause a ripple effect through all referring services. This is where the business process information model comes into play, it allows you to version and evolve the services independently of each other.

The Canonical "Data" Model concept is also sometimes referred to as a Common Information Model (CIM). Both the business process information model (BPIM) and the data focused CDM/CIM models has the same goal: mediation og semantics. However, they are not the same as the two other models are both variations of the common data model approach. The business process information model is about semantic business process integration, not just only semantic data integration.

SOA: Ontology and Taxonomy of Services

As someone misinterpreted the use of the term "service" as an idiom in my last post, to imply that the post was about defining that "a specific service is a single business process step" (rather than being about semantic covenant services); let me point to two 'Architecture Journal' articles defining important terms for discussing service-orientation:

• Ontology and Taxonomy of Services in a Service-Oriented Architecture
• Service-Oriented Modeling for Connected Systems

The articles are about the defining different categories of services and about creating business models and mapping business capabilities to services. The last article covers Motion (video at Channel9), which is now known as Microsoft Services Business Architecture (MSBA).

Think of this when reading the articles:

A service classification scheme should have the "hierarchy" taxomomy form. It should not use a scientific hierarchy nor a tree form. There must be no "is using" or "is dependent on" relation between the services in the taxonomy. There can be no dependency between services in SOA except for composite services realizing a business process by consuming the actual services. Taking a dependency on another service will break the central "services are autonomous" tenet of SOA. Having a service calling another service will just repeat the failure of DCOM/CORBA for distributed system design, using RPC over web-services solves nothing.

Semantic Covenant: The Service is Always Right

This post is about service semantics and how to be “liberal in what you accept” in your service-oriented architecture implementation. Let me start by defining that in this post the term "service" is used for a specific business capability that can be composed and reused in different business processes. I.e. the idiom “service” used here is representing a building block business capability. This is not to say that a service equals a single business process step, it just makes it easier to talk about the topic of this post (which is not BPM): semantic covenants.

A service has a defined contract, but also some implied semantic. The semantic is hard to express through the contract, and the business compositions that utilizes (consumes) the operation can only assume that the service is a covenant, i.e. that the service will do B when provided with input A. Having semantic coupling is just another type of coupling that prevents services to be truly loosely-coupled and that will cause a breaking ripple effect through all consumers of the service when the inevitable thing happens: change.

There is some thinking in the SOA community that to avoid this ripple effect caused by a service contract or semantic change, it is up to the service to do the right thing independent of the information provided by the sender of the message. That is, that the service should behave as if the consumer is always right, and adapt its semantic to suit the sender.

A service can do no such thing, as it is an actual representation of a specific business capability. The capability does what it does, i.e. a credit check is just that – a credit check. It must be up to the business composition of the services to interpret the real-life events that have occurred and their context, and then invoke the right set of operations. Note that the service should still be liberal in what it accepts (message duck-typing), as it is the business event that is important, not the format of the event message.

It is easier to recognize that the service is always right when you think of messages as representing business events and apply EDA thinking such as publish/subscribe instead of a habitual SOA command and control pattern. A service will then subscribe to business events that must trigger the business capability, but the service need not know who the publisher is. Neither need the publisher know who is listening for the business process state changed events (event topics). Thinking in service “consumers” is very connected to having a request/reply, command and control style SOA. 

To simplify the “service is always right” notion, let us call the publisher a “consumer”. The consumer can send any message (business process event, context and state; mail-order analogy) it likes, as it is the publisher. However, as the consumer does not know who the subscribers are, it cannot know or depend on the services having specific semantic. The service cannot depend on who the publisher is, it just accepts that a business event happened that it must process. A business event message never specifies any handling operations, the logic to decide how to handle the event is part of the mechanism that routes messages to services. This ensures that the autonomous and loose coupling tenets of SOA are fulfilled.

When you apply EDA style thinking to service orientation, you’ll see that the service is always right. There cannot be any traditional “the consumer is always right” in relation to service semantic, as the services have no request/reply consumers.

Alas, still the consumer can still be right; it is just the responsibility of the process composition mechanism to accept what ever messages (events) the consumer (publisher) throws at it and invoke the applicable service operations. It is the business process composition mechanism that has to be the semantic covenant.

PS! Note that I have deliberately used the term "composition" here and not even mentioned using an ESB or BPM to implement this. The semantic covenant composition mechanism can simply be implemented as a mashup (composite application) instead of a composite service or an orchestration/saga.

What is WS-Splat anyway?

I just briefly gave an overview of the ten WS-* standards supported by WCF at my talk at NNUG a couple of weeks ago, as there are way too many topics there to cover in just one session. The ten supported standards are:

• WS-Security
• WS-Trust
• WS-SecureConversation
• WS-Policy
• WS-SecurityPolicy
• WS-Addressing
• WS-ReliableMessaging
• WS-AtomicTransaction
• WS-Coordination
• WS-MetadataExchange

Alas, Michele Leroux Bustamante has written an excellent introduction to all the different WS-* standards: Making Sense of all these Crazy Web Service Standards.

Definitely worth reading if you wonder whats the difference between WS-Security and WS-SecureConversation and other important issues. Then again, who wonders about such things...

Flex support for [Flags] enum - not

My last post showed how to get explicit semantics in a WCF contract description using [Flags] enum. As readers of my blog may know, our main service consumer is a Flex 2.x client. As it turns out, Flex has an issue with correctly interpreting such enumerations.

A [Flags] enum looks like this in the WSDL:

<s:simpleType name="DisciplineFlags">
<s:list>
<s:simpleType>
<s:restriction base="s:string">
<s:enumeration value="None" />
<s:enumeration value="Administration" />
. . .

The correct input format for an element that uses the above type is XML whitespace separated values, such as "None Administration". Flex, on the other hand, interprets the list schema construct too litterally, and tries to pass in an array. This causes an exception in the WCF service, as the message cannot be deserialized due to the incorrect XML.

As the Flex guys had no other solution to this problem, I just added a simple "FlexDisciplineFlags" string property to the contract. Flex has to pass a CSV list of the filter enum values, and the service will "cast" the input into the official filter condition:

private void HackFlexIssues(DocumentCardFilter filter)
{
if (String.IsNullOrEmpty(filter.FlexDisciplineFlags)==false)
{
//input string must be a CSV-list of enum items
filter.DisciplineFlags = (DisciplineFlags)Enum.Parse(typeof(DisciplineFlags), filter.FlexDisciplineFlags, false);
}
}

Note that I'm kind of making a workaround for a technical limitation, maybe I'll get sued by Adobe :)

Enum Flags, Specification Pattern & Explicit Semantics

We use the specification pattern for all the dynamic query operations in our WCF service, and this includes using Nullable<T> on all criteria elements and several "multiple choice" criteria based on [Flags] enumerations. To ensure that our contracts convey explicit semantics, we have used long meaningful names in the enumerations instead of the more cryptic codes that are wellknown in the internal system and that are the domain terms used by the biz-people. These internal codes is not, however, very meaningful to the partners that are consuming the WCF service.

As the multi-choice lists sometimes contain 20+ items, I feared that interpreting the [Flags] enum and building the repository query would become huge switch statements with lots of ugly bit-wise "and" logic to deduce which of the 20+ items were specified in the criteria. In addition, I needed a way to add the translated code items to a SQL @parameter list as part of a SQL in clause to actually filter records on the criteria.

As I never embark on implementing something ugly, I decided to look for a simpler way of interpreting the [Flags] enum. Knowing that an enum internally is represented by the integers and not the human-friendly names, I decided to add another internal enum that exactly mirrors the contract enum, just using the internal domain codes, and then cast to the internal enum. But the DataContractSerializer has a simpler approach using the [EnumMember] attribute:

[DataContract]
[Flags]
public enum DisciplineFlags
{
None = 0,
. . .
[EnumMember(Value = "Instrumentation ")]
I = 0x000100,
[EnumMember(Value = "MarineOperations")]
J = 0x000200,
[EnumMember(Value = "Materials")]
M = 0x000400,
[EnumMember(Value = "Navigation")]
N = 0x000800,
//
[EnumMember(Value = "Process")]
P = 0x001000,
[EnumMember(Value = "QualityManagement")]
Q = 0x002000,
[EnumMember(Value = "Piping")]
S = 0x004000,
[EnumMember(Value = "Telecommunications")]
T = 0x008000,
. . .
}

Note that you must apply the [DataContract] attribute to the enum for the [EnumMember] to take effect. If you just use the plain enum as a [DataMember] property, then the internal names will be published in the contract.

The "None" item is there as the default value for the DataContractSerializer when the incoming XML specification contains no flags filter element (xsi:nil="true"). The nullable flags filter must be asserted like this before usage:

if (filter.DisciplineFlags.HasValue
&& filter.DisciplineFlags != DisciplineFlags.None)
{
. . .
}

So now I had the real one-letter domain codes, but how to avoid bit-wise interpretation ? Also, I needed a list of the codes making up the flags combination. The answer is really simple:

string disciplineList = filter.DisciplineFlags.ToString();
string[] inValues = disciplineList.Split(new char[] {','}, StringSplitOptions.RemoveEmptyEntries);

The last problem is that you cannot add the list of values for an in-clause as a @parameter. Just concatenating the CSV-string into the SQL is not an option due to SQL-injection attacks. So a for-loop is needed to add SQL parameters and related values:

for (int i = 0; i < inValues.Length; i++ )
{
string param = "@in" + i;
if (i > 0) sql += ", ";
sql += param;
filterCommand.Parameters.AddWithValue(param, inValues[i].Trim());
}

I would rather have avoided the for-loop, but at least the specification interpreter became much simpler than I expected it to be. No pesky switch statements and bit-wise logic.

See the end of this earlier post for further details about implementing specification filters using TableAdapters.

WCF: Security Architecture and Claims

At my WCF talk at NNUG last night, I didn't have the time to go into the security architecture of WCF in general or into identity model details such as STS, tokens, policies and claims.

To get an overview of these aspects, I recommend reading "The holy grail of Enterprise SOA security" by Matias Woloski.

To learn more about how to actually build an authorization system based on claims, you should read "Building a Claims-Based Security Model in WCF" part 1 and part 2 by Michele Leroux Bustamente.

[UPDATE] Read the Digital Identity for .NET Applications whitepaper by David Chappell to get a good overview of the involved technology.

WCF+EntLib3: Getting started

In addition to the WCF+EntLib3 validation and exception handling posts at Guy Burstein's blog/CodeProject, this info from David Hayden's blog about logging should get you started:

• A new EntLibLoggingProxyTraceListener class. This trace listener is designed to be used in WCF’s System.ServiceModel trace source, which is configured within the <system.diagnostics> section. This trace listener receives WCF messages logged to this trace source, wraps them in an XmlLogEntry class, and forward them to the Logging Application Block where it can be processed according to the application block configuration.
• A new XmlLogEntry class, which derives from LogEntry but includes a new Xml property that preserves the original XML data provided by WCF.
• A new XmlTraceListener class, which derives from the .NET XmlWriterTextWriter class. This class can extract XML data from an XmlLogEntry class

In short, attributes and the interceptor mechanisms of WCF are used to integrate EntLib3 features for validation and exception handling, while logging is just another trace listener. You'll need to understand how to configure all three application blocks to make the WCF integration work. I recommend using the EntLibConfig.exe tool to configure the EntLib3 stuff, and the SvcConfigEditor.exe tool to configure WCF tracing wrt logging. The WCF toolbox is included in the .NET3 SDK.

That is all I have managed to find regarding the WCF features of EntLib3. The online MSDN docs contains mostly "new WCF features added to the ____ Application Block" statements and no code samples.

MSCRM: viewing DHTML source of 'related entity' frames

Michael Höhne has an good post on how to change the default view of related entities in MSCRM v3, in which he shows how to gather the info necessary for creating the OnLoad javascript. One of the needed elements is the ID of a combobox inside the <iframe id="areaActivityHistoryFrame">. However, getting at the source of frames is not that easy in MSIE as Michael has discovered. The same goes for seeing the actual content of a modified dynamic HTML page.

Viewing the 'history' source is actually rather simple, follow these steps:

• Open the view account form
  
• Press CTRL-N on the keyboard to show the browser chrome including the address bar
• Select the 'History' folder to view the related activities
  
• Enter this javascript in the address bar and press enter to view the frame source:
  

javascript: '<xmp>' + document.frames("areaActivityHistoryFrame").document .body.outerHTML + '</xmp>';


Change the name of the <iframe> in the javascript to view the source of other related entity folders. Remember to select the folder first to make the frame source available for the script.

The above script is a slight modification of this script to view the dynamic source of the window.document:

javascript: '<xmp>' + window.document.documentElement.outerHTML + '</xmp>';

The MSIE Developer Toolbar is also useful, except for the issues with frames as described by Michael.

While you're in javascript mode, check out the impressive list of useful javascript snippets for MSCRM at Michael's Stunnware site.

WCF: Exception Handling with Logging

In my 'WCF Exception Shielding using Generics' post there was a "//TODO: add logging as applicable" comment, which I now have implemented using the EntLib3 Logging Application Block.

As part of the error information I wanted to write to the Windows Application Event Log was the actual request message causing the service operation exception. Thus, I googled for a solution for how to get the XML of the incoming message, so that I could log it along with the stack trace. There are a lot of options for getting at the message in WCF:

• Write a custom message interceptor at the channel level
• Use the Message class directly in the service operations definitions
• Use CreateMessage internally to get at the Message class methods to reproduce the behind-the-scenes stuff of WCF

The problem of the first option is that I only want to get at the message XML when handling exceptions. The second option is not an option (duh!) as I want my operations to be strongly typed using my data contracts. The third option was more like what I wanted, but it seemed to be unneccessary complicated code.

The code to get at the message XML from within an operation is quite simple:

private static string GetMessageXml(DefaultMessage request)
{
return OperationContext.Current. RequestContext.RequestMessage.ToString();
}

This approach does not work for operations with no RequestContext, such as one-way operations.

The logging code is added after creating the fault details, this makes it possible to use fault details data in the log. This is how my WCF exception shielding with logging looks like:

public static FaultException<T> NewFaultException<T>(Exception ex)
where T : FaultContracts.DefaultFaultContract, new()
{
return NewFaultException<T>(ex, null);
}

public static FaultException<T> NewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
return CreateNewFaultException<T>(ex, request);
}

private static FaultException<T> CreateNewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
StackFrame stackFrame = new StackFrame(2);
string methodName = stackFrame.GetMethod().Name;

T fault = new T();
Type type = fault.GetType();
type.InvokeMember("SetFaultDetails", BindingFlags.Public BindingFlags.Instance BindingFlags.InvokeMethod, null, fault, new object[] { ex });

string msg = String.Format("An unexpected error occurred in {0}.", methodName);
Dictionary<string, object> props = new Dictionary<string, object>();
props.Add("stackTrace", ex.ToString());
if(request!=null)
{
string xml = GetMessageXml(request);
props.Add("requestMessage", xml);
}
Logger.Write(msg, Category.General, Priority.Highest, (int)EventLogId.Exception, TraceEventType.Error, ex.Message, props);

string reasonText = methodName + " failed, check detailed fault information.";

#if DEBUG
reasonText = methodName + " error: " + ex.ToString();
#endif

return new FaultException<T>(fault, new FaultReason(reasonText));
}

Note that I now use an inner private method to create the new fault to be thrown and thus must skip two stack frames to get the details of the original method called. The DEBUG section is there to make it simple to see full exception details in NUnit or with Fiddler when debugging the service. This DEBUG code must never make it into release builds, as this is not exactly exception shielding.
I have not looked into the WCF app-blocks integration mechanisms added in the april release of EntLib3, that is next on my task list for quieter days. There is not much info about this to be found, but Guy Burstein provides some samples. Btw, you should check out the Patterns and Practices Guidance site, it contains useful stuff and tutorials about EntLib3.

EntLib3 should apply convention over configuration

After using the EntLib3 validation application block (VAB) for a while now, and also some of the new configuration mechanisms such as external configuration source and environmental overrides for different build types; it occurs to me that there is a bit to much XML noise in the config.

EntLib3 should do as in Castle Windsor and MonoRail and Ruby on Rails; apply "convention over configuration". In EntLib3 config files, you have to add mandatory attributes to identify the default configuration when there are multiple config options. Using "first is default" as in the Windsor 'constructor injection' mechanism would make configuration simpler and less error prone (see VAB issue below). I understand the reasons for having a default-identifier as a separate attribute, but with the new config override and merging mechanisms in EntLib3, there should be less need for compulsory config "switching" attributes.

The convention "first is default" would prevent silly omission errors such as not setting the default rule set in VAB less drastic. As it is now, if you forget to set the VAB default rule set, no validation will be applied, neither will there be any "no rules" exception - and your code will run as if all validations passed, even if there are plenty of broken rules in the input.

While I'm at it, the caching of connection strings and service URLs in the Settings class and the service proxies, is also really silly; you will not be able to detect that some configuration is missing until you move the solution to an isolated staging environment that has no access to the databases/service resources referenced in the development environment. Most test environments are not that isolated from the development environment, and such config errors can go undetected for a long time during testing. This is one area where it would be better if Microsoft could make configuration compulsory.