Versions and Schema Semantics in SOA Composite Services

As part of your SOA governance efforts, it is not sufficient to handle just the versioning of the services and the data schemas used by the services. You must also govern the semantics of the data to enable service composability. Composite services are not possible without semantic data integration.

There are two information models in play when composing services - just as there are two different, but related, process models involved in service-oriented solutions. One is the well known common information model (CIM), the other is the related business event message model used to enable semantic business process composition.

The process composition information should be partitioned according to the business process domains to allow for the business process information model (BPIM) to evolve independent of each other. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain. Your BPIM must be federated even if evolved and versioned separately, to enable semantic data mediation and semantic business process integration.

The BPIM contains metadata that models business event messages as projections of CIM data entities. A business process message typically contains a subset of one or more domain objects. E.g. in an order handling system, the real-world event "customer submits order" is a projection of the customer, address, credit and product entities from the CIM. Think classical paper-based mail order schemas or the Excel schemas you use to get your travel expences reimbursed, or rather the form used to apply for a vacation.

Creating an enterprise canonical data model might be feasible, but federated domain models are recommended. You would anyway need to mediate semantics between your system and third-party services that you involve, or on parts of your processes that you chose to outsource.Federated models are required for B2x integrations as you cannot easily enforce your model on the outside world. Using a CDM might work in a centrally controlled EAI hub, but most likely not across departments, organizational and partner boundaries in an extended enterprise.

Information Models: Business Process or Forms-over-Data

Joe McKendrick has a post about the need for doing data governance today:

If data governance is inadequate — information is outdated, out of sync, duplicated, or plain inaccurate — SOA-enabled services and applications will be delivering garbage. That’s a formula for SOA disaster.

He links to the XML to the rescue: Data governance in SOA article by Ed Tittel, which describes how having a common model and doing data governance is imperative for SOA success. The article conveys the same concepts as chapter 4 'XML: The Foundation for Business Data Integration' in David Chappell's seminal book Enterprise Service Bus. David describes the need for having a common XML data format for "expressing data in messages as it flows through an enterprise across the ESB". This common XML format is a specialization for the common data model (CDM) pattern from the classic book Enterprise Integration Patterns by Gregor Hohpe/Bobby Woolf.

I've written many times about the importance for having a common information model (CIM) for the domain objects (business entities) that your data services operates on, and the need for applying master data management (MDM) on those data. David Linthicum and I agreed on this last year and still do.

What I've written even more often about, and what most canonical models do not encompass; is that the data expressed in the messages conveying business events is not simply CIM entities, but are rather projections of one or more of the domain objects. In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands.

E.g. an insurance clerk triggers a business process by registering the real world event "customer has moved" using a projection of customer and address data, not the complete customer aggregate. This action event message triggers a recalculation of the insurance premium due to the relocation of the customer, in addition to updating the customer record - i.e. an underlying business process is executed.

In a traditional forms-over-data application, the whole customer aggregate would be used - but this style would not make for services that can be easily composed to support business process management efforts. The business event data requirements must be simple enough to make composite services possible, including semantic data mediation and integration. In addition, process tailored data make it easier for any involved human operator that have to enter or act on the data. After all, BPM is not all about automation, and human tasks will be central in driving the processes for quite a while.

We need to move to an information model that allows for semantic business process integration, not just semantic business data integration. That is why a business process information model (BPIM) is required, modeling the messages for the action, query and notification events that drive the business processes. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain.

The BPIM is based on the CIM as each service domain model is just a projection of the common data model, and the projection metadata should be goverened as any other SOA artifact.

How "Oslo" relates to BizTalk

The Microsoft "Oslo" initiative is to provide a modeling (UML), management, and hosting platform for delivering model-driven service-oriented composite applications and S+S solutions; plus cloud computing through modeling.

The recently published BizTalk 2009 roadmap shows how "Oslo" features can be utilized also from BizTalk:

In fact, you won’t need to upgrade BizTalk Server to take advantage of "Oslo" – current BizTalk Server 2006 R2 or BizTalk Server 2009 customers can benefit from "Oslo" by being able to leverage and compose existing services into new composite applications. BizTalk Server today provides the ability to service enable LOB systems or trading partners as web services (using WCF supported protocols), which can be composed with the "Oslo" modeling technologies.

As the roadmap shows, WCF is the central enabler for connecting services from different systems into service-oriented solutions. WCF and WF are central to the new "Oslo" modeling tool and repository, allowing users to define and execute business processes (much like BPEL, XLANG). "Oslo" has a strong focus on enabling automation of business processes with strong support for humans as central actors in the processes (like BPEL4People, WS-HumanTask). Add the BAM interceptors for WCF and WF provided by BizTalk, and you have a platform that gives repeatability, consistency and visibility to your business process management efforts.

SOA Information Models and Services Compositions

I've written quite a lot about having a model for the information needed for semantic business process composition in service-oriented solutions, two of the most read and commented are SOA Canonical "Data" Model and Ontology for Business Processes, SOA and Information Models.

The process composition information model is focused on the business events, messages and documents and the semantics of the information that the model encompasses. The model is targeted at the orchestrations level in the ontology, not at the services level; as shown in the figure (click to enlarge):



The service classification scheme has the "hierarchy" taxomomy form. It is not a scientific hierarchy nor a tree form, thus there is no "is using" or "is dependent on" relation between the services - it is just a classification scheme. There can be no dependency between services in SOA except for composite services realizing a business process by consuming the actual services. Taking a dependency on another service will break the central "services are autonomous" tenet of SOA.

A note on the splitting of the service categories: this is a pure metadata ontology based on the taxonomy for categorizing services, it is not a split based on technolog or, teams, and surely not on layers. A service should not call on other services, making a service dependent on another service will lead to repeating the failure of DCOM/CORBA in distributed systems; using web-services and calling it SOA solves nothing. Use the active service pattern or EDA to ensure that your services are truly autonomous and independent of the availability of other services. Isolate service dependency to the composite services, or even better to the consumer mashups, only.

The "business process information model" (BPIM) is what models the business events and documents in the composite services level (orchestration / conversations / sagas). I like to use a domain model as it by definition focuses on the concepts of the problem domain (the business process information artifacts) rather than just the data of the domain. The information model must comprise the different message types for the action, query and notification events that drive the business processes. The model must be canonical for its domain to ensure that the format and semantics of the model is consitent within the domain.

There is some confusion on the difference between the process composition BPIM and the EAI "canonical data model" (CDM), plus the similar concept Common Information Model (CIM). As can be seen from the comments on the two postings and some blog reactions like SOA doesn’t need a Common Information Model, this has made it harder to convey the difference between the process composition information model and the more data-oriented models.


A CIM should be used for the resources data model - which the BPIM is related to. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain. The data in the event document for the real-life action "CustomerHasMoved" is not a complete "Customer" resource entity, it is a message including minimal "AddressChange" data. This data will typically contain both actual event data combined with reference keys to the resources such as customers and addresses in the different service domains that the composition comprises. 

The message data is of course associated with the CIM of the underlying resource services. The BPIM message types are projections of CIM objects; in fact, they are projected compositions of the referenced resources - not just simple compositions of resource objects. These projections must be kept as small as possible, preferably just the resource keys, in addition to event details such as a booking reference number. Keeping the event document data model small has the nice side-effect of making the schema versioning a bit simpler.

The CIM should be the starting point of your modeling efforts, as the documents in the process compositions is based on the service domains. Having Business Process Modeling Notation (BPMN) diagrams can help in modeling, as BPMN encompass process, events, messages and business documents, covering actions, queries and notifications. Other process modeling diagrams can be used, just ensure that the events, messages and data are captured in the model.

It is still important to recognize that having a "one true schema" for process composition or SOA in general is not a recommended practice, hence the term "domain" in the naming. Rather than trying to enforce a common model across all service domains, federated models should be used. When composing services across the domains, mediation and transformation of messages will be required - here a service bus with such capabilities would come in handy.

A thing to recgonize is that the overall model incorporates several "flows": process flows, event flows, message flows, and data state flows (resource lifecycles). Typically, only the process flow is clearly shown in diagrams, while e.g. message flow is only represented by receive and send ports. See this interview with Gregor Hohpe on Conversation Patterns about how more than just the process need to be modeled.

For an excellent primer on architectural aspects of SOA and the proliferation of acronyms, I recommend reading Architecture requirements for Service-Oriented Business Applications by Johan den Haan. Note that he does the classical simplification of focusing on business processes without including a BPIM; it is important to separate the process composition model from the resource model, just as business processes are separated from resource lifecycle processes. The business process events, documents and messages are not just collaterals of the BPMN modeling process, they are important artifacts for semantic mediation in composite services.

[Business Process Integration: "e-Business: organizational and technical foundations", Papazoglou, M. P. & Ribbers, P. (2006)]

Business Process Modeling, Not Only For Automation

Every now and then I see statements like this and it makes me cringe:

"Unless we are only documenting the As Is, business process modeling are made to optimize the business process, notably by using automation."

Aiming for automating processes might be a good strategic goal, but it will get you into a lot of both political and technological issues. How will information workers react to a project that aims at automating their jobs? What percentage of business processes can readily be automated with the available technology today?; think of the BPMN-BPEL chasm. Also read The Case Against BPEL: Why the Language is Less Important Than You Think.

In my opinion, you should do business process modeling with the objective to document the processes, enable visibility into the process performance for operational awareness (BAM, KPI, etc) and to achieve consistency and repeatability in the way the processes are performed - even by humans. Way too many processes that are core to your business (DDD core domain) depends on human interaction, and these processes are not going to be automated using BPM any time soon - however, they might get efficacious support from BPMS solutions.

The ultimate BPM automation system might be Agent Smith: "Never send a human to do a machine's job".

CAB/SCSF: Designing WorkItems From Use Cases

One of the most elusive and complicated parts of CAB is the workitem dependency injection container and all its use case resource collections. SCSF improved on the CAB workitem by adding the WorkItemController as the place to put use case controller logic. The WorkItemController gets a WorkItem instance injected, and many developers never use anything but this default workitem built into the ModuleController. This might lead to a system that is hard to build and maintain due to lack of task isolation because of no good mechanisms for controlling the lifespan of use cases (workitems).

In most projects there is a strong focus on the visual elements of the solution: the views (WorkSpaces, SmartParts, Ribbons, etc) - and CAB makes it quite easy to create a composite UI application. But this doesn't necessarily make for a maintainable, pluggable and loosely coupled composite application / service-oriented business application (SOBA).

It is imperative that the workitem structure and lifespans are analyzed to be able to design the lifecycle management based on the dynamic behavior, events and state of the system's use cases. Without designing a set of workitems and their lifecycle management requirements, your application will not be able to provide users with a task-oriented composite application allowing users to start, work on, switch between, and complete a set of use cases. Without workitems, your system will be just a multi-view forms-over-data application - having no processes for resource life-cycles.

I recommend reading the 'Identifying WorkItems' section of Designing Smart Clients based on CAB and SCSF by Mario Szpuszta, a case-study from Raiffeisen Bank. The case-study shows how to analyze use case diagrams to identify workitems and modules, and provides some good rules for how to do this analysis. It shows how to use root use-cases and pure sub use-cases to identify first-level workitems and sub-workitems, and what I call hub-workitems from sub use-cases that are used from several use-cases. The hub-workitems are often entry points to use-cases such as "find customer".

What I would like to add to the procedure is that adding a dynamic modeling aspect based on the use cases will help with explaining, doing and presenting the workitem analysis.

Dynamic modelling is done using UML activity diagrams or BPMN diagrams, both showing workflow and responsible parties using swimlanes. The main help of using these dynamic diagrams is to get a dynamic (run-time) view of the static use case diagrams to be able to see the lifespan of tasks.

Create business process or workflow diagrams of the use cases. Each process will have a defined start and end, and is a good candidate for becoming a first-level workitem with zero, one or more sub-workitems. The diagram swimlanes are good candidates for identifying sub-workitems of the first-level workitem. Use the process diagrams to identitfy the lifecycle management requirements for workitems in your composite application.

Don't create a CAB application with only one workitem that lives as long as its module, unless your use case is as simple as that.

SOA Canonical Cross-Domain Data Semantics

I have over the last year promoted the need for having a business process information model (BPIM) for semantic mediation and business process integration to enable SOA, not to forget enabling consumers such as mashups to do composition and consumption of the provided services, to deliver value to our users.

There has been some discussions about the need for an enterprise to have federated canonical domain models, but I think more and more people tends to believe that BPIM is an important SOA and BPM enabler. This blog post by Joe McKendrick quotes Hub Vandervoort, CTO of Progress Software, on some common SOA challenges:

He recommended three approaches to the problem, including "getting your transports aligned between business entities so that you can use eventing-oriented mechanisms to communicate across domains"; establishing SLA and security policies that ensure visibility; and establishing a common enterprise data model.

"You have to get your semantics aligned among the members," he said. "And that doesn't have to be a common vocabulary in its entirety, but certainly what we regard as the data in flight, those things that fly between domains and different working groups have to be highly normative."

The important part of Vandervoort's message is that you should not try to make a complete Enterprise Data Model, but rather focus on a common information model (CIM) for the "data in flight", i.e. the events, messages and documents of your business processes, when creating your information model.

Ontology for Business Processes, SOA and Information Models

I have for a long time promoted that you should apply a specific kind of information model (the business process information model - BPIM) when composing services into business processes in service oriented solutions. A BPIM models the messages for the business events that drive the business processes. The main goal of this information model is to enable coordinated loose-coupling (semantic mediation) between the services at the composition level.

There has been a lot of discussions on how the BPIM relates to SOA services and the common information model (CIM), and how the BPM process layer relates to the SOA process layer. In the following ontology I have tried to model the relationships between the documents and the domain objects, Shy Cohen’s service taxonomy & ontology, and Jean-Jacques Dubray’s SOA+BPM ontology. 

The service classification scheme has the "hierarchy" taxomomy form. It is not a scientific hierarchy nor a tree form, thus there is no "is using" or "is dependent on" relation between the services - it is just a classification scheme. There can be no dependency between services in SOA except for composite services realizing a business process by consuming the actual services. Taking a dependency on another service will break the central "services are autonomous" tenet of SOA.

Click to enlarge the ontology:

Note how there are processes at two levels in the model: the event-driven resource lifecycle processes for domain objects (bottom), and the human-driven business processes realized as composed services (top). These two levels are separate bounded contexts with well defined mappings between them: the business process domain and the resource lifecycle domain.

The BPIM is applied at the business process domain level (orchestrations / composites / sagas), while the services operate on the domain objects in the CIM to manage the lifecycle of these resources in accordance with the business events.

Note that the BPIM is related to, and is a subset of/reference to, the resource domain objects in the common information model (CIM). In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. This in like a mail order paper form that contains e.g. some customer data and references to product data. The word "subset" is central, e.g. the “customer has moved” action event has a BPIM document that contains the data pertinent to that specific business process event, not the complete schema of the customer domain object. The resource process service used to act on the “customer has moved” event does of course operate on the actual domain object, but this is invisible to the human workflow process.

The business process messages (action, query, notification) driving the composite services and the resulting business events are central artifacts when designing the services and the information model. I am a strong believer in designing a service-oriented solution by applying "EDA style" thinking to avoid missing out on the events and their documents due to traditional “invoke operations” SOA thinking, getting too much focus on business process flow.

An aspect not shown in the model is that the need for agility increases towards the top, while the cost of change increases towards the bottom. This is an important argument for having processes at two levels; it allows you to contain the frequent changes to the business processes (mashup style compositions) and composite services rather than having to constantly make changes to the services themselves, which could become very costly as the number of service subscribers increases over time. This is something that your business people should readily appreciate.

Design your system to have flexibility in the flow of the business processes and composite services. The diamonds of your business process are the business decisions, and this is where you should implement a business rules engine (BRE) mechanism. Process flow logic is not service business logic, avoid leaking domain logic into the orchestration layer at all cost.

Finally, the model show how claims relate to services at all levels. I have done this to show how a claims-based security model is a cross-cutting concern throughout a service-oriented solution. I strongly recommend designing in process claims right from the beginning, as this makes it easier to learn how access control relates to the business processes and solution domain.

Master Data Management (MDM) relates to the "Resources" in the above ontology. It is imperative that you apply a MDM strategy to your enterprise resource domain objects to avoid inconsistencies and multiple versions of the truth.

I hope this ontology makes my viewpoint on business process driven SOA clear to you. Feel free to comment on this model, and also check out the six figures in JJD’s ontology for models showing different perspectives of resource lifecycles & business events, BPEL, BPMN and human tasks.

See also this related post.

SOA, EDA, CEP, BPM

While I have been in Redmond the last three weeks learning my true color (I'm hyper green btw), there has been some interesting posts by Joe McKendrick on some topics that I've written about the last year:

• Convergence of SOA+EDA+CEP
• SOA and BPM - automation and humans
• Removing "complex" from CEP

Long time readers of my blog knows how I strongly advice applying EDA thinking when designing service-oriented solutions, but I do not foresee CEP to become a commodity for quite some time yet. However, it would be nice if designing systems around business events could provide better insight on how BPM relates to SOA in practical terms, along the ontology presented last december by Jean-Jacques Dubray.

Modeling Business Processes for SOA, don't forget the Information Model

Recently I have done a few architecture reviews at customers struggling to get their SOA efforts going. The common problem of these attempts on service-enabling legacy systems to support agile business processes have been - except from system designers not grasping that SOA is in fact a paradigm shift from invoking-operations composition thinking, and those using RPC over web-services - that they focus solely on business process management (BPM) modeling. From this model they try to identify services without having a taxonomy for classification of services (activity services, information services, composite services), struggling with service granularity and how to identify what goes into the message and data contracts.

The problem is that modeling your business processes typically misses out on the central building block of service-oriented solutions: the data and the semantics of the data as related to the business processes. This is typical for workflow models like the 'UML Activity Diagram'.

The data should represented by a Common Information Model (CIM). The CIM should be the starting point of your modeling efforts, in combination with Business Process Modeling Notation (BPMN) diagrams, as BPMN encompass process, events, messages and business documents. Based on the CIM, you must then design a business process information model (BPIM) for each business process domain. Then you could follow the SOA mantra coined by David Linthicum: data-service-process (much like the TDD red-green-refactor) and create mashups based on composite services. 

Last week Jean-Jacques Dubray posted an article about 'The Seven Fallacies of Business Process Execution'. The article contains a lot of good points about the 'case-tool grail' dreams about business people doing agile modeling in BPMN and then just generating systems that automates the business processes.

The really interesting part of the article is in 'Fallacy #5: Business Process Execution' where Jean-Jacques differentiates business objects (resources) and their lifecycle from the business processes that drives the lifecycle of the resources. The article explains how the data aspects are missing from BPMN even if it is the most important part of the business - just the problem I see so often.

Another point made in the article is that it is the resource lifecycle events that are important when trying to identify the services that your business process will depend on. The business events advance the business data through a state machine until the end of the business process. This is what Jack van Hoof, Nick Malik, myself and others have blogged a lot about this year: it is the business events and their message and data (documents) that are the key artefacts in your modeling efforts when creating a service-oriented architecture.

The article also shows how process service composition (orchestration of service execution and resource lifecycle management; BPEL) is different from user-centric business processes (human workflows and tasks; BPMN, WS-HumanTask, BPEL4People).

The article goes on to discuss how development is the glue that joins the different artifacts that constitutes a SOA. Developers are needed to actually implement the BPMN using the provided services. I strongly agree. Even if you could do BPMN to BPEL code generation, professional testing and QA in a development-staging-production environment is still a must. I will only go as far as to allow business people manage a set of goverened business rules, never directly implement any process (i.e. declaratively configure the diamond shapes in flow charts). Read more about business rules/decision service at James Taylor's blog.

Finally, the need for a durable 'composite service container' for the orchestrations is also something that is often disputed during my SOA reviews. "Command-and-control" developers rarely consider how the availability of autonomous services affects the fault-tolerance in the overall SOA solution. The question I always ask is: "What is the availability of a business process that composes five services that each have 99% availability? Is it 99%?" The answer is of course less: 95%. This is where a container for providing long running + reliable processes and 'resource' state machines comes in handy.

I recommend that you read and, most importantly, understand the concepts and the architecture blueprint presented in the article. I have to give Jean-Jacques a big credit for the good figures he has made to illustrate the taxonomy and ontology presented in the article.

Enterprise Architects, Top-Down SOA

I had hoped that the age of non-programming architects was over, but yesterday I was let down. I attended the monthly meeting at the "Practical SOA forum" at SINTEF where an experienced architect ("20+ years of IT experience") presentet his groups work at a large public service company in Norway. The architect explained the EA modelling work going on to build a service-oriented enterprise, showed the EA tool and portal, documented SOA guidelines, refinement of the model into BPMN (but not into BPEL), etc.

All systematic and professional, but then he went on to complain about having to spend too much time with the developers and hired consultants to help them actually understand and implement the model according to the guidelines and policies - which prevented him from playing (my word) with the model to refine it and expand it into new areas of the business...

No wonder many of the pure top-down approach SOA projects are regarded as failures.

The Information Model for Composite Services is not about Solving Service Versioning

The business process information model (BPIM) is not intended for solving versioning issues for a specific service, that can be solved using message duck-typing or consumer-driven contracts. The BPIM is for mediating semantics between independent and autonomous services, allowing the services to be composed into biz processes and still be able to evolve each service separately. The autonomous SOA tenet should be used to achieve shared services and interchangeagility rather than focusing solely on reusability. A service is not autonomous before you can use the scissors on parts of a business process and replace the service by e.g. outsourcing it.

Indeed, service versioning is important for the evolvability of the service itself, but the BPIM is about the agility of the business processes in your service-oriented solution.

Do not confuse the BPIM with the EAI CDM, they are similar concepts at different architectual layers. The BPIM will contain a subset of the common information model (CIM), but is rather focused on the data+business documents and semantics of the business events and processes that the information model encompasses. Design the BPIM based on the CIM, ensuring that the model is canonical for each business process domain. The BPIM is targeted at the service composition layer, not at the data access services layer.

Related reading: SOA doesn’t need a Common Information Model