As I've written about before, there is a lot of confusion and different interpretations of what a "Common Information Model" (CIM) is. The most common reification of CIM is to imply that it is the same concept as in chapter 4 "XML: The Foundation for Business Data Integration" in David Chappell's seminal book Enterprise Service Bus from 2004. Dave describes the need for having a common XML data format for "expressing data in messages as it flows through an enterprise across the ESB", but never uses the term CIM or describe any modeling approach. This common format will naturally comprise any kind of XML data used in the service bus, including messages, business entities and event data.
The CIM concept in relation to SOA was described by Mike Rosen and Eric Roch in 2006: it defines a common representation of business entity objects that provides a canonical format and unified semantics for a business domain.
The above figure is from Mike Rosen's article Business Architecture and SOA. As can be seen in the figure, CIM comprises a shared information model that is the basis for the documents that flows through the business processes built using the capabilities provided by the services.
Mike explains the role of CIM like this:
[The SOA Business Model] has to manage the sharing of services and information across processes. In other words, it needs to eliminate redundancy, overlap, and gaps between services so that each business capability is implemented once, by the organizational unit that is responsible for that capability. And that those services are used by all the different processes needing those capabilities. In addition, all of the information shared between services must be identified in the common information model. In other words, all services that are related to the same business concepts must use the same information to describe those concepts. Finally, the SOA Business Model must ensure that all of the information passed into and out of the business services (mostly in the form of documents) is defined in the [shared semantic] information model.
I hope this clarifies the definition of what a CIM is: it is a model that comprises shared information entities such as "Customer", but not actual messages like "CustomerHasMoved" defined in the business process information model (BPIM). The message types such as "AddressChange" (document) are part of this logical model as projections of the shared entities.
The "Canonical Schemas" used in messages are part of the service model's standardized service contracts. Note that these schemas are generated from, but are not part of CIM. Generating the canonical schemas from CIM makes enforcing "Schema Centralization" unnecessary, as they already share a common basis. This helps alleviate the negative effects of a one true schema approach; e.g. the versioning ripple effect through the canonical data model that affects all related service contracts - even if not intended.
Do not confuse CIM with the centralized SOA logical data model (LDM) approach. The services speak "CIM" natively. Service messages are not transformed to/from CIM by a message broker as in the LDM schema bus approach. Still, using the message broker approach is handy to include non-conformant services in your SOA solutions. Enforcing a LDM for your information will cause the same problems as the traditional EAI hub-and-spoke canonical data model (CDM).
I recommend reading chapter 5 "Service Context and Common Semantics" in the "Applied SOA: Service-Oriented Architecture and Design Strategies" book by Michael Rosen et al. Figure 5-12, 5-14 and 5-15 in the book shows how a grapical projection DSL for documents (message types) might look.
InfoQ has published an article about SOA Message Type Architecture by Jean-Jacques Dubray. The article shows how to model 'message type' artifacts based on an enterprise common data model using a DSL, and also outlines how the modeled artifacts can be used to generate XML schemas for use in your service contracts. The message type DSL is not for modeling messages, it is just for modeling the types used as message payloads. In WCF terms, a message type is a [DataContract].
Note that even if the message type model contains a set of standardized verbs, the model does not cover business process aspects such as flow and actions. You will still need to analyze the business processes that pass those message types around to model the action, query and notification events that drive the processes. I've written several times about creating such a model that comprise both the business events and the message types, a business process information model (BPIM). I like JJ's approach; I just think that we need to model also the business capabilities and interactions that utilize the message types to get a complete set of artifacts for service contracts.
One difference is that I prefer using a common information model (CIM) as the basis for modeling the message types, rather than an enterprise data model (EDM). It is a lot of effort to create an EDM that covers all information in all systems-of-record in a company; and the moment you have completed the all-encompassing model, your CxO will inform you that parts of the business have been outsourced or that a new business will have to be incorporated, or even just that the CRM system is to be replaced. Change is the only constant. Thus I prefer starting small by creating a CIM that covers only the business entities comprised by the business processes that are about to be service-oriented. As there will be multiple resource domains in your architecture, there will be multiple CIM models as your SOA grows. Federate these domain models by creating context maps for cross-domain capabilities and logic only.
Create a CIM according to the Domain-Driven Design (DDD) principles and use the domain entity objects when implementing the business logic underlying your services. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain. As the BPIM is a projection of the CIM, the service interface messages will have an unambiguous mapping to/from the underlying domain resources (entities and aggregates). Mapping data between the message types and objects will be required in the service implementation (provider container).
The 'message type' artifacts should also be partitioned into bounded contexts according to business area as in Domain-Driven Design; where each service domain comprise a set of cohesive services based on the same underlying CIM, delivering an BPIM for the specific business domain. Having clear domain boundaries make it easier to analyze, model and design, and version the artifacts - it also aids the discoverability by the consumers by providing a clear business context for the provided services.
Your SOA solution will, as it grows, encompass multiple service domains each with its own BPIM. Composite services that compose business capabilities across two or more domains will require mediation between the message models. Even if "transformation avoidance" is considered a SOA best practice, it is unrealistic that you will be able to avoid mediation completely in your service bus (composition container). In addition, you cannot expect to enforce your model upon the extended enterprise, think of e.g. third-party and outsourced capabilities.
The message types exposed by a service domain implicitly become the "canonical schema". They enable better service discoverability, reuse and composition as they all share the same underlying data model - which also ensures that the message payloads have common semantics within the service domain. In combination with business functions, the schemas provide a complete standardized service contract.
As you have seen, the BPIM and CIM approach fits well with DDD. It also fits well with the middle-out SOA approach - including the recommendation to start small and think big picture, not big bang.
As part of your SOA governance efforts, it is not sufficient to handle just the versioning of the services and the data schemas used by the services. You must also govern the semantics of the data to enable service composability. Composite services are not possible without semantic data integration.
There are two information models in play when composing services - just as there are two different, but related, process models involved in service-oriented solutions. One is the well known common information model (CIM), the other is the related business event message model used to enable semantic business process composition.
The process composition information should be partitioned according to the business process domains to allow for the business process information model (BPIM) to evolve independent of each other. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain. Your BPIM must be federated even if evolved and versioned separately, to enable semantic data mediation and semantic business process integration.
The BPIM contains metadata that models business event messages as projections of CIM data entities. A business process message typically contains a subset of one or more domain objects. E.g. in an order handling system, the real-world event "customer submits order" is a projection of the customer, address, credit and product entities from the CIM. Think classical paper-based mail order schemas or the Excel schemas you use to get your travel expences reimbursed, or rather the form used to apply for a vacation.
Creating an enterprise canonical data model might be feasible, but federated domain models are recommended. You would anyway need to mediate semantics between your system and third-party services that you involve, or on parts of your processes that you chose to outsource.Federated models are required for B2x integrations as you cannot easily enforce your model on the outside world. Using a CDM might work in a centrally controlled EAI hub, but most likely not across departments, organizational and partner boundaries in an extended enterprise.
Joe McKendrick has a post about the need for doing data governance today:
If data governance is inadequate — information is outdated, out of sync, duplicated, or plain inaccurate — SOA-enabled services and applications will be delivering garbage. That’s a formula for SOA disaster.
He links to the XML to the rescue: Data governance in SOA article by Ed Tittel, which describes how having a common model and doing data governance is imperative for SOA success. The article conveys the same concepts as chapter 4 'XML: The Foundation for Business Data Integration' in David Chappell's seminal book Enterprise Service Bus. David describes the need for having a common XML data format for "expressing data in messages as it flows through an enterprise across the ESB". This common XML format is a specialization for the common data model (CDM) pattern from the classic book Enterprise Integration Patterns by Gregor Hohpe/Bobby Woolf.
I've written many times about the importance for having a common information model (CIM) for the domain objects (business entities) that your data services operates on, and the need for applying master data management (MDM) on those data. David Linthicum and I agreed on this last year and still do.
What I've written even more often about, and what most canonical models do not encompass; is that the data expressed in the messages conveying business events is not simply CIM entities, but are rather projections of one or more of the domain objects. In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands.
E.g. an insurance clerk triggers a business process by registering the real world event "customer has moved" using a projection of customer and address data, not the complete customer aggregate. This action event message triggers a recalculation of the insurance premium due to the relocation of the customer, in addition to updating the customer record - i.e. an underlying business process is executed.
In a traditional forms-over-data application, the whole customer aggregate would be used - but this style would not make for services that can be easily composed to support business process management efforts. The business event data requirements must be simple enough to make composite services possible, including semantic data mediation and integration. In addition, process tailored data make it easier for any involved human operator that have to enter or act on the data. After all, BPM is not all about automation, and human tasks will be central in driving the processes for quite a while.
We need to move to an information model that allows for semantic business process integration, not just semantic business data integration. That is why a business process information model (BPIM) is required, modeling the messages for the action, query and notification events that drive the business processes. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain.
The BPIM is based on the CIM as each service domain model is just a projection of the common data model, and the projection metadata should be goverened as any other SOA artifact.
I've written quite a lot about having a model for the information needed for semantic business process composition in service-oriented solutions, two of the most read and commented are SOA Canonical "Data" Model and Ontology for Business Processes, SOA and Information Models.
The process composition information model is focused on the business events, messages and documents and the semantics of the information that the model encompasses. The model is targeted at the orchestrations level in the ontology, not at the services level; as shown in the figure (click to enlarge):
The service classification scheme has the "hierarchy" taxomomy form. It is not a scientific hierarchy nor a tree form, thus there is no "is using" or "is dependent on" relation between the services - it is just a classification scheme. There can be no dependency between services in SOA except for composite services realizing a business process by consuming the actual services. Taking a dependency on another service will break the central "services are autonomous" tenet of SOA.
A note on the splitting of the service categories: this is a pure metadata ontology based on the taxonomy for categorizing services, it is not a split based on technolog or, teams, and surely not on layers. A service should not call on other services, making a service dependent on another service will lead to repeating the failure of DCOM/CORBA in distributed systems; using web-services and calling it SOA solves nothing. Use the active service pattern or EDA to ensure that your services are truly autonomous and independent of the availability of other services. Isolate service dependency to the composite services, or even better to the consumer mashups, only.
The "business process information model" (BPIM) is what models the business events and documents in the composite services level (orchestration / conversations / sagas). I like to use a domain model as it by definition focuses on the concepts of the problem domain (the business process information artifacts) rather than just the data of the domain. The information model must comprise the different message types for the action, query and notification events that drive the business processes. The model must be canonical for its domain to ensure that the format and semantics of the model is consitent within the domain.
There is some confusion on the difference between the process composition BPIM and the EAI "canonical data model" (CDM), plus the similar concept Common Information Model (CIM). As can be seen from the comments on the two postings and some blog reactions like SOA doesn’t need a Common Information Model, this has made it harder to convey the difference between the process composition information model and the more data-oriented models.
A CIM should be used for the resources data model - which the BPIM is related to. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain. The data in the event document for the real-life action "CustomerHasMoved" is not a complete "Customer" resource entity, it is a message including minimal "AddressChange" data. This data will typically contain both actual event data combined with reference keys to the resources such as customers and addresses in the different service domains that the composition comprises.
The message data is of course associated with the CIM of the underlying resource services. The BPIM message types are projections of CIM objects; in fact, they are projected compositions of the referenced resources - not just simple compositions of resource objects. These projections must be kept as small as possible, preferably just the resource keys, in addition to event details such as a booking reference number. Keeping the event document data model small has the nice side-effect of making the schema versioning a bit simpler.
The CIM should be the starting point of your modeling efforts, as the documents in the process compositions is based on the service domains. Having Business Process Modeling Notation (BPMN) diagrams can help in modeling, as BPMN encompass process, events, messages and business documents, covering actions, queries and notifications. Other process modeling diagrams can be used, just ensure that the events, messages and data are captured in the model.
It is still important to recognize that having a "one true schema" for process composition or SOA in general is not a recommended practice, hence the term "domain" in the naming. Rather than trying to enforce a common model across all service domains, federated models should be used. When composing services across the domains, mediation and transformation of messages will be required - here a service bus with such capabilities would come in handy.
A thing to recgonize is that the overall model incorporates several "flows": process flows, event flows, message flows, and data state flows (resource lifecycles). Typically, only the process flow is clearly shown in diagrams, while e.g. message flow is only represented by receive and send ports. See this interview with Gregor Hohpe on Conversation Patterns about how more than just the process need to be modeled.
For an excellent primer on architectural aspects of SOA and the proliferation of acronyms, I recommend reading Architecture requirements for Service-Oriented Business Applications by Johan den Haan. Note that he does the classical simplification of focusing on business processes without including a BPIM; it is important to separate the process composition model from the resource model, just as business processes are separated from resource lifecycle processes. The business process events, documents and messages are not just collaterals of the BPMN modeling process, they are important artifacts for semantic mediation in composite services.
[Business Process Integration: "e-Business: organizational and technical foundations", Papazoglou, M. P. & Ribbers, P. (2006)]
I have over the last year promoted the need for having a business process information model (BPIM) for semantic mediation and business process integration to enable SOA, not to forget enabling consumers such as mashups to do composition and consumption of the provided services, to deliver value to our users.
There has been some discussions about the need for an enterprise to have federated canonical domain models, but I think more and more people tends to believe that BPIM is an important SOA and BPM enabler. This blog post by Joe McKendrick quotes Hub Vandervoort, CTO of Progress Software, on some common SOA challenges:
He recommended three approaches to the problem, including "getting your transports aligned between business entities so that you can use eventing-oriented mechanisms to communicate across domains"; establishing SLA and security policies that ensure visibility; and establishing a common enterprise data model.
"You have to get your semantics aligned among the members," he said. "And that doesn't have to be a common vocabulary in its entirety, but certainly what we regard as the data in flight, those things that fly between domains and different working groups have to be highly normative."
The important part of Vandervoort's message is that you should not try to make a complete Enterprise Data Model, but rather focus on a common information model (CIM) for the "data in flight", i.e. the events, messages and documents of your business processes, when creating your information model.
I have for a long time promoted that you should apply a specific kind of information model (the business process information model - BPIM) when composing services into business processes in service oriented solutions. A BPIM models the messages for the business events that drive the business processes. The main goal of this information model is to enable coordinated loose-coupling (semantic mediation) between the services at the composition level.
There has been a lot of discussions on how the BPIM relates to SOA services and the common information model (CIM), and how the BPM process layer relates to the SOA process layer. In the following ontology I have tried to model the relationships between the documents and the domain objects, Shy Cohen’s service taxonomy & ontology, and Jean-Jacques Dubray’s SOA+BPM ontology.
The service classification scheme has the "hierarchy" taxomomy form. It is not a scientific hierarchy nor a tree form, thus there is no "is using" or "is dependent on" relation between the services - it is just a classification scheme. There can be no dependency between services in SOA except for composite services realizing a business process by consuming the actual services. Taking a dependency on another service will break the central "services are autonomous" tenet of SOA.
Click to enlarge the ontology:
Note how there are processes at two levels in the model: the event-driven resource lifecycle processes for domain objects (bottom), and the human-driven business processes realized as composed services (top). These two levels are separate bounded contexts with well defined mappings between them: the business process domain and the resource lifecycle domain.
The BPIM is applied at the business process domain level (orchestrations / composites / sagas), while the services operate on the domain objects in the CIM to manage the lifecycle of these resources in accordance with the business events.
Note that the BPIM is related to, and is a subset of/reference to, the resource domain objects in the common information model (CIM). In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. This in like a mail order paper form that contains e.g. some customer data and references to product data. The word "subset" is central, e.g. the “customer has moved” action event has a BPIM document that contains the data pertinent to that specific business process event, not the complete schema of the customer domain object. The resource process service used to act on the “customer has moved” event does of course operate on the actual domain object, but this is invisible to the human workflow process.
The business process messages (action, query, notification) driving the composite services and the resulting business events are central artifacts when designing the services and the information model. I am a strong believer in designing a service-oriented solution by applying "EDA style" thinking to avoid missing out on the events and their documents due to traditional “invoke operations” SOA thinking, getting too much focus on business process flow.
An aspect not shown in the model is that the need for agility increases towards the top, while the cost of change increases towards the bottom. This is an important argument for having processes at two levels; it allows you to contain the frequent changes to the business processes (mashup style compositions) and composite services rather than having to constantly make changes to the services themselves, which could become very costly as the number of service subscribers increases over time. This is something that your business people should readily appreciate.
Design your system to have flexibility in the flow of the business processes and composite services. The diamonds of your business process are the business decisions, and this is where you should implement a business rules engine (BRE) mechanism. Process flow logic is not service business logic, avoid leaking domain logic into the orchestration layer at all cost.
Finally, the model show how claims relate to services at all levels. I have done this to show how a claims-based security model is a cross-cutting concern throughout a service-oriented solution. I strongly recommend designing in process claims right from the beginning, as this makes it easier to learn how access control relates to the business processes and solution domain.
Master Data Management (MDM) relates to the "Resources" in the above ontology. It is imperative that you apply a MDM strategy to your enterprise resource domain objects to avoid inconsistencies and multiple versions of the truth.
I hope this ontology makes my viewpoint on business process driven SOA clear to you. Feel free to comment on this model, and also check out the six figures in JJD’s ontology for models showing different perspectives of resource lifecycles & business events, BPEL, BPMN and human tasks.
See also this related post.
Recently I have done a few architecture reviews at customers struggling to get their SOA efforts going. The common problem of these attempts on service-enabling legacy systems to support agile business processes have been - except from system designers not grasping that SOA is in fact a paradigm shift from invoking-operations composition thinking, and those using RPC over web-services - that they focus solely on business process management (BPM) modeling. From this model they try to identify services without having a taxonomy for classification of services (activity services, information services, composite services), struggling with service granularity and how to identify what goes into the message and data contracts.
The problem is that modeling your business processes typically misses out on the central building block of service-oriented solutions: the data and the semantics of the data as related to the business processes. This is typical for workflow models like the 'UML Activity Diagram'.
The data should represented by a Common Information Model (CIM). The CIM should be the starting point of your modeling efforts, in combination with Business Process Modeling Notation (BPMN) diagrams, as BPMN encompass process, events, messages and business documents. Based on the CIM, you must then design a business process information model (BPIM) for each business process domain. Then you could follow the SOA mantra coined by David Linthicum: data-service-process (much like the TDD red-green-refactor) and create mashups based on composite services.
Last week Jean-Jacques Dubray posted an article about 'The Seven Fallacies of Business Process Execution'. The article contains a lot of good points about the 'case-tool grail' dreams about business people doing agile modeling in BPMN and then just generating systems that automates the business processes.
The really interesting part of the article is in 'Fallacy #5: Business Process Execution' where Jean-Jacques differentiates business objects (resources) and their lifecycle from the business processes that drives the lifecycle of the resources. The article explains how the data aspects are missing from BPMN even if it is the most important part of the business - just the problem I see so often.
Another point made in the article is that it is the resource lifecycle events that are important when trying to identify the services that your business process will depend on. The business events advance the business data through a state machine until the end of the business process. This is what Jack van Hoof, Nick Malik, myself and others have blogged a lot about this year: it is the business events and their message and data (documents) that are the key artefacts in your modeling efforts when creating a service-oriented architecture.
The article also shows how process service composition (orchestration of service execution and resource lifecycle management; BPEL) is different from user-centric business processes (human workflows and tasks; BPMN, WS-HumanTask, BPEL4People).
The article goes on to discuss how development is the glue that joins the different artifacts that constitutes a SOA. Developers are needed to actually implement the BPMN using the provided services. I strongly agree. Even if you could do BPMN to BPEL code generation, professional testing and QA in a development-staging-production environment is still a must. I will only go as far as to allow business people manage a set of goverened business rules, never directly implement any process (i.e. declaratively configure the diamond shapes in flow charts). Read more about business rules/decision service at James Taylor's blog.
Finally, the need for a durable 'composite service container' for the orchestrations is also something that is often disputed during my SOA reviews. "Command-and-control" developers rarely consider how the availability of autonomous services affects the fault-tolerance in the overall SOA solution. The question I always ask is: "What is the availability of a business process that composes five services that each have 99% availability? Is it 99%?" The answer is of course less: 95%. This is where a container for providing long running + reliable processes and 'resource' state machines comes in handy.
I recommend that you read and, most importantly, understand the concepts and the architecture blueprint presented in the article. I have to give Jean-Jacques a big credit for the good figures he has made to illustrate the taxonomy and ontology presented in the article.
In my quite popular post SOA: Canonical "Data" Model (picked up by Steve Jones, David Linthicum, Jean-Jacques Dubray and others), I used two analogies for the semantic mediation needed when composing different services into business processes: Esperanto and the Babel fish. This post is about how the two approaches differ in their use of the business process information model (BPIM). But first a short recap of the two terms:
Esperanto is a common world language; when you learn it, you can speak with any other person in the world that also known Esperanto. Everyone does of course still have their native language. See how similar this is to services that interact using a common information model (CIM) to express themselves, while every system still use their own domain model inside.
A Babel fish is a small fish that you put in your ear, which is capable of simultaneously translating any known language in the universe to yours, while adapting semantics between galaxies. See how similar this is to services that interact using a common logical data model (LDM) while every system still uses their own domain model to express themselves.
Both these approaches to semantic mediation in service-oriented solutions do use a business process information model (BPIM). They differ only in how they implement the transformation between formats: inside the services or outside the services.
The CIM approach to mediation is to use the BPIM directly in the service contracts and apply the transformation within the provided services [Hohpe/Woolf: Messaging Mapper pattern (477)]. This is a simple and viable approach; the services are self-contained, but require a bit more to implement and test due to the extra mediation requirements. The services will also need to be changed and tested when the BPIM changes. Dan North suggests this approach in his 'A Low-Tech Approach to Understanding SOA' article. This is still the most common approach, especially for those without an ESB.
There is one minor challenge with the CIM approach: chances are that the Esperanto of an alien galaxy will differ from your Esperanto. Thus, when outsourcing an activity service such as 'credit check', you need to adapt to the service provider's business concepts. So even if you use a CIM for all your enterprise services, you will still need to implement some sort of mediation to do the context mapping between the two federated business domains.
The huge advantage of the CIM approach is that it doesn't require any ESB-style intermediary. The services speak "CIM" natively and thus no transformation to/from the common data model is needed. Note that the service implementation must still map between the CIM format and its internal data formats.
The LDM approach involves using an orchestration mechanism to compose business processes from the set of domain specific services. The services just provide their functions independent of who the requesters (consumers) are and how they try to interact with the services. Think of your SOA solution as having Event Driven Architecture (EDA) to free yourself from "invoking operations" and think of business events and messages instead. It is the task of the Babel fish to mediate and transform the messages sent between consumers and providers [Hohpe/Woolf: Message Translator pattern (87)]. The intermediary can be just the service composition mechanism, but an Service Bus is a more robust and flexible mediation and transformation mechanism.
The LDM approach makes the services themselves simpler and easier to implement and test as they now have just a single responsibility. The services need not change when the BPIM changes, that complexity has been shifted to the intermediary. Implementing semantic mediation outside the services is proposed by Jack van Hoof, David Chappell (Pope of ESB), Bobby Woolf and many others.
Dan North has expanded on Arjen Poutsma's SOA 'holiday request form' analogy in his post 'A Low-Tech Approach to Understanding SOA'. Applying the paper form analogy to design the interactions and messages of your business processes is an approach that I really recommend.
What got my attention was the recommendation at the end of the post in the 'Avoid a universal domain model' section: Do not introduce an “enterprise information architecture” / “universal data dictionary” trying to force everyone to use the same domain model. Instead, introduce business concepts. This is effectively the higher-level, ubiquitous language that ties together all of the finer-grained domain models behind each service. The services use the enterprise-level business concepts when interacting, which decouples the service consumer from the service provider and allows them to evolve independently.
This is spot on with my advice for making a business process business process information model (BPIM) and avoid enforcing a "enterprise data model" or "canonical schemas" across your services. Not only does BPIM allow for loosely-coupled, evolvable services; it also allows for the services to be semantic covenants which is important for service composition.
InfoQ has posted an interview with Jim Webber of ThoughtWorks about "Guerilla SOA", where he talks about how to design scalable SOA solutions that are evolvable and allow innovation in the different services that make up the solution.
The advice he offers covers topics that I have written about in my posts these last months:
- Share business messages owned by the business people (business process information model)
- SOA has no operations, only business messages delivered to letter boxes - services exchange messages to fulfill business processes
- Services cannot be invoked, they just receive event+payload messages and decide if and how to process it: "Could you possibly have a look at this message and maybe if it suits you do some processing on it" (semantic covenant: the service is always right)
- Business processes are workflows; you need to model long-lived conversations beyond request/response (SSDL, check out the Soya WCF SSDL toolkit at SourceForge)
This architectual style is coined MEST, as it resembles REST: it is based on message exchange through a uniform interface - the letterbox. As you will see, MEST is a decoupled scalable design pattern along the lines of EDA.
Listen to the interview and read the transcript: Jim Webber on "Guerilla SOA"
The business process information model (BPIM) is not intended for solving versioning issues for a specific service, that can be solved using message duck-typing or consumer-driven contracts. The BPIM is for mediating semantics between independent and autonomous services, allowing the services to be composed into biz processes and still be able to evolve each service separately. The autonomous SOA tenet should be used to achieve shared services and interchangeagility rather than focusing solely on reusability. A service is not autonomous before you can use the scissors on parts of a business process and replace the service by e.g. outsourcing it.
Indeed, service versioning is important for the evolvability of the service itself, but the BPIM is about the agility of the business processes in your service-oriented solution.
Do not confuse the BPIM with the EAI CDM, they are similar concepts at different architectual layers. The BPIM will contain a subset of the common information model (CIM), but is rather focused on the data+business documents and semantics of the business events and processes that the information model encompasses. Design the BPIM based on the CIM, ensuring that the model is canonical for each business process domain. The BPIM is targeted at the service composition layer, not at the data access services layer.
Related reading: SOA doesn’t need a Common Information Model
A business process information model (BPIM) is a model that is focused on the business events and the data+documents+messages needed as part of orchestrating the events into business processes by composing services. A central part of the creating the model is to keep the model as small as possible (like a "least common denominator" of the data), only passing the state of the process as part of the events. The data and business documents represent the state of an instance of a business process (what Jack van Hoof calls a dossier) and is all a business process step should need to know to be able to perform its actions.
It is important to recognize that the BPIM does not encompass all information that some part of a business process might ever want or need to know, it just needs to contain enough reference data. Neither does the BPIM have to contain data about business entites across all applications in your enterprise, i.e. the BPIM "customer" does not need to model the union of all the different customer fields from your CRM and ERP systems. This is why BPIM is different from a common information model (CIM) or the EAI Common Data Model [Hohpe/Woolf CDM pattern (355)].
The business process that need to know more about the referenced data of the dossier, will have to use services to look it up, as part of the service composition (orchestration). This is what is called the "pull model" in the Observer pattern [GoF] [Hohpe/Woolf Event Message (151)]. The business process can also augment the process state by calling services and adding data to the dossier. Thus, the mantra of making a slender model is "Ask For More".
Keeping the information model small is very important wrt to maintenance of the model, afterall the main purpose of the model is to mediate semantics in the service bus and its orchestrations of services. The number of mappings between the different elements (messages+dossiers and services) of a model will grow and change over time, thus having small model elements are crucial to keep the model maintainable. In addition, split your model into separate domains/bounded contexts and make context maps, as this makes the total number of mappings smaller.
The key maintenance advice is to keep the model elements small (smaller mappings) and the number of mappings between model elements to a minimum (less mappings).
A SOA solution is typically implemented by composition of services provided by different systems. The solution might comprise a mix of of in-house services, 3rd party and outsourced services. In addition, the solution might also involve cross-enterprise service compositions.
To be able to efficiently compose the services, a model that ensures shared sematics is needed, and in my last post I described the business process information model (BPIM). David Chappell calls this approach semantic data integration. BPIM is a similar concept to the EAI canonical data model [Hohpe/Woolf CDM pattern (355)], but it is a model with a slightly different purpose at a different architectual layer and it is not just about the data. The BPIM is about the events, messages and data passed between services, not about having a unified superset of the entities within an enterprise.
The BPIM allows for composition of services without having to know and comply with the model of the underlying service logic, all you need to know is the message types. This is a big advantage, and allows for the consumers to be isolated from the details of the consumed services. E.g. all a service consumer need to know is the "AddressChange" data of the "CustomerHasMoved" event, not the complete data schema of the "Customer" service.
BPIM is closely related to the Common Information Model (CIM) concept. Note that the BPIM message types are projected compositions of the referenced CIM resources - not just simple compositions of resource objects. In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain.
The term 'domain' is borrowed from Domain Driven Design (DDD). Domain-driven design is not a technology or a methodology. It is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains. Read stories about how DDD can be applied to a diverse set of architectual problems: Practitioner Experience Reports.
Focusing on the bounded context for modeling the flow, events, messages, data and semantics involved in implementing the core business processes of the domain, should make it easier to come up with a working model. Note again that the information model is about more than just the data. Hence the name business process information model. Arvindra Sehmi and Beat Schwegler used the same term in Service-Oriented Modeling for Connected Systems, an article that also provides details about creating a service model.
Partner/ 3rd party/ outsourced services are not part of the core business domain according to DDD. If they were core processes in your business, how come they are so general that they can be outsourced or bought? Core processes are those that make your business unique and give you a competitive edge. DDD dictates using translators or an "anti-corruption layer" against services/systems that are not within the domain.
Having a canonical schema model at the service layer might be feasible within an enterprise, but should be avoided as this will cause very tight coupling to the One True Schema. Making every service depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. If agility at the service layer is less important for you, then such a service straight-jacket might initially feel good. Trying to make a enterprise data model (EDM) is not a good idea for the same reasons. Steve Jones has a good post about canonical form issues and how you cannot enforce your model upon the world: Single Canonical Form - not for SOA
Federated Business Process Information Models
DDD recommends splitting big, diverse and complex solutions into several bounded contexts. Set explicit boundaries based on e.g. orgranizational units and application usage. A natural boundary in SOA is partner/ 3rd party/ outsourced services. Each set of services that is not under your control and that you cannot enforce your information model upon, is a separate bounded context (domain model).
Note that also within your enterprise service model there will be several other bounded contexts for different domains, each with its own information model that is canonical per domain.
You should identify each model in play on the project and make a context map. A context map describes the points of contact between the domain models, in addition to outlining explicit translation for any communication between the models.
This figure show how a context map is used to show how two domains relate to each other:
The figure and the definitions of "bounded context" and "context map" is taken from the 'Strategic Design' chapter of the book "DOMAIN-DRIVEN DESIGN" by Eric Evans [Addison-Wesley, 2004].
Note how not all elements of a domain model needs to be mapped to other models. Only the interconnected parts needs to have a translation map. E.g. the credit check process is provided by a 3rd party, thus it exists in a separate external information model. Your BPIM needs to have a translation map to the other service to be able to invoke it. Note how similar this is to the purpose of the BPIM itself: translating between business process compositions and the underlying, composed services. Thus, the context map is the basis for modelling a set of Federated Business Process Information Models. A federated BPIM system map shows the integration of multiple enterprise service models, avoiding the pitfall of designing a single canonical data model across a set of different domains.
An important topic when designing service oriented systems is how to enable different services to share semantics to be able to be composed into working solutions. Jack van Hoof has written a good article about this: How to mediate semantics in an EDA. A few weeks later, Nick Malik posted another good read about this topic: Canonical Model, Canonical Schema, and Event Driven SOA. Read Jack's post first.
They both talk about using a canonical data model (CDM) as the Esperanto / Babel fish to map between the format and semantics of the disparate systems taking part in a SOA solution. Note that CDM is not about having a common data model (EAI CDM) or a shared database across all systems in an enterprise, don't get fooled by the "data" in the term 'canonical data model'. CDM is about not making everybody have to speak English, but rather having CDM translators for each native system.
Btw, Gregor Hohpe sometimes use the term 'canonical domain model' on his blog, while using the term 'canonical data model' in the book "Enterprise Integration Patterns" [Hohpe/Woolf CDM pattern (355)]. I think it is better to talk about the business domain rather than about "data", as this help focusing on the business processes rather than databases and other technology. You'd be surprised how many biz people concern themselves with how the data model looks - maybe a leftover from the client-server days, to show that they know what an ER-diagram is? Focus on designing a business process information model (BPIM) for each business process domain.
Trying to enforce a One True Schema across your services (everyone has to speak English) is not a viable path, and it is also a recipe for future maintenance hell. Making every service contract depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. A simple change to e.g. the order entity will cause a ripple effect through all referring services. This is where the business process information model comes into play, it allows you to version and evolve the services independently of each other.
The Canonical "Data" Model concept is also sometimes referred to as a Common Information Model (CIM). Both the business process information model (BPIM) and the data focused CDM/CIM models has the same goal: mediation og semantics. However, they are not the same as the two other models are both variations of the common data model approach. The business process information model is about semantic business process integration, not just only semantic data integration.
This post is about service semantics and how to be “liberal in what you accept” in your service-oriented architecture implementation. Let me start by defining that in this post the term "service" is used for a specific business capability that can be composed and reused in different business processes. I.e. the idiom “service” used here is representing a building block business capability. This is not to say that a service equals a single business process step, it just makes it easier to talk about the topic of this post (which is not BPM): semantic covenants.
A service has a defined contract, but also some implied semantic. The semantic is hard to express through the contract, and the business compositions that utilizes (consumes) the operation can only assume that the service is a covenant, i.e. that the service will do B when provided with input A. Having semantic coupling is just another type of coupling that prevents services to be truly loosely-coupled and that will cause a breaking ripple effect through all consumers of the service when the inevitable thing happens: change.
There is some thinking in the SOA community that to avoid this ripple effect caused by a service contract or semantic change, it is up to the service to do the right thing independent of the information provided by the sender of the message. That is, that the service should behave as if the consumer is always right, and adapt its semantic to suit the sender.
A service can do no such thing, as it is an actual representation of a specific business capability. The capability does what it does, i.e. a credit check is just that – a credit check. It must be up to the business composition of the services to interpret the real-life events that have occurred and their context, and then invoke the right set of operations. Note that the service should still be liberal in what it accepts (message duck-typing), as it is the business event that is important, not the format of the event message.
It is easier to recognize that the service is always right when you think of messages as representing business events and apply EDA thinking such as publish/subscribe instead of a habitual SOA command and control pattern. A service will then subscribe to business events that must trigger the business capability, but the service need not know who the publisher is. Neither need the publisher know who is listening for the business process state changed events (event topics). Thinking in service “consumers” is very connected to having a request/reply, command and control style SOA.
To simplify the “service is always right” notion, let us call the publisher a “consumer”. The consumer can send any message (business process event, context and state; mail-order analogy) it likes, as it is the publisher. However, as the consumer does not know who the subscribers are, it cannot know or depend on the services having specific semantic. The service cannot depend on who the publisher is, it just accepts that a business event happened that it must process. A business event message never specifies any handling operations, the logic to decide how to handle the event is part of the mechanism that routes messages to services. This ensures that the autonomous and loose coupling tenets of SOA are fulfilled.
When you apply EDA style thinking to service orientation, you’ll see that the service is always right. There cannot be any traditional “the consumer is always right” in relation to service semantic, as the services have no request/reply consumers.
Alas, still the consumer can still be right; it is just the responsibility of the process composition mechanism to accept what ever messages (events) the consumer (publisher) throws at it and invoke the applicable service operations. It is the business process composition mechanism that has to be the semantic covenant.
PS! Note that I have deliberately used the term "composition" here and not even mentioned using an ESB or BPM to implement this. The semantic covenant composition mechanism can simply be implemented as a mashup (composite application) instead of a composite service or an orchestration/saga.
The above figure is from Mike Rosen's article Business Architecture and SOA. As can be seen in the figure, CIM comprises a shared information model that is the basis for the documents that flows through the business processes built using the capabilities provided by the services.
