I have over the last year promoted the need for having a business process information model (BPIM) for semantic mediation and business process integration to enable SOA, not to forget enabling consumers such as mashups to do composition and consumption of the provided services, to deliver value to our users.
There has been some discussions about the need for an enterprise to have federated canonical domain models, but I think more and more people tends to believe that BPIM is an important SOA and BPM enabler. This blog post by Joe McKendrick quotes Hub Vandervoort, CTO of Progress Software, on some common SOA challenges:
He recommended three approaches to the problem, including "getting your transports aligned between business entities so that you can use eventing-oriented mechanisms to communicate across domains"; establishing SLA and security policies that ensure visibility; and establishing a common enterprise data model.
"You have to get your semantics aligned among the members," he said. "And that doesn't have to be a common vocabulary in its entirety, but certainly what we regard as the data in flight, those things that fly between domains and different working groups have to be highly normative."
The important part of Vandervoort's message is that you should not try to make a complete Enterprise Data Model, but rather focus on a common information model (CIM) for the "data in flight", i.e. the events, messages and documents of your business processes, when creating your information model.
Showing posts with label EDA. Show all posts
Showing posts with label EDA. Show all posts
Thursday, May 22, 2008
Wednesday, January 30, 2008
Ontology for Business Processes, SOA and Information Models
I have for a long time promoted that you should apply a specific kind of information model (the business process information model - BPIM) when composing services into business processes in service oriented solutions. A BPIM models the messages for the business events that drive the business processes. The main goal of this information model is to enable coordinated loose-coupling (semantic mediation) between the services at the composition level.
There has been a lot of discussions on how the BPIM relates to SOA services and the common information model (CIM), and how the BPM process layer relates to the SOA process layer. In the following ontology I have tried to model the relationships between the documents and the domain objects, Shy Cohen’s service taxonomy & ontology, and Jean-Jacques Dubray’s SOA+BPM ontology.
The service classification scheme has the "hierarchy" taxomomy form. It is not a scientific hierarchy nor a tree form, thus there is no "is using" or "is dependent on" relation between the services - it is just a classification scheme. There can be no dependency between services in SOA except for composite services realizing a business process by consuming the actual services. Taking a dependency on another service will break the central "services are autonomous" tenet of SOA.
Click to enlarge the ontology:
Note how there are processes at two levels in the model: the event-driven resource lifecycle processes for domain objects (bottom), and the human-driven business processes realized as composed services (top). These two levels are separate bounded contexts with well defined mappings between them: the business process domain and the resource lifecycle domain.
The BPIM is applied at the business process domain level (orchestrations / composites / sagas), while the services operate on the domain objects in the CIM to manage the lifecycle of these resources in accordance with the business events.
Note that the BPIM is related to, and is a subset of/reference to, the resource domain objects in the common information model (CIM). In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. This in like a mail order paper form that contains e.g. some customer data and references to product data. The word "subset" is central, e.g. the “customer has moved” action event has a BPIM document that contains the data pertinent to that specific business process event, not the complete schema of the customer domain object. The resource process service used to act on the “customer has moved” event does of course operate on the actual domain object, but this is invisible to the human workflow process.
The business process messages (action, query, notification) driving the composite services and the resulting business events are central artifacts when designing the services and the information model. I am a strong believer in designing a service-oriented solution by applying "EDA style" thinking to avoid missing out on the events and their documents due to traditional “invoke operations” SOA thinking, getting too much focus on business process flow.
An aspect not shown in the model is that the need for agility increases towards the top, while the cost of change increases towards the bottom. This is an important argument for having processes at two levels; it allows you to contain the frequent changes to the business processes (mashup style compositions) and composite services rather than having to constantly make changes to the services themselves, which could become very costly as the number of service subscribers increases over time. This is something that your business people should readily appreciate.
Design your system to have flexibility in the flow of the business processes and composite services. The diamonds of your business process are the business decisions, and this is where you should implement a business rules engine (BRE) mechanism. Process flow logic is not service business logic, avoid leaking domain logic into the orchestration layer at all cost.
Finally, the model show how claims relate to services at all levels. I have done this to show how a claims-based security model is a cross-cutting concern throughout a service-oriented solution. I strongly recommend designing in process claims right from the beginning, as this makes it easier to learn how access control relates to the business processes and solution domain.
Master Data Management (MDM) relates to the "Resources" in the above ontology. It is imperative that you apply a MDM strategy to your enterprise resource domain objects to avoid inconsistencies and multiple versions of the truth.
I hope this ontology makes my viewpoint on business process driven SOA clear to you. Feel free to comment on this model, and also check out the six figures in JJD’s ontology for models showing different perspectives of resource lifecycles & business events, BPEL, BPMN and human tasks.
See also this related post.
There has been a lot of discussions on how the BPIM relates to SOA services and the common information model (CIM), and how the BPM process layer relates to the SOA process layer. In the following ontology I have tried to model the relationships between the documents and the domain objects, Shy Cohen’s service taxonomy & ontology, and Jean-Jacques Dubray’s SOA+BPM ontology.
The service classification scheme has the "hierarchy" taxomomy form. It is not a scientific hierarchy nor a tree form, thus there is no "is using" or "is dependent on" relation between the services - it is just a classification scheme. There can be no dependency between services in SOA except for composite services realizing a business process by consuming the actual services. Taking a dependency on another service will break the central "services are autonomous" tenet of SOA.
Click to enlarge the ontology:
Note how there are processes at two levels in the model: the event-driven resource lifecycle processes for domain objects (bottom), and the human-driven business processes realized as composed services (top). These two levels are separate bounded contexts with well defined mappings between them: the business process domain and the resource lifecycle domain.
The BPIM is applied at the business process domain level (orchestrations / composites / sagas), while the services operate on the domain objects in the CIM to manage the lifecycle of these resources in accordance with the business events.
Note that the BPIM is related to, and is a subset of/reference to, the resource domain objects in the common information model (CIM). In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. This in like a mail order paper form that contains e.g. some customer data and references to product data. The word "subset" is central, e.g. the “customer has moved” action event has a BPIM document that contains the data pertinent to that specific business process event, not the complete schema of the customer domain object. The resource process service used to act on the “customer has moved” event does of course operate on the actual domain object, but this is invisible to the human workflow process.
The business process messages (action, query, notification) driving the composite services and the resulting business events are central artifacts when designing the services and the information model. I am a strong believer in designing a service-oriented solution by applying "EDA style" thinking to avoid missing out on the events and their documents due to traditional “invoke operations” SOA thinking, getting too much focus on business process flow.
An aspect not shown in the model is that the need for agility increases towards the top, while the cost of change increases towards the bottom. This is an important argument for having processes at two levels; it allows you to contain the frequent changes to the business processes (mashup style compositions) and composite services rather than having to constantly make changes to the services themselves, which could become very costly as the number of service subscribers increases over time. This is something that your business people should readily appreciate.
Design your system to have flexibility in the flow of the business processes and composite services. The diamonds of your business process are the business decisions, and this is where you should implement a business rules engine (BRE) mechanism. Process flow logic is not service business logic, avoid leaking domain logic into the orchestration layer at all cost.
Finally, the model show how claims relate to services at all levels. I have done this to show how a claims-based security model is a cross-cutting concern throughout a service-oriented solution. I strongly recommend designing in process claims right from the beginning, as this makes it easier to learn how access control relates to the business processes and solution domain.
Master Data Management (MDM) relates to the "Resources" in the above ontology. It is imperative that you apply a MDM strategy to your enterprise resource domain objects to avoid inconsistencies and multiple versions of the truth.
I hope this ontology makes my viewpoint on business process driven SOA clear to you. Feel free to comment on this model, and also check out the six figures in JJD’s ontology for models showing different perspectives of resource lifecycles & business events, BPEL, BPMN and human tasks.
See also this related post.
Monday, January 28, 2008
SOA, EDA, CEP, BPM
While I have been in Redmond the last three weeks learning my true color (I'm hyper green btw), there has been some interesting posts by Joe McKendrick on some topics that I've written about the last year:
Long time readers of my blog knows how I strongly advice applying EDA thinking when designing service-oriented solutions, but I do not foresee CEP to become a commodity for quite some time yet. However, it would be nice if designing systems around business events could provide better insight on how BPM relates to SOA in practical terms, along the ontology presented last december by Jean-Jacques Dubray.
Sunday, October 07, 2007
Information Model Mediation: Esperanto vs Babel Fish
In my quite popular post SOA: Canonical "Data" Model (picked up by Steve Jones, David Linthicum, Jean-Jacques Dubray and others), I used two analogies for the semantic mediation needed when composing different services into business processes: Esperanto and the Babel fish. This post is about how the two approaches differ in their use of the business process information model (BPIM). But first a short recap of the two terms:
Esperanto is a common world language; when you learn it, you can speak with any other person in the world that also known Esperanto. Everyone does of course still have their native language. See how similar this is to services that interact using a common information model (CIM) to express themselves, while every system still use their own domain model inside.
A Babel fish is a small fish that you put in your ear, which is capable of simultaneously translating any known language in the universe to yours, while adapting semantics between galaxies. See how similar this is to services that interact using a common logical data model (LDM) while every system still uses their own domain model to express themselves.
Both these approaches to semantic mediation in service-oriented solutions do use a business process information model (BPIM). They differ only in how they implement the transformation between formats: inside the services or outside the services.
The CIM approach to mediation is to use the BPIM directly in the service contracts and apply the transformation within the provided services [Hohpe/Woolf: Messaging Mapper pattern (477)]. This is a simple and viable approach; the services are self-contained, but require a bit more to implement and test due to the extra mediation requirements. The services will also need to be changed and tested when the BPIM changes. Dan North suggests this approach in his 'A Low-Tech Approach to Understanding SOA' article. This is still the most common approach, especially for those without an ESB.
There is one minor challenge with the CIM approach: chances are that the Esperanto of an alien galaxy will differ from your Esperanto. Thus, when outsourcing an activity service such as 'credit check', you need to adapt to the service provider's business concepts. So even if you use a CIM for all your enterprise services, you will still need to implement some sort of mediation to do the context mapping between the two federated business domains.
The huge advantage of the CIM approach is that it doesn't require any ESB-style intermediary. The services speak "CIM" natively and thus no transformation to/from the common data model is needed. Note that the service implementation must still map between the CIM format and its internal data formats.
The LDM approach involves using an orchestration mechanism to compose business processes from the set of domain specific services. The services just provide their functions independent of who the requesters (consumers) are and how they try to interact with the services. Think of your SOA solution as having Event Driven Architecture (EDA) to free yourself from "invoking operations" and think of business events and messages instead. It is the task of the Babel fish to mediate and transform the messages sent between consumers and providers [Hohpe/Woolf: Message Translator pattern (87)]. The intermediary can be just the service composition mechanism, but an Service Bus is a more robust and flexible mediation and transformation mechanism.
The LDM approach makes the services themselves simpler and easier to implement and test as they now have just a single responsibility. The services need not change when the BPIM changes, that complexity has been shifted to the intermediary. Implementing semantic mediation outside the services is proposed by Jack van Hoof, David Chappell (Pope of ESB), Bobby Woolf and many others.
Esperanto is a common world language; when you learn it, you can speak with any other person in the world that also known Esperanto. Everyone does of course still have their native language. See how similar this is to services that interact using a common information model (CIM) to express themselves, while every system still use their own domain model inside.
A Babel fish is a small fish that you put in your ear, which is capable of simultaneously translating any known language in the universe to yours, while adapting semantics between galaxies. See how similar this is to services that interact using a common logical data model (LDM) while every system still uses their own domain model to express themselves.
Both these approaches to semantic mediation in service-oriented solutions do use a business process information model (BPIM). They differ only in how they implement the transformation between formats: inside the services or outside the services.
The CIM approach to mediation is to use the BPIM directly in the service contracts and apply the transformation within the provided services [Hohpe/Woolf: Messaging Mapper pattern (477)]. This is a simple and viable approach; the services are self-contained, but require a bit more to implement and test due to the extra mediation requirements. The services will also need to be changed and tested when the BPIM changes. Dan North suggests this approach in his 'A Low-Tech Approach to Understanding SOA' article. This is still the most common approach, especially for those without an ESB.
There is one minor challenge with the CIM approach: chances are that the Esperanto of an alien galaxy will differ from your Esperanto. Thus, when outsourcing an activity service such as 'credit check', you need to adapt to the service provider's business concepts. So even if you use a CIM for all your enterprise services, you will still need to implement some sort of mediation to do the context mapping between the two federated business domains.
The huge advantage of the CIM approach is that it doesn't require any ESB-style intermediary. The services speak "CIM" natively and thus no transformation to/from the common data model is needed. Note that the service implementation must still map between the CIM format and its internal data formats.
The LDM approach involves using an orchestration mechanism to compose business processes from the set of domain specific services. The services just provide their functions independent of who the requesters (consumers) are and how they try to interact with the services. Think of your SOA solution as having Event Driven Architecture (EDA) to free yourself from "invoking operations" and think of business events and messages instead. It is the task of the Babel fish to mediate and transform the messages sent between consumers and providers [Hohpe/Woolf: Message Translator pattern (87)]. The intermediary can be just the service composition mechanism, but an Service Bus is a more robust and flexible mediation and transformation mechanism.
The LDM approach makes the services themselves simpler and easier to implement and test as they now have just a single responsibility. The services need not change when the BPIM changes, that complexity has been shifted to the intermediary. Implementing semantic mediation outside the services is proposed by Jack van Hoof, David Chappell (Pope of ESB), Bobby Woolf and many others.
Wednesday, August 29, 2007
Guerilla SOA, MEST
InfoQ has posted an interview with Jim Webber of ThoughtWorks about "Guerilla SOA", where he talks about how to design scalable SOA solutions that are evolvable and allow innovation in the different services that make up the solution.
The advice he offers covers topics that I have written about in my posts these last months:
Listen to the interview and read the transcript: Jim Webber on "Guerilla SOA"
The advice he offers covers topics that I have written about in my posts these last months:
- Share business messages owned by the business people (business process information model)
- SOA has no operations, only business messages delivered to letter boxes - services exchange messages to fulfill business processes
- Services cannot be invoked, they just receive event+payload messages and decide if and how to process it: "Could you possibly have a look at this message and maybe if it suits you do some processing on it" (semantic covenant: the service is always right)
- Business processes are workflows; you need to model long-lived conversations beyond request/response (SSDL, check out the Soya WCF SSDL toolkit at SourceForge)
Listen to the interview and read the transcript: Jim Webber on "Guerilla SOA"
Wednesday, August 01, 2007
Lean Information Models for Composite Services
A business process information model (BPIM) is a model that is focused on the business events and the data+documents+messages needed as part of orchestrating the events into business processes by composing services. A central part of the creating the model is to keep the model as small as possible (like a "least common denominator" of the data), only passing the state of the process as part of the events. The data and business documents represent the state of an instance of a business process (what Jack van Hoof calls a dossier) and is all a business process step should need to know to be able to perform its actions.
It is important to recognize that the BPIM does not encompass all information that some part of a business process might ever want or need to know, it just needs to contain enough reference data. Neither does the BPIM have to contain data about business entites across all applications in your enterprise, i.e. the BPIM "customer" does not need to model the union of all the different customer fields from your CRM and ERP systems. This is why BPIM is different from a common information model (CIM) or the EAI Common Data Model [Hohpe/Woolf CDM pattern (355)].
The business process that need to know more about the referenced data of the dossier, will have to use services to look it up, as part of the service composition (orchestration). This is what is called the "pull model" in the Observer pattern [GoF] [Hohpe/Woolf Event Message (151)]. The business process can also augment the process state by calling services and adding data to the dossier. Thus, the mantra of making a slender model is "Ask For More".
Keeping the information model small is very important wrt to maintenance of the model, afterall the main purpose of the model is to mediate semantics in the service bus and its orchestrations of services. The number of mappings between the different elements (messages+dossiers and services) of a model will grow and change over time, thus having small model elements are crucial to keep the model maintainable. In addition, split your model into separate domains/bounded contexts and make context maps, as this makes the total number of mappings smaller.
The key maintenance advice is to keep the model elements small (smaller mappings) and the number of mappings between model elements to a minimum (less mappings).
It is important to recognize that the BPIM does not encompass all information that some part of a business process might ever want or need to know, it just needs to contain enough reference data. Neither does the BPIM have to contain data about business entites across all applications in your enterprise, i.e. the BPIM "customer" does not need to model the union of all the different customer fields from your CRM and ERP systems. This is why BPIM is different from a common information model (CIM) or the EAI Common Data Model [Hohpe/Woolf CDM pattern (355)].
The business process that need to know more about the referenced data of the dossier, will have to use services to look it up, as part of the service composition (orchestration). This is what is called the "pull model" in the Observer pattern [GoF] [Hohpe/Woolf Event Message (151)]. The business process can also augment the process state by calling services and adding data to the dossier. Thus, the mantra of making a slender model is "Ask For More".
Keeping the information model small is very important wrt to maintenance of the model, afterall the main purpose of the model is to mediate semantics in the service bus and its orchestrations of services. The number of mappings between the different elements (messages+dossiers and services) of a model will grow and change over time, thus having small model elements are crucial to keep the model maintainable. In addition, split your model into separate domains/bounded contexts and make context maps, as this makes the total number of mappings smaller.
The key maintenance advice is to keep the model elements small (smaller mappings) and the number of mappings between model elements to a minimum (less mappings).
Thursday, July 05, 2007
Composite Services: Information Model, Federated Information Models
A SOA solution is typically implemented by composition of services provided by different systems. The solution might comprise a mix of of in-house services, 3rd party and outsourced services. In addition, the solution might also involve cross-enterprise service compositions.
To be able to efficiently compose the services, a model that ensures shared sematics is needed, and in my last post I described the business process information model (BPIM). David Chappell calls this approach semantic data integration. BPIM is a similar concept to the EAI canonical data model [Hohpe/Woolf CDM pattern (355)], but it is a model with a slightly different purpose at a different architectual layer and it is not just about the data. The BPIM is about the events, messages and data passed between services, not about having a unified superset of the entities within an enterprise.
The BPIM allows for composition of services without having to know and comply with the model of the underlying service logic, all you need to know is the message types. This is a big advantage, and allows for the consumers to be isolated from the details of the consumed services. E.g. all a service consumer need to know is the "AddressChange" data of the "CustomerHasMoved" event, not the complete data schema of the "Customer" service.
BPIM is closely related to the Common Information Model (CIM) concept. Note that the BPIM message types are projected compositions of the referenced CIM resources - not just simple compositions of resource objects. In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain.
The term 'domain' is borrowed from Domain Driven Design (DDD). Domain-driven design is not a technology or a methodology. It is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains. Read stories about how DDD can be applied to a diverse set of architectual problems: Practitioner Experience Reports.
Focusing on the bounded context for modeling the flow, events, messages, data and semantics involved in implementing the core business processes of the domain, should make it easier to come up with a working model. Note again that the information model is about more than just the data. Hence the name business process information model. Arvindra Sehmi and Beat Schwegler used the same term in Service-Oriented Modeling for Connected Systems, an article that also provides details about creating a service model.
Partner/ 3rd party/ outsourced services are not part of the core business domain according to DDD. If they were core processes in your business, how come they are so general that they can be outsourced or bought? Core processes are those that make your business unique and give you a competitive edge. DDD dictates using translators or an "anti-corruption layer" against services/systems that are not within the domain.
Having a canonical schema model at the service layer might be feasible within an enterprise, but should be avoided as this will cause very tight coupling to the One True Schema. Making every service depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. If agility at the service layer is less important for you, then such a service straight-jacket might initially feel good. Trying to make a enterprise data model (EDM) is not a good idea for the same reasons. Steve Jones has a good post about canonical form issues and how you cannot enforce your model upon the world: Single Canonical Form - not for SOA
Federated Business Process Information Models
DDD recommends splitting big, diverse and complex solutions into several bounded contexts. Set explicit boundaries based on e.g. orgranizational units and application usage. A natural boundary in SOA is partner/ 3rd party/ outsourced services. Each set of services that is not under your control and that you cannot enforce your information model upon, is a separate bounded context (domain model).
Note that also within your enterprise service model there will be several other bounded contexts for different domains, each with its own information model that is canonical per domain.
You should identify each model in play on the project and make a context map. A context map describes the points of contact between the domain models, in addition to outlining explicit translation for any communication between the models.
This figure show how a context map is used to show how two domains relate to each other:
The figure and the definitions of "bounded context" and "context map" is taken from the 'Strategic Design' chapter of the book "DOMAIN-DRIVEN DESIGN" by Eric Evans [Addison-Wesley, 2004].
Note how not all elements of a domain model needs to be mapped to other models. Only the interconnected parts needs to have a translation map. E.g. the credit check process is provided by a 3rd party, thus it exists in a separate external information model. Your BPIM needs to have a translation map to the other service to be able to invoke it. Note how similar this is to the purpose of the BPIM itself: translating between business process compositions and the underlying, composed services. Thus, the context map is the basis for modelling a set of Federated Business Process Information Models. A federated BPIM system map shows the integration of multiple enterprise service models, avoiding the pitfall of designing a single canonical data model across a set of different domains.
To be able to efficiently compose the services, a model that ensures shared sematics is needed, and in my last post I described the business process information model (BPIM). David Chappell calls this approach semantic data integration. BPIM is a similar concept to the EAI canonical data model [Hohpe/Woolf CDM pattern (355)], but it is a model with a slightly different purpose at a different architectual layer and it is not just about the data. The BPIM is about the events, messages and data passed between services, not about having a unified superset of the entities within an enterprise.
The BPIM allows for composition of services without having to know and comply with the model of the underlying service logic, all you need to know is the message types. This is a big advantage, and allows for the consumers to be isolated from the details of the consumed services. E.g. all a service consumer need to know is the "AddressChange" data of the "CustomerHasMoved" event, not the complete data schema of the "Customer" service.
BPIM is closely related to the Common Information Model (CIM) concept. Note that the BPIM message types are projected compositions of the referenced CIM resources - not just simple compositions of resource objects. In addition, the messages covers more than just activity data, they also contain queries, notifications (events) and commands. Design the BPIM based on the CIM, ensuring that the model is canonical for each process domain.
The term 'domain' is borrowed from Domain Driven Design (DDD). Domain-driven design is not a technology or a methodology. It is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains. Read stories about how DDD can be applied to a diverse set of architectual problems: Practitioner Experience Reports.
Focusing on the bounded context for modeling the flow, events, messages, data and semantics involved in implementing the core business processes of the domain, should make it easier to come up with a working model. Note again that the information model is about more than just the data. Hence the name business process information model. Arvindra Sehmi and Beat Schwegler used the same term in Service-Oriented Modeling for Connected Systems, an article that also provides details about creating a service model.
Partner/ 3rd party/ outsourced services are not part of the core business domain according to DDD. If they were core processes in your business, how come they are so general that they can be outsourced or bought? Core processes are those that make your business unique and give you a competitive edge. DDD dictates using translators or an "anti-corruption layer" against services/systems that are not within the domain.
Having a canonical schema model at the service layer might be feasible within an enterprise, but should be avoided as this will cause very tight coupling to the One True Schema. Making every service depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. If agility at the service layer is less important for you, then such a service straight-jacket might initially feel good. Trying to make a enterprise data model (EDM) is not a good idea for the same reasons. Steve Jones has a good post about canonical form issues and how you cannot enforce your model upon the world: Single Canonical Form - not for SOA
Federated Business Process Information Models
DDD recommends splitting big, diverse and complex solutions into several bounded contexts. Set explicit boundaries based on e.g. orgranizational units and application usage. A natural boundary in SOA is partner/ 3rd party/ outsourced services. Each set of services that is not under your control and that you cannot enforce your information model upon, is a separate bounded context (domain model).
Note that also within your enterprise service model there will be several other bounded contexts for different domains, each with its own information model that is canonical per domain.
You should identify each model in play on the project and make a context map. A context map describes the points of contact between the domain models, in addition to outlining explicit translation for any communication between the models.
This figure show how a context map is used to show how two domains relate to each other:
The figure and the definitions of "bounded context" and "context map" is taken from the 'Strategic Design' chapter of the book "DOMAIN-DRIVEN DESIGN" by Eric Evans [Addison-Wesley, 2004].Note how not all elements of a domain model needs to be mapped to other models. Only the interconnected parts needs to have a translation map. E.g. the credit check process is provided by a 3rd party, thus it exists in a separate external information model. Your BPIM needs to have a translation map to the other service to be able to invoke it. Note how similar this is to the purpose of the BPIM itself: translating between business process compositions and the underlying, composed services. Thus, the context map is the basis for modelling a set of Federated Business Process Information Models. A federated BPIM system map shows the integration of multiple enterprise service models, avoiding the pitfall of designing a single canonical data model across a set of different domains.
Monday, July 02, 2007
SOA: Canonical "Data" Model
An important topic when designing service oriented systems is how to enable different services to share semantics to be able to be composed into working solutions. Jack van Hoof has written a good article about this: How to mediate semantics in an EDA. A few weeks later, Nick Malik posted another good read about this topic: Canonical Model, Canonical Schema, and Event Driven SOA. Read Jack's post first.
They both talk about using a canonical data model (CDM) as the Esperanto / Babel fish to map between the format and semantics of the disparate systems taking part in a SOA solution. Note that CDM is not about having a common data model (EAI CDM) or a shared database across all systems in an enterprise, don't get fooled by the "data" in the term 'canonical data model'. CDM is about not making everybody have to speak English, but rather having CDM translators for each native system.
Btw, Gregor Hohpe sometimes use the term 'canonical domain model' on his blog, while using the term 'canonical data model' in the book "Enterprise Integration Patterns" [Hohpe/Woolf CDM pattern (355)]. I think it is better to talk about the business domain rather than about "data", as this help focusing on the business processes rather than databases and other technology. You'd be surprised how many biz people concern themselves with how the data model looks - maybe a leftover from the client-server days, to show that they know what an ER-diagram is? Focus on designing a business process information model (BPIM) for each business process domain.
Trying to enforce a One True Schema across your services (everyone has to speak English) is not a viable path, and it is also a recipe for future maintenance hell. Making every service contract depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. A simple change to e.g. the order entity will cause a ripple effect through all referring services. This is where the business process information model comes into play, it allows you to version and evolve the services independently of each other.
The Canonical "Data" Model concept is also sometimes referred to as a Common Information Model (CIM). Both the business process information model (BPIM) and the data focused CDM/CIM models has the same goal: mediation og semantics. However, they are not the same as the two other models are both variations of the common data model approach. The business process information model is about semantic business process integration, not just only semantic data integration.
They both talk about using a canonical data model (CDM) as the Esperanto / Babel fish to map between the format and semantics of the disparate systems taking part in a SOA solution. Note that CDM is not about having a common data model (EAI CDM) or a shared database across all systems in an enterprise, don't get fooled by the "data" in the term 'canonical data model'. CDM is about not making everybody have to speak English, but rather having CDM translators for each native system.
Btw, Gregor Hohpe sometimes use the term 'canonical domain model' on his blog, while using the term 'canonical data model' in the book "Enterprise Integration Patterns" [Hohpe/Woolf CDM pattern (355)]. I think it is better to talk about the business domain rather than about "data", as this help focusing on the business processes rather than databases and other technology. You'd be surprised how many biz people concern themselves with how the data model looks - maybe a leftover from the client-server days, to show that they know what an ER-diagram is? Focus on designing a business process information model (BPIM) for each business process domain.
Trying to enforce a One True Schema across your services (everyone has to speak English) is not a viable path, and it is also a recipe for future maintenance hell. Making every service contract depend on the One True Schema will make it impossible for the services to evolve separately, they will no longer be autonomous. A simple change to e.g. the order entity will cause a ripple effect through all referring services. This is where the business process information model comes into play, it allows you to version and evolve the services independently of each other.
The Canonical "Data" Model concept is also sometimes referred to as a Common Information Model (CIM). Both the business process information model (BPIM) and the data focused CDM/CIM models has the same goal: mediation og semantics. However, they are not the same as the two other models are both variations of the common data model approach. The business process information model is about semantic business process integration, not just only semantic data integration.
Friday, January 19, 2007
WCF: Core categories of data contracts
One of the famous SOA tenets is "services share contract, not class/implementation", meaning that it is the schema of your contract that is the main conveyor of how to consume the operations provided by your service. This has a huge impact on how you should design your contracts to provide for clear, understandable and comprehensive semantics, and also to minimize ambiguity in how to use your service. Contracts that have subtle or vague semantics are just more difficult to use and are thus more error prone. The same applies to contracts that are too flexible.
This post is about how to design data contracts that a simple to use, rather than easy to implement (simple vs easy); and at the same time keeping the number of data contracts to a minimum. The latter is important both for the consumers of your service and for the maintainability of your service. It is also important wrt SOA governance, the less stuff you have to govern, the better. Less schemas, less semantics, less maintenance, less governance.
Data contracts belong to one of these two groupings: altering state and querying information. Generally speaking, operations that modifies your system need to comply with stricter requirements and rules than operations that reads data from your system. This is because operations that can leave your system in an invalid state have greater technical impact on your business than operations that just returns information. Of course, if you disclose the incorrect information, your business could be in serious legal trouble.
The two data contract groupings can be further refined into several categories based on the different needs for expressing contract semantics and for being unambiguous. These five data contract core categories have manifested themselves through several more or less service-oriented solutions that I have implemented:
The term ‘domain object’ also comprises complex objects (aggregate root objects) such as an order or a document card. The term ‘contained’ is used for complex objects. The term ‘composite’ is used for contracts that consists of several domain objects. The term ‘batch update’ includes insert, update and delete actions or a combination of these actions.
Note that I use CRUDy terms in the categories for simplicity (easier for me), to cover any real-life event that affects the state of a domain object. E.g. the “customer has moved” event falls into the “update” category.
A result contract will typically contain a composite structure of domain objects, defined by exactly the same unambiguous data contracts used for insert/update actions. The main reason for defining data contracts in the first place is to promote standardization and reuse across services and operations. To be able to support both the rigid insert/update data contact requirements and the flexible result contract requirements; it becomes a must to separate structure from data, isolating the structure/composition to the result set data contracts. Structural elements in a data contracts implicitly impose subtle semantics: how will the service handle the omission of composite/contained domain objects.
Insert/Update Contracts
It is important that insert/update contracts have little room for ambiguity, especially for complex domain objects. E.g. if the customer data contract contains a collection of addresses, what will happen if a customer update action is performed and no addresses are provided: does it mean that the customer no longer have any addresses or does it just mean that your can update a customers phone number without having to specify the addresses?
Such contained objects must be either A) required or B) specifically handled and by default optional/ignored. Controlled optional elements can be handled the way that the .NET 1.x XmlSerializer handled optional elements: using an extra property to indicate the state of the optional element. The XmlSerializer uses a Boolean XxxSpecified property for each optional element, e.g. OrderShippedDateSpecified.
Rather than using just a Boolean for the contained optional object, I recommend using an enumeration that contains Ignore (default value) and then some other applicable actions; much like cascading actions in SQL Server. The customer data contract should contain both an AddressList collection and an AddressListAction enumeration with e.g. the values Ignore, Replace, Alter, Purge. The point is that the user has to explicitly assign an action on the contained collection, rather than the service just assuming that an empty collection means deletion of the existing children. Assumptions are semantic coupling, and that is something you should strive to avoid.
Note that these 'insert/update' contract recommendations apply to entity/core services, which are not the services you want to expose publicly. Your public services need to reflect the events of your service-oriented business processes, and these "published" services belongs to the 'application to application services' category. By layering your services according to the four service categories, you will be able to expose more specialized operations with smaller contracts. Large contracts imply stronger coupling to the service, and as large contracts are more likely to change, your service will be more subject to breaking changes. Small contracts are simpler contracts, and simple contracts are important for the reusability, reliability, quality and robustness of your service (more about this in "Patterns for High-Integrity Data Consumption and Composition" by Dion Hinchcliffe).
You can still provide a very specific business operation that builds on the core service. E.g. the "customer has moved" event can be supported by an composite operation that takes only the customer key and the new postal address; which internally gets the complete customer, alters the address, and then stores the customer, in a single transaction using the core services. Services at the A2AS layer allows you to be "liberal in what you accept" as they shield the consumers from the details of the core services.
Read/Query Result Contracts
Result data contracts should be able to fit multiple needs and support several views of domain objects and composite result sets. At the same time, a consumer should be able to control how much information that gets returned from the service. E.g. one consumer might not be interested in address information when fetching customer data. Thus, a result data contract will most likely comprise optional elements, and consumers will not fail if some data is not present in the result set.
An empty collection does not normally imply the same ambiguity for reads as it does for insert/update contracts. A consumer will typically assume that if a fetched complex object contains no elements for a contained data contract, then the object does not have any such children; e.g. that a customer has no addresses if the customer AddressList collection is empty. An extra metadata property could be added to the result data contract as an indication of whether an optional element actually contains data even if not returned due to the processed query specification.
Note that ‘not present’ in the result set is not the same as ‘missing’ from the result set, which is clearly an error and should have caused a service fault.
Batch Update/Import Contracts
Batch update contracts are typically used to alter the state of a set of (related) domain objects. E.g. to update the TaskList collection of a project by sending a message that contains the tasks to add, modify and remove as one batch. Batch operations are a good way to avoid having to expose transactions outside your service; package all domain objects that must be altered in a transaction into a single message and perform the update using a single transacted operation.
Note that each data contract must still follow the rules described for ‘insert/update contracts’ even when used as part of a batch contract.
To be ideal objects for batch operations, domain objects should expose a “row-state” property; if they don’t, you need something like ‘Service Data Objects’ to make your batch contracts really simple to use. A message with one collection per action should be the last alternative.
This post is about how to design data contracts that a simple to use, rather than easy to implement (simple vs easy); and at the same time keeping the number of data contracts to a minimum. The latter is important both for the consumers of your service and for the maintainability of your service. It is also important wrt SOA governance, the less stuff you have to govern, the better. Less schemas, less semantics, less maintenance, less governance.
Data contracts belong to one of these two groupings: altering state and querying information. Generally speaking, operations that modifies your system need to comply with stricter requirements and rules than operations that reads data from your system. This is because operations that can leave your system in an invalid state have greater technical impact on your business than operations that just returns information. Of course, if you disclose the incorrect information, your business could be in serious legal trouble.
The two data contract groupings can be further refined into several categories based on the different needs for expressing contract semantics and for being unambiguous. These five data contract core categories have manifested themselves through several more or less service-oriented solutions that I have implemented:
- Insert/update contracts: Typically one contract per domain object. Optional contained data contracts must be avoided or specifically handled.
- Delete contracts: Typically one contract per domain object.
- Specification/criteria contracts: Typically one contract per result contract, but it is not uncommon that a single specification can relate to multiple result contracts. Optional members are perfectly standard; the same applies to nullable criteria. Composite specifications are normal.
- Read/query result contracts: One or more contracts per domain object. Optional contained data contracts are allowed for flexibility and this is a key mechanism for keeping the number of result contracts to a minimum. Composite contracts are also allowed for the same reasons.
- Batch update/import contracts: Typically one contract per domain object batch operation type. Composite contracts are normal. Optional composite or contained contracts must be specifically handled.
The term ‘domain object’ also comprises complex objects (aggregate root objects) such as an order or a document card. The term ‘contained’ is used for complex objects. The term ‘composite’ is used for contracts that consists of several domain objects. The term ‘batch update’ includes insert, update and delete actions or a combination of these actions.
Note that I use CRUDy terms in the categories for simplicity (easier for me), to cover any real-life event that affects the state of a domain object. E.g. the “customer has moved” event falls into the “update” category.
A result contract will typically contain a composite structure of domain objects, defined by exactly the same unambiguous data contracts used for insert/update actions. The main reason for defining data contracts in the first place is to promote standardization and reuse across services and operations. To be able to support both the rigid insert/update data contact requirements and the flexible result contract requirements; it becomes a must to separate structure from data, isolating the structure/composition to the result set data contracts. Structural elements in a data contracts implicitly impose subtle semantics: how will the service handle the omission of composite/contained domain objects.
Insert/Update Contracts
It is important that insert/update contracts have little room for ambiguity, especially for complex domain objects. E.g. if the customer data contract contains a collection of addresses, what will happen if a customer update action is performed and no addresses are provided: does it mean that the customer no longer have any addresses or does it just mean that your can update a customers phone number without having to specify the addresses?
Such contained objects must be either A) required or B) specifically handled and by default optional/ignored. Controlled optional elements can be handled the way that the .NET 1.x XmlSerializer handled optional elements: using an extra property to indicate the state of the optional element. The XmlSerializer uses a Boolean XxxSpecified property for each optional element, e.g. OrderShippedDateSpecified.
Rather than using just a Boolean for the contained optional object, I recommend using an enumeration that contains Ignore (default value) and then some other applicable actions; much like cascading actions in SQL Server. The customer data contract should contain both an AddressList collection and an AddressListAction enumeration with e.g. the values Ignore, Replace, Alter, Purge. The point is that the user has to explicitly assign an action on the contained collection, rather than the service just assuming that an empty collection means deletion of the existing children. Assumptions are semantic coupling, and that is something you should strive to avoid.
Note that these 'insert/update' contract recommendations apply to entity/core services, which are not the services you want to expose publicly. Your public services need to reflect the events of your service-oriented business processes, and these "published" services belongs to the 'application to application services' category. By layering your services according to the four service categories, you will be able to expose more specialized operations with smaller contracts. Large contracts imply stronger coupling to the service, and as large contracts are more likely to change, your service will be more subject to breaking changes. Small contracts are simpler contracts, and simple contracts are important for the reusability, reliability, quality and robustness of your service (more about this in "Patterns for High-Integrity Data Consumption and Composition" by Dion Hinchcliffe).
You can still provide a very specific business operation that builds on the core service. E.g. the "customer has moved" event can be supported by an composite operation that takes only the customer key and the new postal address; which internally gets the complete customer, alters the address, and then stores the customer, in a single transaction using the core services. Services at the A2AS layer allows you to be "liberal in what you accept" as they shield the consumers from the details of the core services.
Read/Query Result Contracts
Result data contracts should be able to fit multiple needs and support several views of domain objects and composite result sets. At the same time, a consumer should be able to control how much information that gets returned from the service. E.g. one consumer might not be interested in address information when fetching customer data. Thus, a result data contract will most likely comprise optional elements, and consumers will not fail if some data is not present in the result set.
An empty collection does not normally imply the same ambiguity for reads as it does for insert/update contracts. A consumer will typically assume that if a fetched complex object contains no elements for a contained data contract, then the object does not have any such children; e.g. that a customer has no addresses if the customer AddressList collection is empty. An extra metadata property could be added to the result data contract as an indication of whether an optional element actually contains data even if not returned due to the processed query specification.
Note that ‘not present’ in the result set is not the same as ‘missing’ from the result set, which is clearly an error and should have caused a service fault.
Batch Update/Import Contracts
Batch update contracts are typically used to alter the state of a set of (related) domain objects. E.g. to update the TaskList collection of a project by sending a message that contains the tasks to add, modify and remove as one batch. Batch operations are a good way to avoid having to expose transactions outside your service; package all domain objects that must be altered in a transaction into a single message and perform the update using a single transacted operation.
Note that each data contract must still follow the rules described for ‘insert/update contracts’ even when used as part of a batch contract.
To be ideal objects for batch operations, domain objects should expose a “row-state” property; if they don’t, you need something like ‘Service Data Objects’ to make your batch contracts really simple to use. A message with one collection per action should be the last alternative.
Friday, December 01, 2006
Service Architecture: JBOWS, SOA or WOA
A few days ago, I got into a discussion of whether the Amazon Web Services are true SOA or just a programming model exposed using web-services. The discussion started when I said that CRUD style operations are not according to SOA best practices; and that operating on parts of a domain object of type aggregate root should be avoided, favoring actions on the complete entity to be exposed as service operations.
Before making my SOA-vs-JBOWS case, a short introduction to WOA: as adhering to all the SOA service design principles and the different WS-* technologies can be quite daunting and complex, a lot of service-oriented solutions have emerged that take a simpler approach (REST, SOAP, POX, etc). Gartner has coined the acronym WOA (Web-Oriented Architecture) for these kinds of solutions that implement and expose services using more pragmatic techniques following the WOA tenets.
First the "what's in an operation name" issue:
It is not best practice to use CRUDy style operation names that conveys only that the domain object state is going to change in the repository (e.g. UpdateCustomerAddress). You should rather use operation names that reveals the event in the business process that caused the action (e.g. CustomerHasMoved). The point is that your service will most likely perform some business logic on the domain object in conjunction with storing it in the database. The operation name should convey the fact that the state of the real-world entity represented by the domain object has changed.
Note that you need not change the operation names very much, sometimes renaming UpdateXxx to ChangeXxx can be sufficient to convey the correct semantics. Focus on creating business process driven services rather than data-driven services.
The discussion I had was about if names such as AddXxx, ModifyXxx and DeleteXxx are CRUDy style names or not. I think they are to closely named after what the code is going to do with the domain object repository rather than reflecting changes to the real-life entity, i.e. it is more a programming model (WOA) than a SOA operation.
Note that CRUDy operations and names can be quite ok, especially for information services, which will still be needed to manage data in your repositories. There will still be a need for creating new customers, even in SOA.
Then on to the operations on different types of domain objects issue:
A common metaphor for designing SOA operation and data contracts is to think of paper forms being passed around to clerks to fulfil a business process. Another metaphor is mail orders, the point is that the 'document' contains all data needed to perform the business process.
Documents can be simple domain objects or aggregate root objects. An order is an example of an aggregate root object - it contains general order data and a set of order items. The order items belong to the order, i.e. it is an identifying relationship and not just a relation. As a general rule, an operation should process a whole document and never mess directly with 'identified items' within the document. Such operations would be very CRUDy style, and breaks the "boundaries are explicit" tenet.
Note that it is perfectly legal to operate on normal relations, such as adding orders to a customer. It is the nature of the domain and its business processes that decide if a relation is identifying or not, thus there is no hard rule to help you decide whether incremental operations are ok or not.
Using the 'document' metaphor, you would implement a 'RegisterOrder' operation to add new orders. You should, however, not implement operations for changing a subset of the order items; you must rather implement a 'ChangeOrder' or a 'CancelOrder' operation.
In my opinion, the Amazon shopping cart is analogous to the above order domain object. Thus, according to SOA best practices, they should not expose operations such as ChartModify that allows you to do incremental changes to the items in the chart. I understand that they do this for simplicity and performance reasons, and that the shopping cart is really not registered until it is submitted; but some developers use the AWS as general best practices for SOA and apply the same programming model blindly.
I don't propose that Amazon Web Services are not properly designed; they are just WOA rather than SOA.
A closing note: there is no "one or the other" between SOA and WOA, in fact they are both central in web 2.0. Read more about it in this report from the Web 2.0 Summit.
Before making my SOA-vs-JBOWS case, a short introduction to WOA: as adhering to all the SOA service design principles and the different WS-* technologies can be quite daunting and complex, a lot of service-oriented solutions have emerged that take a simpler approach (REST, SOAP, POX, etc). Gartner has coined the acronym WOA (Web-Oriented Architecture) for these kinds of solutions that implement and expose services using more pragmatic techniques following the WOA tenets.
First the "what's in an operation name" issue:
It is not best practice to use CRUDy style operation names that conveys only that the domain object state is going to change in the repository (e.g. UpdateCustomerAddress). You should rather use operation names that reveals the event in the business process that caused the action (e.g. CustomerHasMoved). The point is that your service will most likely perform some business logic on the domain object in conjunction with storing it in the database. The operation name should convey the fact that the state of the real-world entity represented by the domain object has changed.
Note that you need not change the operation names very much, sometimes renaming UpdateXxx to ChangeXxx can be sufficient to convey the correct semantics. Focus on creating business process driven services rather than data-driven services.
The discussion I had was about if names such as AddXxx, ModifyXxx and DeleteXxx are CRUDy style names or not. I think they are to closely named after what the code is going to do with the domain object repository rather than reflecting changes to the real-life entity, i.e. it is more a programming model (WOA) than a SOA operation.
Note that CRUDy operations and names can be quite ok, especially for information services, which will still be needed to manage data in your repositories. There will still be a need for creating new customers, even in SOA.
Then on to the operations on different types of domain objects issue:
A common metaphor for designing SOA operation and data contracts is to think of paper forms being passed around to clerks to fulfil a business process. Another metaphor is mail orders, the point is that the 'document' contains all data needed to perform the business process.
Documents can be simple domain objects or aggregate root objects. An order is an example of an aggregate root object - it contains general order data and a set of order items. The order items belong to the order, i.e. it is an identifying relationship and not just a relation. As a general rule, an operation should process a whole document and never mess directly with 'identified items' within the document. Such operations would be very CRUDy style, and breaks the "boundaries are explicit" tenet.
Note that it is perfectly legal to operate on normal relations, such as adding orders to a customer. It is the nature of the domain and its business processes that decide if a relation is identifying or not, thus there is no hard rule to help you decide whether incremental operations are ok or not.
Using the 'document' metaphor, you would implement a 'RegisterOrder' operation to add new orders. You should, however, not implement operations for changing a subset of the order items; you must rather implement a 'ChangeOrder' or a 'CancelOrder' operation.
In my opinion, the Amazon shopping cart is analogous to the above order domain object. Thus, according to SOA best practices, they should not expose operations such as ChartModify that allows you to do incremental changes to the items in the chart. I understand that they do this for simplicity and performance reasons, and that the shopping cart is really not registered until it is submitted; but some developers use the AWS as general best practices for SOA and apply the same programming model blindly.
I don't propose that Amazon Web Services are not properly designed; they are just WOA rather than SOA.
A closing note: there is no "one or the other" between SOA and WOA, in fact they are both central in web 2.0. Read more about it in this report from the Web 2.0 Summit.
Subscribe to:
Posts (Atom)