O dear, what will we do with the data?

In my previous post, I wrote about the possible scenario's to deal with integration with existing data-centric applications. In this blog-post, I want to cover situations, in which we want to digitize the process (with a BPMS/ACMS) and there is data, but no existing data-centric applications to store it.

Let's first investigate the typical data one encounters when analyzing a process that needs to be digitized...

Type	Description
Customer data	Names, addresses, contact persons, etc.
Product/transaction data	Type of service or product, certain service / product specifications, pricing, other conditions and agreements
Decisions	Specific decisions (what what the decision, who took it, when, and why was this decision taken/on what grounds)
Process and task performance data	Durations of process and tasks (or even effort and costs associated with it)
Audit trails	What was done, when, why and by who
Other process linked data	Comments/communication between process participants, certain checklists/QC outcomes

All this data is crucial for the stakeholders around a process:
- The customer wants their service or product first time right, on time delivered
- The process participants need it to know what to do, and do it correctly
- Process management wants to manage the flow, work-distribution, work-load and compliance
- Responsible managers want to be able to report outcomes, performance and compliance
- Accountants / Audit / QA-people want to understand what happened, and see if it was compliant to regulations
- Future generations (Public sector mainly) want to historically research what happened

When we digitize a process, this data is, in the as-is situation, usually scattered in various forms:
- In unstructured forms, on paper, in folders, or in e-mails
- In semi-structured form, in (often) Excel files

And the user requirements are typically:
- We don't want Excel anymore, the new BPMS/ACMS solution should help us to digitize the process
- We want to store and report on all of this data somehow

Many times, I encounter organizations that are mostly paper-based, sometimes with low-volumes. So when implementing a BPMS/ACMS solution, we do not have the luxury of a CRM-system, and ERP-system (or other Product administration), a Datawarehouse. Sure, organizations should have these systems in place, but often, introducing these technologies are not in our scope. Then the inevitable question comes: O dear, what will we do with the data? Ouch.

From a solution architecture point of view, we have a number of paths.

1. Let parts of the data behind in existing office automation
Pro's:
- Simple

Con's:
- Separate files that need to be updated manually (more effort)
- If the data is also stored in other places: risk of becoming out of sync
- Not addressing client needs
- Hard to query

2. Store parts of the data in office automation and integrate with it 
Pro's:
- Data is updated, no need for manually updates
- Less risk of becoming out of sync
- Low cost

Con's:
- Usually quite risky, as business users can access the files and change structures
- Not suitable for large data volumes
- Technically unstable sometimes, and transaction integrity/locking issues
- Hard to query

3. Store parts of the data in structured form in the process instance
This scenario is tricky. My design rule is: keep the process instance very light, as:
- Performance might get bad, when too much data per instance
- Often hard to query on specific items (usually stored as XML/blobs)

- You never know when the data gets cleaned out (you need strict rules on instance archiving)

With very light I mean:
- Customer/transactional data is only temporarily stored in the process instance, but as quick as possible stored in a other structure
- Data that is really process instance specific (a contact person name for this order)
- Only keys to the data (customerid, productid) are stored
- Only data is stored that is needed to execute business rules (preferably retrieved from other structure just in time).

4. Store parts of the data in digital documents (unstructured)
Pro's:
- Simple (you will need a Document Management solution, and often a 'Case' concept)

Con's:
- No reporting on structured data possible (Text mining too difficult)

5. Store parts of the data in a custom database-structure (linked to a case)
Pro's:
- Clear structured place to store data

Con's:
- Depending on complexity of data, can grow large and complex in terms of effort and architecture
- If the BPMS needs to support multiple processes, each with their own set of data, complexity can become even more

What I sometimes encounter is that scope explodes, because customers would like access to this data through a non-BPMS user interface: a new data-centric app is born and included in the project. Usually not a nice thing.

6. Store parts of the data in structured form in a generic Case data - solution
In this scenario, we create (of buy) a component in which one can define meta-data schemes per process-class or case type. In each scheme, one can define possible data-elements, . When creating a process instance or case, the BPMS/ACMS allows users to maintain the defined data-elements. 
Pro's:
- Flexible, quick to configure and deploy

Con's
- Can be hard to query

7. Store parts of the data in a (small/custom) Datawarehouse
For some types of data, I would recommend to create a small custom set of Star-scheme tables with the relevant dimensions. This allows to query on performance of processes and tasks, by dimensions as process / case type, period, organization unit, etc.  In addition, it would be possible the extent the star-scheme with more transactional data (order volume, price, certain decisions) to allow more complex queries ('In period X  what was the total of orders in Euro, in which we refused to deliver')

Conclusion
Data governance can be a tricky question in BPMS/ACMS projects. It not properly handled it can require lots of effort, effort that we rather would focus on the process and BPMS-advantages. If a project is oversold in terms of BPMS/ACMS promises (and we all encounter this from time to time :-)), it can be hard to manage client's expectations and influence the client to select and implement additional components, such as a CRM-system.
As BPMS-specialist I think we need to be honest to the clients on the risks and limitations of the possible scenario's, and help them to make a sound decision.

Need for better standard for integration between BPM/ACM and data-driven applications

In many of the projects that I worked in teams with BPM or ACM (Case Management) technology, we encountered the integration-issues with other data-driven systems.

Typically this would be the result of the following factors:
- The users wanted to perform work on a certain activity, based on a work-item trigger (from a BPM or Case Management solution). They liked the 'inbox' concept (of work to do), sometimes in combination with a Case view.
- As part of that activity, data needed to be retrieved, stored, updated in another existing data-driven application
- The existing application had a fine user-interface (a number of pages dealing with that information), and often exposed through that user-interface were many complex business rules.

The possible paths for the Solution Architecture are not optimal:

Path 1. Manual activity 
Just create a work-item that tells the user to go to the application to perform a certain transaction. Present the right data, if possible, that the user needs to enter/update. Ask the user to report the activity ready when done.

Pro's:
- Very simple, low investment/low risk from a project-scope perspective

Con's:
- No contextual frame (see below)
- Risk that activity is declared 'done', while the transaction was not completed

Path 2. Duplicate the transaction in BPM/ACM, and call a service for the application
Design a user interface (1 or more screens, wizards, etc) for the activity, and implement this in the BPM/ACM solution. Create some type of service for the application, to support the transaction from the BPM/ACM solution (if needed, published on an ESB).

Pro's:
1. Contextual framing of data and functionality
> With a data-driven application, the user needed to assess an activity - what do I need to do - and then translate this to - how do I navigate to the right screens in the application. This takes time and learning, and is not value added (Lean) in the execution of a process. With a BPM/ACM solution, the system 'knows' the context and can present the right contextual frame

2. Uniform user interface

3. Usually better looking than the 'legacy' datadriven app

Con's:
- Duplication of user interface and possibly business rules. See below.
- More complex (development work on two sides)
- Possible transaction faults (requiring roll-backs)

With path 2, a key question is: what are we going to do with the business rules around the data-driven transaction?
A number of scenario's:

2A: All relevant business rules are duplicated in the BPM/ACM layer
Pro's:
- Good user support in the user interface, while working on the transaction (data-checks, lookups, allowed values and value-combinations, etc), reducing or preventing errors or in Lean terms:  Poka Yoke.

Con's:
- Maintaining business rules suddenly requires updates on two places, requiring good business rules governance and risk that business rules will start to differ over time.

2B: Only direct user interface rules duplicated
Only the direct User Interface driven business rules are duplicated, the harder more back end rules are kept in the application, and are used in the service (which may lead to a reject in the service, because certain rules are not met)

Pro's:
- Less business rules to duplicate

Con's:
- Hard to sometimes decide what are 'user interface' rules and back end rues
- A work-item that is completed could lead to a failed transaction, requiring error handling functionality, that will lead to additional work-item loops to the user ('Your work-item lead to errors, please correct'). This is not value added.

2C: No business rules are duplicated
In that case, a rather 'dumb' workitem set of screens are presented. All business rules are checked as part of the service call, on the application side.

Pro's:
- Business rules on one place
- More simple

Con's:
- Can create user confusion, and higher risks that people enter wrong data
- Requires error handling functionality, that will lead to additional work-items loops (not value added, not first-time-right)

Path 3: User interface inclusion
I have actually never seen this in real life (we always dropped it as a viable solution). In this scenario, in some kind of way, the user interface of the application is included in the BPM/ACM user interface. As soon as a user opens a work-item, the appropriate screen from the application is shown (if possible with populated data from the application and the BPM-work item & process instance data). As soon as the user performs a certain triggering activity, either the BPM-system closes the application screen, committing the transaction in the application, or the application detects the transaction-completion and triggers BPM so that the work-item can be reported ready.

Pro's:
- No duplication of screen and business rules
- Contextual framing
- User recognizes the well-known screens of the application

Con's:
- Complex. Most presentation-layer technology has a hard job here
- Triggering to mark the completion (or errorstates) is very complex
- Risk that triggers are lost and application and BPM get out of sync (application has commited data, BPM is not informed)
- User interface variations in the BPM user interface
- Strong coupling

I have seen various implementations for path A and B. But I am never really happy. What would be a great way out, is that the technology vendors come up with a better integration standard: a standard to expose each user interface from their applications (ERP, CRM, etc) as a service, with some type of metadata exchange on functionality, presentation, and business logic/rules. This would allow BPM/ACM systems on design time or run time to present the right user interaction (but in the UI of the BPM/ACM system itself) ,while using all functionality, data and business rules on the application side.

Is JCA supporting this? I am not a Java-technie, but I don't think so. Not to this depth.
Of course, there are still complexities, for example what if the transaction granularity and focus in the back-end application is different from the required transactions from the BPM application?

But a standard in this area would lead to a path 4, that would have a lot of advantages:
- Contextual framing (Value Added)
- Direct checking of business rules (Poka Yoke)
- State consistency (in terms of data, transaction and work-item completion)
- No duplication of business rules and business logic
- Consistent user interface

Key question: are vendors of data-driven applications open for this? Especially vendors that have their own BPM-layers? I doubt it. Old-world thinking Application vendors might think: 'I don't want my application to become some black-box behind a BPM-layer! How will I compete in terms of brand recognition?'.
But perhaps I am too cynical here, and can Application vendors see the value of providing strong building blocks of data/rules centric functionality, that fit in 'plug & play' in modern BPM-based architectures.
We will see...