Friday, October 26, 2007

SOA & BPM? Beware - two-tier solutions might be back...

Lately, I have been seeing a lot of wonderfull features of a BPM technology suite.
Point, click, ah, building applications is so easy. Point to existing web services, pull them in a BPMN flow, create Xforms that talk to web services, no pains!
Really?
Sure, it's easy to create the application, given the current requirements and services.
But what if a service change?

Suddenly I found myself back in early 90's, with 2 tier client server tooling. Oh, what a delight after the green screen software. Widgets, windows, and click click link it to a database table, an SQL query, and tada, the software worked!
Until a requirement changed, we needed to make a small change to the database, and realized 40% of our windows would not work and needed fixing (even though their functionality did not change...).

It's called coupling....

I am assuming that many of these BPM tools have this risk: the risk that we build BPMN and presentation layer stuff, directly connecting to services. Leading to heavy coupling.

The alternative is supported by most tools too, I hope: an intermediate layer, that shields presentation to back-end services. Call it controller or business objects or whatever -
but please use it...

Towards a new TLA: ITPM

Short post, it's late :-)

Interesting article at BPM enterprise about BPM for software development
(link: http://www.bpmenterprise.com/content/c071022a.asp)

What's missing is a key metric: the likelyhood/trackrecord that we deliver on scheduled date.

It's funny - all the effort in various software development methodologies.
In a way it's all (new TLA:) ITPM. IT Process management....

As a BA/BPM specialist, and 15 years of IT development/projectmanagement experience, it often happens that when starting on a new or running project, 20 - 40% of my time is actually spent on process improvement... in the project. E.g. implementing, structuring, refining and improving requirement management process, issue management, testing approach etc. :-).

Is your job also often ITPM?

Monday, October 15, 2007

Towards a new requirements framework

James triggered me again, around Requirements in a BPM context. See http://www.ebizq.net/blogs/decision_management/2007/10/whats_the_difference_between_r.php
I notice in discussions with clients that there is a great deal of confusion about the notion of requirements. I get quite careful when clients start asking "are the requirements done?", "do we have clear requirements"?
Same with terms as "Software requirements" "Specifications" "Business Requirements". When I ask a bit further, the definition, content and context of these terms differ per person, even in the same organisation. If Standish sees 'Clear Requirements" as one of the top-3 CSF's for projects, then it would be quite nice if we all know what we mean by that. How I love clear concepts :-).

Working with a client recently, we discovered together that their change processes were, well, somewhat messy. As change as a process is clearly an element in your BPM strategy: e.g you want to have control over this process as well!, we researched the painpoints in more detail.

One central thema - you guessed it - no shared language around the concept of requirements.

If we don't share the same language on the meta-concepts and the quality criteria around requirements, how would we even start to hope that we understand the requirements in itself?

What we came up with was a high level Requirements Framework, which defines requirement area's. The following diagram is useful:

It defines a framework of types/areas of requirements.

The area's:

1. Business Requirements. As in: requirements you put upon your business.

Strategic Goals & Drivers

These are statements about the intent and plans of the organization. 'Be number 1 in market XYZ, region ABC". "We sell to <target audience> through channels X, Y and Z". "We want to have an operating income of at least XXXX".

Product requirements

All the properties of your products (or services). And the rules of engagement - when are you willing to sell a product? What terms and conditions?

Information requirements

What information do you need when performing your processes? Per process? Per activity? What information do you produce?

How is this information structured? What rules apply (correctness, etc)?

What information is needed to steer your organisation and it's processes?

What information is needed from external stakeholders?

Process requirements

What processes do you need to exist? To serve your customers, deal with suppliers, produce products, steer processes and change processes? What supporting processes?

Which steps in a process? Inputs, outputs, link to goals, etc.

Organisation requirements?

How do you want the organisation to be structured? Which roles? Which authorities? What skill levels?

How many people?

2. Software requirements. As in: requirements you put upon your IT systems?

Functional requirements

WHAT does the application need to be able to do.

Non-functional requirements

Think URPS. Number of users to support, performance, etc.

Architecture requirements

Often put in the same area as non-functionals, but in fact very different. These are requirements from another stakeholder group, and need another area: what (mostly longterm) structures, practices, and technology standards do you require the application to comply with.

A number of notes:

- Each area has it's own value and it's own techniques & best practices to understand, formulate and document the requirements in the area.

- Areas are interrelated. Actually, most interesting requirements you will find on the borders: when performing process A to sell product B with information do we need when in step C. When performing process B, what functionality is required from the IT system?

- In each area you will find hierarchy in terms of detail. In process, for instance, a high level services overview, then drill down in process architecture, process activity flows, etc.

- Functional Requirements is the area that gets most attention. Typical techniques to document requirements in this area are "System Should .....", use cases, business rules documents, etc.

- Note that business rules are types of requirements you will find in more area's. In information: an customer can order multiple products in one order... In process: if insured amount > XYZ manual approval by role ABC is needed. Etc.

- You could call this complete framework "business architecture". I am always a bit carefull with "architecture", since that's an even vaguer concept. My objectives are to have a framework that evryone in a company can understand (and not some ivory tower group).

- The pyramid is also a great tool for knowledge management. If you work somewhere, these are the area's you should understand, to know your own role and added value!


Observation:

I think that methodologies that play a part in changing a business, including software development life cycle methods such as RUP, SDM, etc. should be innovated to methods that understand that requirements can be defined on multiple levels (e.g. system - as in organization and in software system) and that are able to integrate into these broader frameworks. This to prevent the earlier blogged about "task oriented applications" (see http://process-transformation.blogspot.com/2007/10/need-for-new-requirements-approach-and.html).

Saturday, October 13, 2007

Requirements vs Process and Rules

James Taylor reacted to a posting about the importance of good Requirements. He sees process as something else than requirements. And he advocates a clear seperation of the concept of Rule and Requirement.
See http://www.ebizq.net/blogs/decision_management/2007/10/requirements_business_process.php

First on the process part vs requirements.
I see a process as a specific concept in requirements. It can be requirements towards a business:
- This is the way I want our people to perform this process (a business requirement)
Or it can form part of the requirements to a system (a software requirement)
- This is the process I want the system to follow when processing claims.
Or a mix of course.
So, is a process definition a software requirement? Sometimes.

Then the part of the rules: again, i don't fully agree.

First of all - YES. When we are trying to understand how an AS-IS situation is working or a TO-BE situation should be working, we can use various concepts in analysis. What processes do you have? What information flow, inputs, output? What do you do in an activity? And, what rules apply for decisions - why do you refuse this order, how do you calculate that price, why do you decide to escalate this?
Rules analysis are a great tool for understanding. And during requirements analysis, I typically seperate them out of the other requirements, simply because it's a separate set of things/concepts.
But are rules requirements or not? I think rules are, for a start, always a business requirement: "when we are doing business, I want our people to follow this rule".
And, in my opinion, when a rule will get automated, we will need a requirement defining the rule. The rule becomes a software requirement: we want to IT system to perform CHECK XYZ, and based on the results do A or B or C.

Now - before I continue, my requirement constraint: a requirement is part of the WHAT definition of a desired system, and does not define the solution.

I have the suspicion that people are separating rules out of the requirements, because they have the solution already in the back of their minds: a rules engine or BRMS.
This links back to my earlier posting on "pick the right technology for your functionality".
Rules can be put in a BRMS. Or programmed in Java. Or in Cobol. Or asked to a user (continue Y/N?). It depends on the requirements...mainly the non-functional ones.

Are the requirements that define this rules highly violatile? Or should they be maintainable by non-programmers? Well, don't put them deeeeep in code in that case.
Are rules quite fixed? Do you only have Java programmers and no money for a BRMS? Does the business delegate all IT rule keeping to the IT people? Go ahead, bury them in code (but do make sure you defined them as a seperate part in your requirements, and make them traceabile in your code...!)

/**=== Check business Rule XYZ , traceability: BR23.22 =======
if (bla.blo(var1, var2, var3)
{.....}
else
{....}

/** end check BR23.22

Conclusion: each technology has it's business case. BPMS and BRMS as well.

The right technology for the right functionality

A number of posts reminded me about a discussion I had some years ago.

Situation: we were introducing advanced output management technology (e.g. a tool + programming language to create applications that were able to receive XML and generate output (PDF's, prints, etc)).
This technology had a number of advantages:
- Quicker time to market
- Powerfull language
- Easier testing (what you see is what you get programming environment)
- More possibilities in layout

The objective was to reduce the code base in the legacy area considerably. In those huge Cobol programs, many were programmed to generate output (line print, so very '70-ies output). Of course, whenever output needed to change - a new product, new rules, a new contract, quote, etc, Cobol programmers would need to start hacking. They did that for years and saw no issue with that. But the business did - waiting for 2 - 3 months of implementation time, a lot of testing, and that 70-ies layout.
The Cobol group had a lot of resistance towards the new technology. One argument was: well, now we can programm the WHOLE application in one language, but with this technology, we will need multiple skills in multiple languages.
Yep. And I explained that we don't use a hammer to crush coffeebeans to make coffee. We use a grinder. And the hammer we use for nails.

The lesson: when requirements ask for a certain solution, pick the right technology for it.

The same discussion I recognize in a number of postings on BPM (Why Java developers hate BPM tools):
http://weblogs.java.net/blog/johnreynolds/archive/2007/10/why_do_java_dev_1.html
http://www.oreillynet.com/onjava/blog/2007/10/do_i_hate_bpm_no_i_hate_bpm_pr.html?CMP=OTC-FP2116136014&ATT=Do+I+hate+BPM+No+I+hate+BPM+Products

The key question is again: what are the requirements, and what is the best technology to solve something. I am sure most BPM featurs you can easily develop in Java. Hell - you can develop a database with Java, but big chance is you don't....

The key advantages with using BPM technology:
- You can change a process quickly
- The process diagram reads (most of the time) quicker than 10 pages of a programming languages, heck - maybe some people on the business side even understand!
- The whole SOA calling is hidden - it helps you to focus on the process and information flow instead of Java and XML
- Process and activities are measured for you, in a standard way.

Case management - The empowered information worker...

Recently I was at the Cordys Cordial (http://www.cordys.com/) day, a day were they presented the new version of their platform.

A funny statement by (I think John Pyke) was: processes are created when something failed. :-) - now, that's a nice start of BPM... And John also nagged towards SAP, which as an abbreviation would stand for Stone Age Processing or Spent All Profit. Hm.

A good presentation was on the new case management features of the Cordys platform.
These features were added to the platform when they discovered (while doing a project with my company) that certain processes are not that straight forward as hoped for. Aka - forget modelling it in BPMN, the diagram would be too complex and add no value. I sometimes wonder how many processes actually are really that structured, but that's another post...

A small summary on the case management features and thoughts behind Cordys:
- Where in production workflow, each process instance has a certain flow, each case will have a unique sequence of activities
- When a case is started (based on some event), certain activities are automatically opened, based on the type of the case and certain event criteria.
- A user is presented with activities, can open them (role bases), execute them, delegate them or create new activities (select or freely define).
- Status of cases and activities can be tracked, measured (monitoring)
- Three main components were presented:
+ Document matching - used to trigger creation of a case, or to be able to link an incoming document to a running case. Remark: strange - I would not have called this a document matching system. Incoming documents (and other information flows) basically define EVENTS. So - it's a CEP or Event Correlation Engine....
+ Case management - the actual presentation towards the user of all case activities, history, data, etc.
+ activity management - the ability to execute, define, delegate, etc activities.
- Another component is the maintenance area - in which case rules and activities can be defined, including roles etc.

They had a nice slide which defined a sliding scale between pure workflow and pure case management.
- 100% production workflow
- production workflow with small cases
- Flow + case
- Case management, with small flows
- Pure case management
I liked it a lot - since I do think you will encounter many processes where things are mixed. For instance a straight through processing unit, which asks for case management when exceptions occur.

Some remarks:
- Unfortunately, the "activities" are NOT integrated in the current BPM engine. E.g. when I create flows in the BPM engine, I can not mark them as optional or mandatory sets of activities in the case management component. Ideally, in the case environment, as a user (or automatic triggers by certain rules) I want to be able to pick: a listed activity, a free activity, a process fragment or a transaction. This also meant that in the current Cordys setup - the BPM inbox and the case management module are not integrated. Ouch!
- Cordys talked a lot about "the ability to measure status of all cases, allowing you to manage the workforce". Well - since Case flow can not be predicted, in my opinion, workforce management will be difficult. Sure I can see what activities are open NOW, but I have no clarity on the expected activities for these cases, and the required resources...
- What I see as a challenge is case data. When you are doing production workflow, most process instance data that you will need, is clear on design time. Each activity and decision needs certain data - and you will need to make sure it's available in the process instance. But what if you are working with cases, where you do not exactly know what information is needed to supply the knowledge worker with enough information to decide on activities and case closure.
- Another item is where to store this data? In the Cordys solution, all data is stored in unstructured documents. But what to do with all data that is stored in backend systems, which users maybe need to decide? Store in the case management system, or call/get every time the case is opened and viewed?

The demo showed a nice presentation layer which showed the case in full context. It allowed a user to understand the trigger, all done activities, notes, etc. A true process workbench.
The main lesson I started realizing, seeing the demo, was that whatever the type of process support a knowledge worker gets with his or her BPM application - the best would be to provide him or her with this full context. Don't make the mistake of underestimating your users and only give them the microscopic task view (aka inbox with single task - do this, no questions asked), without a clue about the full process!
Empowerment and process ownership are things that should be the basis for process workers..

Friday, October 12, 2007

BPM technology consolidation...

Another one bites the dust...
http://www.ebizq.net/news/8554.html

Oracle to buy BEA.

Hm, interesting. Definitely a step towards further consilidation.
How many process engines will Oracle have now???

Tuesday, October 09, 2007

The need for a new requirements approach - and BPM's role

Ok, it has happened again. Fortunately, it doesn't happen often, but from time to time, I find myself in the unlucky projectworld of "Type 2" , described by the Standish Chaos research as "Over time, Over budget and Less Functionality". Ouch.

Remember the Standish Chaos Report? Also see for instance http://www.projectsmart.co.uk/docs/chaos-report.pdf

It lists succesrates for projects, based on research. And it lists succes and failure factors.
It's findings (So, I am part of the majority with my project :-):
- 16.2% of all projects are Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as initially specified.
- 52.7% - Resolution Type 2, or project challenged: The project is completed and operational but
over-budget, over the time estimate, and offers fewer features and functions than
originally specified.
- 31.1% - Resolution Type 3, or project impaired: The project is cancelled at some point during the development cycle.

The failure factors for projects ending up as a 2 or 3:

The survey participants were also asked about the factors that cause projects to be challenged.
1. Lack of User Input 12.8%
2. Incomplete Requirements & Specifications 12.3%
3. Changing Requirements & Specifications 11.8%

And yes - in this project, for all kinds of reaons, these items are true.

It wasn't me, though, I was not working on the requirement side ;-). But a repeating lesson I learned: good requirements are essential. And in my opinion that needs process-context awareness.

Remember the old days of applicatie analysis and design? I call it feature oriented requirements engineering: talk to users, find out what's difficult in their job, and think up a set of features to cover their issues.
Example: we do sales, but miss information on our clients and status of orders.
Ah! We prototype a nice screen, build you a nice database with clients, and the order status we get through EAI from your order entry system. Problem solved!
Really? No.
You end up with a so-called data & isolated task oriented application. It's not aware of the process.
To be honest, as IT requirements person, I used to like this. Process was difficult, all those exceptions, terminology, discussions, yucky. Features where clear - a screen here, a database there, a business rule there, data and task oriented.

But in these days of evry growing complexity, task and data oriented requirements analysis will not do anymore. Why? Because it leads to suboptimal solutions, which force people to work AROUND an application, to perform the process. (And ask a random user how many systems he or she typically needs to use to execute a front to back process - usually a lot...)

Premises: EVERY application we use, we use in the context of a certain process. So, by carefully analysing the process, we can get to better requirements, more project succes (standish) and much better solutions.

There is a problem though. Most of the current Requirements frameworks are still in the Data/Task oriented mode. Think of...
- Feature driven development
- XP's User stories
- RUP's Use cases
All these items are focused on isolated functionality, that brings value to a user, in a logical transaction. Process is much more than that! So, how do we capture this.
Sure, some methods call for "business modelling", but most of these elements are, to say the least, not very developed.

BPM would be a great vehicle to improve this.
It asks us from an analysis perspective to:
- SCOPE: Define the scope for a to-be process area and supporting IT solution
- GET CONTEXT: Analyse the processes, front to back, including trigger, input, output, steps, information required in a step, data produced in a step, data flow, actors, KPI's, etc
- DEFINE HIGH LEVEL REQUIREMENTS And then dive deeper: per step, we can start thinking about the required IT support.

From this step, we can revert back to for instance use cases - to define the functionality a user needs to complete a logical process step. (So: a use case is not the same thing as a business process!)

My lesson: BPM is there for many reasons. And one of them is to help you improve your understanding of the business context and get to better requirements. Add it to the more IT driven methods, and you have a better chance of getting to project success.