Thursday, February 26, 2009

A new form of business rule mining and validation?

I am fully emerged in a business rules automation project, as a process consultant for rules governance.

Last week I received training in the business rules management suite that is used within the project (Be Informed, a promising dutch player, see their website).
And it brought me back years ago, when I was programming (during my time @ university) in Prolog.
Part of my Prolog assignment back then, was to implement an algorithm (from Quinlan) from the Artificial Intelligence - Machine learning domain (see for instance http://en.wikipedia.org/wiki/Supervised_learning).

It worked basically as follows:

You gave it a set of examples, in which each example contained a set of variables and their data, and an associated classification.
For instance:
(Color=Red, Tiers=True, NrOfTiers=4, Engine=True, Sail=False, CanBeInWater=False) -> Car
(Color=Green, Tiers=True, NrofTiers=2, Engine=False, Sail=False, CanBeInWater=False) -> Bike
(Color=White, Tiers=False, NrOfTiers=n/a, Engine=False, Sail=True, CanBeInWater=True) -> Sailingboat
Etc.

When you ran the algorithm, it was able to analyse the examples and classifications, and come up with a minimum set of rules that matched the key determining variables and classification, e.g.:
(Tiers=True, NrOfTiers=4, Engine=True) -> Car
(Tiers=True, NrOfTiers=2, Engine=False) -> Bike

Back to the project....
Considerable time is currently spent in the project for business rule elicitation. Workshops, study of systems, analysing documents, analysing legislation, etc.
In a recent project someone in my company worked with Process Mining, in which logfiles were analysed, and based on the logfile a process diagram was constructed (+ all kinds of performance data).

Adding algorithm, effort and the concept of process mining, made me look up Rule Mining (see http://en.wikipedia.org/wiki/Business_rule_mining). Strangly enough this differs from the concept of Process mining (it does not analyse data, but code).

And this triggered the following "Wouldn't it be cool" idea:

What if we constructed a strong machine learning algorithm component in a business rules management suite, that was able to analyse great sets of (historic) production data and construct the business rule set that was apparently used?

Example: we have a large CRM database and a Billing system. We create a combined table:
(Customer data), (Order data), (Billing Data) and select certain variables that we want to explain, using business rules based on the remaining variables.
For instance: what are the rules for VAT, discount and Gold membership?

The algorithm would do it's work, and create the minimal rule set...
For instance:
(Country=Netherlands) -> (VAT=19%)
(Orderamount > 200, OrderAmount <300, country =" Germany," customerindustry =" Bank)"> (Discount = 5%)

I see a number of benefits:
- Saving time for rules elicitation
- Retroactively understand what rules where used, and compare them to the existing policies (compliance checks)

Do any of the current BRMS have this feature already? (Time to patent ;-))

For a great overview of machine learning basics:
http://www.informatica.si/PDF/31-3/11_Kotsiantis%20-%20Supervised%20Machine%20Learning%20-%20A%20Review%20of...pdf

2 comments:

James Taylor said...

There is some active research in this area. Indeed the extraction of rules using Rule Induction is pretty common as is the use of various decision tree techniques - decision trees, of course, being a ruleset of a particular type.
JT

Paul Vincent said...

Check out classification algorithms like Quinlan and ID3. Some tools in the past used these for particular rule engines, but the signal:noise was never 100%, and the training data never 100% correct, so end-users generally decided it was quicker to write the rules explicitly, from scratch.

More recently, the advent of increased data (warehouses) has led to a resurgence in Machine Learning ideas: e.g. optimization technologies such as predictive analytics.

[Disclosure: TIBCO Spotfire S+ Miner is one such tool, which has the potential to generate rules for the TIBCO BusinessEvents rule engine...]

Cheers