The Road Towards a Kogito Public API

In the previous post we showed how to use our fancy new Public API, but I threatened promised you we would get back to the design rationale. So, here we are.
This blog post is rather long, so I decided to break down into two parts. It is loosely based on the presentation I gave during my KIE live, but I am adding some details about DataContexts that I did not include during that live stream. 
 
However, if you would rather sit though me blabbering about API design, please do jump to our channel. Otherwise read on.

Seven Red Lines

A long, long time ago, Tiago Dolphine and I were tasked with the goal of designing a public API for our platform. The requirements were easy enough; the API should have been:

  • consistent across all components
  • extensible with new components while retaining consistency
  • flexible in the way components can define their own methods, because no component is totally equal to the others

My immediate reaction was along the lines of:

I don’t know how familiar you are with that video where the expert in red-line-drawing pictured above is being asked to draw “seven red lines, all strictly perpendicular, some with green ink and some with transparent”. But that is kind of like how the requirements above felt at first: that is, very contrasting with each other.

Moreover, ourselves, we wanted to go further and design something that scaled and allowed for multiple types of interactions.

In fact, KIE 7 APIs are designed for a synchronous, Java-based, local workflow.

We wanted the new design to scale even in an asynchronous, polyglot, distributed setting. But in order do that, the API surface should have been minimal and account for communication over the wire.

An Emerging Design

As we recovered from our confusion, we decided that the best strategy to deal with this was to first observe whatever we did so far, and look for patterns. Then evolve those patterns to account for the new use cases that we had in mind.

After all, the code-generation procedure that in Kogito automatically materializes REST endpoints from business assets uses an internal Java API that evolved from our internal requirements.

This API was not completely designed, rather for a large part it just evolved from emerging requirements. The reason why we never called it public is that we wanted to take our time to make it cohesive and consistent.

That time had finally come.

We started by observing how REST endpoints created and interacted with sessions, process instances, DMN models, PMML models.

// Processes
application.
    .get(Processes.class)
    .processById(processId)
.createInstance(...)
.start()

// Rule Units
application.
    .get(RuleUnits.class)
    .create(Class<?>)
.createInstance(...)
.fire()

// PMML
application.
    .get(PredictionModels.class)
    .getPredictionModel(modelName)
.evaluateAll(...)
...

// DMN
application.
    .get(DecisionModels.class)
    .getDecisionModel(modelName)
.evaluateAll(...)
...

As you can see, all these API look different, but not wildly different. So a common pattern was emerging. In general we can see how the API is concerned with getting to a named resource, and then invoking a method over it.

app.get(ModelType.class)
   .getResourceById($identifier)
   .action(values) 

In some cases, before you get to invoke the action() you may need to navigate further to a sub-component within that resource, e.g.:

app.get(ModelType.class)
   .getResourceById($identifier)
   .instances()
   .getInstanceId($instance_id)
   .action(values) 

or:

app.get(ModelType.class)
   .getResourceById($identifier)
   .subComponent()
   .getSubComponentId($instance_id)
   .action(values) 

for instance:

application.
    .get(Processes.class)
    .processById(processId)
    .getInstance(instanceId)
    .getWorkItem(name)
    .update(values)

The method is usually is a “verb” indicating an action to apply over the resource, with a payload that represents the data context over which the action should be applied

Observations

All these method calls have something in common. The breakthrough came when we came to the following 3 realizations:

  1. The fluent API denotes a path
  2. “Verbs” are always at the end of the method chain
  3. Each method is applied to at most one “payload”

Let’s see them in detail.

Observation 1: The fluent API denotes a path

Imagine that we want to invoke a method (a “command”) over a process, or a process instance; or over a decision model, or over a prediction model; or over a rule unit. In order to identify these resources, we always have to pass an identifier.

The identifier is constructed differently for each type of resource, and usually it is not self-contained; i.e. by the identifier alone, it is hard to predict what kind of resource it would point to; for instance, is my.resource the name of a process? Is it the name of a PMML? A DMN maybe?

If you were sent this single string over the wire, how would you decode it? Well, you could create a structured object, instead:

class ProcessInstanceId {
  ProcessId parentProcess;
  String instanceName;
}
class ProcessId {
  String processName;
}

But then, would you represent the type of the object? How would normalize and serialize this into a String? How can you make sure that this String is representable as an object not just in Java, but making sure that is consumable by a client, written, say, in Python?

And here comes the first observation. Consider, for instance:

app.get(Processes.class)
   .get($process_id)
   .instances()
   .get($instance_id)

/processes/$process_id/instances/$instance_id

This method invocation returns an object of type Process<?> over which you will invoke a command, for instance start(variables). Each method call may fail, because $process_id may not exist, it may not have child instances, or the instance $instance_id does not belong to that process.

This eager evaluation of the method chain makes it hard to design an asynchronous API around it. In a trivial remote implementation, each method invocation would correspond to a remote call.

However, if we look at the method chain, we notice that each invocation does not really perform an action, but rather retrieves an object, that retrieves another object, that retrieves yet another object… until the last item on the chain is reached, which is usually an action (a “verb”) like start() or evaluate()

This means that everything that comes before the action is really a path. And a Path is trivially serializable as a string:

/processes/$process_id/instances/$instance_id

This is the reason we introduced the AppRoot object, which provides a fluent, navigable, typed API to a resource, while still retaining the ability to convert into a string, and parseable back into a typed object. 

The validation of such a path can be delayed until the entire path has been formed, and delegated to the action itself. Which brings us to the second observation

Observation 2: “Verbs” are always at the end of the method chain

If the fluent API denotes a Path, we can delay validation and defer it when a command is invoked, . For example, if we don’t check if a process instance identifier exists until we try to start() it, then the start() command may perform that check..

app.get(Processes.class)
   .get($process_id)
   .instances()
   .get($instance_id)
   .start()

In other words, “real” commands are always at the end of a chain of method calls; the method chain only serves the purpose of “finding” the right object instance. In fact, the relation between “commands” and the method chain that precedes them may become clearer if we rewrite it as follows:

abort(app.get(Processes.class)
   .get($process_id)
   .instances()
   .get($instance_id))

or even:

ProcessInstance pi = app.get(Processes.class)
   .get($process_id)
   .instances()
   .get($instance_id);

start(pi, variables)

This apparently small realization has a deep effect: it means that we can separate the definition of the commands from the way we address a process or a process instance.

  • It means that we can identify commands by their name
  • It also means that we may have multiple implementation of such commands

In other words, the following text conveys about the same amount of information as the method chain above:

START   /processes/$process_id/instances/$instance_id    variables

And this makes it representable using pretty much any programming language.

The text above can be seen as a simple specification of one of the “commands” that our system accepts. It includes information about

  1. /processes the type of resource
  2. /$process_id the identifier of that resource
  3. /instances a type sub-component of that resource
  4. /$instance_id the identifier of that sub-component 

a collection of such rows 

COMMAND        /TYPE/$MAIN_ID/$SUB-PATH        PAYLOAD 

forms the description of a service. For instance a service for a DMN engine may be:

EVALUATE    /decisions/$decision_id   evaluation-context

It should be easy to see how this easily map to method calls, REST service requests, or even messages. 

In general, this is the reason why we introduced services; i.e., interfaces that describe the entire API surface of an engine, and multiple interfaces and implementations can be provided for the same engine (for instance a synchronous vs an asynchronous process service).

For instance:

interface DecisionService {
DataContext evaluate(Id identifier, DataContext ctx);
}

but also, possibly:

interface AsyncDecisionService {
Future<DataContext> evaluate(Id identifier, DataContext ctx);
}

In the next part of this post we will see this principle more in detail.

Observation 3: each method is applied to at most one payload (and possibly extra metadata)

Each method is generally applied to an object that sometimes is called a model or variables (processes), sometimes data (RuleUnitData, for rules), sometimes context (DMN and PMML). More generally that is the payload of the request  

The object is generally used to extract values or to send values through a command.

application.
    .get(Processes.class)
    .processById(processId)
    .getInstanceById(instanceId)
    .updateVariables(model)

Now, a frequent pattern for the model object is to convert that object into another type. For instance, it is frequent to project variables of a project to show only “input” variables and “output” variables. In other cases, it may be useful to extract the computation results as a Map and then convert that Map to another type.

In any case, if we consider a distributed, polyglot execution environment, tying the representation of the data that is manipulated through an engine feels very restrictive.

This is why we introduced the DataContext interface, shared across implementations, and requiring the simple contract that a DataContext should be convertible into another type of DataContext through some mapping mechanism.

Conclusion

In this post, I gave an overview of the principles and reasoning that were put into the design of the new public API.

In the next post, I will show what these principles may enable in the future.

In particular we will see how using paths for identifiers makes it easier to adopt it in multiple situations (such as data binding), that in turn enables a lot of convenient use cases. We will also see how the simple service-based API definition scheme may be useful in the future to enable service discovery and distributed execution. 

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments