When I was studying my degree, I recall a wise teacher that repeatedly told us, their beloved pupils, that the most difficult part of a document is the beginning, and I can assure you, dear reader, that he was right, because I cannot figure out a better way to start my first entry on aRead more →

When I was studying my degree, I recall a wise teacher that repeatedly told us, their beloved pupils, that the most difficult part of a document is the beginning, and I can assure you, dear reader, that he was right, because I cannot figure out a better way to start my first entry on a RedHat blog than explaining the title I have chosen for it. The title refers to two entities that in BPMN 7.48.x release has been brought together, hopefully for good: BPMN messages and Kafka.

According to the not always revered BPMN specification, messages “represents the content of a communication between two Participants”, or, as interpreted by JBPM, a message is an object (which in this context means a Java object, either a “primitive” one like Number or String, or an user defined POJO) that is either being received by a process (participant #1) from external world (participant #2) or, obeying the rules of symmetry, being sent from a process to the external world.

Kafka, not to be mistaken with the famous existentialist writer from Prague, is an “event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications”. In more plain language, the middleware that is becoming the facto standard for inter process asynchronous communication in the software business applications world. If still not clear to you, try to think of Kafka as a modern replacement for your old JMS or equivalent message broker pal, but please do not tell anyone I have written that, because as Kafka designers proudly manifest, their fancy toy is doing so much more than that.

As other messaging brokers, Kafka uses a set of channels, called topics, to organize the data being processed. This data consist on a set of persistent records, each of them composed by an optional key and a meaningful value. Most Kafka use cases consist on reading and/or writing records from a set of topics and, as you will have guessed by now, JBPM is not the exception to that rule, so the purpose of this entry is to explain how KIE server sends and receives BPMN messages from and to Kafka broker.

With such spirit, let me briefly explain the functionality that has been implemented . When Kafka support for messages is enabled, for any KIE jar (remember, a KIE Jar contains a set of BPMN process and related artifacts needed to make them work) that is being deployed into a KIE server (because Kafka integration is a KIE server feature, not an JBPM engine one), if any of the process being deployed contains a message definition, depending on the nodes where that message is used, different interactions with Kafka broker will occur.

If the nodes using the message are Start, IntermediateCatch or Boundary events, a subscription to a Kafka topic will be attempted at deployment time. When a Kafka record containing a value with the expected format is received on that topic, the JBPM engine is notified and acts accordingly, so either a new process instance is started or already started processes are resumed (depending on the incumbent node being an Start or an Intermediate Catch respectively)

However, if the nodes using the message are End or IntermediateThrow events, then a process event listener is automatically registered at deployment time, so when a process instance, as part of its execution, reach one of these nodes, a Kafka record containing the message object will be published to a Kafka topic.

Examples

Once description of the functionality has been concluded, let’s illustrate how it really works with a couple of processes. In the first one, a process instance will be started by sending a message from a Kafka Broker. In the second one, a message object containing a POJO will be published into a Kafka Broker when the process is ended.

First example just consist of a start message event that receives the message object, an script task which prints that message object and the end node

The start event node, besides receiving the message named "HelloMessage", assigns the message object to a property named "x", of type com.javierito.Person. A person has a name and an age.

Scrip task just prints the content of "x" in console to verify the message has been correctly received using Java code (the output of toString method).

When this process is deployed to KIE server and Kafka extension is enabled, if we publish {"data":{"name":"Real Betis Balompie","age":113}} on Kafka topic "HelloMessage", then Received event is Person [name=Real Betis Balompie, age=113] is printed in KIE server console.

Second example diagram is even more straightforward than the previous one, it just contains two nodes: start and end message event

In order to fill the message object to be sent, an input assignment is defined to set message object value from property "person", of type com.javierito.Person

And that’s all, when an instance of this process is executed, passing as "person" property a Person instance which name is Real Betis Balompie and age is 113, a cloud event json object {.... "data":{"name":"Real Betis Balompie", "age":113},"type":"com.javierito.Person" ...} is sent to Kafka topic "personMessage"

Hopefully these two simple examples will give you a basic idea of what kind of functionality can be achieved when integrating BPMN messages with Kafka. In next section, you can find a FAQ where certain technical details are discussed

FAQ

• How is Kafka functionality enabled? Kafka functionality is provided at Kie server level. As an optional feature, it is disabled by default and it is implemented using an already existing functionality called Kie Server extension. In order to enable it:
  • For EAP deployments, set system property org.kie.kafka.server.ext.disabled to false
  • In Spring Boot applications, add kieserver.kafka.enabled=true to application properties.
• Why Kafka functionality was not included as part of JBPM engine? Because JBPM engine must not have dependencies with external processes. Kafka broker, as sophisticated as it is, consists on at least one (typically more) external process, which due to its distributed nature relies on Zookeeper, which gives a minimum of two external processes.
• How BPMN knows which Kafka topics should be used? In a nutshell, using message name. More specifically, if no additional configuration is provided, the message name will be assumed to be the topic name. In order to provide a different mapping, system properties must be used for now ( an ongoing discussion regarding the possibility of providing mapping between message and topic in the process itself is happening while I wrote these lines) . The format of these system properties is org.kie.server.jbpm-kafka.ext.topics.<messageName>=<topicName>. So, if you want to map message name “RealBetisBalompie” to topic name “BestFootballClubEver”, you will need to add following system property to Kie Server: org.kie.server.jbpm-kafka.ext.RealBetisBalompie=BestFootballClubEver.
• Why a WorkItemHandlerNotFoundException is getting thrown in my environment when the message node is executed? JBPM has been out for a while and any new functionality needs to keep backward compatibility. Before this feature for Kafka was added to JBPM, when a process sends a message, a WorkItem named “Send Task” is executed. This behavior is still active, which means that in order to avoid the exception, a WorkItemHandler implementation for “send task” needs to be registered . The steps to register a work item handler are described here. If just Kafka functionality is needed, this handler might be a custom one that does nothing (implemented methods will be empty). Keeping this legacy functionality allows both JMS (through registering the proper JMS WorkItemHandler) and Kafka (through enabling the Kie extension) to naturally coexist in the same KIE server instance.
• Which is the expected format for Kafka record value to be consumed by JBPM? Currently, JBPM expects a JSON object that honors cloud event specification (although only “data” field is currently used) and which “data” field contains a JSON object that can be mapped to the Java object optionally defined in structureRef attribute of MessageDefinition. If no such object is defined the java object generated from “data” field will be a java.util.Map. If there is any problem during the parsing procedure, the Kafka record will be ignored. In future we are planning to support also plain JSON objects (not embedded in “data” field) and customer customization of the parsing procedure (so value can contain any format and customer will be able to write custom code that converts its bytes to the java object defined in structureRef)

In the previous blog post Flexible processes in Kogito – Part 1 I talked about the new components and functionalities for flexible processes introduced in Kogito 0.12.0. In this post I will walk you through an example process putting all this in practice. Service Desk process The flexible-process-quarkus example is available in the kogito-examples GitHubRead more →

In the previous blog post Flexible processes in Kogito – Part 1 I talked about the new components and functionalities for flexible processes introduced in Kogito 0.12.0.

In this post I will walk you through an example process putting all this in practice.

Service Desk process

The flexible-process-quarkus example is available in the kogito-examples GitHub repository. In the project README.md file you can find instructions to build and execute the process.

It describes a service desk process where customers open a ticket related to a problem or question about a specific product. 

First, the ticket goes through a triage process where it is assigned to a support team and then a random engineer will be designated. If the system can’t decide which team has to handle the ticket, the support engineer will be assigned manually.

At any moment both, the engineer and the customer, can add comments until any part is satisfied and the case can be considered as Resolved. If for any reason, someone decides to add another comment, the case will be reopened and assigned to the other party until it is resolved again.

Finally, once resolved, the customer will have a task to send a satisfaction questionnaire considering the case as closed.

Overview

This is a diagram of the service desk process:

Create a support case

In order to start the process, an initial supportCase object has to be provided in the request. It will contain information about the product, the customer and a description of the problem itself.

{ 
  "supportCase": {   
    "customer": "Paco the customer",   
    "description": "Kogito is not working for some reason.",   
    "product": {     
      "family": "Middleware",     
      "name": "Kogito"   
    } 
  }
}

The triage

The Triage sub-process is automatically started when the process is instantiated and the support group is decided through the following decision table that takes as the entry the product name and family and returns the supportGroup.

You can see that for the case where the product family is not in the list. The engineer will have to be assigned manually.

Adding comments

The Work case sub-process is also auto-started. Anyone from the customer group can add comments and the same for people from the support group.

As mentioned above, an empty POST request has to be sent in order to create a task that can be used to add a comment. Imagine it as the “Add comment” button that renders the form next.

So, after sending this empty POST you will see something like this:

curl -D - -XPOST -H 'Content-Type:application/json' -H 'Accept:application/json' http://localhost:8080/serviceDesk/b3c75b24-2691-4a76-902c-c9bc29ea076c/ReceiveSupportComment

HTTP/1.1 200 OK
Content-Length: 305
Link: </b3c75b24-2691-4a76-902c-c9bc29ea076c/ReceiveSupportComment/f3b36cf9-3953-43ae-afe6-2a48fea8a79a>; rel='instance'
Content-Type: application/json

{
  "id":"b3c75b24-2691-4a76-902c-c9bc29ea076c",
  "supportCase":{
    "product": {
      "name":"Kogito",
      "family":"Middleware"
    },
    "description":"Kogito is not working for some reason.",
    "engineer":"kelly",
    "customer":"Paco the customer",
    "state":"WAITING_FOR_OWNER",
    "comments":null,
    "questionnaire":null
  },
  "supportGroup":"Kogito"
}

The URL present in the Link HTTP header can be used when rendering the form action. Don’t forget to add the user and the group query parameters. Note that the data sent must be in JSON format. This example assumes the Javascript framework will deal with that.

<form action=”/serviceDesk/b3c75b24-2691-4a76-902c-c9bc29ea076c/ReceiveSupportComment/f3b36cf9-3953-43ae-afe6-2a48fea8a79a?user=kelly&group=support” method=”post”>
  <label for="comment">Comment:</label>
  <input type="text" id="comment" name="comment">
</form>

Case resolved

Let’s say the customer is happy with the resolution provided by the support engineer. The customer decides to click the “Resolve Case” button. 

This button sends an empty POST which triggers the service task. The case is then set as Resolved and a “CaseResolved” event is emitted.

Closing the case

The event will signal the milestone which doesn’t have any Condition. Then, the Questionnaire task is started and assigned to the customer.

The task expects a comment and a numeric evaluation based on the customer’s satisfaction. As a result, the questionnaire is added to the support case and the case is finally closed.

What are flexible processes you might be wondering about? If you are familiar with Case Management you will get it quickly. If not, I suggest you start here. As a brief introduction. In Case Management there is the concept of Case that is opened. Then the knowledge workers will perform tasks or update the Case-relatedRead more →

What are flexible processes you might be wondering about? If you are familiar with Case Management you will get it quickly. If not, I suggest you start here.

As a brief introduction. In Case Management there is the concept of Case that is opened. Then the knowledge workers will perform tasks or update the Case-related information, known as Case File. At a certain point the Case can be considered as closed. Cases might eventually be reopened, depending on the situation.

Case Management is a way to provide traditional processes with more flexibility because the execution path is not rigid. This is an advantage when the process can’t be fully automated or the scope is hard to measure. A typical example is related to healthcare. A patient has a file that is accessible to the doctor and any health professionals assigned to the patient. They take actions, make different decisions and assessments until the patient is diagnosed and treated.

Instead of implementing Case Management as defined in the CMMN specification we decided to use BPMN extensions to provide a similar experience. This is what we call Flexible processes in order to avoid false expectations and misunderstandings.

What’s new?

Ad-Hoc process

Starting with the processes themselves, the first thing to do is to say that your process is an ad-hoc process in the modeler. For that, open the process properties and check the Ad-Hoc flag. This flag tells the modeler that you are defining a flexible process that might or might not have a start node.

Ad-Hoc sub-processes

An Ad-Hoc sub-process is a way to wrap related tasks together. Consider it as the equivalent to Stages in Case Management.

This type of sub-processes have some interesting features like the Activation and Completion Conditions. Activation Conditions control when the tasks within a node can start their execution. On the other hand, Completion Conditions decide when is time to move on to the next node. The same applies to the completion condition after the execution of all the nodes. The sub-process node will wait for the right state to complete and continue to the next node, if any.

These conditions are defined as Java lambdas and they will be re-evaluated each time a process variable is updated.

Milestones

This is a concept directly taken from Case Management. Although the same functionality can be achieved through different BPMN components we considered some users might find it simpler to understand and to apply.

Milestones are a node that can be automatically started by an event with the same name of the node

Additionally, they can have an optional “Condition” input assignment. Use a Java expression to decide when this node is ready to be completed and move forward to the next one.

Imagine the example about the patient. Possible milestones could be “Patient diagnosed” or “Patient recovered”. Each milestone implies that a specific state has been achieved and it’s time to move on to the next one.

AdHoc Fragments

We call AdHoc fragments nodes that don’t have an incoming connection to any other node and are not start nodes. AdHoc fragments may or may not start automatically, depending on the AdHoc Autostart flag.

If the node is automatically started it will be executed when the process is created. However, when this flag is not set, the only way to trigger the node execution is with a signal. Processes receive signals externally through the REST API, internally by using events or programmatically. See more about external signals in the next section.

REST API

For non-flexible processes, the main POST request will trigger the Start node whereas in a flexible process it will just create the process instance. All nodes with the AdHoc Autostart flag will start with the process.

For the case of the AdHoc fragments an empty POST can be sent to start the execution of the node. In the specific case of the HumanTasks where a new task is created, users or API consumers will most likely be interested in the generated task URL. This is possible through the Link HTTP header present in the response. It will contain a relative path with the process instance ID, the task name and the task instance ID.

curl curl -D - -XPOST -H 'Content-Type:application/json' -H 'Accept:application/json' http://localhost:8080/serviceDesk/b3c75b24-2691-4a76-902c-c9bc29ea076c/ReceiveSupportComment

HTTP/1.1 200 OK
Content-Length: 305
Link: </b3c75b24-2691-4a76-902c-c9bc29ea076c/ReceiveSupportComment/f3b36cf9-3953-43ae-afe6-2a48fea8a79a>; rel='instance'
Content-Type: application/json

{
  "id":"b3c75b24-2691-4a76-902c-c9bc29ea076c",
  "supportCase":{
    "product": {
      "name":"Kogito",
      "family":"Middleware"
    },
    "description":"Kogito is not working for some reason.",
    "engineer":"kelly",
    "customer":"Paco the customer",
    "state":"WAITING_FOR_OWNER",
    "comments":null,
    "questionnaire":null
  },
  "supportGroup":"Kogito"
}

Give me an example!

In the next blog post I use all this in an example. Go to Flexible processes in Kogito – Part 2 and see it in action!