Until recently, creating a change for Drools, jBPM, Guvnor, Planner or any of the other Drools modules could prove to be challenging. Not because of the code, but because the monolithic build was big and brittle.
It took quite a few minutes to build the whole thing, not to mention to test it, if it could build successfully at all. Hudson was red more often than it was blue/green and stayed red for weeks. Why? If a test in for example drools-core failed for a few days, hudson didn’t build guvnor (or any of the dozen other modules). Meanwhile people committed changes that broke guvnor but hudson didn’t complain. By the time that drools-core was fixed, multiple other modules were failing. So while we’re fixing one test, other tests were broken and it became very hard to determine which change broke which test.
Who’s to blame for Hudson being red so much? Nobody really. You can’t blame a developer for not running all the tests before committing if running all the tests takes hours. And if some tests are already failing (and hudson is red) before his changes, it’s hard for him to notice that his changes also break the build.
The real culprit was the monolithic build. There’s no need to build drools (the rule engine) from source and test it if you’re just working on guvnor. And we are in luck, because our code isn’t monolithic at all: every module is lightweight (especially just-drools-the-rule-engine or just-jbpm-the-workflow-engine). There are just a lot of modules adding all kinds of functionality on top of those lightweight engines. So, we have split up the build, the git repository and the hudson jobs.
We used have 1 big, monolithic git repository:
- droolsjbpm:
- Contained everything (flow/jBPM was already removed some time ago)
Now we have several smaller repositories, each with it’s own build and it’s own Hudson job:
- droolsjbpm-build-bootstrap
- The parent pom (common stuff for the build)
- droolsjbpm-knowledge
- The public API and the internal API common to drools and jpbm
- drools
- The rule engine modules
- jbpm
- The workflow engine modules
- guvnor
- The web app to manage Drools & jBPM repositories
- drools-planner
- The planning optimizer modules
- droolsjbpm-integration
- Integrating Drools & jBPM with Seam, Spring, Camel, etc
- droolsjbpm-tools
- The eclipse, maven and ant plugins for drools & jbpm
- droolsjbpm-build-distribution
- OSGi bundles, top level assemblies, etc
Every repository can build alone: you don’t need to clone the other repositories or build them from source (but you can of course). Some repositories do depend on SNAPSHOT’s build by other repositories, but those SNAPSHOT’s are deployed by Hudson and therefor available from Nexus.
| Project | Git size | Locally (2th build): mvn install -DskipTests | Hudson (*): mvn install -Dfull |
|---|---|---|---|
| droolsjbpm-build-bootstrap | 1 MB | 3s | 55s |
| droolsjbpm-knowledge | 35 MB | 6s | 1m 53s |
| drools | 160 MB | 38s | ? (red) |
| drools-planner | 142 MB | 9s | 4m 34s |
| jbpm | 21 MB | 31s | 11m 7s |
| droolsjbpm-integration | 38 MB | 23s | 11m 0s |
| guvnor | 131 MB | 58s | 21m 40s |
| droolsjbpm-tools | 24 MB | ?m ?s | ? (red) |
| droolsjbpm-build-distribution | 10 MB | ? (todo) | ? (red) |
| New total | 562 MB | 2m 48s (without tools, dist) | ? |
| Old droolsjbpm | 1200 MB | 3m 18s in JAN 2011 5m 29s in DEC 2010 (without jbpm, tools, dist) | 2h 41m 57s (without jbpm) |
The git history has been successfully migrated and split up across the new git repositories (except for jbpm). Generated files (such as jars and GWT binaries) are removed from the Git history. Therefor, the new total git size is over 50% smaller.
Branches and tags have not been migrated because their monolithic build would not work. The canonical source for branches and tags up to and including 5.2.0.M1 is subversion. The old git droolsjbpm repository will be deleted to avoid confusion.
We’re still ironing some things out (most notably some major refactors on drools) and we’re also working on speeding up the tests for some projects (such as guvnor).
(*) The Hudson times are not reliable, because it is the result of a random build which may or may not needed to download dependencies. However 1 reason why the monolithic full build was so slow, was that adding the eclipse p2 repo in 1 module (droolsjbpm-tools/drools-eclipse) made all the other modules build a lot slower (and now it only affects the modules in droolsjbpm-tools). Wierd.
Summary: Cloning a repository, building it, testing it and sending us a pull request (or patch) now takes a fraction of the time it used to. So try it out! More info in the README file.
