In 2013 we bagged a development project from one of the largest fresh produce company in the world based out of USA to build half a dozen custom workflows and incrementally add more workflows in the future. One of our senior architect went onsite to their location for requirement, environment , Systems dependencies study and from that he arrived with a solution design and implementation plan which includes project sizing, effort, cost & timeline etc.
The client wanted to to build the workflow application in Force.Com platform because their Trade Promotions Management (TPM) system was built in Force.com but we decided not to go with that the reason being it is very proprietary and lot of custom built applications were already in Microsoft .NET stack and it made sense to stick to MS.NET as the support from Microsoft also guaranteed and good.
During the study our team learnt that each workflow request details (I mean the data captured in the workflow request or attributes) are completely different. You can consider a typical example of a workflow requests are (For security reason, I cannot reveal the actual request form but similar)
- Expense claims approval request
- Hiring approval request (Hiring request)
- Leave approval request
- Travel approval request
- Loan approval request
With my experiences from full stack BPM solutions implementation (product evaluation, procurement, training and implementation) in Infosys back in 2006, I realized that we need to build a light weight BPM engine. Commercially off the shelf BPM engine was out of question due to the complexities it brings in, licensing cost, infra requirement, consultants demand their and their exorbitant salary etc! also I wiped out windows workflow foundation which IMHO is a failure framework form Microsoft and SharePoint is more for document workflow hence custom build in .NET was the proposal we put forward and were able to got their IT team approval to move-ahead.
The Design Challenge
Being an ALT.NET advocate, I wanted the design based on SOLID principles (loose coupling, plug & play, scalable etc.) as the client also given enough and reasonable time for the project and they were very co-operative. More than the BPM engine the design challenge is Workflow Request Domain Model (Data model), I wanted to keep one single code base and one single domain model for all the workflow request so the natural choice of the database is Document Database / schema less (No SQL) but the customer felt it is risk because of the first time implementation in the enterprise and not time tested in their environment hence seeking approval from various groups inside the enterprise might take longer time or will not get approval due to lengthy approval process and policies involved in it so as an alternate I suggested to store it in single table in a single column as JSON (like in No SQL) with a compromise that “We cannot query the request object as it involves string parsing or loading the domain model as POCO collections in the client (.NET) which is not advisable and it will not scale as time passes and data grows.”
(Note: The above design is against RDBMS concept but it just works(Ed) as Jason Fried says in his book “Rework”)
Note: We didn’t had any need to query the workflow request hence it was easy to go with the above approach but what-if if the need comes in the later stage? The Answer was “No SQL” i.e. we had a plan of importing the workflow data into the document database like (reporting server / de-normalized database) and fire-up JSON based queries and present the result. Even if we import into the Document database, it will demand strongly typed objects (domain model) which we didn’t had in our design (we used C# dynamics and leveraged DLR) …this is the requirement for RAVENDB whereas MongoDB accepts any JSON/ schema without strongly typed domain model which is an optional one [This is where ElasticSearch came into the picture and our initial design embraced it without any hassle]
I was very specific in the design to cut short the #. Of lines of code, encourage code re-use and leverage DLR (Dynamic Language Runtime) capabilities of the .NET which Microsoft released to promote C# as dynamic language like Python etc. We took following decisions
- Use C# dynamics and no separate domain model or view model for each workflow request and any domain per say.
- All data has to be first class JSON citizen so that in the future we can migrate to Document database (No SQL like MongoDB or Raven DB)
- Use Massive Micro ORM layer which built on C# dynamics (DLR) which will cut the data access code to sub zero.
- For workflow status querying, we designed a standard workflow table which has dedicated fields for direct querying but the entire request object is stored in a single text field JSON so we are free to store anything. This will help us to import into schema less DB like Mongo etc. in the future
- First time to develop in ASP.NET MVC and making our controller act as RESTful end point using content negotiation.
- Build a generic Audit trail engine based on “Event Sourcing Design Pattern”. All the data stored as JSON
- Build workflow engine by leveraging proven design patterns (i.e. Chain of responsibilities, Strategy, Specification)
- Plus all our standard architecture, tools and frameworks i.e. Spring.Net for DI & AOP, JSON.NET for JSON serializing & de-serializing, client side template, ASP.NET controllers for view rendering and RESTfy
We finished the development and thoroughly tested our design; all worked well as expected however we did continuous fine-tuning.
What is the role of Elasticsearch?
We realized the potential of the work we did for this client and we thought why not we create it as a product; also as we learnt from our pre-sales pipeline that many companies live with manual excel based workflows or custom developed rudimentary workflow tools with tightly coupled business rules which is hard to modify. The cost of the workflow products are huge which SME cannot afford and many don’t want to go for cloud based tools as they feel their data is not secured, all of this encouraged us to make this as generic workflow engine!
When we approached the prospects, most wanted the solution in Open Source Platform and not in MS.NET stack plus their need is also querying the workflow requests so we decided to re-write it in JAVA/PLAY framework with MongoDB & My SQL as Backend and Elasticsearch for Querying and reporting engine.
For Elasticserach querying, we ran a scheduled job which will push the workflow request into RabbitMQ queue and a listener will pick up from the queue and push it into the Elasticsearch Server and builds up the index. We wrote bunch of elasticsearch JSON based queries to pull the results and weaved into our UI layer. This is up and running in-house @ Vmoksha, nearly 10 internal workflows built in this design and rolled out to production which caters 150 of our employees on daily basis. The result from elastic search is lightening fast.
What are the other places we use ElasticSearch @ Vmoksha?
Apart from BPM, we heavily use ElasticSearch to store custom index of the data from our Applicant Tracking System (ATS) built in Open Source Rails based tool REDMINE and on top it we wrote Elasticsearch Scalar queries to power our dashboards, also we use ElasticSearch to pull our P&L data from custom SQL server database and present it in our Executive and Financial Dashboard, Our CRM metrics dashboard is powered by Elasticsearch where the data is coming from SQL Server.
Architecture of our (Vmoksha’s) vBPM, Embeddable Human Workflow Service
We have it in two flavors MS.NET & JAVA (PLAY). vBPM expose its functionalities as RESTful services and can be easily accessed from any client i.e. mobile, web, windows etc. and it is technology agnostic. The Input to the vBPM & output from the vBPM is JSON which is very light weight in nature and easy to consume from any type of client. Security is implemented using Json Web Token (JWT)
TBD : ElasticSearch & RabbitMQ are optional components hence not depicted in the diagram. I am working in creating a new diagram and once done this will be updated.
Workflow Designer : vBPM persist the workflow definition in the MySql tables (Workflow & actors table). It will have a rudimentary web based user interface to configure the workflows and steps, alternatively one can setup the workflow by directly entering the data in to the MySql tables.
Workflow Execution Engine : It is a Java class file which implements Chain of responsibility (CoR) design pattern. All of the functionalities of the workflow are exposed as RESTful apis. Once the workflow instances are kicked-in, this will read the workflow meta data and accordingly execute the instance. The actors of the workflow steps will be identified using CoR & Workflow rules.
Business Rules Engine : It is a a Java class file implemented using Strategy Design pattern tied together with CoR through Spring DI & AOP
Activity Monitor : During workflow process execution, the state of the workflow can be queried through Activity Monitor which can be presented in the client layer. This will be exposed as REST api.
ElasticSearch & RabbitMQ is optional components for high performance searching & querying.
SLA Manager : At workflow level & at workflow step level, SLA can be configured. The Workflow execution engine will monitor the cross over of the SLA and raises corresponding alerts and notifications which again will be a configurable template at workflow & step level.
Alerts & Notification : Custom alerts can be sent to the interested parties. Alert templates is a configurable parameter for every workflow and workflow actions. Email & SMS alerts can be sent. The system having a provision to configure any email & sms gateway through provider model design pattern.
External API invoker : The system has a provision to invoke external APIs and use their result as input for workflow routing.