Customer Scenario

Our customer needed to respond quickly to increasing reporting requirements and greater data volumes associated with the COVID pandemic. They needed to accelerate the reporting of resource (staff) utilization in more near time for tracking and planning purposes. 

Their existing system was designed to capture and synchronize data from three different databases. Unfortunately, the system had inefficiencies and performed unnecessary and costly processing. Additionally, the reporting system was only synchronized nightly, when their process would send a complete data set of source data.

Due to the critical nature of the work of their staff, the customer needed to modernize their ill-performing system and move to an architecture and application that better met their current and future needs.

Solution

Big Compass has extensive experience utilizing MuleSoft to build solutions that process and synchronize large data sets. Our approach to meeting our customers' needs relied on establishing a sound solution architecture and MuleSoft's System and Salesforce APIs.  

A thorough application review allowed our team to quickly identify and implement an integration pattern using a decoupled, layered approach replacing the existing large monolithic process. 

We also planned for sizing constraints and reliable message queues to prevent errors or data loss. The abundance of MuleSoft and Salesforce resources and integration examples accelerated the development process. 

The solution Big Compass implemented utilized database system APIs for each of the database instances, with each database having a dedicated Process, Poller, and System API.

Poller applications paginate through all the new or updated records in the database and publish those to an AWS message queue. A Process API facilitated the business and transformation logic and the Salesforce upserts.

API policies were implemented to allow granular SLA and access control over consumers. Paired with PingIdentity as the external identity provider, the solution enforces access control by introspecting the client tokens. 

The solution uses a single job reconciliation application shared by the three interfaces. Orchestration between these components was conducted using AWS SQS (queues). 

The system reports could be produced using the same reporting process as the original system. Additional error reporting was implemented to ensure no updates were lost.

Results

The reports' vital staffing and utilization data is available hourly versus once daily.  The process to synchronize the reporting data is optimized and has reduced processing time, relying on delta detection methods versus processing the entire database.   

The customer can now produce resource reports more frequently, increasing their accuracy and improving decisions. The customer's operations team can also directly address any failed records promptly. 

The MuleSoft API-led solution positions the customer for the future by using standard APIs and MuleSoft connectors. The generalized design of the APIs was intentional, allowing them to be reused or expanded to support future integration needs that utilize the same databases.