Upcoming Event: 
All Things Integration Meetup: Deep Visibility - Serverless Logging with AWS
View all events

Traceability for Process Automation and Improvement in API-led Connectivity

A Big Compass client that we’ve worked was faced with a challenge - they had large volumes of claims transaction that represented the heart of their financial system. Successful submission of these claims for payment was a critical process and important to the business’s financial health.

During a system discussion about process improvement and automation, one of their architects asked, “How would I use logging to identify points of failure in the claims processing system?”

Our answer? Use traceability.

Traceability - The Right Integration Pattern

That answer may seem short considering the complexity of what was asked. Much like the question, the implementation of the solution is more involved than the answer implies.

Implementation relies on leveraging the iPaaS’s internal capabilities to handle the integration pattern and the strict adherence to the company’s standard coding practices. In this case, the client was using MuleSoft as their integration platform. The coding practices to be followed in this instance were ensuring that the ID and the associated metrics were logged for every message.

This approach comes with specific challenges. Your enterprise architects must establish enterprise-wide API communication standards. And they must gain business buy-in on the idea that the logs are useful for insights and needed for driving business decisions.

Our client had multiple systems involved in the business flow that needed to be accounted for in the solution. The source of the data was a legacy system that sent claims daily to the MuleSoft APIs. From there, the APIs invoked an additional group of internal APIs to detect errors and omissions and potential fraud. Only the claims that passed those tests made it to the API that automatically initiated payments. Failed requests were shunted for manual reconciliation, minimizing the number of fraudulent or incorrect claims in the process.

Once our team had gained a solid understanding of the need and the existing process, we were able to design a solution using a tracer identifier, correlationID, that was attached to the request at the originating claims system. MuleSoft received that tracer ID as part of the header as x-correlation-id.

As the requests passed through the APIs, key metrics were logged against the correlationID, along with the typical entry and exit points in the code. Parameters that were logged included systems invoked and responses. The downstream systems flagged problematic requests as either erroneous data or fraud, as appropriate. For example, if a claim was passed through the system for multiple cataract surgeries on the same eye, that request would be flagged.

A standard was established within the team on the key metrics to log against the correlationID. As the MuleSoft application passed the identifier along to the downstream systems, the process of logging critical metrics with the ID was repeated.

How to Implement Traceability for Process Automation and Improvement

The Client Generated Correlation IDs

As noted, the legacy claims system generated a unique ID, correlationID, which followed UUID standards. When the MuleSoft API was called, the header ‘x-correlation-id’ was set with the UUID:


           doc:name=”Is not null”
           doc:id=”ccd4le06-45ef-45e3-bf19-d69ee07c7f14”
           config-ref=”Validation_Config”
           value=’#[attributes.header.”x-correlation-id”]’
           message=”Header ‘x-correlation-id’ is required.”
/>

Mule automatically read the -x-correlation-id header value and set the variable correlationID to the global value.


           doc:name=”Start Log”
           doc:id=”3b21b299-5b68-4333-a54a-057b5439cb9”
           message=”CorrelationId : #[vars.correlationId] : :
           Message : subFlowStarted : : Flowname : #[flow.name]”
/>

Key metrics, like those generated for failed requests from external systems, were logged against correlationID, along with standard logging.


           Doc:name=”Log Rule Engine Response”
           Doc:ID=”3b261c3d-7b4c-46c3-994f-93f18b47a365”
           Message=”CorrelationId : #[vars.correlationId]
           : : Message : response from rule [#[vars.ruleId]] = #[vars.ruleResponse]
           : : FlowName : #[flow.name]”
/>

Outbound Messages

MuleSoft’s HTTP connector has the capability of sending the correlationID to the downstream system natively. The HTTP connect configuration allows for specifying if the correlationID should always be sent along with a customized name of the variable. The connector, in turn, converts the ID into an outbound header called ‘x-correlation-id’. If the receiving system is another Mule application, the same process follows - the header, ‘x-correlation-id’ is converted to an internal variable in the downstream application.


           doc:id=”3fbd02e3-b57a-4c29-8959-595bf99d6e5c”
           config-ref=”HTTP_Request_Configuration_Subscriber-Sapi”
           path=”#[‘${secure::sapi-path.comments}’]”
           sendCorrelationId=”ALWAYS”
           correlationId=”#[correlationId]”>

Depending on the type of implementation, the correlationID can be sent as a global configuration element. In fact, setting it up at a global connector level is a better approach, as it avoids repeated configuration at each HTTP connector level.

Data Visibility and Dashboards

With the key metrics associated with the correlationID, an enterprise-wide log management tool can ingest the logs and give you visibility into the system. Log management tools like Splunk or ELK can ingest the logs and assist in diagnosing failure conditions, monitor business metrics, and create dashboards to visualize and analyze the data. They can also pull together the system-wide logs using the ID to provide a single, end-to-end view of the message flow.

Customized dashboards make critical business metrics easy to understand and aid in driving business decisions. In our client’s case, some of the metrics we pulled into the dashboard panels included:

  • Number of claims per day
  • Number of claims that failed the error detection rules
  • Most crucial fraud detection rules
  • Average payment amount per condition

This is just one example of how traceability and logging can be used. Leveraging the correlationID and logging standards are low hanging fruit that provide a high return on investment. Using these standards, companies can more effectively analyze system performance and important business metrics.

If you’re looking to gain greater insight into the message flow of your integrations, Big Compass can help. We’ll work closely with you to define an effective solution based on the metrics and data important to you and your business and can even help you implement it.

Get the eBook