Beyond Basics – Integration of MuleSoft Logging with Datadog

Written by Muhammad Aqib Arif

Senior Software Engineer - Middleware Practice

Logging is a vital piece of software development as it helps generate a detailed set of events that occur within the applications or servers. Logs can help find the cause of the problem and monitor the application flow end to end. In this blog, we will see how to integrate MuleSoft logging with Datadog to collect the logs of MuleSoft applications and perform certain operations like monitoring, filtration, and projection.

The Datadog Mule Integration is an Agent-based integration which collects metrics from MuleSoft products and uploads them into Datadog. These metrics can be used to take advantage of the out-of-the-box dashboards and monitors or create your visualizations.

What is Datadog?

Datadog is a tracking service for cloud-scale applications, providing monitoring of servers, databases, applications, tools, and services through a SaaS-based data analytics platform.

Follow the steps below to integrate MuleSoft logs with Datadog.

Sign up for Datadog account.

Click on Get Started Free button as shown in the image.

Fill the fields as required or you can sign up with Google

Once your Datadog account is created, have a tour of this tool by visiting different options from the left sidebar.

Now get the API key to integrate Datadog with MuleSoft. Go to the Integration menu from the sidebar and click on API.

Select API Keys panel, then hover over to Key field and copy the key

Now let us open the log4j2.xml file of your mule application from the path src/main/resources.

Add HTTP Appender in tag for pushing the logs to Datadog. Please see the snippet below.

<Appenders>
<Http name="DataDog" url="https://http-intake.logs.datadoghq.com/v1/input?host=${sys:hostName}&ddsource=Mulesoft&service=loggingApp;ddtags=env:Dev;app=dd-chub-poc ">
	<Property name="Content-Type" value="application/json" />
	<Property name="DD-API-KEY" value="<YOUR DD API KEY>" />
	<JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true">
		<KeyValuePair key="host" value="${sys:hostName}" />
		<KeyValuePair key="apiName" value="dd-chub-poc" />
		<KeyValuePair key="appName" value="DataDog Cloudhub POC" />
		<KeyValuePair key="apiType" value="sys" />
		<KeyValuePair key="ddsource" value="mule" />
		<KeyValuePair key="service" value="${sys:dd.service}" />
		<KeyValuePair key="correlationId" value="$${ctx:correlationId:-}" />
	</JSONLayout>
</Http>
</Appenders>

Let us analyze the URL with tags like host, ddsource, service, and env. Also, see the key-value pair for JSONLayout. We’ll see these properties in Datadog with the logs.

Paste the Datadog API Key in value of DD-API-KEY property.

Note: The Datadog API key was copied in step 6

Now add the reference of DataDog Http appender to Loggers element.

<Loggers>
	<AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
	<AsyncLogger name="com.mulesoft.agent" level="INFO" />
		<AsyncRoot level="INFO">
			<AppenderRef ref="DataDog" />
		</AsyncRoot>
</Loggers>

Let us now create a mule flow and add some loggers there. Here, we will add a couple of loggers in my flow and a Request component with the wrong URL, which should throw an error.

Let us now create the deployable jar and deploy it to Cloudhub.

Now click Next -> Finish

Once the deployable jar is ready, let us login to Cloudhub and deploy the application.

Note: Make sure to disable Cloudhub logs

Now click Deploy Application.

Once the application is deployed, Open postman or any other REST Client and send the request a few times to the deployed application. In my case, it is:

Now go back to Datadog

Click on Logs -> Search from the left sidebar

You should notice that the logs have started appearing on the dashboard, as seen in the image below.

Now click on any of the log lines and see the details.

Notice that a detailed description of the error is visible now.

You can also filter the logs based on facet and criteria, as seen in the image below.

So, this is how we can bring all the logs of our applications in one place and monitor them in datadog as needed.

You may be interested to read this article on Smart and Strategic Integration with MuleSoft API which shows how MuleSoft’s Anypoint Platform empowers everyone in the company — from IT to LoB, from a small enterprise to a big organization, from integration to API management — to unlock applications data and devices and go digital.

Though APIs are universal, creating a sustainable API business model is an ever-evolving process that needs strategic approaches. At Royal Cyber, we have developed deep experience in API business models by working with many organizations at various stages of API maturity. We have created this guide to Choose the Right API Business Model that will empower you to select the right business model and get the most from your APIs.

Summary

At Royal Cyber, we are excited to deliver innovations to empower everyone to create connected experiences. With Datadog’s powerful features, we are steering towards the dream of enabling the next level of monitoring– where the logging capabilities can be utilized for the applications and APIs. For more information, you can email us at info@royalcyber.com or visit www.royalcyber.com.

Leave a Reply