We’re in the process of experimenting with Elasticsearch to help us manage logging better than we could with CloudWatch alone.

lambda logs
Lambda logs to CloudWatch are passed to Elasticsearch

It’s easy to add console.log statements to log information from a Lambda function into CloudWatch, but then it can be a challenge to find the information we want. Logs for each Lambda are broken out into different groups, and logs within each group are further broken out into different streams. CloudWatch is a powerful tool but has limited search capabilities. Looking at logs while editing Lambda functions can involve quite a lot of clicking around.

As a result, we’ve started experimenting with logging JSON objects that are streamed into Elasticsearch. For example, our getAccount query, which retrieves information about a specific Workspace, might log the following JSON if a user tries to load the wrong Workspace:

{
  “od_event”: “getAccount”,
  “od_status”: “failure”,
  “od_message”: "Unauthorized: mismatched clientUUID"
  //…
}

We’ve configured a subscription (filtered on the string “od_event” to minimize cruft) from the getAccount CloudWatch stream to a Lambda provided by AWS that inserts the JSON into Elasticsearch.

The benefit of passing the JSON into Elasticsearch, rather than trying to review in CloudWatch, is that the key/value pairs in the JSON are broken out into searchable fields that are easy to filter and count. For example, we can check to find out if we’re seeing an unusual number of errors for a particular user or customer:

elasticsearch
Elasticsearch turns JSON into searchable fields

We’re also building visualizations with Kibana that we aim to cover in a future blog post.

With a new product in early Beta and a team that is building a serverless application using the latest and greatest, logging will to continue to evolve rapidly. For the moment we’re super stoked about how sending logs from CloudWatch to Elasticsearch gives us a great deal more visibility than using CloudWatch alone and, in doing so, allows us to increase development velocity.

Submit a comment

You may also like

Freedom to Change
Freedom to Change
12 April, 2019

Ondema takes advantage of modern web architecture and its ability to save data and re-render components as needed. This ...

The Ondema Technical Interview for Developers
The Ondema Technical Interview for Developers
15 July, 2019

The defining characteristic of software developers is how little we know, and how useless the information we do have can...

From the Military to Software, Agile Processes are Key
From the Military to Software, Agile Processes are Key
6 November, 2019

I had the privilege of serving in the US Marine Corps from 2003-2011 as an AH-1W Super Cobra pilot, an expeditionary uni...