We’re in the process of experimenting with Elasticsearch to help us manage logging better than we could with CloudWatch alone.

lambda logs
Lambda logs to CloudWatch are passed to Elasticsearch

It’s easy to add console.log statements to log information from a Lambda function into CloudWatch, but then it can be a challenge to find the information we want. Logs for each Lambda are broken out into different groups, and logs within each group are further broken out into different streams. CloudWatch is a powerful tool but has limited search capabilities. Looking at logs while editing Lambda functions can involve quite a lot of clicking around.

As a result, we’ve started experimenting with logging JSON objects that are streamed into Elasticsearch. For example, our getAccount query, which retrieves information about a specific Workspace, might log the following JSON if a user tries to load the wrong Workspace:

{
  “od_event”: “getAccount”,
  “od_status”: “failure”,
  “od_message”: "Unauthorized: mismatched clientUUID"
  //…
}

We’ve configured a subscription (filtered on the string “od_event” to minimize cruft) from the getAccount CloudWatch stream to a Lambda provided by AWS that inserts the JSON into Elasticsearch.

The benefit of passing the JSON into Elasticsearch, rather than trying to review in CloudWatch, is that the key/value pairs in the JSON are broken out into searchable fields that are easy to filter and count. For example, we can check to find out if we’re seeing an unusual number of errors for a particular user or customer:

elasticsearch
Elasticsearch turns JSON into searchable fields

We’re also building visualizations with Kibana that we aim to cover in a future blog post.

With a new product in early Beta and a team that is building a serverless application using the latest and greatest, logging will to continue to evolve rapidly. For the moment we’re super stoked about how sending logs from CloudWatch to Elasticsearch gives us a great deal more visibility than using CloudWatch alone and, in doing so, allows us to increase development velocity.

Submit a comment

You may also like

Why Autosave Matters in Shared Applications
Why Autosave Matters in Shared Applications
14 July, 2020

Our goal for the Ondema Workspace is to make it easy for all users within a company to manage, update and view real-time...

A Little Calvinball Goes a Long Way
A Little Calvinball Goes a Long Way
25 February, 2019

“Other kids’ games are such a bore! They gotta have rules and they gotta keep score! Calvinball is better by far!

The Ondema Technical Interview for Developers
The Ondema Technical Interview for Developers
15 July, 2019

The defining characteristic of software developers is how little we know, and how useless the information we do have can...