I know Grafana for a long time, but in my previous career, we used different tools as Datadog or other older tools. We switched from Datadog to Grafana last year to unite with other teams in one central place.
I was looking at how others maintain their dashboards and some teams just edit and store JSON outputs from Dashboards. I heard on Monitorama conference about Jsonnet a few years ago. I thought this would be a better approach than editing JSON files directly.
Grafana has a library called grafonnet-lib that helps you with creating widgets in grafana via jsonnet. You can find prepared components for a dashboard, annotation, graph, singlestat, template, and more. What I find out later was that many options and parts were missing. You need to edit directly or send PRs into the library. I sent a few PRs to fix it. We found more and more missing parameters and I gave up after seeing that many PR were not merged. Today I found that the Grafana team is working on a better solution. In Grafana 7.0 they are generating jsonnet from spec. I hope this will be a good solution to have all parameters available and without the need to fix it by ourselves.
How we structure our dashboards
This is list of files in our repository:
├── Makefile ├── README.md ├── build │ ├── dev │ ├── prod │ └── staging ├── dashboards │ ├── actions.jsonnet │ ├── backend.jsonnet │ ├── mongo.jsonnet │ ├── oncall.jsonnet │ ├── redis.jsonnet │ └── traffic.jsonnet ├── environments │ ├── dev.libsonnet │ ├── prod.libsonnet │ └── staging.libsonnet ├── grafonnet (copy of library) └── templates ├── config.libsonnet ├── header.libsonnet ├── mql.libsonnet
The makefile has these commands to prepare dashboards and command for deploy.
$ make Usage: make <target> Targets: help Display this help clean Clean built dashboards build Build dashboards deploy Deploy dashboards auth Get fresh token from grafana
We use the same dashboards for every environment and there are configuration changes that are stored in folder
environments. We use one grafana instance for all environments but all settings related include
title of dashboards are specific for the environment. Dashboards have similar names as
Team - Environment - Service.
template folder we have
config.libsonnet that is the main config file reused in all dashboards. In the folder we have common parts, such as a header with the message:
# Don\'t edit here. Edit in repository, and make PR there!. We extend code as tutorial with
The command to compile jsonnet into
build/*.json looks like:
jsonnet --ext-code-file "env=environments/$(patsubst %/,%, $(dir $(patsubst build/%, %, $@))).libsonnet" dashboards/$(basename $(notdir $@)).jsonnet > $@
From the files in the
dashboards folder, we create json output that is split to folders by environments. We are following guidance from Grafana Labs at KubeCon: Foolproof Kubernetes Dashboards for Sleep-Deprived On Calls that recommends Hierarchical Dashboards.
Our dashboards are organized in this order just on two levels. We have the main one called
oncall for overview with summary for traffic, databases, etc. and if you click on that in the main dashboard that opens for you a more detailed dashboard with specific information and you can select region, availability zone/domain or application in a template.
We are using a jsonnet standard library to generate the list for templates that are big and easy to maintain.
.addTemplate( template.new( 'id', '$datasource', 'map($app;' + std.manifestJsonEx(dataMap, '') + ')', hide = 2, refresh = 1, ) )
and if we get issues with newlines in a map we can easily fix using standard library too and remove newlines:
.addTemplate( template.new( 'id', '$datasource', 'map($app;' + std.strReplace(std.manifestJsonEx(dataMap, ''),'\n','') + ')', hide = 2, refresh = 1, ) )
Making these changes manually in many places would be exhausting.
The advantage of using jsonnet is more organized code for dashboards. We try to make it dry and if we have some graph type in the dashboard more times, we create a function to generate it with parameters that are different, such as application name. All code is directly in the dashboard jsonnet file. You can move that to separate files if you need it in more places.
The disadvantage of this solution is the learning curve, people don’t know jsonnet, and there isn’t much documentation on how to learn it. But from the year of experience, I see more benefits.