Amon turns Five. A new Year. A new Agent

By Martin Rusev read

A couple of days ago Amon turned 5. I've started the project on 9th January 2011 and have been working on it ever since.

The first piece of Amon I wrote was the metrics collector agent. At the time I was a Python developer and since Python is installed by default on any Linux distro - it seemed like a good choice.

I wrote the initial agent in a month. It worked well and the code remained mostly unchanged for the next few years. Until I decided to extend its functionality that is.

The cracks started showing up when I decided to extend it with plugins for the most popular apps like MySQL, PostgreSQL, Nginx, etc. To do this, I had to install library specific dependencies for each plugin - like psycopg2 for PostgreSQL, MySQL-python for MySQL and so on.

That is the point when things started falling apart. My goal with the collector agent is to support all major distros, including the current stable(LTS) release and the one before. That means - CentOS 6 and 7, Debian 7 and 8 and Ubuntu 14.04.

With the Python agent I had to develop the plugin on my desktop Linux distro, which is Ubuntu and when I am done - to get over each distro from the list above and check how to install the corresponding dependency. I had to do this for every plugin. To solve this pile of technical debt - I've created a Docker based test suite that goes over each distro and checks the dependencies and the output. It looked something like this:

declare -a arr=(ubuntu1404, ubuntu1504, debian7, debian8, centos6, centos7)

for distro in ${arr[@]}
    cp $distro/Dockerfile
    docker build $distro
    rm Dockerfile

You can read in more detail about the Docker test suite in my previous article - Test Driven CAPS(Chef, Ansible, Puppet and Salt) Provisioning with Docker

On top of all that, I had to create something called amonpm or the Amon Plugin Manager. Amonpm was a wrapper around an Ansible Playbook for each plugin. You could do amonpm install mongodb and that command would call ansible-playbook /path/to/mongoplugin/mongo.yml and install all dependencies for the Mongo Plugin.

As you can probably see - it was a huge technical burden at this point and almost killed my enthusiasm for writing new plugins.

I did explore the various options for packaging up a python applications. From virtualenv-tools , pyinstaller, cx_freeze, PEX and Ombnibus, which is a Ruby based project, but can be used for Python as well.

These projects work only to a certain degree. They still rely on the libraries and the Python version installed on the system. All this research was not completely in vain - I am using PEX to package Amon itself and that works well, but only on a brand new server and a small list of supported distros (CentOS 7, Debian 7 and 8 and Ubuntu 14.04).

The new agent

Last summer I realized I can't extend Amon with the Python agent and started looking for alternative languages that solve the dependency problem. I looked into Go and Rust. Decided to stick with Go, because it is definitely more mature.

Go feels restrictive at first, but once you understand the "Go" way of doing things - it becomes a much more enjoyable experience. It took me 2 months coding in Go in my spare time to fully grasp the "Go" way and actually be productive.

A word of advice on Go - don't fight it. If you are coming from Ruby or Python or any other language with different conventions - in Go you can do pretty much anything you were doing in these other languages - by using interface{} and reflect, but the code quickly becomes messy and unreadable.

A couple of days ago I released the first version of the new agent that is now on par in terms of features with the old Python agent, plus a couple of new awesome additions:

  1. A single binary - after going through the Python dependency hell, I can't emphasize enough how refreshing it is to be able to bundle everything in one single binary that just works on any Linux distro.
  2. 10 initial plugins - detailed plugins for all popular apps like MySQL, PostgreSQL, Nginx, Mongo. Just like the old agent - you are not only getting metrics, but also table/index size, missing indexes, etc.
  3. Can run service/health checks locally - the old agent was not able to run service/health checks and you had to do that with the remote engine. That is fine and actually can save you a lot of time, but with the remote engine - you have to install salt-minion on each server, which takes some memory, plus anyone that has login details for your Amon instance can alter your infrastructure.

    This is not an actual issue, because Amon is a self-hosted app and you can easily restrict access to the Amon URL with Nginx and serveral allow, deny rules.

    Back to the topic - with the new agent, you can list all the checks in a config file, they will be executed and the information sent to Amon for further analysis.
    ["check-dns.rb -d",
    "check-disk-usage.rb -w 80 -c 90",
  4. One line custom plugins - print "request.per_second:100|gauge. That is a custom plugin right there. You can write them in any language you are comfortable with. To make it work - you just have to print each metric on a new line in STDOUT and the agent will get the output, parse, analyze and send that to Amon.
    # Custom Plugin format
    # If you want to group multiple values on a single chart:
    print "|gauge"
    print "connections.waiting:3|gauge"
    print "connections.max:40|gauge"
    # The agent currently supports two types: gauge and counter
    print "|gauge"
    print "logged_in_users:50|counter"
    # In Python - use print
    print "|gauge"
    # In Ruby - use puts
    puts "|gauge"
    # In Bash - use echo or printf
    echo "|gauge\n"
    printf "|gauge\n"
  5. Everything runs in parallel - and I mean everything. Every plugin and every check for every plugin has its own Goroutine. That means, you can monitor as many things as you want, collecting all these metrics will always take the same amount of time (Around 2-3 seconds.)
  6. Memory and CPU efficient - the new agent fits nicely in 10MB RAM, which makes it convenient for monitoring a small VM or a Docker container and even an IoT device in the near future.

What is next?

This release is just the beginning.A few things I have in mind for the near future.

  1. Statsd - Statsd in 2016 is absolutely crucial. This is something coming very very soon. You will have a statsd daemon running on every machine with an Amon agent. It will aggregate the data, calculate the percentile and send the data to Amon for charting.
  2. Remote configuration - the remote engine runs on Salt, which can do pretty much any configuration change you desire on your hosts. In a next release, you will be able to enable plugins directly from the web interface. For example: Salt will create a read-only user, grant permissions to your MySQL database, put the login details in a config file and restart the agent. That was something you could do with the old agent and amonpm locally: (\)
  3. Remote custom plugins - the new format for creating custom plugins is really simple, but I want to take it one step further. Just like with health/service checks - you will be able to define a custom metrics plugin directly through the web interface. Salt will create the file for you, put it in the appropriate location, restart the agent and you will be able to see the data right away. You could do that by hand or with a CAPS tool(Chef, Ansible, Puppet and Salt), but I think it will be so much nicer to keep all Amon related data and configuration in Amon itself.
  4. Windows, FreeBSD and MacOS support - now that I don't have to think about dependencies anymore, I want to use the same codebase and support all these operating systems. Windows is first with the others hopefully coming shortly after that.

Amon Turns 5

As I mentioned in the beginning of this post - I started working on Amon on 9th January 2011. It was a side project until May 2014, when I decided to take the plunge and focus 100% of my time on it. 5 years is a really long time in the software industry, where new things come up every few months and other become obsolute just as quickly. Thanks to your support - I am able to work on Amon full-time and keep it up to par with the biggest open source and commercial tools.

Today I am just as excited about Amon as I was the day I started. From the outside viewer - monitoring seems like a boring and mostly solved problem, but from my point - things are looking really primitive in 2016. My goal is to eliminate all terminal interactions and to have a smooth Web UI only experience. With the new agent, I am much closer to that goal than ever before.

P.S If you are new to Amon and read the whole post - I want to thank you for your time and attention with this promo code: AMONTURNSFIVE. You can use it to purchase any self-hosted license with a 10% discount.