SaltStack - Review and how it fares against Ansible and Puppet?

By Martin Rusev read

I have been a devoted Ansible user for over an year now. I really like what Ansible can do and how powerful and simple the project is for automating almost any server related task. You can pickup Ansible and be productive in a couple of hours. The docs are great and there are tons of builtin modules that you can use and do anything from installing a package, deploying a new instance on your cloud provider to manipulating databases and creating users with specific permissions.

With all that said - no software is perfect and Ansible is no exception. It is a great project, but still have some shortcomings. The things I personally dislike about Ansbile at the moment is the SSH default, lack of proper Windows support and the way it scales. My idea in this post is to cover each one of these points from the Salt perspective.

Salt - First Impressions

Salt is a massive project and it requires a huge time investment to get into. This becomes instantly obvious when you start browsing the documentation. The docs are well written and extensive, but you have to know what you are looking for. Salt has it is own way of doing things and my past experience with Ansible, Puppet and Chef was not as useful as I thought it will be.

At least in my opinion Salt is not recommended for people who are just getting into devops and automated systems in general.

One of things that make Salt even harder for newcomers is the fact that they have a special "smart" term for all the important parts in the project - there are salt states, the salt mine, pillar, grains, formulas, etc.

Before getting any further I want to explain each one of these terms.

As they say, a picture is worth a thousand words:

Salt works similarly to Puppet and Chef. It has one or more masters that send commands to multiple agents/minions. The biggest difference between Salt and all the other tools is the communication protocol. Salt uses encrypted Zeromq connection with all the data serialized as msgpack binary.

In the past I've had the pleasure of working with Zeromq. One of the first versions of Amon was using zeromq as an alternative to the default http implementation for sending app metrics and logs. The amount of data and the speed at which I was able to push it was nothing short of exceptional. If you ever have to work on a library that has to efficiently send data over the network - ZeroMQ is an excellent choice.

Compared to Ansible, Puppet and Chef which use either SSH or http as a connection protocol, Salt is really fast and scales really well. Scaling is one of the main issues I have with Ansible and the reason I gave Salt a shot. Ansible "scales" in two ways, first to compensate for the slowness of SSH it keeps a persistent connection with the hosts. SSH is great and secure, but it was not created for real time communication from one server to many others. To keep all these persistent connections open Ansible forks itself and you have one Ansible instance running for each host which could lead to unpredictable crashes and memory overload.

Salt was designed from the ground up for real time communication with as many servers as needed. It is the first tool that actively encourages that behaviour. Puppet and Ansible are "static" in comparison - you write a playbook/manifest and then execute them and it doesn't matter if they are fast or not, because most playbooks are I/O bound. When I was learning Salt, instead of writing static salt states(playbooks/manifests) and then executing them, I was doing cool things like this:

salt -G 'os:Debian' pkg.install vim
salt "*" pkg.upgrade # upgrades all packages on all servers
salt -G 'os:CentOS' service.restart httpd

Salt - Up and Running in 5 minutes

There are a couple of detailed tutorials on the SaltStack website, but they try to cover as much ground as possible and require significant time investment. I am yet to find a tutorial that skips the details, which are important at later stage, but irrelevant when you just want to try Salt.
In this part I will try to cover the absolute bare minimum you need to use Salt. I am going to use Ubuntu as reference. The process is almost identical on Debian and CentOS

  1. Install and configure the salt-master

Add the salt repository

sudo apt-get install -y software-properties-common python-software-properties
sudo add-apt-repository ppa:saltstack/salt
sudo apt-get update
sudo apt-get install salt-master

Master configuration, again this is all you need for the master to work vim /etc/salt/master

interface: 0.0.0.0
max_open_files: 100000

Restart the master sudo /etc/init.d/salt-master

  1. Install the salt-minion

Add the salt repository

sudo apt-get install -y software-properties-common python-software-properties
sudo add-apt-repository ppa:saltstack/salt
sudo apt-get update
sudo apt-get install salt-minion

Minion configuration vim /etc/salt/minion

master: master_ip

Restart the minion sudo /etc/init.d/salt-minion

  1. Accept the minion key
salt-key --list
salt-key --accept-all

  1. All Done. Execute commands on the minion
salt debian8 pkg.install vim
salt debian8 pkg.upgrade # upgrades all packages
salt debian8 grains.items # Display facts about the host
salt debian8 cmd.run "uname -a" 

What is next?

Even if Salt had only this feature - securely executing one line commands on your servers, in real time, it would be perfectly fine by me. But Salt doesn't stop there.Like all the other automation tools you can organize these commands in a Salt State. Just like in Ansible, there are written in YAML with Jinja2 for rendering and things like variables and loops, etc.

To get started with states, you have to add a file_roots variable in your config file (/etc/salt/master). Salt is going to search for the State(.sls) files in this directory:

mkdir -p /salt/states
# /etc/salt/master
interface: 0.0.0.0
max_open_files: 100000
file_roots:
  base:
    - /salt/states
sudo /etc/init.d/salt-master restart

Salt has something called the top file or top.sls which is used as a catalog. The top file defines what state should be applied to which server.

subl /salt/states/top.sls
base:
  '*': # These modules are going to be applied to all minions 
    - core # This is /salt/states/core.sls
  'webserver':
    - apache # /salt/states/apache.sls
  'db':
    - postgres # /salt/states/postgres.sls
  'os:Ubuntu':
    - match: grain
    - repos.ubuntu # /salt/states/repos/ubuntu.sls

Execute those states:

# Applies the states defined in top.sls
salt "*" state.highstate

The top file is the officially recommended way for applying Salt States. It is really powerful, but unnecessary and complex for deployments with less than 10 instances and for people that are just getting started with Salt.

They don't highlight this in the documentation, but you can skip this file completely and use Salt in an Ansible kind of way.

Salt - The Ansible way

# /salt/states/common.sls 
common:
  pkg.installed:
    - pkgs:
      - git
      - zsh
      - build-essential
      - gcc
      - python-dev
      - ruby
# Applies the /salt/states/common.sls file to the debian8 minion
salt debian8 state.sls common

Executing on multiple nodes:

# /etc/salt/master
.............
nodegroups:
  django: 'digitalocean-ubuntu1404' # The names of the minions, separated with comma
  db: 'linode-postgres, linode-postgres-replica'
  mongo: 'digitalocean-mongo'
# Applies the /salt/states/common.sls file to the django server
salt django state.sls common

Writing Salt States

The Salt State syntax looks a little odd at first and I definitely needed some time to get used to it, especially when writing more complicate states with jinja variables:

npm:
  pkg.installed

yaml:
  npm.installed:
    - require:
      - pkg: npm

Just like Ansible, Salt has a huge list of built-in modules that you can use out of the box. Modules ranging from the basic os stuff like installing packages, manipulating cron jobs, extracting archives, to more specific like creating mongodb users, pulling git repositories or sending SMTP or Hipchat notifications. The list is really extensive and is going to save you a lot of time and effort if you had to do all these things by hand.

One other thing that will save you even more time is the official collection of Salt States on Github. It took me a while to discover those. Both the modules and the states repository are buried deep in the documentation and in my opinion they should be put front and center. On Github you will find over 150 ready to go Salt States for all the popular apps and services - PHP, MySQL, Nodejs, Postfix, Apache, Docker, etc.

You can use these to install the libraries you need, but you can also use them as a starting point - for learning best practices, how to use variables and loops, how to write complex states with multiple folders and config files. All these are covered to some extent in the documentation, but I found it easier to learn from real world examples.

One thing I wasn't particularly happy about is that to use these states, you have to add the path for each one in the /salt/etc/master config file and restart the master. Salt support folders and initially I fought that I can clone the repos and then do the following salt debian8 docker-formula.docker.

cd /salt/states
git clone git@github.com:saltstack-formulas/docker-formula.git
git clone git@github.com:saltstack-formulas/php-formula.git
# /salt/etc/master 
interface: 0.0.0.0
max_open_files: 100000
file_roots:
  base:
    - /salt/states/base
    - /salt/states/base/formulas/docker-formula
    - /salt/states/base/formulas/php-formula
sudo /etc/init.d/salt-master restart
salt debian8 state.sls php

Conclusion

Salt is a great tool with very a steep learning curve, even for people who are not new to the devops movement. Prior to Salt I have been using Chef, Puppet and recently Ansible for almost 7 years and somehow my first 2 days with Salt were really frustrating I was not able to do much. One of the main reasons for this frustration was the documentation, which heavily emphasizes on enterprise usage and "cool" terms.

With Salt the documentation and the tutorials are in reverse - you don't learn how to execute a single state on 1 node until later in the docs. First, there is the information about grouping nodes and states in the top.sls file, globbing with grains(-G "OS:Ubuntu) and pillars and highstate. These are all great ideas, but you can't really appreciate them until you understand the basics.

I really hope you enjoyed this post, please let me know in the comments below what you think about Salt and if you would like to know more I will gladly share my experience.