Devops Discoveries

Salt git integration without gitfs

SaltStack has some pretty cool git integration. Unfortunately it also has quite a few bugs, especially when using gitfs for pillars.

These issues can be annoying at small scale, but they can become very important as you add more minions. To work around these I looked for ways I could simplify our salt/git integration and now that it’s complete I couldn’t be happier.

With a post-receive hook on my gitlab server and a salt master that is also a minion, the salt server updates it’s file root’s directory from git without the salt-master process having to do any interfacing with git at all. As a result applying states through our environment of nearly 200 minions is faster and more reliable than it ever was with gitfs.

I even have some features that I never had with gitfs, like automatic environments based on branches. Here’s how it works.

My salt master has the following state applied. This state ensures that the salt-master service is running. It gets the list of branches from the git remote and makes sure that that branch is cloned into a directory under /srv/salt/. It also manages a file in /etc/salt/master.d/roots.conf which defines each environment that has been cloned and restarts the salt-master process when the file changes. This uses one git repository for both states and pillars, so states are in the repo/states directory and pillars are in the repo/pillar directory.

With just this and a schedule you already have an okay salt-git integration. But with a little more work you can take it to the next step and make it event driven on git push.

If you’re using gitlab for your salt repository, you can create a post-recieve script by putting a file in /var/opt/gitlab/git-data/repositories/salt/salt.git/custom_hooks/post-receive.

#!/usr/bin/env bash                                                        
 
while read branch; do                                                      
        branchname=$(cut -d "/" -f 3 <<< "${branch}")                      
        sudo salt-call event.send salt/push branch=${branchname}           
done                                                                       

Now in your salt master config, add a reactor:

Add the reactor file in your git repo.

And add a bit more logic to the salt-master-git.sls to handle the individual branch being pushed. With this logic if the pillar salt_git_branches is included in the state run, the state will only update that branch. If it is not included, the state will update all branches, and clean up old deleted branches. This saves some time which is important when it’s being called by a post-recieve hook.

Now enjoy the best of both worlds. Automatic integration between salt and git and the reliability and speed of a simple file_roots configuration.