The final blog in this 4 part series is all about creating an automatic self-healing infrastructure, preventing accidental and malicious changes to your infrastructure using SaltStack Config. We’re going to protect nginx, mainly because its pretty common, and also keeps it simple for those of us new to SaltStack (me included!)

If you’ve been following the 3 previous blogs in this series, you’ll know all about SaltStack Config! You’ll know about its capabilities, how to attach a minion, how to deploy software and packages, and how to manage configs within those packages. But how do you ensure that your state stays consistent between your job runs? Enter beacons and reactors.

Before we start with the hand on side of things, I think its best I explain how this works.

We can define a beacon on a minion which sends an event whenever a particular activity occurs. This activity can be file changes, system load, the status of a service, a user logging in or even network and disk utilisation. This event is sent back to the salt master where the ‘reactor’ lives. The reactor, does exactly what you’d expect, reacts and provides an output. Whether that be restoring a file back to its preferred state (the one that is stored on the master), restarting a service, alerting, the list goes on.

For my example though, I’m going to make a change to the index.html file. To remind you, this is generated by the jinja file which adds some dynamic content. So by default, it monitors the file every 5 seconds but is smart enough to disable monitoring whenever a job is ran which is pretty cool. Already far smarter than me!

So lets get started, first up we need to write a state file to install the beacon on the minion. For this we need an additional package called ‘pyinotify’, so we’ll declare that in the state file too

Saltenv: base

Path: /beacon/beacon.sls

# Name: beacon.sls
# Description: Install Beacon on Minion to monitor index.html

needed-pkgs:
  pkg.installed:
    - pkgs:
# Name: beacon.sls
# Description: Install Beacon on Minion to monitor index.html

needed-pkgs:
  pkg.installed:
    - pkgs:
      - python3-pyinotify
      
install_beacon:
  file.managed:
    - name: /etc/salt/minion.d/beacon.conf
    - source: salt://beacon/files/beacon.conf
    - makedirs: True

restart_minion_service:
  service.running:
    - name: salt-minion
    - watch:
      - install_beacon

Next, we need to tell salt to deploy the beacon, essentially copying the file down from the master to the minion, and allowing it to create the directories it needs to make that happen (makedirs)

Next we need to create a beacons.sls file to define what we want to self-heal. Lets create this:

Saltenv: base

Path: /beacon/files/beacon.conf

# Name: beacon.conf
# Description: Beacon to deploy to minion to protect changes to index.html and nginx service

beacons:
  inotify:
    - files: 
        /var/www/sam/index.html:
          mask: 
            - modify
            - close_write
    - disable_during_state_run: True
  service:
    - services:
        nginx:
          onchangeonly: True
          uncleanshutdown: /run/nginx.pid
    - disable_during_state_run: True
    

There are a few interesting bits to this, firstly you can see where we’re using the inotify component and which file we’re monitoring, in this case /var/www/sam/index.html. This is the file that has our content in, so any changes to it will show publically.

The mask: element defines which actions generate an event, so we’ve kept it simple here and just chosen modify (which is self explanatory) and close_write, which is where a writeable file is closed.

We have also added a line to ensure that this event isn’t created during a state run, as this would be a legitimate change to the file if it came from the master.

In the service section, you can see that we’re monitoring the nginx service in this case, this is native and doesn’t require the inotify package, hence when it’s not contained within the itnotify block.

Lets create a job, much the same as we have done before to push the beacon to the minion.

Testing

The above code will create an event when the index.html file is changed within the /var/www/sam folder. But we need to test that the beacon actually works. Lets jump onto the server and see whether an event will be created when we open the file.

Firstly ssh onto your ubuntu server: ssh [email protected], change to the folder and open the file making a change and saving it. Once that is changed jump back into the saltstack console and head over to Activity > Completed. You should see an event that has been created by the minion and fired over to the saltmaster. It should look something like this:

If, like me, you like to actually see the event being thrown in real-time, SSH on to your saltmaster, and run:

sudo salt-run state.event pretty=True

You’ll see something along these lines:

and it’s much the same when I try and stop the nginx service either via the saltmaster or directly on the ubuntu server itself:

So now we know that the beacons are working, it’s time to get Salt to fix this using event-driven automation!

Reactor

Unlike the beacon which runs on the minion, the reactor runs on the saltmaster, waiting for the events to be generated however the creation of the reactor is exactly the same. My skills are pretty basic and I must thank one of my VMware colleagues for help here (Rob Robinson – thanks for the pointers!) but I got it working

Saltenv: base

Path: /reactor/master.sls

# Name: master.sls
# Description: Used to deploy reactor to the saltmaster

copy_reactor:
  file.managed:
    - name: /etc/salt/master.d/reactor.conf
    - source: salt://reactor/files/reactor.conf

restart_master:
  service.running:
    - name: salt-master
    - watch:
      - copy_reactor

This state file’s job is to push the reactor.conf file to the relevant location on the saltmaster. Despite it being the saltmaster, it still runs a minion just like all the other servers, and therefore we can deploy to it in exactly the same way. Once we’ve copied the file down, we restart the salt-master service so the reactor is ready.

Saltenv: base

Path /reactor/files/reactor.conf

reactor:
  - 'salt/beacon/*/inotify//var/www/sam/index.html':
    - salt://reactor/files/fileReactor.sls
  - 'salt/beacon/*/service/nginx':
    - salt://reactor/files/serviceReactor.sls

This is pretty self-explanatory. Line #1 tells you this is a reactor config file, lines #2 and #3 tells salt to use inotify when a change is made to the /var/www/sam/index.html and where to find the state file to fix it. Lines #4 and #5 tells Salt that we’re watching the nginx service and where the state file is to respond to an event about the nginx server.

Lets have a look at the fileReactor.sls file

# Name: fileReactor.sls
# Description: Reactor file to tell Salt to restore original index.html

fix_index_html:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - arg: 
      - reactor.files.webconfigFix

Another state file, but this time its describing on how how to restore the index.html if an event is created. Line #6 is particularly interesting, essentially this is defining the target of the reactor, using the id passed from the beacon event to get the correct minion id. The argument then calls the fix which restores the original file stored on the master.

Saltenv: base

Path: /reactor/files/webconfigFix.sls

# Name: webconfigFix.sls
# Used to fix index.html in the event a modification is made

# Sleep to give time for me to demo
sleep_20:
  cmd.run:
    - name: sleep 20

# Copy the file to the minion
deploy_index_html:
  file.managed:
    - name: /var/www/sam/index.html
    - source: salt://webconfig/files/index.html.jinja
    - template: jinja
    - require:
      - sleep 20
      
# Restart the nginx service
restart_nginx:
  service.running:
    - name: nginx
      watch:
      - file: deploy_index_html

The first block sleeps for 20 seconds, you don’t actually need it, but if I didn’t add this in, you’d never see Salt fix the config – it’s that fast.

The second block copies the ‘fixed’ file from saltmaster, the exact one we wrote in the 2nd blog post.

Lastly, the 3rd block restart the nginx service referring to the 2nd block to tell nginx which file to use. If you only want to watch files – those 3 files will do the business.

Lets have a look at the reactor that restarts the nginx service for completeness

Saltenv: base

Path: /reactor/files/serviceReactors.sls

# Name: serviceReactor.sls
# Description: Reactor file to tell Salt to restart nginx service

{% if data['nginx']['running'] == False %}
start_service:
  local.service.start:
    - tgt: {{ data['id'] }}
    - arg:
      - nginx
{% endif %}

You’ll notice that this uses a little logic before the start_service block, which just defines whether the nginx service is in a running state. If it isn’t (i.e. =false) then it proceeds with restarting it based on the id passed from the beacon event, so it knows which minion to run this on. We could easily hardcode the minion id here, but we want our automation to be scalable!

So what does this all look like in practice you ask? The first video is where an evil hacker has got into my ubuntu server and tries to lay claim to my wonderful html, lets call this ficticious person Nico Vibert.

And just to round this off, look what happens when the evil hacker tries to take my website down by stopping the nginx service

For those interested in the files for this series, they can be found on my GitHub. Having quite enjoyed playing with Salt, I hope to produce some more focused around using Salt in a Windows environment.