SaltStack SDB Interface

The SDB (Simple Data Base) interface is designed to store and retrieve data that, unlike pillars and grains, is not necessarily minion-specific. It is a generic database interface for SaltStack.

We will show how we can make use of SDB for storing and retrieving passwords.

SDB Configuration

The SDB interface requires a profile to be set up in the master configuration file.

/etc/salt/master.d/passwords.conf

The configuration stanza includes the name/ID that the profile will be referred to as, a driver setting, and any other arguments that are necessary for the SDB module that will be used.

 pwd:
     driver: json
     data: /srv/salt/common/pwd.json

We will store the data in the JSON format and make use of the sdb execution module to get, set and delete values from this file. These three methods can be implemented as follows in the python script /srv/salt/_sdb/json.py

'''
SDB module for JSON

Like all sdb modules, the JSON module requires a configuration profile
to be configured in either the minion or, as in our implementation,
in the master configuration file (/etc/salt/master.d/passwords.conf).
This profile requires very little:

    .. code-block:: yaml

      pwd:
        driver: json
        data: /srv/salt/common/pwd.json

The ``driver`` refers to the json module and ``data`` is the path
to the JSON file that contains the data.

This file should be saved as salt/_sdb/json.py

.. code-block:: yaml

    user: sdb://pwd/user1

CLI Example:

    .. code-block:: bash

        sudo salt-run sdb.get sdb://pwd/user1
'''
from __future__ import absolute_import
from salt.exceptions import CommandExecutionError
import salt.utils
import json

__func_alias__ = {
    'set_': 'set'
}

def _read_json(profile):
    '''
    Return the content of a JSON file
    '''
    try:
        with salt.utils.fopen(profile['data'], 'r') as fp_:
            return json.load(fp_)
        except IOError as exc:
            raise CommandExecutionError(exc)
        except KeyError as exc:
            raise CommandExecutionError(
                '{0} needs to be configured'.format(exc))
        except ValueError as exc:
            raise CommandExecutionError(
                'There was an error with the JSON data: {0}'.format(exc))

def get(key, profile=None):
    '''
    Get a value from a JSON file
    '''
    json_data = _read_json(profile)
    return json_data.get(key, {})

def set_(key, value, profile=None):
    '''
    Set a key/value pair in a JSON file
    '''
    json_data = _read_json(profile)
    json_data[key] = value

    try:
        with salt.utils.fopen(profile['data'], 'w') as fp_:
            json.dump(json_data, fp_, indent=2, sort_keys=True)
    except IOError as exc:
        raise CommandExecutionError(exc)

    return get(key, profile)

You can now store the hashed passwords in the JSON data file

{
  "user1": "$5$tEpxpTHeP...0128tglwMKE.X9b88fO4x0",
  "user2": "$5$n4XiZajqf...P3BrvFM5hYq.UazR4dHxl8"
}

and quering SDB to get the hashed strings:

$ sudo salt-run sdb.get sdb://pwd/user1

Of course the SDB query can be coded in a pillar .sls file.

users:
  user1:
    fullname: User One
    uid: 2000
    gid: 1000
    password: {{ salt['sdb.get']('sdb://pwd/user1') }}

SaltStack Execution Modules

SaltStack (or Salt, for short) is a Python-based open-source configuration management software and remote execution engine. It supports the “Infrastructure as Code” approach (the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools) to deployment and cloud management.

I’ll describe in this short article how you can extend the Salt functionalities through the addition of a new execution module.

Just to clarify, you don’t need to be able to write Python or other code to use Salt in the normal usage case. Adding extensions to Salt is an “advanced”, but quite interesting topic.

A bit of theory

A Salt execution module is a Python (2.6+) or Cython module, though some specificities do exist, placed in a directory called _modules at the root of the Salt file server, usually /srv/salt.

An execution module usually define a __virtual__() function, to determine whether the requirements for that module are met, and the __virtualname__ string variable, that is used by the documentation build system to know the virtual name of a module without calling the __virtual__ function.

The following example from the official documentation should clarify this point.

A huge and consistent corpus of libraries and functions are packaged in the Salt framework. You can speed up and highly simplify the development of your modules by making use of those resources.

# Some examples of import:
import salt.utils
import salt.utils.itertools
import salt.utils.url
import salt.fileserver
from salt.utils.odict import OrderedDict

See the official documentation for more information, or… wait for a future article.

As an example, we can write a simple module returning some information about the CPU architecture of a Linux host.

Step by step development

We start by importing some Python and Salt libraries and by defining the  __virtualname__ variable and a __virtual__() function.

# Import Python libs
import logging
# Import Salt libs
from salt.exceptions import CommandExecutionError

__virtualname__ = 'cpuinfo'

def __virtual__():
    '''
    Only run on Linux systems
    '''
    if __grains__['kernel'] != 'Linux':
        return (False,
            'The {0} execution module cannot be loaded: '
            'only available on Linux systems.'.format(
            __virtualname__))
    return __virtualname__

As you can see, __virtual__() returns  False  when the operating system is not Linux. This means that the module cpuinfo will be only available for Linux minions and hidden otherwise.

Salt comes with an interface to derive information about the underlying system. This is called the grains interface, because it presents salt with grains of information. Grains are collected for the operating system, domain name, IP address, kernel, OS type, memory, and many other system properties. The __grains__ dictionary contains the grains data generated by the minion that is currently being worked with.

It’s time now to implement the logic of our module.

log = logging.getLogger(__name__)

def _verify_run(out):
    '''
    Crash to the log if command execution was not
    successful.
    '''
    if out.get('retcode', 0) and out['stderr']:
        log.debug('Return code: {0}'.format(
            out.get('retcode')))
        log.debug('Error output\n{0}'.format(
            out.get('stderr', 'N/A')))
        raise CommandExecutionError(out['stderr'])

def _lscpu():
    '''
    Get available CPU information.
    '''
    try:
        out = __salt__['cmd.run_all']("lscpu")
    except:
        return None

    _verify_run(out)

    data = dict()
    for descr, value in [elm.split(":", 1) \
        for elm in out['stdout'].split(os.linesep)]:
            data[descr.strip()] = value.strip()

    cpus = data.get('CPU(s)')
    sockets = data.get('Socket(s)')
    cores = data.get('Core(s) per socket')
    return (cpus, sockets, cores)

Note that the functions _lscpu() and _verify_run() have their names starting with an underscore (Python weak “internal use” indicator) and thus, by convention, will not be exported by Salt to the public interface.

The Salt method cmd.run_all is used here to execute an external binary (lscpu) and grasp its standard output and error.

The function _verify_run() aims to catch any system error and, when necessary, abort the module execution. This code snippet shows the usage of the Python exceptions in Salt. We raise here a CommandExecutionError exception, declared in the Salt library salt.exceptions if a system error has occurred.

To end our module we implement a function which just calls _lscpu() and parses the user command line arguments (if any), or the module extra arguments, when out module is called by another script. A CommandExecutionError exception is raised for any invalid argument passed to our function.

def lscpu(*args):
    (cpus, sockets, cores) = _lscpu()
    infos = {
        'cores': cores,
        'logicals': cpus,
        'sockets': sockets
    }

    if not args:
        return infos

    try:
        ret = dict((arg, infos[arg]) for arg in args)
    except:
        raise CommandExecutionError(
            'Invalid flag passed to {0}.proc'.format(
            __virtualname__))
    return ret

This function lscpu() is public and will be available on all the Linux minions managed by Salt. Any public method that you define in a module can be invoked by prefixing its name with the corresponding virtual module (cpuinfo in our case):

salt '*' cpuinfo.lscpu

or, if you just need the number of logical CPUs:

salt '*' cpuinfo.lscpu logicals

We have extended Salt.

The final module

When we put all of the preceding code together, we end up with the following code:

'''
SaltStack module returning some information about the CPU
architecture.  This module parses the output of the command lscpu.
'''
# Import Python libs
import logging
# Import salt libs
from salt.exceptions import CommandExecutionError

__virtualname__ = 'cpuinfo'

def __virtual__():
    '''
    Only run on Linux systems
    '''
    if __grains__['kernel'] != 'Linux':
        return (False,
            'The {0} execution module cannot be loaded: '
            'only available on Linux systems.'.format(
            __virtualname__))
    return __virtualname__

log = logging.getLogger(__name__)

def _verify_run(out):
    '''
    Crash to the log if command execution was not
    successful.
    '''
    if out.get('retcode', 0) and out['stderr']:
        log.debug('Return code: {0}'.format(
            out.get('retcode')))
        log.debug('Error output\n{0}'.format(
            out.get('stderr', 'N/A')))
        raise CommandExecutionError(out['stderr'])

def _lscpu():
    '''
    Get available CPU information.
    '''
    try:
        out = __salt__['cmd.run_all']("lscpu")
    except:
        return None
    _verify_run(out)

    data = dict()
    for descr, value in [elm.split(":", 1) \
        for elm in out['stdout'].split(os.linesep)]:
        data[descr.strip()] = value.strip()

    cpus = data.get('CPU(s)')
    sockets = data.get('Socket(s)')
    cores = data.get('Core(s) per socket')

    return (cpus, sockets, cores)

def lscpu(*args):
    '''
    Return the number of core, logical, and CPU sockets,
    by parsing the lscpu command and following back to
    /proc/cpuinfo when this tool is not available.

    CLI Example:

        .. code-block:: bash

            salt '*' cpuinfo.lscpu
            salt '*' cpuinfo.lscpu logicals
    '''
    (cpus, sockets, cores) = _lscpu()
    infos = {
        'cores': cores,
        'logicals': cpus,
        'sockets': sockets
    }
    if not args:
        return infos
    try:
        ret = dict((arg, infos[arg]) for arg in args)
    except:
        raise CommandExecutionError(
            'Invalid flag passed to {0}.proc'.format(
            __virtualname__))
    return ret

You can find other examples in this GitHub page.