Extending Ansible – modules - D2C — Platform for configuring, deploying and scaling apps with Docker

Under the hood of d2c.io service we use Ansible a lot: from cloud VM creation and provisioning to Docker containers and user apps orchestration.

If you missed previous articles of our “Extending Ansible” series, you should definitely read them to get initial overview of the difference between plugins and modules. The most important one is: plugins are always executed on the localhost (Ansible control host), but modules are executed on remote systems (managed hosts).

Main purpose of plugins is to influence playbook workflow and to add data manipulation capabilities (load/modify/select); As for modules – they are supposed to broaden range of supported target systems/services that Ansible can manage. For example, implement module vultr to provision servers at Vultr cloud, implement module mywifiauth_user to manage users of your proprietary office WiFi authentication system.

Константин Суворов

Ansible ninja

Module execution workflow

Module is a small piece of software that:

is executed on remote host
may take input parameters (via file)
prints result to stdout in JSON format

Basic execution workflow:

Ansible takes task from execution queue, detects module name.
If there is an action plugin with this name, executes it (see “Plugins, p.1” article).
Plugin may do some preparation work (e.g. process templates and deliver files to target host).
Prepares parameters file (based on task parameters).
Depending on module type:
- if module doesn’t utilise “Ansible Framework”, Ansible copies parameter file and module’s script to target host as is
- if module is based on AnsibleModule, Ansible packs parameters, module’s code and required libraries into single Python-file (Ansiballz) and deliveres it to target host. This technique is required to make pipelining mode available (see article about speeding up Ansible).
Ansible executes module or Ansiballz package.
Module do it’s job on remote host.
Ansible reads module output as JSON-object.

In general you use remote servers/appliances as target hosts: e.g. user module manages users on the host where you run it. But some modules are usually targeted to localhost (using connection: local, local_action or delegate_to: localhost), e.g. cloud modules like ec2 which require cloud access credentials defined on localhost. Module wait_for is also usually called on localhost to wait for remote TCP-endpoints to be available (for example SSH-server after host provisioning).

Simple module

You can write module in any language. Starting with Ansible 2.2 you can even use binary executables as modules. Let’s make a simple module in bash:

#!/bin/bash

echo '{"changed":false,"date":"'$(date)'"}'

Save this code into ./library/bash_mod.sh and try it:

$ ansible localhost -m bash_mod
localhost | SUCCESS => {
    "changed": false,
    "date": "среда, 6 сентября 2017 г. 17:00:32 (MSK)"
}

Ansible parses module’s standard output to get its result. For this very reason your module should produce valid JSON-object, so don’t try to debug modules with print statements! You can use several properties of resulting object to alter Ansible behaviour. One of such properties is changed. For example if you change bash_mod code to return "changed": true and execute it, you’ll see that the output is now yellow (task has changed something) instead of green (everything is ok, nothing to do).

Input parameters

There are several ways to get input parameters in your module:

Via file with key=value pairs separated with spaces. This method is used for modules (scripts) written in interpreted languages. The path to this file is passed as the only parameter to your script. Our bash_mod is an example of such module.
Via file with JSON-object. This method is used for binary modules, modules based on AnsibleModule and modules in interpreted languages that have WANT_JSON keyword in its bodies. E.g. we can add # WANT_JSON comment into our bash_mod to switch to this method.
Via JSON-object injection. This method is used for modules in interpreted languages that have <<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>> marker in its bodies. This marker is replaced with JSON-object inplace and module is sent to remote host for execution.

If you develop module with AnsibleModule class, you don’t have to worry about parameters – this base class will do all required under the hood work for you.

AnsibleModule class

To simplify module development Ansible provides AnsibleModule base class. It provides all necessary functions to handle parameters (input process, type checking, require/supported values checking) and some additional helpers (file manipulation, hash-checking, etc).

Let’s take a look at ping module as basic example:

from ansible.module_utils.basic import AnsibleModule

def main():
    module = AnsibleModule(
        argument_spec=dict(
            data=dict(required=False, default=None),
        ),
        supports_check_mode=True
    )
    result = dict(ping='pong')
    if module.params['data']:
        if module.params['data'] == 'crash':
            raise Exception("boom")
        result['ping'] = module.params['data']
    module.exit_json(**result)

if __name__ == '__main__':
    main()

You can see a module with a single optional parameter data that supports check mode (--checkoption in Ansible). It will return pong or value of data parameter if any. Also it will raise an exception if you pass crash as input data.

If you want to dive deeper, you can check other modules’ code. Ansible is an open source project, so you can learn by looking in to the code.

Custom module example

One of possible use cases for custom modules is a wrapper for shell command. If there is a task you do with cli command on different remote hosts and in differen parts of your playbook/project, you may want to wrap it into idempotent module with check-mode support to make your playbooks’ code clean and easy to read. We’ll make a bit artificial but demonstrative example – a module to set OS volume level:

#!/usr/bin/python
# -*- coding: utf-8 -*-

DOCUMENTATION = '''
---
module: osx_volume
short_description: Set OS X volume level
description:
   - Set OS X volume level or mute flag
options:
    level:
        description:
            - Volume level to be applied
        aliases:
            - volume
        required: false
    muted:
        description:
            - Set mute on/off
        required: false
author:
    - Konstantin Suvorov
'''

EXAMPLES = '''
- name: Set volume to 25
  osx_volume:
    level: 25
- name: Mute
  osx_volume:
    muted: yes
'''

from ansible.module_utils.basic import AnsibleModule
from subprocess import call, check_output

def get_volume():
    level = check_output(['osascript','-e','output volume of (get volume settings)']).strip()
    muted = check_output(['osascript','-e','output muted of (get volume settings)']).strip()
    muted = (muted.lower() == "true")
    return (int(level), muted)

def set_volume(level=None, muted=None):
    if level is not None:
        call(['osascript','-e','set volume output volume {}'.format(level)])
    if muted is not None:
        mute_str = 'true' if muted else 'false'
        call(['osascript','-e','set volume output muted {}'.format(mute_str)])
    return get_volume()

def main():
    module = AnsibleModule(
        argument_spec=dict(
            level=dict(type='int', required=False, default=None, aliases=['volume']),
            muted=dict(type='bool', required=False, default=None)
        ),
        supports_check_mode=True
    )
    req_level = module.params['level']
    req_muted = module.params['muted']

    l, m = get_volume()
    result = dict(level=(req_level if req_level is not None else l),
                  muted=(req_muted if req_muted is not None else m),
                  changed=False)

    if req_level is not None and l != req_level:
        result['changed'] = True
    elif req_muted is not None and m != req_muted:
        result['changed'] = True

    if module.check_mode or not result['changed']:
        module.exit_json(**result)

    new_l, new_m = set_volume(level=req_level, muted=req_muted)

    if req_level is not None and new_l != req_level:
        module.fail_json(msg="Failed to set requested volume level {} (actual {})!".format(req_level, new_l))
    if req_muted is not None and new_m != req_muted:
        module.fail_json(msg="Failed to set requested mute flag {} (actual {})!".format(req_muted, new_m))

    module.exit_json(**result)

if __name__ == '__main__':
    main()

This is a module for MacOS to get/set volume level via osascript command. It also get/set mutesetting. Module supports check mode (dry run), so it will return changed: true state if change is required but will not apply this changes. The module is idempotent, so it will not do anything and return changed: false if there is nothing to change. The code is stright forward and uses AnsibleModule discussed above.

Debugging modules

The simpliest way that requires zero setup is to run Ansible with ANSIBLE_KEEP_REMOTE_FILES=1environment variable set and with -vvv verbosity option. This way Ansible will not cleanup files/packages on target hosts after execution. Increased verbosity level allows you to note the path to that files, so you can login to remote host in question via SSH, switch to that folder (e.g. cd /tmp/ansible-tmp-1488291604.43-129413612218427) and execute/change/debug your module.

If your module is based on AnsibleModule, there are a couple of debug commands exist:

explode – unpack Ansiballz file into temporary folder
execute – start module from temporary folder (with modified code)

For example for ping module:

./ping.py – execute original module from Ansiballz package
./ping.py explode – extract package content into debug_dir
./ping.py execute – execute module from debug_dir (with all modifications you applied)

Another way to debug module is to use special test-module tool. It tries to mimic Ansible’s behaviour in handling module packing and parameters passing but without using Ansible workflow. This way you can run rapid tests on your local machine and execute test-module under a debugger.

Modules distribution

For Ansible to be able to find your module, it should be available inside a directory on Ansible control host within search path. By default it is a ./library directory near your playbook. But you can change this setting via ansible.cfg or environment variable.

If you use roles, you can ship your module with role. Make a library subdirectory and place your module there: e.g. ./roles/myrole/library/mymodule.py. This way if you apply myrolesomewhere during a playbook, mymodule become available in the same playbook. This can be a dummy role just for module distribution (without tasks/main.yml file).

Module documentation

It’s useful to document your modules! If your module is writted in Python and you have DOCUMENTATION and EXAMPLES strings in supported format (see module example above), then the documentation for your module will be available via ansible-doc standard tool.

This tool can also generate module snippets ready to be copy-pasted into your playbook. For example:

$ ansible-doc -s postgresql_db
- name: Add or remove PostgreSQL databases from a remote host.
  action: postgresql_db
      encoding               # Encoding of the database
      lc_collate             # Collation order (LC_COLLATE) to use in the data
      lc_ctype               # Character classification (LC_CTYPE) to use in t
      login_host             # Host running the database
      login_password         # The password used to authenticate with
      login_unix_socket      # Path to a Unix domain socket for local connecti
      login_user             # The username used to authenticate with
      name=                  # name of the database to add or remove
      owner                  # Name of the role to set as owner of the databas
      port                   # Database port to connect to.
      ssl_mode               # Determines whether or with what priority a secu
      ssl_rootcert           # Specifies the name of a file containing SSL cer
      state                  # The database state
      template               # Template used to create the database

Seems it’s time to wrap up our article about modules. If you have read all articles from this series, you are ready to extend Ansible in any direction!