Under the hood of d2c.io service we use Ansible a lot: from cloud VM creation and provisioning to Docker containers and user apps orchestration.

Ansible is very flexible by its nature. It is written in Python and has modular design – you can easily extend functionality with custom plugins and modules. Plugins add new functions or change behaviour of local Ansible controller host, whereas modules (being executed remotely) allow support of new software and hardware to be managed. And on top of this, you usually don’t need to do any preconfiguration or setup to make plugins or modules work, they can be tiny Python scripts in special directories within your playbooks.
Let’s look at this playbook:
--- - hosts: localhost vars: foo: - a - b - c tasks: - copy: content: "{{ foo | shuffle }}" dest: /tmp/test
In this case copy
is a module and shuffle
is a Jinja2 filter, provided by filter plugin.
Note: all plugins in Ansible are executed in context of local host (control host)! For example, a common mistake is an attempt to get remote environment variables with env
lookup plugin:
- name: show current user shell: echo {{ lookup('env','USER') }}
In this example lookup will always return the name of user, who executed ansible
process on control host, no matter what is the target remote host for this task.
Konstantin Suvorov
Ansible ninja
Plugin types
Here is an overview of available plugin types (in alphabetical order, for Ansible 2.3.x).
Action
Action plugins are a kind of wrappers for modules. Ansible executes them just before corresponding module is executed remotely. They are usually used to preprocess input data or postprocess results.
This is typical workflow for mymodule
task:
- Ansible searches for
mymodule
action plugin and executes it locally; - Preparation steps are done (e.g. templates evaluation);
- Green light for module execution is given
mymodule
module is executed on remote host;- Result of execution is returned to control host
- Context is switched back to
mymodule
action plugin; - Data postprocessing is done (e.g. filters are applied).
If there is no mymodule
action plugin, Ansible uses base action plugin class.
Cache
Cache plugins are used to connect fact caching backends. By defaul memory
backend is used, meaning gathered facts are stored in-memory during playbook execution. There are several plugins available out of the box: jsonfile
, memcached
, pickle
, redis
, yaml
.
External (persistent) facts cache is usually populated by scheduled facts-gathering playbooks for hundreds of hosts. But in “business-logic” playbooks facts gathering is turned off to speed up execution.
Callback
Callback plugins allow to react on events generated by Ansible core. For example execution log you see in terminal is a result of default
stdout callback plugin that catches events and prints info to standard output. Another example is notification callback plugin slack
– you can enable it an recieve messages about playbook runs into your Slack channel.
Connection
Connection plugins allow Ansible to interact with different target systems: ssh
– for Unix, winrm
– for Windows, docker
– to execute modules inside Docker containers. The most used are ssh
(the default) and local
, which is used to execute modules in local context of Ansible controller.
Filter
Filter plugins add custom Jinja2 filters. Jinja2 template engine is used to work with variables in Ansible. You can use built-in or ansible supplied filters. If you need more, just write a filter plugin.
Lookup
Lookup plugins are used to fetch data from external sources and to make loops.
For example to fetch a key from etcd
you can use {{ lookup('etcd', 'foo') }}
.
To iterate over some command’s stdout, you can use lines
plugin:
- debug: msg: "{{ item }}" with_lines: cat /etc/passwd
In this example cat /etc/passwd
is executed on Ansible controller and debug
is called for every line of its output.
You can create loops with any lookup plugin using with_<plugin-name>:
. When you write with_items
you actually use items
lookup plugin.
The list of available plugins is easy to check via repository. Pay attention to version number, this link is for Ansible 2.3.x.
Shell
Shell plugins allow Ansible to work with different shells on target boxes. For example, bash
or csh
. For Windows targets powershell
plugin is used.
Strategy
Strategy plugins define task execution order for remote hosts. There are three plugins available out of the box:
linear
(the default) – Ansible executes one task on every host in a play, when task is completed on every host, Ansible takes the next taskfree
– Ansible executes tasks as fast as it can. One host can complete a play in a minute and others – in 10 minutes.debug
– based onlinear
– if there is a failed task, interactive debug prompt appears. You can check current variables state, modify variables and tasks parameters and retry the failed task. Documentation.
Terminal
Terminal plugins allow Ansible to use different CLIs. This plugins are used with network appliances (switches, routers) because they are very different to usual linux shells.
Test
Test plugins add Jinja2 tests that you can use in conditional statements. There are built-in and additional tests.
Vars
Vars plugins allow custom behaviour for host and group vars loading process. They are rarely used.
Plugin examples
Let’s make some own plugins from simple ones to more advanced.
Test
At d2c.io we interact with EC2 instances a lot. Imagine you should select running instances from a list. You can use this expression:
{{ ec2.instances | selectattr('state','equalto','running') | list }}
Or make a tini test-plugin (place it into ./test_plugins/ec2.py):
class TestModule: def tests(self): return { 'ec2_running': lambda i: i['state'] == 'running' }
And use:
{{ ec2.instances | select('ec2_running') | list }}
I’ve simplified example a bit, but you can see that instead of making long and complex Jinja2 expression we can program required logic into custom test plugin and keep our code clean and readable.
You can use test filters in when
statements as well:
when: my_instance | ec2_running
The task will be executed only when my_instance
is in running
state.
Tests can have parameters. For example, built-in test divisibleby.
Filter
Filters are used to modify variables. For example there was no tools to work with dates inside Ansible for a long time. And if you want to make some time-based decisions in your playbooks, you can use this filter (place into ./filter_plugins/add_date.py):
import datetime class FilterModule(object): def filters(self): return { 'add_time': lambda dt, **kwargs: dt + datetime.timedelta(**kwargs) }
With this filter you can peek into future:
- debug: msg: "Current time +20 mins {{ ansible_date_time.iso8601[:19] | to_datetime(fmt) | add_time(minutes=20) }}" vars: fmt: "%Y-%m-%dT%H:%M:%S"
Action
Action plugins are useful when you want to modify data on its way to module’s input or filter module’s result. Another use case for action plugins is when you need only local action to happen: for example, debug
module is not actually a module, but just an action plugin and is always executed locally no matter in what host context it is mentioned.
As an example we will modify setup module behaviour. This module is used to gather facts about remote systems and is very handy as ad-hoc
command:
ansible all -i myinventory -m setup
There is filter
parameter for this module to reduce output. But this filter can be applied only to top-level keys in the result. So you can’t display only tz
fields to get server timezone, or specify ipv4
to list only IPv4 fields.
Let’s make an action plugin wrapper (place into ./action_plugins/setup.py):
from ansible.plugins.action import ActionBase class ActionModule(ActionBase): def run(self, tmp=None, task_vars=None): def filter_dict(obj, filter): res = dict() for k, v in obj.items(): if filter in k: res[k] = v elif isinstance(v, dict): val = filter_dict(v, filter) if val is not None and val != dict(): res[k] = val return res result = super(ActionModule, self).run(tmp, task_vars) query = self._task.args.get('query', None) module_args = self._task.args.copy() if query: module_args.pop('query') module_return = self._execute_module(module_name='setup', module_args=module_args, task_vars=task_vars, tmp=tmp) if not module_return.get('failed') and query: return dict(ansible_facts=filter_dict(module_return['ansible_facts'], query)) else: return module_return
Minimal requirements for action plugin is to inherit ActionBase
class and define run
method.
In our example:
- We define helper function
filter_dict
, that applies filter to all keys, not just top-level ones; - Execute
run
from parent class in case it does something useful; - Get value of the new
query
parameter: if it’s there, we pop it from parameters list, otherwisesetup
module will fail (it knows nothing aboutquery
parameter); - Execute original
setup
module to gather facts; - If everything OK and there was a
query
, we callfilter_dict
helper to reduce result, otherwise return result as is.
That’s it. We added new functionality to existing module without touching its code. If setup
module gets updates with new Ansible releases our extension will still work on top of them.
Callback
Callback plugins allow to monitor for differents events that Ansible emits during playbook execution and react on them. The most common use cases for callbacks are logging and notification.
The list of callback plugins available out of the box you can check in the repository.
For example, there are mail
, slack
, hipchat
notification plugins.
To alter terminal output you can use minimal
or json
plugins. You can set configuration option:
[defaults] stdout_callback = json
This way Ansible will not print human-readable log during execution, but will print long JSON object with the information about playbook run. This can be useful to setup cron jobs on your CI
/CD
servers.
For example to execute playbook and count changed hosts we can use:
ANSIBLE_STDOUT_CALLBACK=json ansible-playbook myplaybook.yml | jq ‘.stats | map(select(.changed > 0)) | length’
Let’s make simple notification plugin that sends popup message via system GUI manager when playbook execution is finished (place into ./callback_plugins/notify_me.py):
from ansible.plugins.callback import CallbackBase from subprocess import call from platform import system as get_system_name class CallbackModule(CallbackBase): CALLBACK_VERSION = 2.0 CALLBACK_TYPE = 'notification' CALLBACK_NAME = 'notify_me' CALLBACK_NEEDS_WHITELIST = True def v2_playbook_on_stats(self, stats): def notify(msg,is_error=False): sys_name = get_system_name() if sys_name == 'Darwin': sound = "Basso" if is_error else "default" call(["osascript", "-e", 'display notification "{}" with title "Ansible" sound name "{}"'. format(msg,sound)]) elif sys_name == 'Linux': icon = "dialog-warning" if is_error else "dialog-info" rc = call(["notify-send", "-i", icon, "Ansible", msg]) print "error code {}".format(rc) hosts = stats.processed.keys() failed_hosts = [] for h in hosts: t = stats.summarize(h) if t['unreachable'] + t['failures'] > 0: failed_hosts.append(h) if len(failed_hosts) > 0: notify("Failed hosts: {}".format(" ".join(failed_hosts)),True) else: notify("Job's done!")
There’s an attempt to make this plugin MacOS/Linux cross-platform 🙂
We make a subclass of CallbackBase
and define v2_playbook_on_stats
method – it’s being called upon final playbook report is ready. Default stdout plugin prints PLAY RECAP
table with this method. We also define notify
helper function that tries to send a message via OS notification manager.
In v2_playbook_on_stats
body we check all hosts list for hosts with errors (unreachable
or failures
) – if there are any send error notification with list of failed hosts, otherwise send “good” notification with “Job’s done!” message.
Pay attention to CALLBACK_NEEDS_WHITELIST = True
– this setting tells Ansible to enable this plugin only when it is explicitly whitelisted. This way our plugin is ready to use, but will send notifications only when we ask Ansible to do so. You usually don’t want this functionality for playbook development, but want to enable it for long-running jobs when you send terminal to background:
ANSIBLE_CALLBACK_WHITELIST=notify_me ansible-playbook test.yml
Full list of methods(events) which you can handle in callback plugins you can find in the repository.
Нам понадобится вспомогательная функция notify
, которая в зависимости от платформы пытается отправить пользователю оповещение.
Seems like it is enough information and examples to start playing with. We will try to cover more plugin types in the following article. Stay tuned…