Ansible Best practices

I've been using Ansible since it came out and made a lot of mistakes on the way. So here are my collected best practices.

This page is always changing.

Example playbook repo structure

Lately I've been creating more and more standalone playbook repos for various projects.

Here's one example for mastodon.

mastodon/
    tasks/
      common-tasks.yml
    rbenv.yml
    vars/
      common.yml
    inventory/
      default/
        hosts
        host_vars/
        group_vars/
    bootstrap.yml
    db.yml
    backend.yml
    frontend.yml

And I find that using more includes in tasks makes for easier conversion to roles down the line when that becomes necessary.

So backend.yml is the one you run from the root mastodon/ dir and it might look like this.

---

- hosts: mastodon-backend
    become: True

    vars_files:
    - vars/common.yml

    roles:
    - ...

    vars:
    mastodon_yum_packages:
      - ...

    tasks:
    - include: tasks/common-tasks.yml
    - include: tasks/rbenv.yml

Keep It Simple, Stupid

Ansible is not a magic bullet. If you have a complex service consider creating a package for it first to leverage other install solutions like rpm, dpkg, python setuptools or whatever is available.

Task separation

How to separate tasks? There's no easy answer but my suggestion is to make separation based on how you want to run Ansible.

For example let's say you want to deploy new code with Ansible whenever it's checked into a git repo. Then separate the deployment process from the rest. There's no need to deploy OS packages, systemd service files or configuration files if you've only made code changes to the main app.

Same goes for the rest, if you're only changing configuration there's no need to checkout git code or install systemd services.

It gets tedious running Ansible if every time it has to go through all the motions. So separate based on how you wish to run Ansible.

Handlers

Something that jumped up and bit me unexpectedly was the fact that handlers run in the order you define them, not in the order you specify them in the notify-statement.

become_user

The user you become with become_user cannot have a home directory readable by world.

So this;

become_user: nagios

Requires this;

$ sudo chmod 0750 ~nagios

Roles and input

An important point as your deploys become more complex is variable precedence.

So besides keeping track of that I also recommend overriding default variables in roles when you include the role, as if passing input arguments to a role. Of course that doesn't mean I don't specify these variables elsewhere in many of my deploys. This is really debatable.

roles:
- role: nginx
  nginx_var1: foo
  nginx_var2: bar

Never encrypt host_vars or group_vars

I prefer if encrypted vault files are clearly marked so you know when you need to use --ask-vault-pass instead of being surprised by an error.

Using encrypted host or group vars means you need to use --ask-vault-pass in many contexts where you might not need it. Better have separate vault files that are included on demand in playbooks.

For example

vars_files:
- vars/prod-vault.yml

Create agnostic roles

This was less obvious for me when I started using Ansible than it is now, but worth mentioning that roles should never contain platform/client/project-specific info. They should be usable on any system.

Another good way of looking at it is; your roles should be publishable on github in the future without revealing sensitive data.

Rely more on host groups

Even if they only contain one or two systems, it makes your setup more clear.

[database:children]
database-masters
database-arbitrator

[database-masters]
db01
db02

[database-arbitrator]
db03

Write playbook deployments in steps

For medium and large deployments I tend to make steps and document the run order in a README file.

playbooks/projectX/prod-bootstrap.yml
playbooks/projectX/prod-setup.yml
playbooks/projectX/prod-configure.yml

So for configuration changes you don't have to clone git repos or install RPM packages.

Another example would be a very dynamic set of playbooks where you can change the deployment environment by setting a value to prod or test.

bootstrap.yml
setup.yml
configure.yml

Each would include the following file.

- hosts: all

    vars_files:
    - vars/common.yml
    - "vars/{{env}}.yml"

And you'd run a playbook like this to specify which settings file is used.

ansible-playbook -e 'env=prod' -i hosts bootstrap.yml

Use inventory to switch between environments

My preferred method over the one described above is to use different inventories.

First create ingentories in their own directories so you can also have isolated group_vars.

$ mkdir -p inventory/my_env/group_vars

That way other environments can easily be added in other git branches without conflicting with each other.

The beginning of the inventory will define the environment name.

[all:vars]
site_environment=my_env

[servers]
node01

This can then be used in playbooks to include certain vars files. Note that I always start with a default environment that is normally intended for Vagrant and local testing. So this line will default to the default environment if site_environment is not set.

---

- hosts: all

  vars_files:
    - ["vars/{{ site_environment }}.yml", 'vars/default.yml']

Use virtualenv

With 1.x and certainly in the transition to 2.x this was very important to me.

Now though I could easily recommend using your systems package manager. Homebrew, yum and apt all include the latest release of ansible.

Personally I use pip install --user ansible and have $HOME/.local/bin in my $PATH, or I install from the Fedora package manager.

Prefer template over lineinfile or copy

I don't think I've ever actually needed lineinfile or copy modules.

Lineinfile seems terribly unpredictable to me, even if you use a regexp. There's a risk of garbage insertions into files compared to template where you control the entire file.

Copy is ok for binary files, which I try to avoid having in git. So template wins mostly because I can add the ansible_managed.

Don't forget ansible_managed

Speaking of the ansible_managed, don't forget to use it whenever possible.

Don't forget old files

Using the ansible_managed one might argue that a cleanup playbook should also be available, to uninstall anything you might have done.

I wish I could say I adhere to this but principally I believe in it.

Don't forget spacing

YAML, and jinja templates, allow for spacing so use it to make your work easier to read and follow.

We're not helping anyone by saving bytes.

Create contained playbook directories

I realized this relatively late but I prefer creating self-contained directories, of playbooks, vars, templates and so on, for every project or environment I manage in ansible.

With their own hosts inventory and perhaps even their own ansible.cfg if anything needs to be overridden from the global one.

Here's an example.

ansible/
    roles/
    playbooks/
    playbooks/Client
    playbooks/Client/project/
    playbooks/Client/project/projekt.yml
    playbooks/Client/project/ansible.cfg
    playbooks/Client/project/templates/
    playbooks/Client/project/group_vars/

Ansible use is influenced by your workflow so it will be very personal and depending on what sort of organization you work in. I started using Ansible in a consultant organization with many clients who sometimes only used Ansible in small parts of their delivery.

Separate your roles into their own repo

I never liked Ansible Galaxy because I don't like the idea of trusting some users ansible playbooks and then also having to keep up with any changes made to their roles in the future.

So because of that I maintain a separate repo of custom roles. Anything from simple roles like installing nginx to complex roles that manage an entire part of an in-house application deploy.

Keep them agnostic and keep them separate and you might even be able to share them online.

Don't use Ansible Galaxy

Personal opinion but I'm not going to run foreign playbooks without first checking what they do.

And even then, I need to keep up with changes made to them.

So I'd rather keep that responsibility within the organisation.

Use the global ansible.cfg

I started out putting an ansible.cfg everywhere in my repos, wherever I would happen to run ansible commands from.

This grew out of hand so now I advocate using ~/.ansible.cfg instead.

Of course with this method it's good if everyone within an organisation is united on which ansible_managed string to use.

Troubleshooting

With strange issues I've found -vvv argument and the environment variable ANSIBLE_KEEP_REMOTE_FILES=1 helpful.

Also to keep track of which user is running the module with become.