Ansible batching is an enormous hack

So let’s say you want to do something that would benefit from batching, like, say, installing a bunch of packages on Debian system. In case you haven’t done this or thought about this recently, this command:

apt-get install foo

takes a fairly long time, even if foo isn’t a real package. If you’re installing a bunch of packages, running

apt-get install bar
apt-get install baz

gives you a pretty ugly constant overhead for each operation. On the other hand,

apt-get install bar baz qux

means you only incur apt-get’s bootstrapping and locking overhead once. So batching is great, especially if you’re developing deployment software and want to be running lots of tests.

Ansible, if you haven’t heard of it, is a deployment/orchestration/configuration management tool that many people like very much. You tell Ansible you want a system to have packages on it sort of like this:

---
- name: Install dependency
  apt: pkg=foo

and you can do batch operations like so:

---
- name: Install dependencies
  apt: pkg={{item}}
  with_items:
    - foo
    - bar
    - baz

and everything just works out. If you’ve used Ansible some more though, you may have realized that only package management works this way. Other things don’t seem to support batching.

Well, let’s look at the implementation! It’s all in Python, and a very simple, readable style at that. If you were going to choose a deployment system for readability and hackability, Ansible would be the one. So go to your Ansible checkout, or github, and open up ‘library/packaging/apt’. Here’s the module’s interface definition:

    module = AnsibleModule(
      argument_spec = dict(
        state = dict(default='installed', choices=['installed', 'latest', 'removed', 'absent', 'present']),
        update_cache = dict(default=False, aliases=['update-cache'], type='bool'),
        cache_valid_time = dict(type='int'),
        purge = dict(default=False, type='bool'),
        package = dict(default=None, aliases=['pkg', 'name']),
        default_release = dict(default=None, aliases=['default-release']),
        install_recommends = dict(default='yes', aliases=['install-recommends'], type='bool'),
        force = dict(default='no', type='bool'),
        upgrade = dict(choices=['yes', 'safe', 'full', 'dist']),
        dpkg_options = dict(default=DPKG_OPTIONS)
      ),
      mutually_exclusive = [['package', 'upgrade']],
      required_one_of = [['package', 'upgrade', 'update_cache']],
      supports_check_mode = True
    )

So the variable we’re looking at is called package. How is it used? First, there’s this:

    p = module.params

Then there’s this:

    packages = p['package'].split(',')

wait- what!? How did that happen- this isn’t how other modules work. Well, you should look at lib/ansible/runner/__init__.py:

        if len(items) and utils.is_list_of_strings(items) and self.module_name in [ 'apt', 'yum', 'pkgng' ]:
            # hack for apt, yum, and pkgng so that with_items maps back into a single module call
            use_these_items = []
            for x in items:
                inject['item'] = x
                if not self.conditional or utils.check_conditional(self.conditional, self.basedir, inject, fail_on_undefined=self.error_on_undefined_vars):
                    use_these_items.append(x)
            inject['item'] = ",".join(use_these_items)
            items = None

Suddenly, much is clear. This, by the way, also works to batch installs:

---
- name: Install dependencies
  apt: pkg=foo,bar,baz

There are also other modules with slightly different ad-hoc batching, like mysql_user’s handling of privilege lists:

def privileges_unpack(priv):
  """ Take a privileges string, typically passed as a parameter, and unserialize
  it into a dictionary, the same format as privileges_get() above. We have this
  custom format to avoid using YAML/JSON strings inside YAML playbooks. Example
  of a privileges string:

   mydb.*:INSERT,UPDATE/anotherdb.*:SELECT/yetanother.*:ALL

  The privilege USAGE stands for no privileges, so we add that in on *.* if it's
  not specified in the string, as MySQL will always provide this by default.
  """
  output = {}
  for item in priv.split('/'):

This, as you may have guessed, does not work with with_items.