How To Rewrite A Project

(Succesfully)

This is motivated by watching several teams fail to complete a rewrite at my day job, my own mixed experiences with big rewrites, and my love for high quality snark[1].

Big rewrites are a tricky thing. People have written lots of things about how they are terrible and fraught. They are right! Your first choice should be to do something other than rewrite it, like writing integration tests[2]. Integration tests are significantly less fun than redoing everything from scratch, and occasionally you get stuck with a codebase so awful that it doesn’t matter what you do as prep, you’re still going to have to rewrite the thing to keep people sane.

The Pitfalls

The most common failure modes of a rewrite are “too unfinished to live”, and “deprecated by ongoing developments”. The first is where you’ve made progress, but not enough to use, and after some amount of time that makes everyone upset, you all just abandon the thing. “Deprecated by ongoing developments” is when ‘emergency patches’ made to the original codebase, but not necessarily ported back to the hot-new-thing, become significant enough that no one trusts or loves the hot-new-thing enough to keep it going.

You should design your rewrite around making these failure modes as unlikely as possible. It’s going to take some sacrifices though.

Stop Having Fun

The goal, at all times, needs to be shipping. Improvement is how you get stuck. If I had a nickel for every time people wandered off on a yak shaving quest during a rewrite or refactor that got them totally stuck I’d have another cup of coffee.

Ship It Piecemeal

If you’re not rewriting in a new language, or you are, but you’re writing a server backend, an excellent trick is to port over API endpoints piecemeal, until the original piece of software is just a facade that calls the new. Then you can delete the old.

This is great for several political reasons. First, it means you can put the new software into production pretty quickly. All software does trivial things- port those things first. As soon as it’s production code, it’s difficult to delete or abandon. This saves you from bit-rot and enthusiasm drop after you’ve all been pulled away to deal with some unplanned nonsense. It also means you’ll get a chance to sort out the dreaded “unknown-unknowns”, where you discover that the new logging library is is a tire fire, or that you don’t know how to not leak memory in your new language, before it’s an emergency.

If you’re not blessed with the ability to use a shim, you should still sit around and scheme up a way to use the new and old system in parallel, because it’s likely you will. This is a time to break out the clever hacks and the bad puns.

Port It Automatically

If you can, this is the best way. Strongly structured things, like deployment projects, are best for this. You get a system that works at the end. Will it be all-singing, all-dancing, filled with candy and Best Practices?

It will not. It will work though, which means you can continue making it better, instead of going back to whatever was so awful that you decided it was impossible to improve.

Do not write a gnarly script to convert it, and then check in the results. Your first try is going to be wrong. So is your second, and so on. Check in the scripts, check in usage instructions, add a .gitignore thing to ignore the automatic output for now. Or your version control system will be useless, and filled with thousands and thousands of lines of garbage.

You Will Not Get To Finish Uninterrupted

The Iron Law of commercial software development is There Will Be Interruptions. Plan for it. There’s going to be a hotfix that pulls you away.

Keep Expectations Low

Everyone hates $oldthing. We’re all mostly sold on $newthing. There will be flaws. Try not to let it all drown in an orgy of grumpiness whenever you discover that Go’s garbage collector is not quite as sophisticated as Java’s G1.

[1]: there are always two sides

[2]: the great thing about integration tests is that they’re sort of portable, and ideally less brittle and more useful than the tautological unit tests favored by cargo-cult TDD-ists. If you’re committed to rewriting something, maybe do integration tests first.

Ansible v2 is now testable

https://github.com/ansible/ansible/tree/devel/v2/hacking

“source v2/hacking/env-setup” to see if you’ve broken all of your finicky monkey-patching action plugins.

Exchange…

If you are having trouble sending plain-text email to Exchange webmail or Outlook users, you should try switching your linebreaks from \r\n to \n. This will convince Exchange to show text, instead of a blank white page.

Yes, I do know that the standards specify CRLF for email.

Ansible batching is an enormous hack

So let’s say you want to do something that would benefit from batching, like, say, installing a bunch of packages on Debian system. In case you haven’t done this or thought about this recently, this command:

apt-get install foo

takes a fairly long time, even if foo isn’t a real package. If you’re installing a bunch of packages, running

apt-get install bar
apt-get install baz

gives you a pretty ugly constant overhead for each operation. On the other hand,

apt-get install bar baz qux

means you only incur apt-get’s bootstrapping and locking overhead once. So batching is great, especially if you’re developing deployment software and want to be running lots of tests.

Ansible, if you haven’t heard of it, is a deployment/orchestration/configuration management tool that many people like very much. You tell Ansible you want a system to have packages on it sort of like this:

---
- name: Install dependency
  apt: pkg=foo

and you can do batch operations like so:

---
- name: Install dependencies
  apt: pkg={{item}}
  with_items:
    - foo
    - bar
    - baz

and everything just works out. If you’ve used Ansible some more though, you may have realized that only package management works this way. Other things don’t seem to support batching.

Well, let’s look at the implementation! It’s all in Python, and a very simple, readable style at that. If you were going to choose a deployment system for readability and hackability, Ansible would be the one. So go to your Ansible checkout, or github, and open up ‘library/packaging/apt’. Here’s the module’s interface definition:

    module = AnsibleModule(
      argument_spec = dict(
        state = dict(default='installed', choices=['installed', 'latest', 'removed', 'absent', 'present']),
        update_cache = dict(default=False, aliases=['update-cache'], type='bool'),
        cache_valid_time = dict(type='int'),
        purge = dict(default=False, type='bool'),
        package = dict(default=None, aliases=['pkg', 'name']),
        default_release = dict(default=None, aliases=['default-release']),
        install_recommends = dict(default='yes', aliases=['install-recommends'], type='bool'),
        force = dict(default='no', type='bool'),
        upgrade = dict(choices=['yes', 'safe', 'full', 'dist']),
        dpkg_options = dict(default=DPKG_OPTIONS)
      ),
      mutually_exclusive = [['package', 'upgrade']],
      required_one_of = [['package', 'upgrade', 'update_cache']],
      supports_check_mode = True
    )

So the variable we’re looking at is called package. How is it used? First, there’s this:

    p = module.params

Then there’s this:

    packages = p['package'].split(',')

wait- what!? How did that happen- this isn’t how other modules work. Well, you should look at lib/ansible/runner/__init__.py:

        if len(items) and utils.is_list_of_strings(items) and self.module_name in [ 'apt', 'yum', 'pkgng' ]:
            # hack for apt, yum, and pkgng so that with_items maps back into a single module call
            use_these_items = []
            for x in items:
                inject['item'] = x
                if not self.conditional or utils.check_conditional(self.conditional, self.basedir, inject, fail_on_undefined=self.error_on_undefined_vars):
                    use_these_items.append(x)
            inject['item'] = ",".join(use_these_items)
            items = None

Suddenly, much is clear. This, by the way, also works to batch installs:

---
- name: Install dependencies
  apt: pkg=foo,bar,baz

There are also other modules with slightly different ad-hoc batching, like mysql_user’s handling of privilege lists:

def privileges_unpack(priv):
  """ Take a privileges string, typically passed as a parameter, and unserialize
  it into a dictionary, the same format as privileges_get() above. We have this
  custom format to avoid using YAML/JSON strings inside YAML playbooks. Example
  of a privileges string:

   mydb.*:INSERT,UPDATE/anotherdb.*:SELECT/yetanother.*:ALL

  The privilege USAGE stands for no privileges, so we add that in on *.* if it's
  not specified in the string, as MySQL will always provide this by default.
  """
  output = {}
  for item in priv.split('/'):

This, as you may have guessed, does not work with with_items.

Two solutions

So, you have a problem.

Your problem is that you have so much code that you cannot use your source control tool in a timely fashion, because it wants to check everything before doing things, and everything is too large a thing. Many people, when faced with this problem, would consider partitioning the code into logical segments, therefore changing ‘everything’ into ‘a reasonably sized thing’ for most or all cases of thing.

Alternatively, you could short circuit your version control’s “check the state of the code” functionality so that it is probably correct and much faster as long as you have a reliable background daemon running.

Then, because ‘everyone’ is too large and ‘everything’ is too big, you could lazily load data, giving you an ad-hoc logical partitioning that’s probably OK unless the network is down or slow or somebody ran find because they’re the kind of old-school asshole who does that.

Congratulations.