using Chef in 2018

As a bit of a caveat, I know it’s a little weird to be doing a bunch of Chef investment in 2018. I’m in sort of a spot though, where I’m mostly doing support for an application that is deeply container incompatible. So Chef it is.

You need support tooling outside of Chef

Chef does configuration management, but it can’t really do orchestration. By that I mean, it is not a good fit for doing actions in a sequence across a fleet of machines. This turns out to be kind of important in a lot of scenarios. I recommend Ansible for this, it’s simple, it’s very good at doing things in order, in parallel, it has a ton of support for a wide variety of services, and it doesn’t require an agent if you want to do things directly on hosts.

Using Chef and Ansible can feel like a lot of overlap, but knife is not an adequate substitute. If you’re using Chef, you probably also have a lot of investment in cookbooks & configuration, and porting to Ansible would probably be a nightmare. So don’t do it! The key is to keep all the host configuration stuff in Chef, and not start doing half of it in Ansible, and half in Chef. That would be bad.

You also need an automation platform. Jenkins is the king here, I’m not a huge fan, but it’s fine. There are a billion other competitors, too. You need to be testing all your cookbooks, separately, on every commit. You need automation to push cookbooks out after they pass their tests, and you need automation to push out Chef configuration objects- roles, environments, etc. It’s a ton of work to set up, it is an ongoing investment to maintain, it is totally worth it. It is a lever that you will regret not building.

Your automation platform should be as git-based as possible. By that, I mean that changes should go through pull requests, not be done by someone with admin privileges in the GUI. That is how you get weird breakage and snowflake jobs, and those are bad.

Your automation platform should be doing the orchestration things, too. This is always hard from a security perspective, I recommend at least separate dev and prod automation services- different accounts, or servers, or something- and a many-tiered system might be even better. Development automation should be grassroots and self-serve, production automation should be change-controlled, if that makes sense.

Cookbooks

I think the application-cookbook, the top-level wrapper cookbook pattern, is the way to go. That is, servers have one cookbook in their runlist, and it is in charge of everything. That cookbook may just wrap a couple of other cookbooks, but that’s fine. That cookbook can have tests. The recipes it calls will come in a defined order, that you could test. So no more things that break in production because the final run list composed in a weird way, and something happened before it should have.

Toggling features should be done via attributes, not by adding recipes. If that means that at the bottom, it’s a series of if guards around include_recipe statements inside of the cookbook, that’s fine. It will still work better, be more testable, more composable, and safer than setting things in runlists.

A common anti-pattern for cookbook authors is allowing every string they write out to be overridden in attributes. It is true that this approach means that changes do not require cookbook changes. On the other hand, because cookbooks are integration-testable, but environments and node attributes are not, this pushes responsibility for things working to an awkward runtime problem. Environments, roles, nodes, they should have big picture settings, like feature flags, FQDNs, big picture things that people can audit. They should not have a cascade of related attributes coordinating paths that daemons use.

Chef cookbooks can be inaccessible to developers and administrators, and attribute-explosion can seem like a way to isolate cookbook writing to an expert community, but I haven’t really seen it work out. Reconfiguration of fundamentally needs an integration testing story, which cookbooks have.

Community cookbooks should be used sparingly. Many, at base, are apt install and a configuration file, but in a way that does not play nice with other things. Some drag in terrifyng dependencies. Others want to compile the world at runtime. Don’t be too afraid of NIH.

The sous-chef cookbooks are generally a safer bet than the official Chef cookbooks.

Recipes

Pushing logic out of the top-level DSL and into resources is a good idea, and worth doing pretty aggressively. Recipes that are a straight-line list of resources tend to be robust, things with lots of if statements and iterations tend to be less robust.

Writing to attributes is probably bad. There’s node.run_state for collecting things from the whole run.

Chef-Vault

If you’re going to use it, you should keep in mind that you’ll want locking around item updates. So knife vault refresh and node provisioning can’t happen at the same time, and nodes have to be provisioned one at a time. You can work around this by bootstrapping nodes with empty runlists, then doing a single knife vault refresh to bulk update the vault, and then filling out node runlists, but this is also pretty complex. You could also just ignore the problem and work around it with extra out of band `knife vault refreshes occasionally. It will still be a lurking thing, though.

Test Kitchen

Test Kitchen is the best thing about Chef. You should use it everywhere. I’m a little down on the Dokken stuff, kitchen-ec2 gets you a lot more fidelity, and with per-minute billing, it’s super affordable. That’s not a great answer for the hobby-project set, but most Chef is not really hobby-project tech.

Foodcritic

Foodcritic is worth running everywhere. You’ll probably want to ignore a couple of metadata.rb related nitpicks, but there are several classes of bugs and bad ideas it will find faster than full integration tests.