My Todo List 2016-11-29

With the US Thanksgiving holiday this past week, I decided to take some time and write down all the projects that are still on my overall todo list. To lead off with a disclaimer: this is not a promise I’ll ever get to any of these or that they will happen in any particular order. This is mostly for my own benefit but hopefully it provides some value to others, if only documenting what still needs fixing. Also this is just my own list and does not reflect the opinions of Chef as a project or Chef Software as a company, and is neither in a specific order nor exhaustive.

If any or all of these projects sound interesting to you and you would like to see them happen sooner, maybe consider checking out my job search page?

Finish Porting Foodcritic Rules to Rubocop-Chef

This is in-progress now but still has a ways to go. You can see the current status in the rubocop-chef README. The goal is to replace as much of Foodcritic as possible with Rubocop so that we can get features like multiple message levels, autocorrect, and a unified configuration file. It will eventually be integrated with the cookstyle project as a one-stop-shop for opinions on how to write Chef cookbooks if you want to use an existing style guide.

Secrets Management Abstraction/API

I’ve talked about this a bunch and it’s very close to the top of my list right now. Basically I want to build a modular API for accessing secrets from Chef code so that cookbooks can be written to use “any” storage system (chef-vault, Hashicorp Vault, S3/KMS, etc). The DSL elements will probably look very similar to my citadel cookbook but with an internal configuration system integrated with Chef and plugable backends.

Deeper Integration Between Chef and HashiVault

I wrote up a lot of words about this, but it is still vaporware. Would be nice to spend some cycles making at least some parts of this exist.

RAM Usage Analysis for poise-profiler

This has been sitting half-finished for a while now and needs to get over the finish line. poise-profiler already provides a lot of timing data for Chef performance, I want to add memory usage to that as well to see when a recipe or resource is allocating a large amount of stuff (usually search results) and then not using it.

Meta-Helper Library

A common stumbling block for new and experienced Chef users alike is adding helper methods to Chef. Many of the old patterns in the ecosystem (like Chef::Recipe.send(:include, ...)) are both dangerous and unnecessary. A library cookbook to provide an API to declare various kinds of helper methods could both simplify the process and help document the tradeoffs of the different kinds of helpers.

Implement RFC 33

Chef RFC 33 adds support for a simplified folder structure for cookbooks with the aim of making small wrapper cookbooks or other short one-offs feel less boiler-plate-y. I implemented this long ago, but the branch bit-rotted before we could agree on the RFC so it needs to be started from scratch by now.

Implement RFC 36

Chef RFC 36 adds support for writing Chef recipes and some other Chef-related files in languages other than Ruby (or JSON). Specifically I want to add things like YAML and TOML support for attributes files and maybe some day Python/JavaScript support for recipe code. Similar to 33, there are a few old implementation of this but any new attempt will have to start over.

Implement RFC 75

Chef RFC 75 adds support for assigning more than one policy to a node to allow use of the Policyfile workflow in some situations where having a single compiled object is problematic. I’ve now tried implementing this twice, only to find the approach I used was fatally flawed, but I still think the feature is important and should exist.

Improve Test-Kitchen’s Concurrent Mode

With the surge in Chef releases thanks to the new monthly cadence, CI build times have been rising rapidly. For most of my cookbooks I currently run four test platforms across 18 Chef versions, and Travis only allows for 5 of those to run concurrently at most. Test Kitchen does support its own concurrent execution mode to speed things up, but currently the output ends up intermixed between the concurrent runs making it almost unreadable. For interactive use this can be okay because you just repeat a failed command to see the error message, but in a CI system this isn’t an option. An improved system could buffer the Test Kitchen output to files and then display it all together when an instance completes the command. Not a ton of work, but anything involving threads (which TK’s concurrent mode is built on) is always deceptively difficult.

Use docker exec for Kitchen-Docker

Currently the kitchen-docker plugin launches sshd inside the container and then uses the normal Test Kitchen SSH machinery to connect to the new instance. This works okay, but it requires a fair bit of complexity around authentication and sudo access that can sometimes make testing more difficult. In the years since the plugin was first written, Docker has added their own remote execution system in form of docker exec. Switching to this would allow for fewer moving pieces and simplify the default Dockerfile.

Improve Kitchen-Docker Build Performance

The default Dockerfile used by kitchen-docker is unfortunately slow to build sometimes. Much of this is due to the creation of many intermediary layers which could be collapsed together. I’ve already done a lot of this for my own use but it should be ported back out to the rest of the community.

New Cookbook for Supervisord

The current supervisord cookbook is in need of a major upgrade and retrofit. This can take advantage of the improved Python resources from poise-python and the patterns built as part of the new poise-monit cookbook. This would also include adding supervisord support to poise-service, allowing it to be used with any cookbook based around that service abstraction.

Resource-based Java Cookbook

The current Java cookbook has been a source of much frustration in the Chef world. I’ve already got a lot of tools built for managing language runtimes with Chef as part of the poise-python/ruby/javascript work, though undoubtedly Java will bring its own unique complications.

More Providers for Python and Ruby

The poise-python and poise-ruby cookbooks support multiple providers for how to install their respective language runtimes, but they could both stand to add some more options. On the Python side, source installs still need to be added (refactored from the existing source install support for Ruby) and some common PPAs like Deadsnakes should be included. For Ruby, the Brightbox Ruby packages would be a good option for many.

Improved SCL Handling on RHEL

The recent-ish 2.0 release of poise-langauages fixed support for Software Collections on CentOS, but unfortunately broke many RHEL users. It follows the official RedHat instructions to configure the RHSCL repositories through RedHat’s subscription system, but this flat out doesn’t work for most cloud images and development systems. Cross-installing the CentOS SCLs on RHEL seems like a reasonable fix, as well as a more flexible :rhscl provider that works on at least some more of the ways RHEL can be run.

Finish Refactoring Halite Spec Helpers

As part of the Halite project, I built out a lot of helper methods for writing Chef unit tests for complex cookbooks. Most of these are not specifically tied to Halite so I’ve started splitting them out to a new gem: poise-spec. These rest of the helpers need to be split out and then Halite’s helper module should be rewritten to use and extend poise-spec.

Migrate the Chef Documentation

Now that we’ve started to clean up the Sphinx documentation for Chef, I want to move the docs for Chef itself into Chef’s code repository. This will make it easier to require Chef changes to include doc updates in the same way as we currently require tests, as well as making it easier to version the docs alongside Chef. Some complexity will be needed to figure out how to integrate the Chef docs with the rest of the Chef’s product documentation so it can all be published to but I’m sure we can work through that.

Integrate YARD and Sphinx

The current Chef docs are written using an excellent framework called Sphinx. It has lots of tools and helpers for writing high-quality narrative documentation, but most of it’s reference documentation tools are aimed at the Python and C worlds where Sphinx comes from originally. The gold standard for auto-extracted code/reference documentation in the Ruby world is YARD, and we’ve already begun adding more YARD comment markup throughout Chef and the broader ecosystem. The missing link between them is a way to integrate YARD data into Sphinx docs directly. A related project here is using YARD data to generate cookbook READMEs like other Sphinx documents.

Rewrite Poise Using New APIs

My Poise helper library has aged reasonably but enough has changed in Chef core to make a major upgrade look more and more inviting. This will probably not happen until ~April 2017 (i.e. the Chef 13.0 release), when it will be more reasonable to drop support for Chef <12.5. Notably the new resource property APIs will allow major refactoring of Poise’s helper methods and a lot fewer uses of define_method.

Other Stuff

Outside of the Chef world I’ve got a few projects kicking around in my brain. I’m a strong believer that Kubernetes is going to win the container orchestration wars, but it currently has pretty much nothing for testing and static analysis. Ditto on Terraform for infrastructure management (though there is a kitchen-terraform plugin that looks very interesting for the future). Someone bringing the same “teaching ops folks how to software” approach I’ve aimed at the Chef community is probably going to be needed in the long run.


I’ve got a big backlog of projects in the Chef world (and beyond) and if you want to help make them happen you should hire me.

Back to articles