Vagrant, Chef (installed via omnibus), and the PostgreSQL and Database cookbooks (re: COOK-1406)

blake.willardTechnical Tips2 Comments

While building the development environments for a code base that would eventually be deployed to an auto-scaling AWS environment, a client request pivoted the database tier architecture away from MySQL to PostgreSQL. This pivot was timely given Amazon’s most recent RDS offering, announced as recently as re:Invent. Awesome! Now to integrate PostgreSQL with the existing Vagrant/Chef development environment deployments; not as awesome! What follows is a summary of and solution to several common pitfalls experienced by those deploying PostgreSQL with Chef (omnibus installer). Note: this post assumes some familiarity with Chef.

A quick overview of how our deployment gets PostgreSQL up and running on guests is in order. (Note: we’re using a vagrant-compatible Ubuntu Precise cloud image)

The Database cookbook

What use is automating the binary installation of your database client and server if you can’t also automate your database and database user creation? That’s where the Database cookbook comes into play. The Database cookbook will use the target db architecture’s cookbook’s ::ruby recipe, i.e. mysql::ruby or postgresql::ruby, to do its dirty work. The ::ruby recipes compile and install the proper ruby gems that other recipes can then use to interface with the database. Indeed, if you checkout the database::postgresql recipe it is literally just:

include_recipe "postgresql::ruby"

With this in mind, a typical Chef-driven database installation goes something like this:

# your custom cookbook's database recipe
include_recipe "postgresql::server"
include_recipe "database::postgresql"
conn_info = {
  :host => "localhost",
  :username => "postgres",
  :password => node['postgresql']['password']['postgres']
postgresql_database_user "example_user" do
  connection conn_info
  password "example_password"
  action :create
postgresql_database "example_db" do
  connection conn_info
  action :create
postgresql_database_user "example_user" do
  connection conn_info
  database_name "example_db"
  privileges [:all]
  action :grant

Each postgresql_database* resource is provided by the Database cookbook and leverages the pg gem.


Documentation for both the Postgresql cookbook (see under the subsection entitled “Ruby”) and the Database cookbook (see under the subsection entitled “Resources/Providers”, second bullet) reference ticket COOK-1406, citing this issue breaks usage of these cookbooks when Chef is installed on the guest via the omnibus installer. Guess which method we were using. To make matters worse, the ticket’s comment section meanders across two other issues in addition to the issue that the ticket originated from. Let’s navigate each to get an understanding of what’s going on.

The original issue

Chef, when installed via omnibus, bundles all of its own libraries and Ruby run-time and throws them under the /opt directory. This lets an omnibus Chef install live sovereign on any system it’s deployed to, blissfully unaware of the differences that arise between multiple platform’s system libraries. However, this method can’t account for every possible use case and thus can’t bundle every possible library or header that may be needed during a chef run. So, when the bundled assets aren’t sufficient, Chef will look to system resources to compensate. In the case of building the pg gem, Chef has to utilize libraries installed by the PostgreSQL client on the system itself, namely /usr/lib/ – the PostgreSQL C API. Unfortunately this library and Chef’s embedded run-time use conflicting OpenSSL libraries, a problem remedied by creating a special version of the PostgreSQL client library for use only by Chef, linked and compiled against all of Chef’s bundled libraries as opposed to system libraries. Contrary to the previously cited documentation and most fortunately for anyone who’s made it this far, this special library and re-linking is taken care of for you since commit b26a0dd to the PostgreSQL cookbook repository over a year ago! Using a sufficiently modern version of the PostgreSQL cookbook (i.e. anything >2.0.2) will include this fix and you can continue on using Chef omnibus install and database::postgresql together. It is important to note, however, that the error message provided by the original ticket’s reporter will still appear in your Chef run’s output; despite this, everything should still work as expected.

The second issue

In the same comment thread and presumably after the originating issue had already been solved, a user reports he is still marred by the libpq problem. The posted error message is strikingly similar to the original issue, with one important exception; the Chef run couldn’t find `make`! Oops. This one’s easy however; say a Chef run needs ruby gems to do Chefy things. Ruby gems need build and compilation tools to do Ruby…y things. These tools must be available during Chef’s compile phase because they are being used by Chef itself as opposed to being used by the system after the Chef run has completed. The build-essential cookbook ships with an attribute that let’s you ensure, via your default attributes, its tools are available during the appropriate phase:

"default_attributes": {
  "build_essential": {
    "compiletime": true

The postgresql::ruby recipe will attempt to make sure this attribute is set for you. If, however, you still encounter the commentor’s issue declare the attribute explicitly as shown above. See here for more information regarding the usage of this attribute.

The final issue

It seems COOK-1406 keeps on kicking. If you run into this issue it is likely your chef build order is wacky. If possible, try to keep your run-list as implicit as possible; i.e. look into using Roles or Wrapper cookbooks rather than specifying a run-list in your roles. This allows you to declare only what you directly depend on placing responsibility of additional dependency resolution on those battle tested community cookbooks you’ve downloaded. If you’d like to take it a step further, tools such as Berkshelf or Librarian will handle this sort of thing for you.


To avoid the pitfalls of COOK-1406:

  • use a sufficiently modern version of the PostgreSQL cookbook when installing Chef via omnibus – ignore the error message that will appear;
  • use a version of the Build Essentials cookbook that supports use during Chef’s compile phase;
  • use Roles or Wrapper cookbooks, Berkshelf, Librarian, or some other solution to ease dependency headaches and avoid debugging sessions.

2 Comments on “Vagrant, Chef (installed via omnibus), and the PostgreSQL and Database cookbooks (re: COOK-1406)”

  1. Thanks for sharing the in-depth analysis of why this issue keeps cropping up. I’ve spent the past few days going through this particular issue in a few circles.

    I’m new to chef, and there seem to be many patterns and idealogies about how to organize dependencies. Some eschew roles entirely, while others embrace them. Then you note to use Roles but not run_lists in your roles.

    In particular I recently was working with the repository and still running into this issue. Compared with that pattern (heavy use of roles, but run_lists in the roles) how would you suggest varying it in order to not run into the “wacky order list” you mention?

    Thanks again!

  2. 8 December 2014, this issue is still happening, and NONE of the recommendations here clear the condition.

Leave a Reply

Your email address will not be published. Required fields are marked *