Author Archives: Michael F. Weinberg

Modular SparkleFormation cfn-init Configsets

Originally published for AWS Advent 2016

This post lays out a modular, programmatic pattern for using CloudFormation Configsets in SparkleFormation codebases. This technique may be beneficial to:

  • Current SparkleFormation users looking to streamline EC2 instance provisioning
  • Current CloudFormation users looking to manage code instead of JSON/YAML files
  • Other AWS users needing an Infrastructure as Code solution for EC2 instance provisioning

Configsets are a CloudFormation specific EC2 feature that allow you to configure a set of instructions for cfn-init to run upon instance creation. Configsets group collections of specialized resources, providing a simple solution for basic system setup and configuration. An instance can use one or many Configsets, which are executed in a predictable order.

Because cfn-init is triggered on the instance itself, it is an excellent solution for Autoscaling Group instance provisioning, a scenario where external provisioners cannot easily discover underlying instances, or respond to scaling events.

SparkleFormation is a powerful Ruby library for composing CloudFormation templates, as well as orchestration templates for other cloud providers.

The Pattern

Many CloudFormation examples include a set of cfn-init instructions in the instance Metadata using the config key. This is an effective way to configure instances for a single template, but in an infrastructure codebase, doing this for each service template is repetitious and introduces the potential for divergent approaches to the same problem in different templates. If no config key is provided, cfn-init will automatically attempt to run a default Configset. Configsets in CloudFormation templates are represented as an array. This pattern leverages Ruby’s concat method to construct a default Configset in SparkleFormation’s compilation step. This allows us to use Configsets to manage the instance Metadata in a modular fashion.

To start any Instance or Launch Config resources should include an empty array as the default Configset in their metadata, like so:


sparkleformation/templates/example_instance.rb:
resources do
  example_ec2_instance do
    type 'AWS::EC2::Instance'
    metadata('AWS::CloudFormation::Init') do
      _camel_keys_set(:auto_disable)
      configSets do
        default [ ]
      end
    properties do
      ...
    end
  end
end

Additionally, the Instance or Launch Config UserData should run the cfn-init command. A best practice is to place this in a SparkleFormation registry entry. A barebones example:

sparkleformation/registry/cfn_init_user_data.rb:
SfnRegistry.register(:cfn_init_user_data) do
  user_data(
    base64!(
      join!(
        "#!/bin/bash\n",
        "apt-get update\n",
        "apt-get -y install python-setuptools\n",
        "easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
        '/usr/local/bin/cfn-init -v --region ',
        region!,
        ' -s ',
        stack_name!,
        " -r ExampleEc2Instance --role ",
        ref!(:cfn_role),
        "\n"
      )
    )
  )
end
---
sparkleformation/templates/example_instance.rb:
resources do
  example_ec2_instance do
    type 'AWS::EC2::Instance'
    metadata('AWS::CloudFormation::Init') do
      _camel_keys_set(:auto_disable)
      configSets do
        default [ ]
      end    
    properties do
      ...
      user_data registry!(:cfn_init_user_data)
    end
  end
end

With the above code, cfn-init will run the empty default Configset. Using modular registry entries, we can expand this Configset to meet our needs. Each registry file should add the defined configuration to the default Configset, like this:

sparkleformation/registry/example_config_a.rb:
SfnRegistry.register(:example_config_a) do
  metadata('AWS::CloudFormation::Init') do
    _camel_keys_set(:auto_disable)
    configSets do |sets|
      sets.default.concat(['example_config_a'])
    end
    example_config_a do
      packages(:some_package) do
        apt do
          some_package '1.0.0'
        end
      end
      files('/path/to/file') do
        content 'Content string to write to file.'
      end
    end
  end
end

A registry entry can also include more than one config block:

sparkleformation/registry/example_config_b.rb:
SfnRegistry.register(:example_config_b) do
  metadata('AWS::CloudFormation::Init') do
    _camel_keys_set(:auto_disable)
    configSets do |sets|
      sets.default.concat(['example_config_b_1, example_config_b_2'])
    end
    example_config_b_1 do
      packages(:another_package) do
        yum do
          another_package '1.0.0'
        end
      end
      files('/path/to/json_config') do
        content do
          foo do
            bar 'baz'
          end
        end  
      end
    end
    example_config_b_2 do
      system_user do
        groups ['group_w']
        uid 500
        homeDir '/opt/system_user'
      end
    end
  end
end

Calling these registry entries in the template will add them to the default Configset in the order they are called:

sparkleformation/templates/example_instance.rb:
resources do
  example_ec2_instance do
    type 'AWS::EC2::Instance'
    metadata('AWS::CloudFormation::Init') do
      _camel_keys_set(:auto_disable)
      configSets do
        default [ ]
      end
    registry!(:example_config_b)
    registry!(:example_config_a)
    properties do
      ...
      user_data registry!(:cfn_init_user_data)
    end
  end
end

Note that other approaches to extending the array will also work:

sets.default += [ 'key_to_add' ]
sets.default.push('key_to_add')
sets.default << 'key_to_add', etc.

Use Cases

Extending the default Configset rather than setting the config key directly makes it easy to build out cfn-init instructions in a flexible, modular fashion. Modular Configsets, in turn, create opportunities for better Infrastructure as Code workflows. Some examples:

Development Instances

This cfn-init pattern is not a substitute for full-fledged configuration management solutions (Chef, Puppet, Ansible, Salt, etc.), but for experimental or development instances cfn-init can provide _just enough_ configuration management without the increased overhead or complexity of a full CM tool.

I use the Chef users cookbook to manage users across my AWS infrastructure. Consequently, I very rarely make use of AWS EC2 keypairs, but I do need a solution to access an instance without Chef. My preferred solution is to use cfn-init to fetch my public keys from Github and add them to the default ubuntu (or ec2-user) user. The registry for this:

SfnRegistry.register(:github_ssh_user) do
  metadata('AWS::CloudFormation::Init') do
    _camel_keys_set(:auto_disable)
    configSets do |sets|
      sets.default += ['install_curl', 'github_ssh_user']
    end
    install_curl do
      packages do
        apt do
          curl ''
        end
      end
    end
    github_ssh_user do
      commands('set_ssh_keys') do
        command join!(
          'sudo mkdir -p /home/ubuntu/.ssh && sudo curl https://github.com/',
          ref!(:github_user),
          '.keys >> /home/ubuntu/.ssh/authorized_keys'
        )
      end
    end
  end
end

In the template, I just set a github_user parameter and include the registry, and I get access to an instance in any region without needing to do any key setup or configuration management.

parameters(:github_user) do
  type 'String'
  default ENV['USER']
end
resources do
  example_ec2_instance do
    type 'AWS::EC2::Instance'
    metadata('AWS::CloudFormation::Init') do
      _camel_keys_set(:auto_disable)
      configSets do
        default [ ]
      end    
    registry!(:github_ssh_user)
    ...
  end
end

This could also be paired with a configuration management registry entry and the Github user setup can be limited to development:

resources do
  example_ec2_instance do
    type 'AWS::EC2::Instance'
    metadata('AWS::CloudFormation::Init') do
      _camel_keys_set(:auto_disable)
      configSets do
        default [ ]
      end    
    if ENV['development']
      registry!(:github_ssh_user)
    else
      registry!(:configuration_management)
    end
    ...
  end
end

Compiling this with the environment variable development=true will include the Github Configset, in any other case it will run the full configuration management.

In addition to being a handy shortcut, this approach is useful for on-boarding other users/teams to an Infrastructure codebase and workflow. Even with no additional automation in place, it encourages system provisioning using a code-based workflow, and provides a groundwork to layer additional automation on top of.

Incremental Automation Adoption

Extending the development example, a modular Configset pattern is helpful for incrementally introducing automation. Attempting to introduce automation and configuration management to an infrastructure that is actively being architected can be very frustrating—each new component require not just understanding the component and its initial configuration, but also determining how best to automate and abstract that into code. This can lead to expedient, compromise implementations that add to technical debt, as they aren't flexible enough to support emergent needs.

An incremental approach can mitigate these issues, while maintaining a focus on code and automation. Well understood components are fully automated, while some emergent features are initially implemented with a mixture of automation and manual experimentation. For example, an engineer approaching a new service might perform some baseline user setup and package installation via an infrastructure codebase, but configure the service manually while determining the ideal configuration. Once that configuration matures, the automation resources necessary to achieve it are included in the codebase.

CloudFormation Configsets are effective options for package installation and are also good for fetching private assets from S3 buckets. An engineer might use a Configset to setup her user on a development instance, along with the baseline package dependencies and a tarball of private assets. By working with the infrastructure codebase from the outset, she has the advantage of knowing that any related AWS components are provisioned and configured as they would be in a production environment, so she can iterate directly on service configuration. As the service matures, the Configset instructions that handled user and package installation may be replaced by more sophisticated configuration management tooling, but this is a simple one-line change in the template.

Organization Wide Defaults

In organizations where multiple engineers or teams contribute discrete application components in the same infrastructure, adopting standard approaches across the organization is very helpful. Standardization often hinges on common libraries that are easy to include across a variety of contexts. The default Configset pattern makes it easy to share registry entries across an organization, whether in a shared repository or internally published gems. Once an organizational pattern is codified in a registry entry, including it is a single line in the template.

This is especially useful in organizations where certain infrastructure-wide responsibilities are owned by a subset of engineers (e.g. Security or SRE teams). These groups can publish a gem (SparklePack) containing a universal configuration covering their concerns that the wider group of engineers can include by default, essentially offering these in an Infrastructure as a Service model. Monitoring, Security, and Service Discovery are all good examples of the type of universal concerns that can be solved this way.

Conclusion

cfn-init Configsets can be a powerful tool for Infrastructure as Code workflows, especially when used in a modular, programmatic approach. The default Configset pattern in SparkleFormation provides an easy to implement, consistent approach to managing Configsets across an organization--either with a single codebase or vendored in as gems/SparklePacks. Teams looking to increase the flexibility of their AWS instance provisioning should consider this pattern, and a programmatic tool such as SparkleFormation.

For working examples, please checkout this repo.

52 Foods Week Fifty Two: Barhi Dates

Two years ago, we escaped rainy Portland to celebrate New Year’s Eve in sunny Palm Springs. As I mentioned earlier this year, our trip involved a stop at Shields Date Garden, where we sampled many a date, including the wonderfully sweet Barhi. Barhis are small and round, with a slightly parchment-like skin surrounding extremely sweet flesh. They are also quite fragile, easily crushed into a discolored mush. As with the Deglet Noors, I was able to get a package of Barhis from Leja Farms via Siegfried Dates at the Davis Farmers Market. I knew immediately that I would use these for date milkshakes, an Indio, CA specialty I’d been longing to recreate.

The date milkshakes we had at Shields used a special mix of dates, date sugar, and (as I recall) date ice cream. Lacking both the know-how and the variety of date products to follow in their footsteps, I improvised a recipe, looking to date complementing flavors of vanilla and bourbon to elevate my shake.

I began by pitting and chopping a cup of dates and covering them with four ounces of Maker’s Mark bourbon. I chose Maker’s Mark for its sweetness. Other wheated bourbons would work, too, but I think a rye heavy bourbon would add a discordant spiciness.

Soaking in Bourbon

I put four or five scoops of vanilla ice cream into a blender along with four ounces of whole milk. I added the whiskey and dates, then fired up the blender.

Add Dates and Bourbon

The shakes were looking too thin, so I added a few more scoops of ice cream and blended some more.

A Little More Ice Cream

After the second round of ice cream, the shakes blended up nicely. They poured a little thinner than my ideal, but still had some good ice cream chunks. The dates, unfortunately, mostly sank to the bottom and required regular stirring while drinking. The bourbon was very pronounced, but the shake was still smooth, creamy, and datey.

Barhi and Boubon Shakes

In the future, I would probably forgo the milk entirely, since the bourbon provides plenty of liquid, and the alcohol causes the ice cream to melt faster anyway. I would also macerate the dates in the bourbon for a day in advance, and maybe try to work a little date sugar into the mix to bring the date flavor to the fore. More photos of the date shake process are here.

This is the last Fifty Two Foods post, and fittingly I’m writing it on the last day of the year. Thanks for reading. I hope that some of these posts either introduced you to a new food or a new cooking idea. I count seven foods I hadn’t heard of before I wrote about them, and around half the weeks featured things I’d never cooked before. I’ve also realized that without bacon and its cousins, I would have a very hard time coming up with meals. I hope your 2012 is filled with good food; I know mine will be.

52 Foods Week Fifty One: Guinea Fowl

Back in January, I shared the delicious chicken we got from Cache Creek Meat Company. Since then I’ve frequently stopped by their table for both simple and exotic animals, only to be turned away because they were sold out. Seeking to remedy this before the year ended, I leapt out of bed and went straight to the Farmers’ Market, a few weeks ago, getting there in time to have my pick of beasts. As luck would have it, they were flush with species, and I had my choice of chicken, duck, guinea fowl, and rabbit. Having just stocked the freezer with our CSA share, I could not bring home every animal that I would have enjoyed. I went with the one option I’d never had before: the guinea fowl.

A small, sartorially varied beast, the guinea fowl has a nice balance of dark and light meat, with a slim breast and long legs. Mine—the largest available—was nearly two pounds of bird. I consulted the Silver Spoon for ideas, and settled on a plan involving bacon, herbs, and a pot full of persimmons, potatoes, and onions.

Fowl with Stuffings

I began by cutting in half a few slices of Llano Seco’s wonderfully thick cut bacon. I placed two pieces in the bird’s cavity, along with sprigs of rosemary and thyme. Then I draped a few more pieces of bacon over the guinea fowl’s breast.

Gird with Bacon

I pinned the bacon with some skewers, then trimmed the ends to allow for easy browning of the bird.

Trim Skewers

I brushed our large Le Creuset with oil and did my best to brown the bird allover. Meanwhile, I chopped the onions, persimmons, and potatoes that would cook with it.

Browning the Bird

Once the bird was as well browned as I could manage in the deep pot, I threw in the onions and cooked them until they were translucent.

Sautéing Onions

I returned the bird to the pot, and surrounded it with the onions, persimmons, and potatoes. I’ve seriously embraced persimmons this year, enjoying them both raw and roasted. I particularly enjoy them cooked along with meat in a large pot, with or without potatoes.

Bird in Pot with Onions, Persimmons, and Potatoes

I covered the pot, and put it in the oven at 325°F. After 40 minutes, I removed the bacon from the sides of the bird and the cavity, then left it to cook longer with the lid off. At this point, the bird was still quite pale, but cooking with the lid removed would allow it to darken and let the skin crisp up a little.

After 45 Minutes, Remove Bacon

I sliced the bacon into lardons, and finished them in a pan on the stovetop. After another 15 minutes, the guinea fowl was done, and I removed it from the oven and the pot and let it rest.

Cook Another 15 for Color

I tossed the lardons into the pot with the vegetables, stirred them together, and scooped them into a serving bowl with a slotted spoon. Before disposing of the liquid in the pot, I drizzled a little back on top of the vegetables. Then I carved up the guinea fowl.

Carved Bird and Vegetables

The guinea fowl was very moist and tender. While the light meat was about the color I expected, the legs were impressively dark, even a little ruddy. They were also a little sinewy, but quite tasty, and the breast was succulent, needing just a little salt and carrying a bit of flavor from the bacon. The roast potatoes and persimmons were a good complement—a wonderful mix of sweetness from the fruit and saltiness from the bacon. This was a fun bird to buy and cook.

Guinea Fowl for Serving

All the photos are here.

52 Food Week Fifty: Lamb Shank

As I mentioned last week, we recently signed up for a meat CSA that delivers a nice mix of beef, chicken, and lamb every month. Lamb shanks have so far been a regular inclusion, which delights me to no end, since they are one thing that I’m almost guaranteed to order if they’re available at a restaurant (unless there is rabbit, in which case Thumper usually wins). I had never made lamb shanks at home, but my hand was forced and it was time to attempt one of my favorite dishes.

I did some searching for a recipe that sounded like the Italian-ish preparation to which I’m partial. I finally happened upon an NPR article with a variation on an Alice Waters’ dish that looked both approachable and traditional. I figured that the doyenne of fresh, local cuisine would not lead me astray. I cut the recipe in half to accommodate our household of two and got cracking.

I began by trimming the excess fat and the membrane from the outside of the shanks. Next I rubbed them with salt and pepper, and browned them all over in a hot pan of olive oil.

Browned Lamb Shanks

Once they were browned, I poured off most of the oil, and tossed in some onion, carrot, rosemary, crushed red pepper, and a bay leaf.

Veggies and Spices

Once these were soft, I deglazed the pan with some white wine and a tomato, then put the lamb shanks back in, along with a cup of beef broth. I covered it and put it in the oven, set to 325°F.

Lamb Shanks Ready for Braising

After about two hours of cooking, I removed the lid to let the shanks brown for the final 20 minutes.

About Two Hours Later

Meanwhile, I mixed up a little gremolata of parsley, garlic, and lemon zest.

Gremolata

When the shanks were done, I removed them from the pan, then poured the vegetables into the food processor and puréed them. I then returned the purée to the pan, got it up to a nice simmer, and put the lamb shanks back in for a minute or two.

Shanks in the Sauce

We served the lamb shanks over couscous with the sauce and gremolata on top. This is a fantastic recipe for what I consider the most typical lamb shank preparation. The meat was extremely tender and falling off the bone. We didn’t even need knives. The gremolata adds a nice kick from the garlic and lemon zest. If you’ve got lamb shanks and you’re not sure what to do with them, this is it.

Lamb Shanks Are Served

All the photos are here.

52 Foods Week Forty Nine: Ground Beef

As promised, it’s time for burgers. Burgers may be my favorite meal. In Portland, I did a pretty good job of maintaining at least a burger a week habit, and before we moved to Davis, I made a bucket list that was largely driven by a desire for burgers I’d heard about but hadn’t had yet. Over the last couple years, I’ve tried to perfect my technique to deliver my ideal, medium-rare burger.

We recently joined a meat CSA called the Foragers. Each month we receive a mix of beef, lamb, and chicken. We invited a friend over, and busted out a one pound package of 85% lean ground beef. I’ve found that 1/3 pound burgers are the perfect size—meaty and filling, without being overwhelming. I generally prefer something closer to 75-80% lean beef, but I’m not too picky, as long as it isn’t very lean.

IMG_9205

I divided the beef into equal portions and formed some thick patties.

Weighing Burgers

Then I pressed some salt and pepper into each side of the patties.

Salt and Pepper

I put some bacon grease in a cast iron skillet and sautéed some onions.

Onions in Bacon Grease

When the onions were done, I put the burgers in. I use a frequent flipping method, flipping every minute. As detailed by A Hamburger Today, this method helps retain moisture and reduce the ratio of overcooked to medium rare meat.

Burgers in the Skillet

I find that nine minutes is about the perfect cooking time for medium-rare. When they were done, I removed them to the buns. I like mayo and a bit of dijon on mine. I added a nice dollop of chèvre to the burger.

Chèvre on the Burger

Since tomatoes are very out-of-season, and persimmons are very in season, I added some sliced Fuyu persimmons.

Persimmons on Top

Finally, I added some butter leaf lettuce and the sautéed onions.

Onions and Lettuce

Everyone got to make the burger the way she liked it.

Everyone Is Ready

The goat cheese and persimmon were a killer combo, a little sweet and much better than a December tomato would have been. Though I enjoy a wide variety of cheeses on my burgers, I’ve found recently that soft cheeses, such as chèvre or bleu cheese, added after the burger is done cooking, are really doing it for me. All the photos are here.