Puppet Slow With Processing Directory with ‘recurse => true’

The Problem

Puppet allows deploying a structure of directories and files conveniently using the file resource.

Using the attributes ‘source => …’ and ‘recurse => true’, it is possible to create a copy of a set of files stored in the puppet module on target servers.

Unfortunately, using these attributes can slow down puppet significantly when the target directory on the server contains many files.

In my case, the time it took to apply a catalog jumped from under 5 seconds to more than 300 s.

The Solution

If you are lucky, the related puppet bug #9671 is closed in your version of puppet (not as of time of this writing).

If not, I found the following strategy to be effective:

  • Replicate your base directory structure using plain file resources (without recurse => true).
  • For subdirectories, do either:
    • Create a resource using ‘recurse => true’, for directories which contain many files/subdirectories in your puppet project.
    • Create more plain file resources for directories, which contain many files on the server.

Using this strategy, I was able to reduce the time to apply mentioned catalog below 10 s, while maintaining most of the convenience offered by the ‘recurse’ attribute.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s