Local gem cache to accelerate bundler

I’ve been finding that bundle install is really slow for me. It downloads the entire specs file from rubygems.org every time I run it. Maybe that’s because I’m using rvm, and it can’t find previously downloaded specs. Or something. I haven’t investigated.

And Gemfile.lock is constantly causing conflicts when I switch between branches. Which is a Right Royal PITA.

After some not insignificant frustration, I realised that I could do something like this in Gemfile. It is after all just ruby code:

Gemfile

# compute the latest downloaded gemspec file
def latest_spec_date
  Gem::Specification.dirs.
    flat_map{|spec_dir| Pathname(spec_dir).children}.
    map{|p| p.stat.mtime}.
    sort.
    last.
    to_date
end

# can override this using
#  GEM_SPEC_AGE=-1 bundle install
def spec_cutoff_date
  Date.today - (ENV['GEM_SPEC_AGE'] || 14).to_i
end

if latest_spec_date < spec_cutoff_date
  # more than two weeks old, so fetch
  source 'https://rubygems.org'
else
  # use local cache
  source 'file:///var/cache/gems'
end

gem 'fastandand'
gem 'hashery'
gem 'rubyzip'
gem 'nokogiri'
gem 'redis'
gem 'sequel'
gem 'pry'
gem 'rb-inotify'

group :test do
  gem 'rspec'
  gem 'builder'
  gem 'faker'
  gem 'sqlite3' if File.exist? '/usr/include/sqlite3.h'
end

Asidedly, this is also handy because one of the boxes this is deployed to does not have sqlite installed. Which may sound weird, but I don’t have root access on that box. And getting things installed is a lot more painful than just writing some code.

And now all I need is a way to keep /var/cache/gems up to date and with the correct files and directories:

/usr/local/bin/update-gem-cache

#! /bin/bash

cache_dir=/var/cache/gems

mkdir -p ${cache_dir}/gems
mkdir -p ${cache_dir}/specifications

# clean up broken links
# find -L ${cache_dir} -type l -exec rm {} \;

# link existing gems and gemspecs
# find /usr/local/rvm/ -name '*.gemspec' -exec ln -s {} ${cache_dir}/specifications/ 2>/dev/null \;
# find /usr/local/rvm/ -name '*.gem'     -exec ln -s {} ${cache_dir}/gems/ 2>/dev/null \;

# copy works better than link because otherwise bundle won't update in some cases
find /usr/local/rvm/ -name '*.gemspec' -exec cp -u {} ${cache_dir}/specifications/ \;
find /usr/local/rvm/ -name '*.gem'     -exec cp -u {} ${cache_dir}/gems/ \;

# generate the index files needed by gem and bundle
gem generate_index --no-legacy --modern -d ${cache_dir}/

# update only works when /var/cache/gems/specs.4.8 exists
#gem generate_index --no-legacy --modern --update -d /var/cache/gems

Now bundle install talks to the local cache, and it’s fast.

There's enough for everyone

गते गते पारगते पारसंगते बोधि स्वाहा गते गते पारगते पारसंगते बोधि स्वाहा

Comments