There's enough for everyone

गते गते पारगते पारसंगते बोधि स्वाहा गते गते पारगते पारसंगते बोधि स्वाहा

Local gem cache to accelerate bundler

I’ve been finding that bundle install is really slow for me. It downloads the entire specs file from rubygems.org every time I run it. Maybe that’s because I’m using rvm, and it can’t find previously downloaded specs. Or something. I haven’t investigated.

And Gemfile.lock is constantly causing conflicts when I switch between branches. Which is a Right Royal PITA.

After some not insignificant frustration, I realised that I could do something like this in Gemfile. It is after all just ruby code:


Gemfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# compute the latest downloaded gemspec file
def latest_spec_date
  Gem::Specification.dirs.
    flat_map{|spec_dir| Pathname(spec_dir).children}.
    map{|p| p.stat.mtime}.
    sort.
    last.
    to_date
end

# can override this using
#  GEM_SPEC_AGE=-1 bundle install
def spec_cutoff_date
  Date.today - (ENV['GEM_SPEC_AGE'] || 14).to_i
end

if latest_spec_date < spec_cutoff_date
  # more than two weeks old, so fetch
  source 'https://rubygems.org'
else
  # use local cache
  source 'file:///var/cache/gems'
end

gem 'fastandand'
gem 'hashery'
gem 'rubyzip'
gem 'nokogiri'
gem 'redis'
gem 'sequel'
gem 'pry'
gem 'rb-inotify'

group :test do
  gem 'rspec'
  gem 'builder'
  gem 'faker'
  gem 'sqlite3' if File.exist? '/usr/include/sqlite3.h'
end

Asidedly, this is also handy because one of the boxes this is deployed to does not have sqlite installed. Which may sound weird, but I don’t have root access on that box. And getting things installed is a lot more painful than just writing some code.

And now all I need is a way to keep /var/cache/gems up to date and with the correct files and directories:

/usr/local/bin/update-gem-cache
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#! /bin/bash

cache_dir=/var/cache/gems

mkdir -p ${cache_dir}/gems
mkdir -p ${cache_dir}/specifications

# clean up broken links
# find -L ${cache_dir} -type l -exec rm {} \;

# link existing gems and gemspecs
# find /usr/local/rvm/ -name '*.gemspec' -exec ln -s {} ${cache_dir}/specifications/ 2>/dev/null \;
# find /usr/local/rvm/ -name '*.gem'     -exec ln -s {} ${cache_dir}/gems/ 2>/dev/null \;

# copy works better than link because otherwise bundle won't update in some cases
find /usr/local/rvm/ -name '*.gemspec' -exec cp -u {} ${cache_dir}/specifications/ \;
find /usr/local/rvm/ -name '*.gem'     -exec cp -u {} ${cache_dir}/gems/ \;

# generate the index files needed by gem and bundle
gem generate_index --no-legacy --modern -d ${cache_dir}/

# update only works when /var/cache/gems/specs.4.8 exists
#gem generate_index --no-legacy --modern --update -d /var/cache/gems

Now bundle install talks to the local cache, and it’s fast.

Comments