Running parallel test on local and CI


 

– One problem with long running tests is that team starts to ignore them. It’s also a blocker for deployment and a productivity debt for developers to switch context back and forth between builds.

– Parallelizing tests is the obvious way to make them run significantly faster. Either you are running your test suite locally or remotely on CI servers, there’s good chance that you have multi-core machines. So, why’s not?

– Usually your test suites have integration-, API- and UI-level tests. Unit-level tests already run quite fast, so they’re unlikely to be the bottleneck in your builds. Integration and UI level tests are usually the culprits.

– Running test suite in parallel means spawning multiple instance of the server app. One of the biggest hassles is to bootstrap server app with shared dependencies such as test databases, caching layer (memcache, redis), etc… and those varies with each individual app.

– In the scope of this article, I’m running a stack including rails sitting on top of MySQL and redis. The test suite are rspec and cucumber

Tools

parallel_test gem (https://github.com/grosser/parallel_tests)

From their README:

Speedup Test::Unit + RSpec + Cucumber by running parallel on multiple CPUs (or cores).
ParallelTests splits tests into even groups(by number of tests or runtime) and runs each group in a single process with its own database.

Config

– In your database.yml

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

– For redis, either we turn it off or add a prefix namespace to the key

    # In config/environments/test.rb
    config.cache_store = :null_store

or

    config.cache_store = :redis_store, {
      :host       => AppConfig.redis.host,
      :port       => AppConfig.redis.port,
      :password   => AppConfig.redis.password,
      :db         => AppConfig.redis.cache_db,
      :namespace  => AppConfig.redis.namespace || ENV['TEST_ENV_NUMBER'],
    }

– Running tests is very straight forward

rake parallel:create        # Create additional database
rake parallel:prepare       # Copy development schema
rake parallel:spec          # RSpec
rake parallel:features      # Cucumber

– On CI, we can also pass the profile name into each test, and serialize the output for better readability.

bin/parallel_cucumber -o "--profile ci" --serialize-stdout features
bin/parallel_rspec -o "--profile ci" --serialize-stdout spec

Coming up next: Test perf benchmark

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s