Dev - Shopify's All Purpose Development Tool

by Kevin Peng

Consistency is important across development environments; inconsistencies can lead to debugging nightmares and incorrect local behaviour. Even with the existing tools like chruby, bundler and homebrew to manage dependencies, setup can be a multi-step esoteric process and it can be difficult to outline the processes that achieve desired consistency.

At Shopify, we have hundreds of developers working on hundreds of active projects. Many of these projects, for a while, had their own workflows, setup instructions, ways of running tests and so on.

We've created a tool called dev to provide us with a unified workflow across a variety of projects. It provides a way to specify and automate installation of all the dependencies and includes workflow items required to boot the project on macOS, from XCode to rake db:migrate.

Introducing Dev

Dev is a command-line tool, mostly implemented in ruby, with some shell integration supporting bash, zsh, and fish. It provides a set of commands to interact with the various projects under a GitHub account including cloning, bootstrapping, running tests, booting servers, and shell integration to activate certain dependencies (e.g. particular ruby or node versions) when entering a project directory.

Dev out of the box

Almost all active projects at Shopify are dev-enabled. If a developer wants to start working on one of these projects, they need to run only one command, dev up, to completely setup and update their development environment for the project. This includes everything from installing XCode to running pending database migrations.

During the dev up process, dev executes a sequential list of tasks. Tasks check for some condition or desired state (e.g. some package directory exists), and executes some code to meet that condition, if not already met. This way, initial setup (e.g. install ruby) and ongoing maintenance (e.g. run bundle install) are unified in one action. Dev provides a library of tasks to choose from, and each project defines its own dependencies in its dev.yml file.

dev.yml acts as a spec for a project. It defines what tasks need to be run during the up process, and contains everything that dev needs to know about the project. To start using dev with your project, you need to add a file named dev.yml to the root directory of the project, and start defining the spec for your project. We have a command, dev init that will help to initialize the dev.yml for a project, based on some preset templates. This is a segment of an example dev.yml file from a ruby project:

# up hash defines the sequence of tasks
up:
  - homebrew:
    - shellcheck
  - ruby: 2.3.3
  - bundler

# list of command aliases
commands:
  test: bin/testunit
  style: "bundle exec rubocop ; find . -name '*.sh' | grep -vE 'chruby|nvm' | xargs shellcheck"

Running dev up in this project on a new machine will:

  • Install homebrew
  • Install shellcheck from homebrew
  • Install ruby-install from homebrew
  • Install ruby 2.3.3 using ruby-install
  • Activate ruby 2.3.1 using dev’s built-in chruby
  • Install bundler using ruby 2.3.1
  • Run bundle install

Running dev up again, later, after modifying the project’s Gemfile will:

  • Run bundle install

Dev acts as a version manager for languages, switching to the versions defined in the dev.yml file as the user moves around between projects. Custom command aliases can also be defined in the dev.yml files as shown above, defining commands for checking style and testing.

In addition to aiding with development environment setup, dev provides several commands to help improve navigation. Dev can be used to manage your projects in your file system. We encourage a GOPATH-compatible file structure (~/src/github.com/account/project) but allow users to customize their file structure too.

Projects are added to your file structure using dev clone <account>/<project>, which clones from github, and dev will attempt to infer where it should save your project. Then you can navigate to the project from anywhere in the command line using the dev cd command, fuzzy matching included.

Within a project, dev open can be used to open the browser to specific pages. All projects, dev enabled or not, have access to the command dev open github which opens the github page for the project, while dev open pr opens the existing pull request for your branch or opens a page to create a new pull request. Custom project open commands can be defined in the dev.yml file. For example, the following section creates the command dev open production:

# dev.yml
open: 
  production: https://www.shopify.com

A Note About Datastores

At Shopify, we run all the datastores our applications depend on (e.g. mysql, redis, memcached, elasticsearch, zookeeper, …) in a little sidekick VM managed by some other software we’ve written (‘railgun’). We think Docker For Mac can also probably fill this need, but there’s no integration built in yet.

Datastores can just be provisioned onto the host using the homebrew task, but to get version / data isolation per project will currently require some extra configuration.

Conclusion

Dev's goal was to improve the life of all developers using it. It's easy to get started with dev at Shopify by initializing some dev.yml files. Dev defines a clear idiom for managing development environments and navigating between many projects.

A tool like dev works best when the entire organization standardizes on it. If the following are true for you, you may get some value out taking inspiration from dev:

  • Contributing to another team’s project is more difficult because of setup difficulty;
  • You have many projects with non-standard setup instructions in various READMEs, or half-functional bootstrap scripts that go unmaintained by daily users;
  • It’s feasible to expect all developers to install a new tool and learn to use it.

Thanks for reading!