A popular paradigm to consider when creating or retooling a complex development environment is bundling it in a virtual machine (VM). Tools such as Vagrant make this easy and appealing. This post will attempt to explore some of the pros and cons of development environments contained within a virtual machine.

Basic Concepts

Let's explore some basic concepts regarding development environments in VMs. Ideally these should be considered thoroughly before starting the implementation:

1. The VM provider: Will this be a local hypervisor (ex: kvm, VirtualBox), or a remote one (ex: VMWare, OpenStack)?
2. The VM provisioner: Generally, a layer of configurations need to be applied to your virtual machine. This could be as simple as a bash script, or as complex as running a configuration management tool such as Chef, Puppet or Ansible.
3. The VM base OS: You'll have to pick which operating system the guest VM is running; ideally this would be the same as in production.
4. Delineation between host and guest: What stays on the host, as opposed to living on the guest VM? Where does the code live?

A Concrete Example

This article will explore some advantages as well as some challenges with this approach. We'll look at a specific implementation of a development environment for a company I worked for. Specifically:

1. VM Provider: The VM provider chosen was the default Vagrant provider, VirtualBox. This is the easiest to implement, as VirtualBox has support for many host operating systems and is proven software.
2. VM Provisioner: Chef was picked primarily because of its ability to have automated pull capabilities from a central server. This allows the development environment to grow and receive updates which will automatically be rolled out to all developers, without requiring end-users to run any commands to pick up the updates.
3. VM base OS: We decided to go with RedHat Enterprise Linux since this was the chosen OS for the production environment.
4. Host/guest split: This is the most complicated decision. In our implementation, the host OS runs the IDE and hosts the code. Keeping the code on the host OS ensures that any mishaps with the VM does not result in code loss. All other build tools and runtimes (JDK, Gradle, NodeJS, Tomcat) run within the guest OS. Source code management (git) is installed both on the host and the guest, for extra flexibility.

Extra nice-to-haves

A few more nice-to-haves were implemented:

• Automatic SSH key forwarding: the host OS' SSH key is automatically forwarded to the VM, allowing git identities to be preserved across boundaries.
• Automatic X11 forwarding: GUI applications can be started from the VM and forwarded to the host OS, if needed
• Automatic shell variable setup: A file was created by convention and copied to the VM on every start, allowing a user to preserve configurations/shell variables even after a VM destroy/create.

Advantages

Fast onboarding

One of the main advantages of this approach is the speed of first-time setup. For a new developer coming along, the setup process is essentially 2 commands: 1 to setup his host OS (install Vagrant/VirtualBox/Git and get the necessary files for setup), and 1 to bring the VM up.

After only a few minutes (target time being 3-5 minutes at most), the developer has a fully functional development environment.

Ability to destroy and recreate quickly

Based on what we saw in the previous step, a developer is able to quickly recreate his development environment. If something goes wrong during development, many developers will choose to simply recreate the development environment instead of spending an hour troubleshooting the system.

Isolation from the host machine

Another advantage is isolution from the host machine. Many development tools may conflict with the host OS, for example installing or upgrading python packages using the system-level python may have undesired consequences. Later trying to undo those changes can be really problematic. Hosting everything within a VM lets developers experiment freely without fear of damaging their host OS configurations or packages.

Support of multiple host OSes

One of the requirements we had was to support multiple host OSes. Most developers traditionally did development on Linux workstations, so we had to support RedHat and Ubuntu. MacBooks were also being introduced, so support for Mac was a must. And finally, several QA engineers only had a Windows laptop but were still expected to develop tests, so Windows support was a nice to have.

Trying to tool or document all these setup steps for 4 different OSes would be near impossible. With a VM, this problem goes away almost completely.

Ease of troubleshooting

And finally, because everyone is using the same development OS and all resources have been setup the same way, when a developer needs support, that support can be provided much more efficiently. No more finding out which version of Java they installed or where their global npm node_modules folder is - everything is at the same place and configured identically for every developer.

Re-use during CI

Since the development environment is a virtual machine, it can be re-used during continuous integration for the build slaves. That way, CI processes run on the identical environment as developers, removing the "it builds on my box, why doesn't it build on CI?" problem! Specifically using vagrant, you can simply pick another provider, for example OpenStack instead of VirtualBox, and run your build slaves on the cloud.

Challenges

NFS mount for code share

The previous article on Shopify Developer Environments touched on this slighty: since the IDE and the code was on the host machine, but all build and runtime tools were on the development environment VM, this meant the code had to be shared. The default VirtualBox shared filesystem is notoriously slow, so the current fastest way to share code is with an NFS mount. This works and is pretty fast, but nowhere near as fast as having the code and the tools on the same physical drive.

Additionally, this extra NFS layer added some problems when using tools such as file watchers, and also exposed some buggy NFS implementations in macOS.

NAT/port forwarding

Since our code is now compiled and running in the VM, this means we need to reach out to a different local IP address to connect to it. This is fine locally, however if a coworker wants to reach your VM's private IP, it won't be reachable. Which is where NAT'ing (to enable VM -> outside connectivity) and port forwarding (to enable outside -> VM connectivity) need to be configured properly.

These can all be configured, however it introduces more complexity to the runtime environments as applications (and developers!) now need to work in a NAT'd environment and be aware of private and public IP addresses.

Chef registration

Finally, one specific downside of Chef is that each node that receives configurations must be registered centrally with the Chef server using a private key. We went with a convention of using a known VM name appended to the host machine's hostname to ensure uniquess across the organization, and auto-registration on VM up. However if a developer uncleanly deletes the VM, or if a workstation changes hostnames, the registration with the Chef server can be left behind, causing headaches for the developer.

Conclusion

Overall, this approach has been successful, although not perfect. The trade-offs can mostly be tooled around, and the advantages generally give us enough benefits to continue with this appraoch. However, work is always ongoing to iterate and improve on this approach. Hopefully this article has been useful to share our thoughts on VMs for developer environments!

Consistency is important across development environments; inconsistencies can lead to debugging nightmares and incorrect local behaviour. Even with the existing tools like chruby, bundler and homebrew to manage dependencies, setup can be a multi-step esoteric process and it can be difficult to outline the processes that achieve desired consistency.

At Shopify, we have hundreds of developers working on hundreds of active projects. Many of these projects, for a while, had their own workflows, setup instructions, ways of running tests and so on.

We've created a tool called dev to provide us with a unified workflow across a variety of projects. It provides a way to specify and automate installation of all the dependencies and includes workflow items required to boot the project on macOS, from XCode to rake db:migrate.

Introducing Dev

Dev is a command-line tool, mostly implemented in ruby, with some shell integration supporting bash, zsh, and fish. It provides a set of commands to interact with the various projects under a GitHub account including cloning, bootstrapping, running tests, booting servers, and shell integration to activate certain dependencies (e.g. particular ruby or node versions) when entering a project directory.

Dev out of the box

Almost all active projects at Shopify are dev-enabled. If a developer wants to start working on one of these projects, they need to run only one command, dev up, to completely setup and update their development environment for the project. This includes everything from installing XCode to running pending database migrations.

During the dev up process, dev executes a sequential list of tasks. Tasks check for some condition or desired state (e.g. some package directory exists), and executes some code to meet that condition, if not already met. This way, initial setup (e.g. install ruby) and ongoing maintenance (e.g. run bundle install) are unified in one action. Dev provides a library of tasks to choose from, and each project defines its own dependencies in its dev.yml file.

dev.yml acts as a spec for a project. It defines what tasks need to be run during the up process, and contains everything that dev needs to know about the project. To start using dev with your project, you need to add a file named dev.yml to the root directory of the project, and start defining the spec for your project. We have a command, dev init that will help to initialize the dev.yml for a project, based on some preset templates. This is a segment of an example dev.yml file from a ruby project:

# up hash defines the sequence of tasks
up:
  - homebrew:
    - shellcheck
  - ruby: 2.3.3
  - bundler

# list of command aliases
commands:
  test: bin/testunit
  style: "bundle exec rubocop ; find . -name '*.sh' | grep -vE 'chruby|nvm' | xargs shellcheck"

Running dev up in this project on a new machine will:

• Install homebrew
• Install shellcheck from homebrew
• Install ruby-install from homebrew
• Install ruby 2.3.3 using ruby-install
• Activate ruby 2.3.1 using dev’s built-in chruby
• Install bundler using ruby 2.3.1
• Run bundle install

Running dev up again, later, after modifying the project’s Gemfile will:

• Run bundle install

Dev acts as a version manager for languages, switching to the versions defined in the dev.yml file as the user moves around between projects. Custom command aliases can also be defined in the dev.yml files as shown above, defining commands for checking style and testing.

Navigating with Dev

In addition to aiding with development environment setup, dev provides several commands to help improve navigation. Dev can be used to manage your projects in your file system. We encourage a GOPATH-compatible file structure (~/src/github.com/account/project) but allow users to customize their file structure too.

Projects are added to your file structure using dev clone <account>/<project>, which clones from github, and dev will attempt to infer where it should save your project. Then you can navigate to the project from anywhere in the command line using the dev cd command, fuzzy matching included.

Within a project, dev open can be used to open the browser to specific pages. All projects, dev enabled or not, have access to the command dev open github which opens the github page for the project, while dev open pr opens the existing pull request for your branch or opens a page to create a new pull request. Custom project open commands can be defined in the dev.yml file. For example, the following section creates the command dev open production:

# dev.yml
open: 
  production: https://www.shopify.com

A Note About Datastores

At Shopify, we run all the datastores our applications depend on (e.g. mysql, redis, memcached, elasticsearch, zookeeper, …) in a little sidekick VM managed by some other software we’ve written (‘railgun’). We think Docker For Mac can also probably fill this need, but there’s no integration built in yet.

Datastores can just be provisioned onto the host using the homebrew task, but to get version / data isolation per project will currently require some extra configuration.

Conclusion

Dev's goal was to improve the life of all developers using it. It's easy to get started with dev at Shopify by initializing some dev.yml files. Dev defines a clear idiom for managing development environments and navigating between many projects.

A tool like dev works best when the entire organization standardizes on it. If the following are true for you, you may get some value out taking inspiration from dev:

• Contributing to another team’s project is more difficult because of setup difficulty;
• You have many projects with non-standard setup instructions in various READMEs, or half-functional bootstrap scripts that go unmaintained by daily users;
• It’s feasible to expect all developers to install a new tool and learn to use it.

Thanks for reading!

John Arthorne of Shopify talks about Chat Ops and Developer Productivity

Dan Benamy of Datadog talks about the developer environment and tools at Datadog.

In a previous post, we discussed the systems we have had throughout the history of Shopify. In this follow up post we'll talk about the system we run today. Finally, we will have another post that includes a few forward thinking questions about developer operations at Shopify.

Dev

Dev is a command-line tool, mostly implemented in Ruby, with some shell integration supporting bash, zsh, and fish. It provides a set of commands to interact with the various projects under a GitHub account, including cloning, bootstrapping, running tests, booting servers, and shell integration to activate certain dependencies (e.g. particular Ruby, Go versions, or Node versions etc) when entering a project directory.

Almost all active projects at Shopify are dev-enabled. If a developer wants to start working on one of these projects, they need to run only one command, dev up, to completely setup and update their development environment for the project. This includes everything from installing XCode to running pending database migrations.

When we were making Dev, we had other previous tools in our recent memory. We looked at what we liked and what we didn’t. For example, we knew that a VM didn’t work well with code, but did with services which often caused the most issue. We created Railgun, an Xhyve based (https://github.com/mist64/xhyve) virtual machine that houses services such as MySQL, Redis, and Nginx. While creating a tool to interact with Railgun, we ended up developing Dev.

Core Philosophies

We decided on a few base philosophies from the beginning:
dev clone repo_name will clone the repository.

This will be our only interaction with Git as peoples’ flows vary, we want people to learn and remember Git, and cloning the repo allows us to control where repos are cloned and therefore lets us deterministically control the current directory

• dev up will setup all dependencies so that a project is able to run
• dev run or dev server will run the required systems to make a project run
• dev cd [name] will change directory to a project
• If nothing needs to be done, dev up should take under 1s and ideally under 100ms

These philosophies allowed us to maintain a great, consistent, and fast experience across all projects that specify a dev.yml

Dev.yml - the spec

dev.yml acts as a spec for a project. It defines what tasks need to be run during the up process and contains everything that dev needs to know about the project.
This is a segment of an example dev.yml file from a Ruby project:

# up hash defines the sequence of tasks
up:
  - homebrew:
    - shellcheck
  - ruby: 2.4.1
  - bundler

# list of command aliases
commands:
  test: bin/testunit
  style: "bundle exec rubocop ; find . -name '*.sh' | grep -vE 'chruby|nvm' | xargs shellcheck"

Running dev up in this project on a new machine will:

• Install homebrew
• Install shellcheck from homebrew
• Install ruby-install from homebrew (in the Ruby task)
• Install ruby 2.4.1 using ruby-install and a pre-compiled binary
• Activate ruby 2.4.1 using dev’s built-in chruby
• Install bundler using ruby 2.4.1
• Run bundle install
• Running dev up again, later, after modifying the project’s Gemfile will:
• Run bundle install, if anything has changed

Current state of Environment

As shown above, Dev has taken a little bit of each of the previous iterations. A Virtual Machine was used to house services such as MySQL and Redis, but the code remained on the system. This separation allowed us to split the focus from “what is needed to run this code?” to what dependencies are needed and what infrastructure is needed on the VM. This split made things a lot faster, and also allowed us to throw away infrastructure if they ever had an issue.

Currently, the system we have is optimized in a few key locations. Let’s first talk about the system as a state machine. We have 3 key states: Stopped, Started, and Broken.

• Stopped: This is the default state when everything is off.
• Started: This is the state we want to get to and where everything should work.
• Broken: This is what happens when anything goes wrong.

While each transition should be as quick as possible, we should be able to detect that a “start” will be successful or not as early as possible. Consequently, going from “This is broken” or “This is probably broken” should be as quick as possible - rm -rf everything and start again. This keeps complexity down and allows the possible states to be minimized for simplicity.

Our current system using Railgun and Dev is fairly reliable, accomplishes the above goals effectively, and requires little maintenance. The output is fairly straight forward, communicates intentions and information well, and is reliable most of the time. On the other hand, deviating outside of our normal configuration is difficult and not easy at all (near impossible with how you can change what Railgun runs). While this is not an issue on the surface, it does restrict what can be run and reduces the ability to innovate.

Going over these past experiences allowed us to define a few key goals of an environment:

1. Speed: We need to be able to change states quickly
2. Reliability: We should be deterministic, be reproducible, and handle error states
3. Simplicity: It should be simple to use and well documented
4. (Nice to have if 1-3 are solved) Production parity: It is beneficial if it can be similar to production for education and debugging purposes

With these past experiences, new set of goals, and understanding of the weak points of our current system we set out to find a better solution.

In a follow up post, we will talk a little bit about how we're using these experiences to think forward to the future.

Development operations at Shopify has a long history that winds its way through various systems, technologies, and iterations. Over the course of these iterations we have learned a number of lessons, experienced ups and downs, and ultimately ended up with a system that performs well.

In this post, we’ll discuss the systems we have had throughout the history of Shopify. In a follow up post we'll talk about the system we run today and we will have another post that includes a few forward thinking questions about developer operations at Shopify.

Historical Systems

Throughout Shopify’s history, we have used 4 different systems to create and maintain developer systems. Each one has its own personality and optimized for different needs. We’ll discuss each system, explain some pros and cons of each system, and why we decided to switch from this platform.

Artisan Systems

In the earlier days of Shopify we had a few dozen developers and no automated environment setups. This system was quite ad hoc with the entire system being described in a hard coded list. When a developer started they would be handed a list of dependencies to install (e.g. install MySQL, install Homebrew, install Ruby version X however you want etc). There were some bin/setup scripts to automate some of this, but these bash scripts often broke. Moreover, as dependencies changed new sub-dependencies would be added without realizing it, so the developer would have to decrypt cryptic messages to determine what they need to install to continue.

This system was quite error prone and meant that developers had divergent systems if they chose to install things differently. On the other hand, developers often learned quite a lot about the system, how it worked, and the macOS operating system (our OS of choice to this day).

As the company grew, this system became cumbersome to execute and we often spent a long time setting laptops up. Finally, this cost became too much so we decided to consolidate this process using Boxen.

Boxen

Boxen (https://github.com/boxen/our-boxen/) is a tool that is implemented on top of Puppet (https://puppet.com/) and open source system configuration tool. The intention of Boxen is to wrap some of the more complicated parts of Puppet and allow repositories to include customization scripts.

During our usage of Boxen, we saw the first signs of benefit from consolidating this configuration - developers could help each other a lot more easily with similar setups, we could merge changes to the setup and it would apply everywhere, etc. This allowed us to more easily maintain and deploy developer systems and we were able to move faster.

Unfortunately, macOS is a complex system and running Puppet only added to that complexity. As the system began to run on more and more systems, we saw problems with things that users changed, and things that conflicted with Boxen/Puppet configuration. What made matters worse is that Puppet was not a well known or understood tool and we did not expect developers to learn it. This meant that when something went wrong people could not debug the system themselves. This is what caused us to seek a different alternative.

Looking at the issues we experienced with Boxen, it was clear to us that we had to allow developers to modify the system (they’ll do this regardless), have a consolidated experience, have a system that people understood, and we also desired a more similar configuration to production. This led us to Vagrant.

Vagrant

Vagrant (https://www.vagrantup.com/) was our answer to Boxen. Vagrant is a virtualized system that allows projects to be run within the confines of a virtual machine (VM). An included configuration file can allow a developer to simply start with a ‘vagrant up’ command which would initialize a production-similar system with all dependencies and code. With this setup, all code and dependencies ran inside of a VM which allowed the developer to modify their system in any way without impacting the VM itself. This was a solution that hit all of our requirements of consolidation, production similarity, consolidated configuration, and a system that people understood (more on this in a moment).

Vagrant, for Shopify, was generated from our production Chef (https://www.chef.io/chef/) configurations. Our entire operations team understood how this system worked, so we were able to modify and change the system readily. This also meant that we had a production-development parity. Unfortunately, our production environments often did not take our Vagrant environments into account and we consistently broke developer laptops. Moreover, as the development team grew there became more of a disparity between the number of operations engineers who knew Chef and the number of developers.

We also had other issues with Vagrant. With the code hosted in the VM but the text editors and IDEs on the host machine, we were required to mount an NFS drive from the VM on the local machine. This ended up causing issues with syncing and file changes were often not picked up which resulted in a lot of lost productivity and time in debugging errors from the system.

We ended up deciding that Vagrant was not a proper solution due to the NFS issues and the constant breakage. We sought a new solution and researched a new alternative. From this time, dev and Railgun were created.

In the previous posts we talked about our past experiences with Artisinal Systems, Boxen, and Vagrant as well as our current system using Railgun and Dev. In this post, we're going to talk about some future plans and give a few forward thoughts.

Railgun and Dev accomplish most goals most of the time. We do not allow enough room for innovation or differing tech stacks. How can we allow more teams to innovate while making the normal tech stack glaringly easy to use? We have a few ideas which we’ll discuss below.

Minikube

Minikube is a system built on top of Kubernetes, a container scheduling system. With our current operations being built on top of Kubernetes, this seemed like a good opportunity to consolidate developer and production environments, educate our engineering department about Kubernetes, and give customizability to environments.

This presented us with a unique opportunity to provide some alleviation from our previous issues while also giving a perfect spot for people to learn about Kubernetes. We rolled out a beta version of this system to a number of users to collect data, learn about weak points, and provide more overall context of the system.

Unfortunately, the beta resulted in poor performance in a number of aspects. We experienced issues in a number of areas, which are listed below:

• Previously, we mentioned that the “Broken” to “Stopped” state needs to be fast, however with Minikube we could not accomplish this. It would sometimes take far too long to return to a default state and this unfortunately happened too often. One reason for this is that Minikube used Kubenetes default behaviour to try and “self heal” while we have no issues with simply killing the system and rebooting from scratch. There were also far more failure scenarios.
• The first boot was exceptionally painful. Minikube relied a lot on external factors that we could not control and for which Mac provided little guidance. This often caused the initial install to fail and be very painful.
• Dynamic Host Configuration Protocol (DHCP) failed often and caused conflicts with other resources trying to use the same system (namely Railgun conflicted a lot).
• Pulling containers took a lot time, as did Minikube ISO images. This caused a lot of contention and would not work well with our remote developers.

In the end, we decided to call our own sunk cost fallacy and abandoned the effort. However, The system was left in place for us to use in infrastructure related experiments.

Online Developer Environment

Minikube provided us with a good entry point into Kubernetes, but failed on setup due to time constraints and flaky scenarios. Using this experience, we prototyped a remote developer environment where the services such as MySQL and Redis would run in the cloud. This experiment proved an overall success, but brings a lot of uncertainty. What is the impact on the network round trip? What happens to developers using a poor internet connection? We take away offline capabilities - though we deemed this acceptable. In the end, we kept the work as an experiment and decided to work on Railgun V2 instead.

Railgun V2

Railgun provides a lot of benefit with little downside. If we look at our goals, production equivalence is a nice to have after the other goals so we decided to work down this path some more. Railgun V2 is being designed around Docker instead of a custom virtual machine. Docker will provide us an easy way to use any containers that we build into our Google Cloud Container Registry. This means that the default environment and services will be available by default and other environments are able to be used granted they are uploaded to our container registry. So far, we even gained a few seconds in performance in boot.

Conclusion

In the end, it becomes apparent that scaling a developer environment is not a trivial task. Each system is uncontrolled and brings with it a varied setup. Going into a solution, we need to keep in mind that we need to make the experience quick and easy for the everyday developer, but keep in mind that our own maintenance burden cannot suffer either. While our current solution using Dev and Railgun is good, we can aim to be better and will be working towards a more customizable, quick, and well communicated system.

• What is this for?
• Who will build this, who will be responsible for it?
• High Level Goals of the Website
• What are “Case Studies”?
• Site Layout

What is this for?

Developer Acceleration, or Productivity, has no home or place to find information. This website will be the central hub on which to share and collaborate on guidelines, blogs, and other developer productivity information with the developer productivity community.

Who will build this, who will be responsible for it?

• Shopify will be paying for the creation of the website and the costs for the first year at minimum
• The site will be administered by a collection of volunteers from various organizations. (Interested in helping out? Ping @julian on Slack!)
• This is a website for the community, it will not be branded as Shopify (save for maybe a sponsored by icon in the footer). We want the community to own it, but that doesn't mean we won't invest our time in it either

High Level Goals of the Website

• Be a place where people can learn about developer productivity
• Case Studies and best practices of different developer productivity teams that can help small to large companies build out their tools
• Showcase the diversity of developer productivity in both the people and teams
  • Case studies will be about different teams - whether it’s one person dedicated, a small team, a department - there’s a natural progression in team size that seems to occur and we want to showcase this progression
  • Case studies will be about a variety of people of different backgrounds, genders, ethnicities, etc
  • Different companies have different tech stacks. What's the same? What's different?
• Be a place where we can post blog articles for all our different site sections (CI/Testing, Mobile, Developer environments, etc), tagged appropriately, appearing on both the home page and their respective site sections
• Allow the entire developer productivity community to collaborate together. Administrators from various companies will be chosen to moderate and maintain the website. Everyone else will be able to sign up to contribute to the website (sort of like a pull request for content)
• Host an event schedule for online meetups (and share recordings and materials afterwards) as well as other relevant events, such as a conference.

What are “Case Studies”?

Companies are at different sizes. Some are a few people, some are hundreds, some are thousands - what does developer productivity mean at these sizes?

Through community effort, we can document some of the best practices we learned in a sort of “case study”. These are self-documented guides and resources that teams at various companies used. These can be interviews of team members. These can be styleguides and resources. It's really up to the team.

Think of it like a “life story” for developer productivity at a company

Site Layout

There will be a few sections, some of which will need to support categorization (like style guide, blog, etc).

Here is a map:

• Homebase
  • Case Studies
  • Recent Blog posts
  • Upcoming events
  • Recent Past Events
• Join us on Slack (chat.devproductivity.io, uses https://github.com/rauchg/slackin and can include widgets on any page)
• Case Studies/Best Practices
  • Categorized by companies
  • Further categorized by topics, like seen above
  • Possibly more categories like this doc
• Blog posts
  • Tagged by categories
• Event Calendar and materials
  • Likely not a month calendar, not enough events
  • Upcoming events (?)
  • Past events (and links to materials, recordings)
• Useful materials, links, etc