The Cheeky Monkey Media Blog

A few words from the apes, monkeys, and various primates that make up the Cheeky Monkey Super Squad.

In this post, I’ll cover the improvements we’ve made to our developer workflow at Cheeky Monkey Media. At the center of these improvements is a git branching strategy, but I’ll also talk a bit about code review, dev environments, and tools.

The Dangers of the Jungle

A lot of things can hurt you in the jungle. That’s why monkeys watch — and pick the bugs off— each other’s backs! Monkey troops are organized in a way that makes them as strong as they can be. As development teams, we should create processes and workflows that do the same thing.

The quality of our code, the speed of our production, and therefore the value of our services can be greatly enhanced if we follow some nature-tested survival strategies. In fact, if you organize things well, a better result is almost automatic.

Let’s break it on down.

Git Branching

Git has been around since 2005. It was created by Linus Torvalds (the creator of Linux), just in case you were curious.

I don’t know why, but I think I love it more than is natural. Git makes me “gitty”… (sorry).

Many web-based repo management tools exist to help teams collaborate on code using Git. Probably the most well-known is GitHub. Many of these tools are good, but they seem to be designed for an older style of git workflow, one where developers make changes to code, commit to master, and push that off to wherever it’s going.

Pantheon built an amazing platform that supports this kind of workflow with the valuable benefit of wrapping testing environments around it. This is a huge convenience to the average workflow. It allows you to push changes to Pantheon on the master branch, but only to a “dev” environment where testing can be carried out by the developers. From there you can push to a “test” environment where further Quality Assurance (QA) testing can be done. After that, you can do User Acceptance Testing (UAT) and get signoff from the client before you finally push to “live”.

However, there are some inherent problems with this workflow.

For example, what if we have some stuff in the master branch pipeline that fails QA testing AND we have a hotfix that we NEED to push out today? We can’t just push our hotfix into the workflow then pass the previously rejected commit and on to live. It gets… tricky. What about lots of devs working on lots of different things and an active QA cycle?

This system does a lot for the quality of work that gets delivered to the client, but there’s another way, I think a better way, and Pantheon supports this way beautifully too!

In fact, there’s a git workflow that can be used with any host that will improve code quality, production velocity, and developer sanity!

I’ll illustrate it in terms of the Pantheon platform and “multidev” environments.

What Hurt With Older Work Flow Processes?

  • Different development processes for different projects depending on where they were hosted. This affects the time to spin up or roll onto a project.
  • Different deployment processes for different projects depend on where they were hosted. The many manual steps make deployment stressful.
  • No standardized code review process. This made code review hard and time-consuming.

How to Fix It

  • Streamline the production pipeline.
  • Make code review easy.
  • Automate processes where possible.
  • Be production environment agnostic.

The Repository

The first thing we needed was a central repository that held all our projects. This way developers could have a one-stop shop for project checkouts and less confusion about what lived where.

This central repo needed to support our branching strategy, which treats the master like the pristine jewel it should always be.

It needed to make code review as pain-free as possible.

After some research and comparison to the tools we were already using, the solution for us turned out to be Bitbucket. We could branch and review the way we wanted to and the UX on Pull Requests and Code Review is super slick, right down to providing a link right in the terminal to create a new Pull Request for your last push to the repo! Nice touch!

The Git Branching Strategy

Once we had the platform, we set up our projects in a style similar to “WunderFlow”:

WunderFlow is a git workflow that tries to make it easier to have multiple ongoing development tracks simultaneously while still allowing clean releases and steady hotfixes. It also makes it easy to show any unfinished work to customers.

Cheeky Monkey Media manages projects of various sizes as well as long-term maintenance and support, which means we needed a process that would work for both teeny-tiny and large projects. By standardizing on this workflow we can follow the same development processes on small projects with small teams as well as scale up to projects the size of weightwatchers.com, which has a team of about a dozen developers and truly continuous integration.

How It Works

I say we use a style “similar” to WunderFlow. It differs a bit in that we think of the ‘master’ branch as the production branch, and create a ‘stage’ branch for QA and UAT. The principles remain the same except that ‘master’ is always as clean as possible, its last commit is always the latest released code, and nothing more.

So we have 3 main branches tied to 3 environments:

develop => develop (dev testing)

stage => stage (QA and UAT)

master => live (production)

There are 4 other types of branches that may be used through a development cycle:

Feature: (feature/PRJ-123–short-description)

  • The branch name contains the ticket id and a short description prefixed by ‘feature/’
  • A feature branch is for standard work. We try to plan our tickets so each task is a discrete chunk of work that can stand alone in a feature branch. We want the branch to be as small and contained as possible for easy code review, and so that commits are easily identifiable, should the need arise.
  • Feature branches are generally cut from master unless they’re part of an epic.

Epic: (epic/epic-name)

  • An epic branch can be used for working on a larger set of features that depend on each other.
  • An epic branch is cut from the master.
  • Feature branches for the epic are cut from the epic. This assumes the epic is as clean as the master but also contains what belongs in the epic for dependents.

Hotfix: (hotfix/PRJ-321–short-description)

  • Branch name contains the ticket id and a short description prefixed by ‘hotfix/’
  • A hotfix branch is exactly like a feature branch, except it doesn’t have to wait to be released with a stage branch at the end of the development/release cycle. It can get code reviewed, tested on dev, QA/UATd and released live to master as fast as it’s possible to do so. It will not be held up by any obstacles.
  • Hotfix branches are cut from master. Presumably, a bug has been found in production, which calls for an immediate fix.

Staging: (staging/staging-7.53.2.0)

  • Branch name is staging plus a release version number. This can be any versioning scheme you like. We’re using what may or may not be a semantic versioning scheme. Major.Minor.Patch. But, since we work with Drupal, we use Drupal Major.Drupal Minor.Feature.Patch.
  • Staging branches do a kind of double duty. First they can be merged to the ‘stage’ branch and deployed to the stage environment at any time. This facilitates QA/UAT throughout the release cycle. Second, the staging branch, once all features are signed off, can enter “code freeze” in preparation for a release to master.
  • Staging branches are cut from master at the start of a new release cycle.

* It’s possible, due to release schedules, to have multiple staging/release branches queued up with active development happening on them simultaneously. Each can enter code freeze in their own time and be released according to some schedule. This type of workflow makes that possible without interfering with any other aspect of development.

Code Review

This branching strategy gives us plenty of opportunity to get multiple developers’ eyes on the code. The first and strongest point for this to happen is on the develop branch.

DEFINITION: A Pull Request (PR) is a request by a developer to merge new code into a main branch of a repo. It presents the opportunity for code to be reviewed by other developers as well is the repository maintainer.

Here’s the process in a nutshell:

  1. Developer gets a ticket. PRJ-123
  2. Developer pulls latest master and creates a feature branch. feature/PRJ-123–some-feature
  3. Developer does what developers do and commits code to the feature branch.
  4. Developer does ‘git push origin’ to the central (bitbucket) repo. Bitbucket provides a handy link to create a Pull Request (PR). (Command + Double Click) on the link in the terminal message. This opens a “create pull request” window*.
  5. Developer tweaks the PR settings and clicks the “Create Pull Request” button to create a PR against the develop branch.
  6. Developer moves ticket to “Dev Pull Request” status, then grabs another ticket and moves on.

*PRO TIP on bitbucket Pull Requests. Set develop branch as your default branch and your new PRs will automatically be against develop. Also set up your repo to have Default Reviewers, and add your team there. This way you won’t have to manually add reviewers every time you create a PR. There are lots of other benefits to the Bitbucket platform that I can’t cover here.

At this point the PR is ready for code review and a few more PRs can collect throughout the day. Code review can happen at any time developers have a few moments to look at code, but we also schedule a small amount of time after our morning stand-up or scrum meeting to do some code review and keep things moving along.

If code needs attention quickly because of a pending release or hotfix, developers can notify the team in chat that they’d like a review. The point is, code review is important but can wait a little. We don’t need the mentality of, “I just finished this code, it needs to get pushed through to production as soon as possible.”

If you set up an efficient cadence of development, review, test, and release, things will become smoother and less stressful.

We’re looking for 2 approvals on a develop PR before it gets merged to the develop branch. At any time, the develop branch can be deployed to the develop environment for dev testing. This can even be automated, and can be done daily, every merge, or whenever the lead dev wants. But, generally, merges to the environment branches and deployments to those environments are handled by a lead dev on the project.

From there, the tickets are moved to Dev Testing status and developers can go test their work on the environment. If that checks out, they do a new PR of the same feature branch against the current staging/release branch. That gets one more quick look for conflicts or anything else that may stand out, before the lead dev merges that to the staging branch. That can be deployed to the stage environment for QA/UAT, again, at the lead dev’s discretion.

At the end of the sprint or release cycle, if everything in staging is approved, staging can be merged to master. Master can be tagged with the release version id and pushed to production.

Pantheon

This workflow is quite different from the one Pantheon built their testing and deployment tools around, but that’s ok. This workflow integrates perfectly with Pantheon because of 2 key things.

Thing One: Multidev

Pantheon already gave us a way to test development of different features before merging things into the main branch. These could be used in a way we might think of as epics in our workflow. But we think of multidev specifically as test environments and we always have at least 2.

The first multidev environment is for the develop branch. This is the environment developers use to test their code. We do test in local environments that are configured as closely as possible to our production environments, but dev testing on develop looks for a few of specific things.

  1. First we will see if something in the environment configuration does happen to cause issues with our new code.
  2. Second we will see if something in the configuration of the project causes issues with our new code.
  3. Third we will see if something in other new code causes issues with our new code.

Usually a developer won’t find an issue with his testing on the develop branch, but if any of the above do occur, we’ve caught it very early in the development cycle where it affects far fewer people, and can be corrected without slowing down the later stages of the process or bothering anyone else.

Of course, a developer who tests his own code has a natural bias to test it in a way that’s most likely going to work. The developer knows what it’s “supposed to” do, and does manual tests accordingly. There is a high probability these manual tests will pass, and a low probability something unexpected will throw a monkey wrench into the expected outcome.

We could spend a lot of time talking about automated testing which will help catch things we might miss as primates, but we’ll save that for another time.

The next step, though, will throw some curveballs at this new code and it’s done on the second multidev environment and the “stage” branch.

You will recall that we have a “staging/staging-7.53.2.0” branch for the next version release of our project. This can be deployed to the “stage” environment by way of a “stage” branch tied to the multidev. It’s on this branch where our Quality Assurance testers can test things in a way the developer may not have considered. It’s also where we can demo the work to the Client and have them test it out. They may try unexpected things, or have general User Experience feedback. Again, if something comes up here, we’ve still caught it early enough in the process to prevent issues on production, and to not bog down the production process too much.

Thing Two: Tagging the Pantheon Live Release

So, the secret sauce to using our multidev testing environments and effectively bypassing Pantheon’s built in “dev” and “test” tabs is in how they designate code as “live” code.

I discovered this in their documentation about Hotfixes (which they strongly caution AGAINST using). They caution, I believe, because you get into one of those “tricky” places with git that I alluded to at the top of this post. Using our git branching strategy (or pure WunderFlow), everything is so clean and slick that a hotfix causes no stress whatsoever.

Essentially, when it’s time for us to release code to live on pantheon, we simply merge our ‘staging/staging-[version number]’ branch to the master branch, git tag it with our release version (for our own reference), and then git tag it with ‘pantheon_live_[number]`.

It’s this `pantheon_live_[number]` tag that indicates what code should be live on Pantheon. If you run the command `git tag` on your pantheon repo, you’ll see a lot of these tags. The one that represents your current live code is the one with the highest number at the end. So all we have to do is find the latest one and increment by one, tag our master branch with that, and push it to Pantheon.

Conclusion

Just like monkey troops organize themselves in a way to watch each other’s backs, we’ve done the same with our developers. We’ve organized our development workflows in a way that we can support each other with code review. (That’s a bit like picking bugs off each other’s backs, isn’t it?)

Code Review, has many benefits of course, not the least of which is the mentoring of Jr. Developers by more experienced members of the team.

We’re also set up to “fail fast and early” with testing happening on develop and stage environments. But we’re structured in a way that we can recover and repair these issues very rapidly and with no confusion, even if something happens to slip through to production.

Let’s list all the benefits:

  1. Code Review and Testing helps developers get better at what they do, faster.
  2. Code Review and Testing makes a better product, faster.
  3. By organizing our teams and processes this way, the team can be a real team, supporting each other, passing the ball, coaching at every coachable moment.
  4. The risk of putting something into production that is unstable, or not expected by the client is greatly reduced.
  5. The stress of managing and deploying code, even on a very large scale, is almost eliminated.
  6. Did I mention that this is fast, really agile development? Using our Docker Containers a new developer can clone any of our projects from bitbucket, run `docker-compose up -d`, and start developing immediately.
  7. Because our processes are always the same on every project, we can automate a lot of tasks with scripts.
  8. It’s adaptable. We can configure our deployment scripts to connect to any host, so the deployment process for the developer or lead is the same, regardless of where the code ends up.
  9. Developers that aren’t deploying code, always pull from and push to our central repos at Bitbucket. No hunting for different repos on different projects.
  10. We can use other automation tools to support automated Unit Testing, Behavioral Testing, and Visual Regression Testing. It’s just an extension of our core processes.
  11. … I could go on, but the title says 10. 😉
Posted in General