Click here to monitor SSC

Tony Davis is an Editor with Red Gate Software, based in Cambridge (UK), specializing in databases, and especially SQL Server. He edits articles and writes editorials for both the Simple-talk.com and SQLServerCentral.com websites and newsletters, with a combined audience of over 1.5 million subscribers. You can sample his short-form writing at either his Simple-Talk.com blog or his SQLServerCentral.com author page. As the editor behind most of the SQL Server books published by Red Gate, he spends much of his time helping others express what they know about SQL Server. He is also the lead author of the book, SQL Server Transaction Log Management. In his spare time, he enjoys running, football, contemporary fiction and real ale.

The Body in the Trunk

Published 25 April 2014 11:58 am

When one considers that the primary purpose of a modern Source Control system is to allow branches and subsequent merges, with all the freedom that allows to development teams, it seems odd to see it suggested that it is best practice to avoid the very features that distinguish a source control system from a Wiki.

Ross Snyder, describing ‘Continuous Delivery’ (CD) at Etsy, boasts of a simple deployment process that is rapid to the point that they can do 25 deployments per day. They treat “failure to deploy” as a “P1″ incident, the same level of severity as the whole site going down! In a conventional source control architecture, one can imagine that to support this would require a carefully constructed network of branches, one per feature or “user story”, merging the changes back to “master” when complete and tested, in order exert fine-grained over which features go live when, and so avoid delayed deployments. The downside, of course, is that without strict procedures and a very well-drilled team, it can becomes a recipe for “merge hell”.

So how do CD proponents handle all this branching, and more importantly the potentially-messy merging processes, that allow for such relentless deployment? The answer appears to be that they don’t. The CD orthodoxy is in favor of doing all of the development work ‘to the trunk’, thereby providing a ‘single, canonical version of code’. In this approach, as described in the Continuous Delivery book, a much better answer (than branching and merging) is to “develop new features incrementally and to commit them to the trunk in version control on a regular and frequent basis. This keeps the software working and integrated at all times.”

Ross’s presentation makes brief reference to “putting changes behind config flags”. The idea, then, seems to be that the application and database sit in the VCS, side by side in the trunk, where all developers work. No branching. They get around the delayed deployments issue by using ‘feature flags’ within the application code. They wrap every feature in code that prevents it from running until enabled via configuration. It means the team can deploy all features, even incomplete ones, only enabling each feature for the user as it becomes ready.

The idea of using toggles and flags to prevent the features from running is not new. Experience tells us that the problem with them is that they require a lot of discipline. The cause of the Knight Capital debacle, for example, is believed to misuse of what was essentially a ‘toggle’ system, where an archaic module for buying stocks was ‘toggled out’, forgotten about, and the toggle subsequently reused it for something else. It is a great temptation for developers to try to circumvent a system with bureaucratic overhead without being conscious of potential repercussions.

Of course, merges can be painful but if your source control system gives you “merge hell”, is it better to fix it, or move to another, rather than bend your development methodology to the inadequacies of your current system?

9 Responses to “The Body in the Trunk”

  1. Keith Rowley says:

    I wonder if “how” you develop should depend a lot on “what” you are developing. Thus different source control methodologies would be appropriate for developing different types of products.

  2. paschott says:

    We do something kind of in reverse. We work in branches, but regularly (daily) pull the mainline branch into the story branches. That makes merging back into mainline a bit easier. We do the feature toggles as well, though not _everything_ can depend on those. We still need to be careful when developing code that any changes are either done in their own story branches or are coded in such a way that they could be released at any point without breaking things. As we’re not to that point yet, we develop in feature or story level branches and merge those into mainline as we’re ready to release.

    There’s definitely some discipline required to do this well no matter what method you use. Keeping some sort of record of what’s in each branch/release helps. And none of this replaces good testing methods. Just because you have that code in a stored proc doesn’t mean it should be pushed to production without QA testing.

    Keith also brings up a good point that the how and what go together. Not everything necessarily fits into the “deploy often” bucket. Some projects will be quite complex and release many pieces all at once. Others will consist of many smaller projects that will each be released as they’re done/tested.
    The branching strategy will likely be different for those cases.

  3. Alex_Kuznetsov says:

    If your top priority is to “do 25 deployments per day”, then I guess you have to do whatever it takes to accomplish that. However, the price tag may be quite high – branching and merging really boosts productivity, so developing without branches may be very slow/inefficient.

    Suppose, for example, that Pete and Jake are developing a car. Pete is adding something simple which should succeed: a sun-roof. Jake is experimenting with an idea to make the car fly. So, both Pete and Jake spend precious time integrating their changes, testing, for example, that a car can open a sunroof and unfold wings at the same time.

    This integration testing is essential: tons of bugs are caused by poorly integrated new features. Think of MERGE and filtered indexes not completely working with each other at the release of SQL 2008. We cannot ensure high quality without it.

    Eventually this flying car idea does not work out, and we cannot just “git branch -D CarsCanFly”. Instead, we have to spend considerable time removing all the code related to this dropped feature. This includes the code in the sun-roof module, which was added for the sole purpose of making sure both sun-roof and foldable wings work together.

    This all can be quite expensive, and slow us down. So, when we choose the ability to deploy very quickly, we just need to fully understand the costs involved.

    • paschott says:

      Great points, Alex. However, if we have an Integration branch of some sort to test these things (separate from mainline), it is much easier to revert that or rebase it from mainline, delete the “carscanfly” branch, and re-merge everything else for testing.

      We’ve definitely run into similar issues in the past and it’s hard to communicate the idea that the DB branch contains all changes merged into it and they _will_ go to production when pushed if not managed carefully. That’s a very different concept from “just push this piece of code for this one part of the app” that exists elsewhere with code.

  4. Louis Somers says:

    There is a distinction between entirely new files and classes, versus existing functionality that needs to be adapted.

    Often there is no problem with adding new generic stuff directly in the trunk, for example adding new entities to a schema. However when changes alter existing code, branching off while frequently merging the trunk back into the branch, and reintegrating once it’s stable, is more common. The advantage of the trunk is that different teams (front-end / back-end) can work on the same new features in parallel, maintaining one single version of truth. While it is possible to branch from a branch and merge between branches, it all adds to complexity and unwanted discipline.

    That said, there is still danger of breaking anything that was set up in a generic way, for example an ORM could complain that mappings are missing, even though nothing is using the new entities yet. They may also show up in generic report generating tools etc… so one should know the architecture well before making such decisions.

    Doing everything in the trunk seems to me only realistic if refactoring is never needed and the team is relatively small. The addition of a sophisticated feature-enable-flag system is from my perspective “branching in disguise”.

    • paschott says:

      Louis, the “branching in disguise” of a feature-enable-flag type system actually allows for a lot of iterative development. Several large SaaS companies use that to incrementally roll out new features to gather feedback, metrics, or perhaps to throttle the usage (among other reasons). We use that and it allows us to push the “done” work while not necessarily exposing it until we’re ready or limiting it to get better feedback before a wider release. Perhaps that piece of work is part of a larger set of work and really shouldn’t be turned on without those other pieces but is otherwise complete.

      That said, we don’t do all of our work in the main branch. We make a branch for the work, regularly merge our mainline into those smaller branches, then merge back to mainline when done and tested. It will still be released “dark” so our customers don’t see it right away, but it’s there and ready to use when we flip the switch.

      I agree about creating branches to do the smaller units of work. That way leads quickly to madness without really, really high levels of discipline. Even with that, it’s easy to put something in mainline that shouldn’t go right away – procs with changed parameters or different result sets come to mind. That gets crucial if you’re using some sort of DB source control like Red Gate or SSDT. Those push your entire DB when you release unless you intervene along the way.

      • Louis Somers says:

        Actually the longer I think about it the more I see sense in a feature toggle system. It could come in handy when a problem suddenly surfaces in production (some late nights come to mind after customers suddenly started installing OS updates with breaking changes on their clients). But using them for work in progress would not make me feel comfortable.

        That said, I’ve never worked on any continuously deployed project. I think that continuous integration with decent code coverage is already quite a challenge. And since our creative manual testers always manage to break stuff in ways I never would imagine, I’d hesitate to set CD as a goal for the company. I get the feeling it would suit a culture where late nights and pizza are taken for granted. Though I’m skeptical about feature toggling bidding enough guarantees, I do agree that it will help lower some risk.

        • paschott says:

          We actually shoot for CD to release our code. It doesn’t really translate into late nights and pizza because it’s not seen by the customers immediately and is thoroughly tested before it’s actually released into the mainline branch. :) Of course, just because we shoot for it doesn’t mean that we hit it, but we’re really, really close to being able to just flip a bit and let it go.

  5. Cody says:

    I think I need lots more real-life examples to pass any judgement.

    I imagine this kind of thing is useful for web-scale companies that can deploy to 0.1% of their web services, using users as guinea pigs, and dealing with the issues as they arise because it’s not like they are going to up and quit your service (i.e. Twitter, Facebook, etc).

    Meanwhile the changes they have made are either back-end and not visible, or front-end and nobody cares about them.

    But on the other end of the scale the applications I work on (usually related to accounting etc) your changes include alterations to the database, web services, backing classes and application code that all go hand-in-hand.

    What are they doing?
    - Creating a configuration option for each change. By the end of the month you have 100 additional configuration options that will never be removed from the code but are inextricably dependent on each other as programming progresses.
    - Adding new parameters to existing stored procedures? And doubling the logic inside to deal with both? And if you use them, doubling the unit tests to include the now two+ sets of returned results.
    - Creating double logic in the application code to deal with both variants of the stored procedure also?
    - And the backing objects? I don’t even want to begin to imagine…

    Oh wait I know this pattern! My_Code calls Stored_Procedure, and the configuration option Use_My_Code_2 now redirects My_Code to My_Code_2 and calls Stored_Procedure_2 with its new options! Awesome!

    I’ve seen that in code plenty of times… code that’s a decade old, or a few months old, it doesn’t matter. It’s always garbage.

    Sorry colour me unconvinced. But I’m starting to understand why the programmers at web-scale companies demand such extravagant wages… perhaps it’s because they have to deal with that crap day in day out :-)

Leave a Reply