Do you suppose Zkybase has similar goals?

Thanks,
Joshua
Founder, Wrale Ltd

Interesting article. Please also have a look how we look at this subject via a tooling perspective:

Looking forward to your feedback.

Best regards,

Mark

In short a cowboy coder is someone who is not a team player and doesn’t even care about the team. Just his job and the paycheck it entails.

Your analogy is more like saying in a society without any heavy handed morality rapists aren’t that bad as they increase the population size.

I think that kids are pretty demanding nowadays. My mom used to tell me that when she was a kid, she was happy playing with a stick or a ball. But by the time I was a kid, I was into computers and such so a stick wouldn’t cut it. I think now the kids are going wild crafting things in Minecraft, building huge castles, and all that stuff, and so now Lego Mindstorms isn’t interesting to them.

Several people have written to me telling me that Minecraft modding provided the right gateway for their kids. I believe it.

1) I agree–Facebook’s capability is based on their processes and infrastructure, among other things (e.g. culture, tools). Hopefully that came out in the post.

2) One of the things Chuck mentioned on push karma was that for a while he knew (like you) which developers pushed risky code. But as the org grew, it became impossible to track mentally. But Facebook’s approach is a little different than what you describe. The FB release engineering team has five members at the moment, and they process up to 300 changes every single day. So they can’t really review each change with a fine-toothed comb. Instead they rely on Phabricator to aggregate the data that I mention in the article and display a simple bar chart that gives a quick understanding of the risk. As long as the risk is low and the developer is on the IRC, they cherry-pick the change. Of course, the detailed information is at hand should they need to drill down for whatever reason.

3) I agree with you that while the cost of the test infrastructure might be high, the cost of not having sufficient capacity on hand might be higher still. It’s up to each org to do the math.

Thanks again for your insights.

The fact that your team is having to deal with a monolithic source-repository with scores of developers working on the project checking-in their code does make it complicated and therefore a challenging proposition, and I see the need to reach out to Chuck Rossi in trying to see how Facebook solved their problem.

1. Yes, it is interesting indeed that Facebook is able to manage releases at massive scales BUT it is also true that they are able to do so, because of the confidence that they have built in both their processes and infrastructure design to get to that stage. Facebook’s release team releases code utilizing Release Trains (from reading what’s publicly available) – So, on days that they have major releases scheduled for come rain or hale, the releases go through and the minor releases (and any code that missed the release train deadline) released M,W,Th – Fridays being the no surprises day of-course.
Facebook deploys and test code – ‘major or minor’ internally before sending out to end-users. Since not much is known about how Facebook handles hot-fixes, I would assume the same process may/may-not be followed to patch the issue that does make it into Production.

2. For the trunk to comply with Facebook’s definition of ‘release-able’ I agree with your assessment of them having tight controls to maintain it. CI will only you take a distance, and a battery of tests(kept updated) on the CI build (done on a defined schedule, I would assume AND/OR on-demand by the RE team) in a staging server will flag any issues that it uncovers and the fact use of the ‘karma rating’ of the developer(s) checking-in their code.

At one of the organizations that I managed the release processes, my team (RM) would review each change with a fine comb having the authority to approve/dis-approve a change request thus come morning when the changes needed supporting we already had reviewed and approved changes AND a list of reviewed and unapproved changes that would either have updates from the development teams over-night OR during the course of the day before their releases were to be deployed; the un-approved changes would then be re-reviewed by my onsite team and be either approved or rejected with comments. Following this process led to 0.08% of changes requiring back-out in Production (an average of 8,000-10,000 releases /yr – combination of releases in all environments). While I did not have a ‘karma rating’ system in place, my interaction with the developers on a daily basis made my team and I well aware of the team/developer(s) delivering bad quality code.

I have in my previous engagements required of the developer or developer-delegate be available before a production deployment done and only when they would complete a test-plan on the newly deployed code have the code actually released to the live-traffic (load-balancers help ) – should a developer not show-up for their code-deployment, the code would be held back with the developer/team requiring a good justification / business reason to be unavailable for the deployment/release. So, the fact that Facebook also requires of the developer(s) responsible for their checked-in code to be available does not surprise me at all. With regards to the speed/quality tradeoff – that problem could be solved easily in a onsite/offshore environment, where-in the team at one of the locations is available to cover the release.

3. Performance and Capacity planning are well-known but then not many an organizations have it implemented as part of their release process – I know, because I have either worked at a few that did and some that did not. Its organizations that have their code highly distributed to make it easy to develop/maintain/release/support that have to cherry-pick which application(s) should go through with the P&C tests. Only when the P&C team signs-off on the release should it be made available to be deployed to production and released to live-traffic.

Infrastructure does not come cheap, and not many an organizations can afford to invest in elaborate staging/test environments like Facebook does BUT if most-of your business if not all of it comes from your website/application having HA and performing well within the SLA’s – business teams should not hold back. All that is needed, simply put is to show to the business team the benefits they stand to reap on their investments.

Fair question. The answer is that we’re trying to get high quality weekly and daily releases working for a large web app. Nothing too special in the goal there.

We do have a monolithic source repo, and Facebook does as well. So we wanted some pointers on how they manage to avoid broken builds with 700 people committing against it.

While I agree that the concepts above are largely well-known, there are some interesting points and subtleties to consider.

1) It’s interesting that Facebook is able to use more or less standard prescriptions to manage releases at massive scale. A best practice could fall down well before 1B users without losing its claim to being a best practice. Facebook’s experience says that these practices–layered appropriately–scale just fine to releasing to at least 1B users.

2) Another interesting point is the foundational role of cherry picking and code review in Facebook’s process. The usual advice doesn’t mention cherry picking at all. Instead it says to keep the trunk releasable so you can deploy when you like.

Facebook would agree that the trunk should be kept in good working order, but they have a strong definition of “releasable” that requires an approach that goes beyond simply keeping the build green. Whenever Facebook pushes out a change, the engineer responsible for that change needs to be hanging around to make sure that he’s ready to help if something goes wrong.

If you simply keep the trunk green, then anytime you want to do a daily release, you have to have the whole team around, because presumably everybody’s been committing code. That’s hard. So there’s this speed/quality tradeoff: you either wait for the end of a sprint to do your releases (speed hit), or else you drop the requirement that every engineer has to be hanging out to shepherd their change through to production (quality hit).

3) Test parallelization is clearly a well-known concept, but the level of aggressiveness described in the post is not as well-known. (Happy to be educated on that point.) In a lot of the reading I’ve done, people talk about primarily running unit tests as part of the commit stage of the CI build, with a handful of strategically chosen integration and/or acceptance tests in the mix to detect known risks. The Continuous Delivery book, for example, presents that approach. And while I think it’s a fine approach, the Facebook extreme parallelization approach means that you don’t have to give up either speed or quality when you commit: you can run the entire test suite (unit, integration, acceptance) on every single commit. I think this idea may be cost prohibitive for some orgs, so that could be why it doesn’t seem to come up.