I spend quite a lot of time at conferences, and it consistently bothers me that they are so often focused on one particular function: development, testing, UX, systems administration. The point of continuous delivery is to accelerate the rate at which we can learn from each other – and from our customers. That requires everyone involved in the delivery process (including users, product owners and entrepreneurs) to collaborate throughout. So why isn’t there a conference which focuses on flow – the emergent property of great teams?
I am not going to do a ton of book reviews on this blog (I have one more planned for next month). I’ll only bother posting reviews of books that I believe are both excellent and relevant to Continuous Delivery. This book easily satisfies both criteria. Full disclosure: Gene gave me a draft of this book for free for reviewing purposes.
Told from the perspective of newly-minted VP of IT Operations Bill Palmer, it describes the turnaround of failing auto parts company Parts Unlimited. This is to be achieved through the delivery of the eponymous Phoenix Project, a classic “too big to fail” software project designed to build a system which will revive the fortunes of the company.
In his new book, Antifragile, Nassim Taleb discusses the behaviour of complex systems and distinguishes three kinds: those that are fragile, those that are robust or resilient, and those that are antifragile. These types of systems differ in how they respond to volatility: “The fragile wants tranquility, the antifragile grows from disorder, and the robust doesn’t care too much.” (p20) Taleb argues that we want to create systems that are antifragile – that are designed to take advantage of volatility. I think this concept is incredibly powerful when applied to systems and organizational architecture.
“it’s possible for good people, in perversely designed systems, to casually perpetrate acts of great harm on strangers, sometimes without ever realising it.” — Ben Goldacre, Bad Pharma, p. xi
In a fit of rage caused by reading yet another email in which one of our customers proposed creating a “devops team” so as to “implement” devops, I tweeted that “THERE IS NO SUCH THING AS A DEVOPS TEAM.” Like all slogans, there’s plenty of heat to go with the light, so here’s the scoop: the Devops movement addresses the dysfunction that results from organizations composed of functional silos. Thus, creating another functional silo that sits between dev and ops is clearly a poor (and ironic) way to try and solve these problems. Devops proposes instead strategies to create better collaboration between functional silos, or doing away with the functional silos altogether and creating cross-functional teams (or some combination of these approaches).
This is the third in a series of interviews on continuous delivery, this time with Elisabeth Hendrickson. You can see the first one, with Jesse Robbins, on the ThoughtWorks Studios Blog, and the second, with John Allspaw, here. These interviews will eventually be put together – along with some additional exclusive material – into a set of LiveLessons, with the royalties going to Black Girls Code. If you want to be notified when the final product is released, please sign up for my newsletter.
Elisabeth has held positions as a tester, developer, manager, and quality engineering director in a variety of companies ranging from small startups to multi-national enterprises. She has been a member of the Agile community since 2003, served on the board of directors of the Agile Alliance in 2007/2008, and is currently one of the co-organizers of the Agile Alliance Functional Testing Tools program. Elisabeth won the prestigious Gordon Pask award in 2010 and is the author of two books: There’s Always a Duck and the forthcoming Explore It! Reduce Risk and Increase Confidence with Exploratory Testing published by the Pragmatic Programmers. She is the founder of Quality Tree Software, runs Agilistry Studio, and is the creator of entaggle.com. You can find her at testobsessed.com and @testobsessed
In the video, Elisabeth answers the following questions:
What’s been your experience of the evolution of the testing and QA role in the IT industry?
Why doesn’t separation of duties between engineers and testers work? (01:49)
What are the problems with the hierarchies you sometimes see within testing organizations? (04:30)
What’s wrong with the approach of writing and executing manual test scripts? (07:53)
Does record and playback help to solve some of these problems? (08:48)
So the current paradigm for building quality in to software is acceptance-test driven development. Talk us through ATDD. (10:04)
What does what in ATDD? (11:19)
What proportion of time is spent on production code vs test code? (14:15)
How do you get from a typical “enterprise” testing strategy to ATDD? (17:30)
When implementing automated acceptance tests, where do you start? (21:31)
How has continuous deployment changed the way you approach software delivery? (23:43)
How do you go from releasing once every 3-6 months to releasing once a day? (26:36)
The case of Apple sometimes comes up in discussions around continuous delivery and the lean startup. For example, Richard Durnalldescribed Apple’s strategy to me on Twitter as follows:
Brilliant and unwavering product vision from a few amazing folks going to market infrequently with huge ceremony.
That seems to be the exact opposite of what both lean startup and continuous delivery preach. It’s hard to know what happens behind the scenes at Apple because of their notorious secrecy about their product development process. Furthermore, there’s no way of knowing if the information we do have accurately represents what really goes on there. But if we look at Apple’s history, there are a couple of examples that strongly suggest that the principles behind continuous delivery and the lean startup were very much in play in Apple’s early years.
This is the second in a series of interviews on continuous delivery, this time with John Allspaw, one of my co-authors on the Devops Cookbook. You can see the first one, with Jesse Robbins, on the ThoughtWorks Studios Blog. These interviews will eventually be put together – along with some additional exclusive material – into a set of LiveLessons, with the royalties going to Black Girls Code. If you want to be notified when the final product is released, please sign up for my newsletter.
John Allspaw has worked in systems operations for over fourteen years in biotech, government and online media. He started out tuning parallel clusters running vehicle crash simulations for the U.S. government, and then moved on to the Internet in 1997. He built the backing infrastructures at Salon, InfoWorld, Friendster, and Flickr. He is now SVP of Tech Operations at Etsy, and is the author of The Art of Capacity Planning and Web Operations published by O’Reilly. He blogs at kitchensoap.com
In the video, John answers the following questions:
What is devops? Why now?
What did Flickr do differently that worked? How did you apply that at Etsy?
How did you take Etsy from painful releases to continuous deployment?
How about larger organizations? Does continuous delivery impose more risk?
What’s the role of operations in an organization that wants to practice devops?
Is there still room for specialization in the devops model?
What advice would you give developers who want to do continuous deployment?
How do you reduce the risk of releases?
How do you create resilient systems on the web?
How do you deal with databases in the world of continuous delivery?
Translations:中文 | 한국말
There’s a lot of dogma in the religious wars around software development practices and methodologies. Are phase-gate methodologies effective at managing the risk of software development, or just risk management kabuki? Does TDD really make for higher quality software? Is pair programming a superior replacement for code review or just a way to inflate consulting rates? I’m going to argue that while scientific evidence to decide these claims is lacking, there are two general principles which can help us choose good practices while at the same time improving the value of the software we deliver: reduce cycle time and increase feedback.
There’s been a lotof controversy generated by Voke’s Agile Realities report. SDTimes asked me to comment for their article covering the report, and so I got to read it in full. Obviously Voke want your money to read the whole thing so I am constrained by fair use as to what I can reproduce in this review, but my general conclusion is that the report represents a hatchet job on agile whose most important conclusions aren’t sustained by the (interesting and revealing) data they have gathered.
At DevOpsDays Mountain View I was lucky enough to get some time with Michael Rembetsy, Director of Operations Engineering at Etsy, which manages to be PCI-DSS compliant while practicing continuous deployment. In this short interview, he describes how they do it.
Here are some of the key points from the interview:
Etsy deploy to production 25-50 times a day;
They built a new, segmented PCI-DSS compliant environment for their payment systems – “we built a whole separate Etsy, essentially”;
Segregation of duties doesn’t mean people can’t talk to each other. In fact, collaboration is an essential element of effective risk management. So everybody in the PCI environment still follows devops principles – there’s no physical separation of the teams, and Etsy have hired more people into the various roles (dev, sysadmin, dba, manager, networking) to facilitate collaboration;
In the payments environment they “still have to follow the rules: a developer still doesn’t have access to a production database”, but they’ll have dbas working alongside them who they can ask for help, and graphs showing metrics from the database;
They use a similar deployment process in their PCI environment to their non-PCI environment, but it includes much more logging on who did what when, and they have roles for QA pusher and production pusher;
Developers can run most of the test framework in their PCI development environment, and they have access to production logs via Splunk in read only mode;
They have a ticketing system that all changes have to go through, but they can push out a change from dev to production in less than an hour if necessary with their normal, non-emergency change management process;
Final words of advice: “Make sure that the team is on board and realizes that yes, there’s going to be some constraints, but … if you all work together you’re going to be able to continuously deliver the product that you want even into a secure environment.”
