Fail Fast to be Awesome
Working with eight year old kids to build a robot is an interesting experience. Introducing bright, motivated kids to engineering meant teaching them to fail.
The kids watched a video of the space shuttle moving from its maintenance hanger to the launch pad. Then they built a four foot tall rocket model from Lego bricks.
- They ran out of bricks (frustrated)
- The model collapsed at about three feet (dispair)
- They started checking its structural integrity regularly (break through!)
Ultimately it took three practice sessions. They were over time and over budget. But what had they learnded?
- It’s OK to be wrong, but better to be wrong with a purpose (set out to learn something)
- Test early and often (tight feedback loops)
- Clarify expectations (understand requirements)
Today, they are consummate hackers when programming their robots to solve problems. They break problems into small, testable peices. They set up trials, identify and exploit opportunities. Most of all they realize the need to do this over and over in pursuit of perfection.
Eight year old kids building a robot that can follow a maze and self correct.
Excellence does not preclude failure. It requires the careful management of of rapid failure.
A New way of Working
When dealing process improvement, the fundamentals of science we learned in high school apply:
- Form a hypothesis
- Collect and interpret data
The magic comes when we repeat this until we get a process that maximizes throughput and quality while minimizing cost to the organization as a whole.
To paraphrase Jim Collins in *[http://www.jimcollins.com/article_topics/articles/good-to-great.html](Good to Great)*: Process improvement is a cycle of effective incremental change: evolution, not revolution.
Feedback early and often from business stakeholders will:
- Improves quality
- Reduces overhead
- Improves due date performance
- Lowers stress
The DevOps team set up the necessary equipment for the development, staging and production websites at the beginning of the project.
The Project Manager (PM) scheduled twice weekly 30-minute, show-and-tell meetings.
The Solutions Manager (SM) and Business Analyst (BA) worked with the Developer to turn requirements into a list of tasks order by priority.
In the first show-and-tell meeting, the Developer showed a working, but incomplete, version of the site for review by the PM and SM. Note that the PM was well positioned to represent the meaning of the requirements and clarify any misunderstandings.
Every subsequent Tuesday and Friday, for a strict 30 minutes, the Developer demonstrated progress based on the tasks in the backlog. This caused changes in work behavior for the Developer:
- Instead of looking at the job as a single deadline, they deliver value to the team on a cadence
- Effectively speaking/listening to the business regularly instead of living “in a cave”
- Proactively setting expectations. Earning trust means no surprises.
After each meeting, the PM and SM cataloged problems identified during the review. Problems were not just those for the Developer, but also clarifications needed from the PM or BA on requirements.
Quality was obviously improved. This is best seen when the Account Manager reviewed the site and was stunned he could only find one issue.
Overhead was reduced. This was best demonstrated by the severe reduction in ad hoc meetings.
Due date performance improved. We were on time.
Stress was much lower. The number of times we had released the site to production systems in support of the show-and-tell meant there was no worry about final delivery.
Expanding the “development team” to include the PM was critical to the success of the project. Regular meetings with the person brought a two way accountability between development and the business that cannot be effectively maintained through email.
The assumption that one of our Creative people could not participate from the beginning was challenged. The next experimental run of this process included Creative from the beginning. Silos are seen as inhibitions to effective communication.
The assumption that we could not bring an Account Manager in early was challenged. The next experimental run of this process included an AM early and often.
We questioned the assumption that we could never show an unfinished project to the customer. This is scheduled for discussion again after the next experimental run.
SMs and PMs are demanding to run their projects in this new way. Developers are eager to participate. The Creative department has been an excellent partner
The second run of the experiment was done on a highly visible project with a very demanding customer. The Account Manager let us know the other day:
“That new process was amazing. When I saw the new site there were only a few small bugs. This was great, I want to know how it works!”
The answer to “how it works” is simple. Identify silos, build communication, share responsibility, enjoy shared success.
Technology enables and enhances rather than causes success. In this case we leverage the same configuration management technology from May’s post that allows developers to quickly switch between projects.
Configuration Management allows us to launch an identical machine into the cloud at the beginning of the project. On this machine is the “dev” environment where show-and-tell occurs. The developer pushes new features to “dev” for anyone on the team to review multiple times per day. This is known as “continous delivery.”
When the PM decides stakeholders can be shown the project, they push a button and the current “dev” code is moved to “stage.” The stakeholder reviews the changes and lists needed corrections. The separation between environments allows change to occur at a pace appropriate to the purpose.
On the go-live date, another button is pushed by our Group Director of Digital and the approved “stage” code is moved to the production environment. Because delivery of the exact same code has been practiced over and over in the “dev” and “stage” environments, the risk of unforeseen issues is extremely low.
The build system we use is Atlassian’s Bamboo. Communication about changes to environments are made in Hip Chat in real time by Bamboo. Information is not delayed because someone is in a meeting.
Note that the Operations team lacks a role. This means:
- The PM need not manage any one from the Operations team
- The Operations team is not in meetings, they are working to improve the platform and delivery mechanisms.
- Operations is no longer the last step when time has run out and there is a mad scramble to do the minimum.
Notice the following important roles enabled by the technology:
- The developer is responsible and enabled to deliver change at any time
- The ability to involve the stakeholders rests with the best perspective: The project manager.
- Accepting that it is OK to get things wrong and learn from them leads to great success.
- The fact that code has been deployed dozens of time to the cloud means that making changes is simple, repeatable and reliable.
It is engineering for repeatabilty that will be explored in the next installment of the DevOps digest.