How your engineering team can get more from incident reviews

Another problem with the format was that it didn’t create natural openings in conversation for other engineers to jump in. By focusing on a single event with set questions, we often overlooked commonalities and trends across incidents. This made it difficult for other engineers to understand what they could take away from these meetings and how the lessons might apply to themselves and their team.

We decided to start experimenting.

• We looked at the main improvements we wanted to see:
• We wanted everyone to feel truly safe
• We wanted to open up the conversation, encouraging open dialogue and collective learning
• We wanted to observe patterns and move beyond isolated incidents to observe recurring trends
• We wanted our weekly meeting to feel valuable, to be a weekly highlight for engineers

Facilitation is a learned and practised skill – hard to perfect but hugely rewarding when done right.

The three desired outcomes as goals are:

• Time flies
• Everyone stays engaged
• Everyone grows, even the facilitator

We set a goal to generate at least four strong talking points: two based on the particular incident under review and two that addressed common themes. At the start of every meeting we would launch a poll with each of our talking points and ask participants to vote for the topic they’d most like to discuss. We’d start with the topic that topped the poll and go from there.

The question and answer format was not working, that was clear. So how could we guide constructive conversations that brought about real learning? It has always been team practice to assess submitted reports before each review, to make sure they meet our standards for a good report. We decided to capitalize on this process and use this review as an opportunity to extract common themes across multiple incidents.

To make sure these techniques were working we started sending automated surveys to our incident review Slack channel after each review. The poll asked participants to anonymously agree or disagree with the following statements:

• This meeting was a good use of my time
• I think we discussed the right things
• This meeting felt safe and constructive
• I learned something new in this meeting

The progress we’ve seen to date has been really promising. We still have our outstanding problems, like how to tie these learnings into actionable items on our roadmap, or how to keep attendance strong in particularly busy weeks. But the aim of our program is to build an honest, accountable, and constructive culture within teams that care about our customer experience.

The way your organization processes failure is a mechanism of change – if that mechanism is broken, boring, or painful, then your organization can’t reach its full potential.