Using Experimentation to Evolve our Story Kick Off Process

What were the challenges we were facing?

With an ever-expanding number of product teams, QA Engineers moving between teams and new QA Engineers joining with differing levels of experience, it became challenging to ensure our existing Story Kick Off process was delivering a high standard of quality across all of the product teams.

So, a small group of QA Engineers joined together as a focus group that met on a regular basis.  We decided it would be best to have a common approach to Story Kick Offs to ensure we had a tried and tested approach to quality that would be easy to onboard new starters with, plus would be followed by all teams.  

Firstly, why do we do Story Kick Offs?

In order to find a common approach that would ensure we had met our expectations, we decided to come up with a statement of purpose for the reason why we do a Story Kick Off. This is what we came up with:

Story Kick Off: To ensure a shared understanding of the purpose, impact, and risks for the piece of work so that we know what Done looks like.

Next up, we looked at the pros and cons we experienced with doing story kick offs currently.  This generally had the same positives and pain points even though the focus group were QAs from different product teams.

The main positive was:

• Discussing and writing ‘Acceptance Tests’ in BDD format (GIVEN/WHEN/THEN) gives us a shared understanding of the expected outcomes of the story.

We can check these outcomes during the Story Sign Off to ensure that we have seen the main functionality of the story working before the code is committed.  We can also use these as a guide when automating our test coverage.

The main pain point was:

• The availability of the PO, QA, and Developer (when the story is ready to be started) to get together and do a Story Kick Off and write the Acceptance Tests, which can end up being a blocker to a Developer being able to start working on a story.

We came up with 5 different ideas to experiment with

As you can see, with the main pain point being lack of availability to do Story Kick Off’s it was quite easy to come up with a few different ideas to take back to our teams to experiment with.  They were as follows:

1. ‘Not everything needs a Kick Off’ – to discuss each Story and decide if it would really benefit from a Kick Off.  Do all different Story types need us to write Acceptance Tests?
2. ‘Write the Acceptance Tests together in Sprint Refinement’ – the whole team write the Acceptance Tests together during the Sprint Refinement session.
3. ‘Write the Acceptance Tests together in Sprint Planning’ – the whole team write the Acceptance Tests together during the Sprint Planning session.
4. ‘Pre-seed / Write skeleton Acceptance Tests in Sprint Refinement’ – writing skeleton Acceptance Tests in Sprint Refinement to then be finished during the Story Kick Off.
5. ‘Write Acceptance Tests in QA’s own time’ – the QA to write the Acceptance Tests in their own time before the Story has it’s Kick Off.

We then went back to our product teams and explained which experiment we were doing and why we needed a common approach and how we felt it could address pain points within our own teams.  Our teams were on-board with the experiments and agreed to run them for 4 Sprints – which we would then follow-up by doing a Retro with each team as a chance to reflect.

How did the different experiments compare for the teams?

We wanted to make sure that we had both quantitative and qualitative data from our experiments so that we could compare our results across the different teams.  We came up with these questions to send out to our team members to be answered before the Retro with a 1 – 5 scoring system:

1. I felt the changes had a positive impact on Kick Off.
2. The changes impacted my time positively.
3. I knew what I needed to do for the Story.
4. We missed something because of these changes.
5. Risk was introduced to our work.
6. I felt the changes had a positive impact on sign off.
7. Would you want to carry on with this change?

We agreed to then bring the results along to our Retro’s for discussion and have the remainder of the Retro in the “What went well” / “What didn’t go well” format.

The “Retro of Retros”

Once we had completed all of our Retros it was on to the “Retro of Retros” – comparing our experiment results across product teams. Our results were overwhelmingly positive when it came to writing our Acceptance Tests together as a group, whether it was in Sprint Refinement or Sprint Planning sessions.  We felt that writing them together met our expectation of a shared understanding and actually mitigated the need for a Story Kick Off when the story was picked up by a Developer as it was ready to start immediately.
We didn’t feel that the QA Engineer writing Acceptance Tests on their own supported our expectation that quality is a whole team responsibility and could lead to the QA function becoming siloed. Also, it is a lot of work for one person to be expected to do and could potentially fall down when the QA Engineer is away.
Another positive change was the “not everything needs a Kick Off” experiment – flagging different Story types as not requiring a Kick Off.  This was supported by documentation, with examples and explanations, outlining different Story types and whether they need Acceptance Tests written.

Our new process for Story Kick Offs

We agreed that teams would be able to choose when the Acceptance Tests were written, as different Sprint meetings will work better for different teams, as long as it’s by the time the Sprint starts. For teams working in Kanban/milestones we state this needs to be done by the start of the milestone.

We do this by ensuring:

• All the Stories in the Sprint that need Acceptance Tests have got them.  (To determine if a Story needs Acceptance Tests we can refer to our document on ‘Acceptance Test Approaches by Story type’).
• Acceptance Tests are written and reviewed by the whole team to ensure a shared understanding.

Advantages over our previous process

• Kick Offs are no longer necessary – Acceptance Test writing happens earlier – in Sprint Planning or Sprint Refinement.
• Clear definitions are given for which Stories need Acceptance Tests, and which don’t.

The rollout to all product teams

Next up was the rollout to the rest of the product teams.  We presented our new process to all of the QA Engineers during one of our regular QA Community sessions and explained:

• Our need for a common approach.
• The statement of purpose we had come up with.
• The different experiments we did.
• The results we received via our individual team Retro’s.

We asked all of the QA Engineers for their thoughts on these changes and how they thought their product teams would feel?  We then explained our conclusions and requested all QAs to introduce our new process in their product teams.  To aid the rollout we have documented the approach and setup a Slack channel for any support needed.  We have also booked in a Retro in a few months’ time to see how the teams are taking to the changes.  Our QA Manager will also be following the teams progress and be on hand to help teams as and when needed.