When I first joined a rapidly growing startup as the solo tester for the web team, I made every mistake in the metrics book. Fresh from an enterprise background, I tried to measure everything. Six months and countless spreadsheets later, I had plenty of data but no real insights. Here’s how I turned it around.
The Metrics Mess: Where I Started
My initial dashboard included:
- Test case execution rates
- Code coverage percentages
- Number of bugs found per feature
- Automation coverage
- Time spent testing
- Regression test results
- API test results
- UI test results
- Performance test metrics
- And so much more…
Each metric seemed essential. Each one took time to track. None of them actually helped us make better decisions.
The Breaking Point
The turning point came during a release meeting. Our CEO asked a simple question: Are we confident to deploy? Despite all my metrics, I couldn’t give a clear answer. That’s when I realised something had to change.
The Transformation
Step 1: Focusing on What Matters
I stripped everything back to one crucial DORA metric: Change Failure Rate. But I simplified it:
- Is the feature working in production?
- Did customers report any issues in the first 24 hours?
- Did we need any hotfixes?
Just these three questions told us more about our quality than all my previous metrics combined.
Step 2: Building Back Better
From this foundation, I gradually added only metrics that helped answer specific questions:
Are we getting better at catching issues before deployment?
- Track customer-reported issues in new features
- Compare with issues caught in testing
- Note patterns in what we miss
Are we maintaining stability while moving fast?
- Monitor critical user journeys
- Track recovery time when things break
- Note deployment frequency against failure rates
Lessons Learned
- Start with the Questions Instead of What can we measure?, ask What do we need to know?
- Embrace Simplicity A simple metric that drives actions beats a complex one that nobody understands.
- Use What You Have I stopped trying to implement enterprise tools and started using:
- Our existing issue tracker
- Simple spreadsheets
- CI/CD pipeline data
- Customer support tickets
- Focus on Trends Absolute numbers matter less than patterns and changes over time.
Today’s Approach
Now my metrics dashboard fits on one page:
- Deployment success rate (simplified Change Failure Rate)
- Critical path stability
- Customer-reported issues in new features
Each metric:
- Takes minutes to update
- Drives specific actions
- Helps us make real decisions
- Maps to DORA metrics without the complexity
Key Takeaways
- Less is More You don’t need dozens of metrics to make good decisions.
- Context Matters A simple metric with good context beats a perfect metric with none.
- Evolution Over Perfection Start simple, learn what works, and evolve gradually.
Moving Forward
If you’re struggling with metrics overload:
- List all your current metrics
- For each one, ask: What decisions does this help us make?
- Keep only those that have a clear answer
- Start rebuilding from there
Remember: Your job is to help deliver quality software, not create perfect metrics.
I’ve just released a new eBook with 7 pitfalls to avoid in this situation, get it here:
