Jonathan's Blog

Pillars of Data-Driven Software Engineering


Jonathan Fries

Jonathan Fries

I work for Techtonic as the leader of the software delivery group. I am a Certified Scrum Product Owner.

Pillars of Data-Driven Software Engineering

Posted by Jonathan Fries on .

Pillars of Data-Driven Software Engineering

Posted by Jonathan Fries on .

I've spent quite a while in companies where metrics and KPIs are important: they are valuable tools, when used correctly. Everyone is more empowered when they see the business succeeding and see their own role in making it happen.

But I chose the image for the header of this post intentionally - it shows people working together, not a bunch of numbers or a pie chart. This is because people must engage with the data, and ideally, they should be the ones requesting more data to do their jobs better. This only happens if you take them into account at the beginning of any such program and figure out what makes them want to succeed, and what they need to do their jobs better.

When it comes to KPIs and metrics, I've seen problems crop up with them that usually fall into 3 big categories:

  1. The creation and roll-out is ham-handed and focuses on management without listening to people, understanding what they need or want, and addressing their fears. People will be afraid - engage the fear or risk the disengagement that will follow.
  2. What happens if a business, department, or team has an off year/quarter/month? How can you keep team members engaged? How do you keep your KPIs or data-based management from feeling like a flogging?
  3. How do you balance the contributions of teams and individuals? How do you strike a balance between team metrics that measure (and reward) team success vs. those that recognize individual contributions? Both are important.

Because I've been getting exposure to OKRs through some client work recently, I think it is worth looking into how a successful metrics/OKR system can be constructed and avoid the problems up above.

I have created this diagram to show how the elements should work together:

Here is a quick overview of the 5 major components:

  1. Company Vision is critical. It really comes from outside and is separate from the pillars, but it influences them strongly and should determine (especially in the case of KPIs and OKRs) what they are. Company vision is the 'Why' of the company and you should see this strongly reflected in KPIs and OKRs.
  2. KPIs (Key Performance Indicators) measure the ongoing business performance of the organization, including its profitability and how it achieves its vision. If your KPIs miss one of these marks (I've seen it) then your employees will be disconnected from it.
  3. OKRs (Objectives and Key Results) - these are measurable goals that are more transitory than KPIs. OKRs should be a measure of what you are doing NOW (this month, this quarter, this year) in order to achieve and improve results. OKRs done well should result in improved KPIs - at their best they show us that we have selected the right work and done it well.
  4. Software Metrics - this has proved elusive in organizations that I have worked for. What is the measure of good software development? What is the measure of a great developer? We're currently starting to use GitPrime - now part of Pluralsight, internally and with some clients. Good engineering metrics should result in agreed upon set of standards for software engineers, a high bar for quality of work, and they should drive us to deliver more and better features, so that we can do more valuable work.
  5. Positive Behavioral Metrics - I will spend some time outside the bulleted list on why this is important and how this works. But if you're skimming this then you only need to know that the orange arrows tell you why this is important. When you have a down quarter, when a team struggles, what gives people energy and lifts them up so they can deliver anyway? What makes them carry on? What makes them feel that it is worth it to do so? This is driven by positive behavioral metrics.

Don't buy it? Wishing for comprehensive behavioral metrics? Let me explain why positive is so key here:

  1. You never have to look far to hear negative behavioral information. It's there everyone knows about it, the focus here is to gather up the positive stuff to act as a counter balance.
  2. Your other pillars - KPIs, OKRs, Software Metrics leave plenty of room for the realistic and pessimistic. If there are issues, they will show up there.
  3. If you have a real, true behavioral problem on your team somewhere, you know it. You need to deal with it. Waiting around for a mid-year or annual review is wrong. The lack of presence of some type of negative ledger should create MORE urgency for you as a manager - not less. Deal with your drains, negatrons, and nasty actors - do it now.
  4. HR already has systems and paperwork for negative behavioral problems. You don't really need another one.
  5. An empty ledger says a lot - if someone has no items in their positive behavioral tracking system. Essentially the system for sending out positive vibes, you can assume that they are not a force for inspiration and energy on a team.
  6. Link positive behavior to your company vision and guiding principles whenever possible. The arrows in the diagram show everything flowing out from your company vision, and this is how it should be. Your vision and mission (or whatever you call it) should be your barometer and benchmark for what you do. But, when people succeed, if you link this back to parts of your mission or principles, this can help to reinforce why what they did was important, and later on you can look at where and how your principles are being re-inforced, and see if there are any gaps worth thinking about.

Challenges with 'Comprehensive' Behavioral Metrics

In one of the organizations that was very metrics focused, we had a metrics section for behavioral traits. This was not exclusively about positive behaviors.

The rating's were from 1 (not at all cool in any way) to 5 (godlike).

There were a lot of 3's and 4's handed out by managers. Which was correct and how the scale was designed, but it lead, almost exclusively to problems.

Problem 1: Most people view themselves as 4s or 5s. As a result, the reactions that people had to pretty realistic, generally positive reviews (say a 3.5) was one of disappointment, "You mean I'm not a 5?" When the system was set up so that almost no one could ever get a 5.

Did anyone ever move up from being a 3 to a 5? A few, but it didn't have anything to do with the review process or the behavioral traits metrics. It had to do with great managers and mentors making a real difference in people's careers, so that they were uplifted and excited to come to work. Getting a 3 (out of 5) on a performance review never does that.

Problem 2: By the time you were handing out a 2 or a 1, it was too late and you should have fired that person already. I can think of only one exception to that, and it was a 2.5, not a 1.

This is why I have a pretty dim view of behavioral metrics, and see that focusing on the positive is the most valuable way to come at this.

The diagram up above is how I see this all working together. I am currently working with Weekdone for OKRs and I like how you can essentially set and use this at the granularity you prefer. You can have team OKRs and you can have individual OKRs, and they can roll up or not.

I think this is an excellent pivot point between the team/individual and it allows you to customize this to your needs.

For software development I think a lot of your engineering metrics are going to be on an individual level (# of commits, accepting pull requests, etc).

This allows you to keep KPIs at the global or divisional level, where I think it is good to have a strong focus on the team and the sum of the parts working together.

How To Get It All Done

"Sure," you say, "that is a nice diagram Jonathan, but who has time for all of this stuff?"

Or perhaps you say something less flattering.

This is a fair criticism, our time and our energy are our most precious commodities as leaders, so a bunch of extra paper pushing does not help anyone.

Here is a cost effective way to approach these, get started, and grow it as you see which is most valuable for you.

  1. KPIs - Most KPIs should be an accounting function, and those that are not either should be, or they should be baked into the software product you develop as a key management function, end of story. Beware of special cases and subdivisions that make reporting on KPIs more work. Everyone should get the same measurement, that is what makes it a KPI. Large organizations with true business divisions can subdivide. If that is you, great. If it isn't you, don't spend your time one it.
  2. OKRs - Experiment and scale as necessary. I mentioned using Weekdone, and I think it is a fine tool with a nice free-to-use option for getting started. You could also create some basic OKRs for teams for a quarter and use a spreadsheet to track them. That would be a fine way to start. Just be sure you're getting the spreadsheet out periodically and discussing it.
  3. Engineering Metrics - Here you need a tool. There are a number of these now, and I do not have experience with all of them, but engaging your development team in understanding their work better is the key here - fine a tool that helps you engage them in understanding their work and how to get more done. What are the roadblocks? What works well? What is challenging? To the extent that it is possible: get the teams themselves to engage with and suggest ways to use these numbers. Homegrown metrics that are seen by the team as serving them or their fellow devs is the maxim you should go with. Make suggestions, sure. But the more this can be owned by the people building stuff, the better off you'll be. Here are few tools that are available in this space, I do not know all of them well:
  1. Positive Behavioral Metrics - I think a tool is helpful here, but not 100% necessary. A tool helps you to spread the positive energy to more people because it can do automatic emails, texts, or messages. Those messages give you a boost, show your employees and bosses that you spread that energy, and gives all of them a boost too. Just sending an email to 1 person and tracking the info in a spreadsheet doesn't do those other things. I've had to build this tool myself at several companies and I have basic source code to do it. I am currently developing an open source solution, and when I get it done I will share. There are tools for employee recognition out there, I do not know how well any of them solve the core problem of multi-person recognition, up-chain recognition, general positive energy flow in organizations, and principle-based recognition. Those are the core tenets of what my solution does (and has always done).

In the absence of a great tool, you simply have to find ways to spread as much positive energy as possible and make sure you are up-reporting (sending your kudos to the person and that person's boss or boss's boss or both). Not the world's worst assignment. As one CEO that I worked for said to me, "One of the best parts of my job is going around recognizing people for all the great things they do." So, sending her emails about my team's successes was not hard for me to do. It made her feel good, it made my employees feel good, and it made me feel good. That's some positive energy.

If your method doesn't immediately enable superb reporting, try to keep a record somewhere - you can even use email history.

Jonathan Fries

Jonathan Fries

I work for Techtonic as the leader of the software delivery group. I am a Certified Scrum Product Owner.

View Comments...