We love to measure things.  How many steps we take in a day, how many emails we have in our inbox, how many miles to work, how long it took to commute and the weather.  As human beings we like to measure things up to understand them better.  This apple is 5 inches in diameter and this one is 4 inches in diameter.  From that we make judgments like this apple is bigger so therefore I’m getting more for my money.  We measure for the sake of measuring.

Metrics are used to decide if something is of value or performing well.  So off we go creating metrics and measurements for it.  Let’s say we are measuring a subway system.  When we think of a subway system, we think about measuring the number of riders, number of trains, percentage of trains that are on time, total fees paid by riders, average distance traveled by riders and more.  There are a ton of metrics that could be used to measure a subway system.  I’m sure you could think of at least a dozen or more without really trying.  We try to measure everything - even if that measurement isn’t of value.  The apple may be bigger but does it taste better?

We even baseline and track them over time.  Ridership this year was 10% better then last year.  This year’s crop of apples is 5% bigger then last year.  Of course that leads to some pretty pictures in PowerPoint and nice graphs to display on your wall.   We even budget or forecast metrics and make guesses on what the metrics will be next week or even 5 years from now.  Those trending graphs on PowerPoint slide look awesome, but can we do anything with them?  Do they help us add value to our system, product or project?

Metrics get the crap analyzed out of them.  Then there is the universal truth “numbers can be made to support anything”.  Complexity and mind numbing calculations produce the results we are looking for to convince ourselves we are performing well or on the right track.  Or worse, we dive into the black hole of analysis paralysis – stuck trying to divine the future out of metrics and measurements that are incomplete or inaccurate.

We fail to step back and ask the question “Is this metric of value?”. 

A few months ago I took trip to meet a client.  I got an upgrade from the “Steerage” section to the one inch more in front of your knees “Not Quite Steerage” section.  Felt pretty special until I realized the upgrade basically allows me to exit the plane faster and get a bag of peanuts.  I grab the luggage and head out to the rental cars to pickup my car.  The rental car agent let’s me know I’m the lucky recipient of a brand new SUV.  This upgrade was really an upgrade by giving me more leg and luggage room.  Off I went to visit my client.  After I parked the SUV, I exited the vehicle and hit the lock button on the key fob.  Nothing happened.  You usually get that beeping noise. I tried again and again.  Still no sound.  I tried to insert the key into the door to lock it manually but that didn’t work either.  With all my luggage in the SUV made me a little worried leaving it unlocked.

So I called the rental car company.  The agent was very pleasant and informed me my rental vehicle was now locked and that it had been locked and unlocked about a dozen times.  So I go out to the SUV to check to see if it really is locked.  It was not locked.  Just a bit frustrating!   I call back and relay the whole story over again to a new agent.  This agent informs me that the vehicle locks and unlocks by proximity to the keys I’m holding in my hand.  When the keys are about 9 feet away of the vehicle it will automatically lock.  When I get within 9 feet of the vehicle it will unlock.  The agent again informs me the vehicle has been locked and unlocked over a dozen times in the last hour.  I let the agent know there was no beeping sound to indicate it locked.  The agent agent lets me me know it’s a feature that it makes no noise when locking.  That “feature” sounded more like a defect and drove me nuts.

After the whole thing was straightened out something dawned on me.  The rental car agent counted the number of times the vehicle was locked and unlocked.  Yes they knew the current status of the vehicle which was pretty cool.  But I’m asking myself “Why would you count the number of times the SUV was locked and locked?”.  Seems like good information, but I’m not sure how it really helped me with my problem of the phantom locking and unlocking. 

So what would you do with that metric?  Could it be used to improve the rental car experience?  I couldn’t really make that one work in my head.  My expectations of the rental car experience are that that car I reserved is there (the upgrade was kind of a pleasant surprise), the price is reasonable and the vehicle works.  So how does the number of times it was locked and unlocked add value?  How does it meet my expectations? 

Don’t create metrics for the sake of metrics – they should be meaningful and actionable.  If the metric isn’t going to help you achieve better service or meeting expectations of your customers, why bother with it at all?  You’re just wasting time measuring something that is completely unusable.  Creating metrics that have no value pull project team members away from critical work into measuring things that don’t matter.   Pick the metrics and measurements carefully.  Be sure your metrics are created in a way to create business value and improve the process, product or project.

Are there standard metrics every Business Analysts should be measured?  Nope.  Every project is different in that is has a different outcome and business value.  Sure projects are typically measured by cost, hours, schedule and materials.  Those are important for ensuring a project is moving forward – but they lack measurement of whether or not the project added business value. 

I get asked “should I measure this for all Business Analysts?” at lot.  It takes what it takes.  The value of business analysis is building trust and relationships within the organization to create a common understanding to the problem or opportunity and the solution.  The larger the organization or the greater number of stakeholders involved in the problem / opportunity can slow the process down.  So a Business Analyst working on a large project could take months to build a common understanding and agreed upon solution.  While a Business Analyst working on a smaller project would only take a few days.  The point here is that business analysis is very subjective and people based at it’s core.  “You have until Tuesday next week to agree” isn’t always the best approach.  You need to understand why they don’t agree and if the definition of the problem or opportunity needs to change.

Can you imagine a measurement like “How many days did you spend on defining scope?”.  Is this useful in any way?  No way.  Some projects are small and low in complexity.  Other projects are huge and significantly complex.  Does speed in defining scope add business value?  Granted doing anything faster is a good thing – efficiency is good as long as the product quality isn’t sacrificed.  Now compare the two projects – one small and one big.  Can you do anything with the number of days it took to define scope? 

The other thing about measurements is that they drive behavior.  If you measure speed or number of days everyone is focused on making that number as low as possible.  Now everyone is focused on speed but not completeness or quality.  That leads to a disastrous outcome.

In a nutshell:

• Measure what adds business value – Can it be used make the project, product or service better?
• If you are using metrics for comparison, does the metric make sense when compared to different projects, products and services?
• Keep it simple – get metrics fast without a lot of work so time isn’t taken away from the project, project or service.
• Ensure the metric is creating the behaviors you want to see – don’t’ create metrics that drive speed when you really need quality and completeness
• Don’t measure everything – stick to 3 – 5 metrics

And that SUV I rented?  More on that in a future blog on process improvements.

Check out Bob’s blog for more good stuff on business analysis and sign up for our newsletter today.