[Blog header] Are your compliance training metrics 'vanity metrics'? Here's how to tell.

5 minute read

Are your compliance training metrics 'vanity metrics'? Here's how to tell.

November 9, 2021 3:00:00 AM CST / by Ricardo Pellafone

Vanity metrics are metrics that sound impressive but don’t really impact your core goals. And they will RUIN your program.

Basically, vanity metrics confuse popularity with results. They're seductive because they make it sound like cool stuff is happening. BUT they're destructive because that cool stuff doesn't actually matter—and they lead you to doing more "cool stuff" and less "stuff that actually works."

So today I'm gonna unpack what vanity metrics are, why your compliance program will get dumber if you end up using them, and how The Princess Bride provides the perfect way to determine whether your current metrics are the real deal.

Let's dive in.

 

"Did it work," not "did we like it."

One of our core goals at Broadcat is "making money." It's not our sole goal, of course, but it's kind of an existential one.

Vanity metrics for us would be “website hits” and “LinkedIn shares,” because those things make us feel good but aren't money. Our real metrics are sales, not how many people read our blog or share our content on social media. Those metrics can be useful as a lead-in to sales—if you hate us or don't know about us, you're not going to buy from us—but they aren't a substitute for sales.

Likewise, in the compliance world your core goal is “reducing risk,” because if you fail to do that your compliance program ceases to exist. (You might still have a job, but you are not actually doing compliance.)

Vanity metrics for a compliance program are things like "training satisfaction" and "employee survey feedback" because they make you feel good but don't reduce risk. Again, they can act as a lead-in to reduced risk—you're probably not going to get much done in terms of risk reduction if people just HATE your training, for example—but they are not a substitute for reduced risk.

Your real metrics are in monitoring your risky business processes, not seeing whether employees like the things you are doing.

Now, of course, there’s nothing wrong with being popular. We like when people share our work, and you should want your employees to like what you do, and it's fine to track those things if you want to. 

But here’s the rub: those things don’t necessarily lead to what you really want. It’s very possible to have employees be totally psyched about your compliance program and also have it be totally useless—and vanity metrics are one way to fall into that trap.

 

Why vanity metrics are destructive.

If you use vanity metrics, your program will start to get fluffier and dumber over time—even if you start off doing really gritty, operational stuff.

That’s because you tend to optimize what you measure. I know that sounds like consultant-talk, so here’s what it looks like in practice:

CCO
So, after releasing our new Code of Conduct training, 85% of our employees reported that they would recommend the training to a friend at another company.

BOARD MEMBER
That sounds good!

BOARD MEMBER (THINKING)
Wait, I need to add value here.

BOARD MEMBER
On second thought, that's a good number—but you could do better. How can you get that to 90% in the next two quarters? I think that’s my challenge to you.

CCO
Uh … well, that’s not really the point.

BOARD MEMBER
Well, that's what the chart you gave me shows.

CCO
Right, I know, but our goal is reducing risk. I mean, if we wanted to chase those stats, I guess we could just give everyone candy and show them The Princess Bride instead of training. Everyone likes candy and The Princess Bride.

BOARD MEMBER
Well, I leave that up to you.

CCO
No, I was kidding, that would increase our risk.

BOARD MEMBER (FLIPPING THROUGH PAPERS)
Where are the numbers on that?

CCO
Um, that's not in the deck.

BOARD MEMBER
Well, I read an article on the internet about how compliance needs to be data-driven, so let’s just focus on what we can measure. I’m just using the data you gave me. 

CCO
I’ll have to get back to you.

That’s a slight exaggeration ... but not much. Because if you give someone numbers, they will want you to make those numbers better.

And honestly, that’s fair: if you’re presenting metrics, you must think they measure what really matters, so it makes sense that your leaders would ask you to get even better at them.

But if you're presenting vanity metrics, things get bad fast.

That's because vanity metrics don’t correspond to your actual goal. So if you use them, you will get tasked with doing things to make them better—meaning you'll chase employee perceptions and satisfaction instead of operational excellence.

And that will destroy your program.

(And the first part to go will be your compliance training, because training is the part of your program that is most susceptible to this stuff. That's why there's so much silly nonsense in compliance training—because it's being created in response to vanity metrics that reward silly nonsense.)

 

How to tell if you have vanity metrics: The Princess Bride test.

The good news is that while defining great metrics is hard—you have to really understand your business process—identifying vanity metrics is pretty easy.

And here’s how to do it: it’s a simple, two-step process.

First, take what you did and your metric and put it into a sentence that has this structure: “After doing [thing you did], we found [thing we measured].” Like these examples:

After making a new compliance workflow for engineering, we found that 85% of our engineers would recommend it to a co-worker.

After making our training 15 minutes shorter, we found that 95% of our employees thought the training was less burdensome than normal compliance training.

After rolling out our new process on invoice review, we found a 40% reduction in invoices that were approved with red flags in them.

Second, take that sentence you just made and change the beginning so it reads like this: “After we showed our employees the movie The Princess Bride, we found [thing we measured].”

If the sentence still makes sense, it’s a vanity metric.

Here, let’s use the same examples:

After we showed our employees the movie The Princess Bride, we found that 85% of our engineers would recommend it to a co-worker.

After we showed our employees the movie The Princess Bride, we found that 95% of our employees thought it was less burdensome than normal compliance training.

After we showed our employees the movie The Princess Bride, we found a 40% reduction in invoices that were approved with red flags in them.

The first two are vanity metrics. The third is a real one—it describes a specific business outcome that reduced risk, and that's why it doesn't make sense when we put The Princess Bride in there.

The reason the first two are vanity metrics are because the goal of the thing you did (reducing business risk) and what you measured (how people felt about what you did) are so logically disconnected that you can swap out showing people a movie and it still makes sense.

And while that might be OK if you were measuring something where the baseline goal was "people should like it," that's not your goal.

Your job is to reduce risk, not be popular. If you can do both, that's terrific—but don't get them confused.

 

Don't despair.

In closing, let me say this: passing The Princess Bride test does not mean you are using good metrics. For example, a pass rate on a compliance quiz would pass The Princess Bride test, and we are notably skeptical of the value of that kind of thing, because abstract knowledge does not necessarily lead to behavior.

But first things first.

Passing The Princess Bride test *does* mean you are at least avoiding the absolute worst metrics (the kind that will lead you to drift away from operationalization 🙀 ), and that’s a great place to start.

 

Broadcat mascot peeking out of Clubhouse door with sign saying "Join the Club!"

We've got the tools to help you improve your *real* metrics. Book a free consultation to learn how we can help!

Book a free consultation!

Ricardo Pellafone

Written by

Ricardo Pellafone

As Broadcat’s Founder and CEO, Ricardo is responsible for setting and obsessively refining Broadcat’s core methodology, approach, and vision of practical, useful compliance.