Want to evaluate whether changes in your employee survey results are significant?
PeopleNav has developed a FREE simple to use Excel based calculator to help anyone working in employee surveys check the significance of changes in their survey results. You can download the calculator at the bottom of this article.
Cut through the optics: Why you should look at statistical significance in your EX survey
I have come across many organisations who place a lot of emphasis on optical changes in their employee survey results. The problem with this is that the optical improvement (or decline) they see could just be down to chance skew in data rather than any real change.
In one particularly bad example, I heard how a leader who had a portion of his bonus tied to an employee survey score ended up penalised because his survey results went down a couple of %pts. Had those results been properly evaluated for significance then it would have shown the decline is not significant and really call into question the fairness of negatively impacting his bonus.
To illustrate why you should look at statistical significance, take a look at the chart below showing employee survey results across two surveys:
A small handful of HR professionals are savvy enough to understand this chart. They know that in the relatively small teams of Finance and Marketing each person has the potential to change the results by several percentage points. This means the changes we see in the scores need to be interpreted with due caution. Meanwhile at the overall company level a change of 2% points might look small but among 5,000 people it is statistically significant. That suggests something in the overall company employee experience has likely improved since the last survey - and it's unlikely due to chance.
Without a proper understanding around the significance of the trends in this chart, the same company could erroneously feel disappointed with its overall improvement (despite it being very significant) and give too much attention to Finance and Marketing (despite the trends not being large enough to reach significance).
Building HR's numerical credibility in the boardroom
Knowing the significance of your survey scores gives you credibility in the boardroom. Compare these two fictional conversations that you could have during a presentation to your Executive Leadership Team on your latest pulse survey results:
Conversation one:
You: "Our truck drivers are 5%pts more satisfied with Safety practices here since our last survey"
CEO: "Is that significant?"
You: "Erm. I'm not sure. It might be."
*Awkward pause*
Alternative conversation:
You: "Our truck drivers' 5%pts increase in satisfaction with our Safety practices is statistically significant - suggesting we've genuinely achieved real progress in this important area."
CEO: "That's fantastic. What actions were taken since the last survey?"
You: "We revamped our new Safety and Wellbeing at Work campaign for Truck drivers and so these results suggest it's working"
CEO: "Shouldn't we hurry up and roll that out to our Warehouse employees too?"
Ok so I admit that these two conversations are somewhat contrived, but hopefully you can imagine how much more robust the presentation given in the second conversation would be when you have statistically evaluated a trend for its significance.
Some employee survey technology providers are letting their customers down
I was spurred to provide this tool after hearing from several clients that a big player in employee survey technology responds to customer questions about statistical significance with "We don't talk about statistical differences here. We talk about 'felt' difference - so, does it feel different to you?". It's disappointing to hear this is happening in the market from such a notable EX technology firm. This tech provider company would be better telling their customers "unfortunately we don't know" instead of giving a misleading pseudo science response designed to deflect the question.
If you've found yourself in this situation with your employee survey provider then feel free to use the tool below to get the answer you're looking for.
Download the FREE employee survey statistical significance calculator here:
If you have any questions about this tool or other statistical tests and analyses you can conduct on your employee survey, then drop an email to contact@peoplenav.co.uk
Comments