Control Freak

All too often I see results of a survey displayed where someone says ’so 66% of people are in favour of this change so we should get on and do it’ and I find myself resisting the urge to jump up and say ‘that’s nonsense, don’t you know about statistical significance?’. This happens so often with information that is used for important management decisions that it scares me. Here’s an explanation of just what I’m on about.

The basics

The best way to find out what a group of people think is to ask everyone and get an answer from everyone. However when only some people answer there is a mathematical way of working out just how reliable any answer is, that is Statistical Significance. This method works out from the sample taken, how many people would say the same thing if you asked everyone and just how likely that answer is to be correct. It cannot give you an exact answer, but it can give you a range within which the actual answer lies and you have to determine whether that range is too big to be of any use.

It can also tell you how many replies you need to get in order to get a significant result. It cannot tell you how many to ask since not everyone will reply and that is down to psychology.

The theory

Statisticians over the centuries have noticed that the answers to surveys (amongst other things) fit into patterns of distribution and that with enough answers you can work out what that distribution pattern would be. The more answers you get the more certain you can be about the distribution pattern.

You can never be 100% certain what everyone thinks but with enough answers you can get close. So most statisticians work on the basis of trying to be 95% certain about an answer. However there are times when you might want to be 90% certain or even 99% certain. This degree of certainty about the answer is called the Confidence Level. For most management statistics a 95% Confidence Level is fine. If you are dealing with things like infection rates you might want to use 99%.

The significance of any answer depends on three things:

• Population. The total number of people that you could ask if you asked everyone.
• Sample. The number of people from whom you actually got a response.
• Percentage. The percentage who gave a particular answer to a particular question.

What you get back is a spread of percentages, which is the measure of statistical significance, known as the Confidence Interval. So for example if you did a survey and 75% answered yes then the calculation might tell you that if you asked everyone the actual number who said yes would be between 35% and 115%. However with a larger sample size it might tell you that if you asked everyone the number who said yes would be between 70% and 80%. As you can see the first result is meaningless but the second is very useful.

There are some interesting points to note about the way this spread changes:

• The larger the sample size the smaller the spread. But it is not linear so calculate it, don’t try to guess it.
• For a larger population then a smaller sample relative to the population is needed to get the same degree of spread.
• The closer the percentage of people who give one answer gets to 50% (above or below) then the wider the spread becomes.
• The spread is the same whether you take a positive or negative answer. So for example if 75% said yes and 25% said no then you would get the same answer which ever way you did the calculation.

Examples

There are actually two ways you can use this calculation. One is after you have the answer and the other is before you send a survey. Both of these assume a Confidence Level of 95%.

Example 1 – “ How significant is this answer?

Assume we have a population of 3000 people and we send a survey to which a sample of 300 reply and out of those replies 225 (percentage 75%) say yes to a particular question.

When we do the calculation we find out that if we were to ask everyone the same question then the percentage who would say yes would be between 71.35% and 79.65%. So we can be certain that the majority agree.

The actual calculation returns a Confidence Interval of 4.65% so the lower limit is given by 75% – 4.65% = 71.35% and the upper limit is given by 75% + 4.65% = 79.65%.

If however we only got replies from 30 and of those 22 (73%) said yes then when we do the calculation we find out that if we asked everyone the same question then the number who said yes would be between 58% and 89%. Still a majority but a much wider spread of possible results.

Example 2 – “ How many do I need to ask?

Assume we have a population of 3,000,000 then we can do the reverse calculation to find out what sample size we need to get answers from to get a spread of just 1% on any answer they give. In this case it is 9573.

As the spread changes depending on the percentage who answer, this calculation assumes the worst case, which is that 50% give one particular answer. If we did get 9573 replies and more or less than 50% gave the same answer then the spread would be shorter.

The calculation

The actual calculation is too complicated to explain (and I’ve forgotten it) but you can visit the web site

http://www.surveysystem.com/sscalc.htm

which will do the calculation for you both ways. You can also save this page and do the calculation offline if you want as the code that performs the calculation is all stored with the page.

jay Leadership