Calculate the Margin of Error and Its Significance

To understand the results of any survey, you must calculate the margin of error and the statistical significance of any difference found.  See below for a basic explanation of the relevant statistic procedures. To calculate the margin of error and its statistical significance, or to view the actual formulas, click on the appropriate link in the box to the right.

To calculate the margin of error …

To understand how accurate your survey results are, one must calculate how much error is likely given the size of the sample you are surveying in relationship to the total population. The margin of error is the amount of error one could expect to find, due to just chance, above or below the actual figure obtained in the survey results. To calculate an acceptable margin of error, you must first select the confidence level you want for your results, and you must enter the sample size you are surveying and the total size of the population from which the sample is drawn.

An important guideline is the confidence interval or confidence level. That tells you just how confident you can be that the error rate you find reflects simply errors due to chance or sampling variation and not actual differences in the total population of interest. If you want to be very sure of your findings – say, if life-or-death decisions are to made as a result – then you will want a 99 percent confidence level. That means that you can be 99 percent confident that there is only a so-many percent likelihood (your margin of error) that the differences do not reflect actual differences but are due to chance. If you are looking for directional advice, you may want to go with a 90 percent confidence interval, as that will give you many more statistically significant results. Most commonly, a 95 percent confidence level is used.

The sample size is the number of people you are surveying. A sample is a portion of a total population. Sometimes you know the actual number of the total population, say if you are surveying a percentage of your customers or association members. When you don’t know the total but you know it’s more than the number you are surveying, then you will want to select an infinite population.

Most of the time, you will probably want to calculate a margin of error for the percentage or proportion of the sample choosing a particular answer in the survey. Sometimes, however, when you’ve asked your respondents for a number – such the number of hours they are on the Internet – then you will want to look at the average or mean number, and calculating the margin of error for a mean is somewhat different. To do this, you still must know the confidence level you desire, the sample size and the population size, as you do for calculating margin of error with proportional or percentage data. But you also must know the standard deviation for your data set. The standard deviation is the square root of the variance, which is based on the sum of squared differences between each score and the mean. Most calculators and spreadsheet programs can calculate the standard deviation for your data set.

Once you know your margin of error, you can say this about your data: If one were to pull 100 samples from the population and ask each group the same questions, you can be certain that 95 percent of the time (or whatever your confidence level) you will get answers that are within five percent (or whatever your margin of error) of the answers you got this time.

To test whether the difference between two results is significant …

Two proportions or percentages, or two means, may be far apart in actual numbers but not so far apart as to be statistically significant if they come from samples of far different sizes. To be sure the difference is due to your independent variable – the issue or action you are testing to see if it makes a difference in your dependent variable – you must test the statistic to make sure the difference could not be the result of chance or sampling variation.

A simple way to determine whether two proportions or percentages are truly different is to conduct a z-test. To calculate the z-score, one needs to know the desired confidence level, the two actual percentages, the size of the two samples used and the size of the total population. One also should be aware of whether one sample is included in the other sample (called a subsample), which occurs, for instance, when a sub-group is being compared with the total. For instance, when one wants to compare the percentage of one group answering a particular question yes with the percentage of the total sample answering yes, then the z-score must be calculated differently than when one compares the percentages of two different groups.

The best way to determine whether two means, or averages, are truly different is to conduct a t-test. To calculate the t-statistic for means, one must know the desired confidence level, the two means, the two sample sizes, the total population size, and the standard deviations for each mean.

When the z-score for two proportions or the t-score for two means are high enough, then one can say with some (90%, 95% or 99%) confidence that the difference between the two is due to the action of the independent variable on the dependent variable and not simply due to chance or sampling variation.