Most of us answer survey questionnaires now and then. If we take a survey seriously, we probably feel that the responses we provide capture how we really feel. Most of us are unaware of how sensitive our answers can be to the way a question is asked.

There are various reasons why it can be a good idea to ask the public about their beliefs, their preferences, their likely behaviour, and so on. Social scientists, including economists, do it regularly. Reasons include to help evaluate policies or projects, or just to understand people. Governments also do it, to help decide which policies to adopt, and how to implement them.

Most people who are not social scientists have no idea how tricky this task of asking people can be. Here is an illustration.

Suppose that the government wishes to foster water conservation by encouraging the adoption of a package of water saving technologies in the home. People who adopt these measures will have lower water bills because of lower water consumption, but the government wants to provide an additional push, and provides a one-off discount on the water bill for households that install them.

Proposal 1: Would you support a proposal that rich households (e.g. with family income over $300,000 per year) should receive a higher discount than poor households (e.g. with family income below $50,000 per year)? Assume that the level of water saving is the same with or without this proposal.

If you are like most people, you would not support that proposal. To most of us, it seems unreasonable to favour rich people. They are already better off, so why should they get an additional benefit?

Now suppose instead that the government takes a different approach to the scheme. In that first scheme, the default position was that the household did not adopt the technologies, and a discount was provided if they did. In the alternative scheme, the view is that everybody should install the technologies, so the default position is that every household does. Everybody receives a one-off reduction in their water bill (the same reduction for everybody, consistent with your preference under the first proposal) to help offset the installation costs. Those households that cannot demonstrate that they have installed the technologies will be hit with a surcharge on their bill.

Proposal 2: Would you support a proposal that poor households should pay just as large a surcharge as rich households? Assume that the level of water saving is the same with or without this proposal.

You probably reject that proposal too. It feels unreasonable, doesn’t it?

If you are one of the majority who answers ‘no’ to both proposals, the bad news is that this is completely illogical. Saying ‘no’ to the first proposal is exactly equivalent to saying ‘yes’ to the second, and answering ‘no’ to the second proposal is equivalent to answering ‘yes’ to the first proporal.

To illustrate: if you say ‘no’ to the first proposal, you are saying that the savings on the water bills of poor people from installing the technology should be just as large as for rich people.

If you say ‘no’ to the second proposal, you are saying that the savings on the water bills of poor people from installing the technology should be less than for rich people. (They would get charged less for not installing, so the cost avoided if they do install would be less.)

(If you are finding it hard to get your head around that, there is a numerical example at the end that should make it clear.)

Why did you answer so illogically? Most people can’t answer this question. They just feel confused.

If you were in the illogical majority and answered ‘no’ to both proposals, go back and look again at both proposals and decide which one do you want to change to ‘yes’.

The amazing thing is that you probably don’t want to change either answer. They both still feel absolutely right, even though you now know that one gives the opposite result to the other.

If we did a public survey on this, we would get completely different results depending on how we described the system – as based on discounts or surcharges.

So which result would be the true representation of how people feel about these issues? My proposed answer to this question is neither/both. The worrying thing for a researcher is that the two contradictory outcomes can both be an accurate reflection of how people feel, depending on the wording and context of the question. It’s as if the real world changes depending on how you ask the question.

This dependence of people’s answers on the wording and context of questions is called a ‘framing effect’. Skilled survey developers who wish to get reasonably accurate results are well aware of the risk of framing effects (although sometimes it is hard to decide which frame is preferred, as in the above example). Less skilled/experienced survey developers – perhaps people who are not social scientists and are just having a go at a survey – often make awful mistakes.

Where does that leave us? Amongst other things, I think it means that we need to be cautious and humble in interpreting the results of surveys and other forms of public consultation. Remember that the way that people express their preferences and beliefs can sometimes be highly sensitive to how the question is asked, and be prepared to take any particular survey result with at least a pinch of salt. I always like to see the exact wording of a survey before I put too much weight on its findings.

It also highlights the importance of not taking the task of developing a survey lightly. If you are responsible for a survey, make sure you use an expert or get expert help and become well-informed about the pitfalls.

Numerical example

Water bills before policy change: Poor people $300; Rich people $500

Scheme 1 (subsidy)

Water bills if we answer ‘no’ to proposal 1, and the subsidy to both groups is $100.

• If they don’t adopt: Poor $300; Rich $500
• If they do adopt: Poor $200; Rich $400

Scheme 2 (surcharge)

The discount given to everyone, on the assumption that everyone adopts, is $100. If we answer ‘yes’ to proposal 2, the surcharge to both groups for not adopting is $100.

• If they don’t adopt: Poor $300; Rich $500
• If they do adopt: Poor $200; Rich $400

Notice that all the bills are the same as before, whether they adopt or not. So answering ‘no’ to proposal 1 is identical to answering ‘yes’ to proposal 2. If we answer ‘no’ to proposal 2, the bills for people who don’t adopt might be, say, $250 and $500, so rich people get a larger benefit from adopting. That corresponds to answering ‘yes’ to proposal 1.

Further reading

Bütler, M. and Maréchal, M.A. (2007). Framing effects in political decision making: evidence from a natural voting experiment, University of St. Gallen Department of Economics working paper, number 2007-04. IDEAS page for this paper

Kahneman, D. (2011). Thinking, Fast and Slow, Farrar, Straus and Giroux, New York.

Kragt, M. & Bennett, J. W. (2012). Attribute framing in choice experiments: How do attribute level descriptions affect value estimates? Environmental and Resource Economics, 51, 43-59. IDEAS page for this paper

Pannell, P.B.W. and Pannell, D.J. (1999). Introduction to Social Surveying: Pitfalls, Potential Problems and Preferred Practices, SEA Working Paper 99/04.