Chapter 9 Questionnaire Design
Learning Objectives
AFTER READING THIS CHAPTER, YOU SHOULD BE ABLE TO:
9.1Apply the concepts of reliability and validity to questionnaire design
9.2Distinguish among nominal, ordinal, and interval/ratio data
9.3Create different types of marketing research survey questions
9.4Write programming instructions for a questionnaire
9.5Explain the dos and don’ts of questionnaire design
In the last three years, how many times have you donated to charity?
Did you have trouble answering this question? If so, you are not alone. This is because the question has several flaws that make it confusing and difficult to answer.
First, the question asks you to think back three years, which is not easy for most people. The researcher should consider how far back in time most people are able to remember. In this case, the respondent may find it easier to think back one year instead of three.
Second, the question does not specify what “donating to charity” means. Is giving a loonie in exchange for a poppy on Remembrance Day donating to charity? Or how
148
about giving $5 to a homeless person, or donating one’s time through volunteer work, or donating food? Or, by “donating to charity,” does the researcher mean “donating money to a registered charity”?
Finally, if the researcher meant “donating money to a registered charity,” there is still something missing from the question: the minimum amount that was donated. This is because, when the data are collected, there may be two respondents who donated 10 times to charity in the last year. However, one of these respondents donated $10 each time, while the other donated $100 each time, which is a big difference. The characteristics of these two respondents (their respective professions, income levels, lifestyles, etc.) may be worthwhile to investigate further, but the researcher would not know to look more closely if the dollar amounts were not indicated.
Throughout this chapter, you will be shown examples of different types of survey questions. By the end, you should be able to recognize when a question could be improved and, more importantly, how to improve it in order to get meaningful data.
Jon Callegher, Ted Langschmidt. (2020). Marketing Research for Marketers (1st Canadian Edition) [Texidium version]. Retrieved from http://texidium.com
9.1 Reliability and Validity in Questionnaire Design
9.1 Apply the concepts of reliability and validity to questionnaire design
We begin this chapter with a discussion of two essential concepts in the design of questionnaires: reliability and validity. While it is easy enough to memorize the definitions of these terms, students and even seasoned researchers sometimes fail to apply them. Consequently, the answers to certain questions or even to an entire questionnaire may be incomplete or—worse—useless. This results in an unfortunate waste of time for the respondent, the research team, and the client.
Therefore, as a future marketer, it is important that you consider issues related to reliability and validity not only when approving questionnaires, but also when designing your own questions for research purposes.
While reliability and validity apply to writing questionnaires for interviews and focus groups, this chapter will focus on applying the concepts to the careful design of a survey. In order to make accurate statistical calculations, a questionnaire must be structured in way that collects information extremely consistently. This means that the researcher must take the time to “get it right” before administering a survey to potentially thousands of people.
Reliability
In marketing research, reliability means that a question has been worded in a way that leaves no room for an alternative interpretation and allows data to be collected in a consistent way. The “charity” question in our introduction is an example of a question that is not reliable because respondents will have different interpretations of what “charity” means. As a result, people’s answers (the data) will be inconsistent and therefore cannot be trusted (or relied upon) when making business decisions.
reliability
Ensuring that a question has been worded in a way that leaves no room for an alternative interpretation and allows data to be collected in a consistent way.
Similarly, imagine that a researcher asked you to take a bite out of a new candy bar, roll it around in your mouth, and swallow it. The researcher then asked you:
“So, did you like the taste and texture of the candy bar?”
double-barrelled question
A single question that asks about more than one issue, but only allows for one answer.
This is a Yes or No question, but you were asked about two different things (taste and texture). What if you liked the taste but disliked the texture, or vice versa? In research, this kind of question is referred to as double-barrelled. The data collected149from this question will be unusable since the researcher has no way of knowing whether respondents answered Yes or No in regard to the candy bar’s taste, its texture, or both.
Reliability also applies to the administration of the survey. It is important that the survey is presented so that all respondents will complete it in a consistent manner.
This is challenging because an online survey may be completed by some respondents on a desktop computer, by others on a tablet, and by others on a smartphone. Each of these devices will display the survey differently and device-specific patterns may appear, such as how long it took respondents to complete the survey, the amount of text they entered, and whether or not they dropped out of the survey. If a survey is completed by respondents on various devices, then the client should be made aware of this in the final report.
Validity
While reliability means that questions are asked in a way that gets consistent responses, validity means that what is supposed to be measured is actually being measured. This requires the researcher to recall the research objective and research questions (for a refresher, review “Stating Marketing Research Objectives and Research Questions” in Chapter 2—The Marketing Research Process).
validity
Ensuring that what is supposed to be measured is actually being measured. This involves remembering the research objective so that the right questions are being asked and the right answer choices are being provided.
For example, imagine that a used-car dealership wanted to know what customers consider most important when shopping for a used car. The dealership asked the following question:
What is your most important consideration when shopping for a used vehicle?
Extended warranty
Colour
Availability of financing
Safety features
Gas mileage
Other (specify)
Now, this question is simple and clearly worded. It leaves no room for an alternative interpretation. In other words, the question is reliable.
However, the question is not valid. Why? To find out, you may have to read it over several times, perhaps aloud to yourself or to another person. If you still don’t find anything wrong with the question, then try answering it yourself. It’s at this point that you may notice that some obvious answer choices are missing. That is, while the options Extended warranty, Colour, Availability of financing, Safety features, and Gas mileage are included, in addition to Other (specify) to capture any less obvious answers, some obvious answer choices are missing.
The missing answer choices are:
Price
Make
Model
Year
Condition
Service history
If the dealership asks the original question and then makes a business decision based on the results, the decision is based on incomplete results. The dealership did not measure what was supposed to be measured, which is a customer’s top consideration among all popular considerations. The results, therefore, are not valid.150
If you were shopping for a used car, what would be your top consideration? When designing this question for a survey, the researcher must create an exhaustive list of answer choices so that the respondent feels represented and business decisiosn are based on full information.
Taking another example, try to answer the following question:
When determining a post-secondary institution to attend, which of the following are important to you? Select all that apply.
It has a variety of programs.
The tuition fees are affordable.
It has an online component.
It has a student residence.
It has fitness facilities on campus.
Other (specify)
If you understood the concept of validity, then you should notice that this question is not valid. It is missing some answer choices that are of great importance to many students, including:
The location is convenient for me.
It has a good reputation.
It offers me a bursary or a scholarship.
It has accessible learning services.
Finally, validity can apply to the entire questionnaire. For example, imagine that a marketer for Orville Redenbacher popcorn was interested in how shoppers decided which snacks to purchase before “family movie night” at home. If the marketer designed a survey that only asked about shoppers’ choices of salty snacks like popcorn, chips, and nuts, would the results be valid? The answer is no because snacks also include sweets like candy, chocolate, and ice cream.
Ensuring Reliability and Validity in Questionnaire Design
During your career as a marketer, you may be required to approve a questionnaire that was designed by an MRP. As well, you may be required to design a questionnaire yourself for, say, a customer feedback survey or an internal staff survey.
While a questionnaire should gather the right information accurately, it should also be designed in a way that encourages respondents to complete it. In marketing research, the percentage of respondents in a sample that actually completed the survey is called the response rate. For example, if 1,000 people were emailed a survey link and 340 completed it entirely (that is, they did not start and then drop out), then the response rate was 34%. When survey questions are both reliable and valid, respondents are less likely to be confused or frustrated and more likely to complete a survey to its end, leading to a higher response rate.
response rate
The percentage of respondents in a sample that actually completed the survey.
Whether you are signing off on a questionnaire or designing it, below are some guidelines for ensuring that your survey is reliable and valid, makes the most of your respondents’ time, and increases the likelihood of a higher response rate.
Prepare an Introduction
A reliable survey begins with an introduction that is succinct, unambiguous, and provides the following information:
The purpose of the survey
Why the respondent’s input matters
How long it should take to complete the survey
The reward (incentive) for completing the survey
Assurance of confidentiality and, only if possible, anonymity
151
Below is an example of an introduction for an online survey:
You are invited to participate in this important study of people who work in the tech industry.
This is a province-wide study in partnership with the Government of Ontario and a variety of technology-sector associations.
Our objective is to explore important work-related issues and propose solutions to challenges, including how to attract and retain employees in the tech sector. We would greatly appreciate your help.
The survey should take you 20 minutes to complete. Your responses will be kept strictly confidential.
As a thank you for completing this survey to the end, you will be invited to enter a draw to win 1 of 5 smart watches.
Ready?
Please click CONTINUE to begin.
Remember the Research Objective
To ensure validity of your survey, you must decide what specific information is needed to make a business decision and then keep your questionnaire focused on gathering it. By sticking to the research objective, you are respecting the respondents’ time and increasing the likelihood that they will complete the survey in its entirety, which improves the survey’s reliability.
Below are some questions that can help you remain focused:
Who will be answering my questions?
What do I really want to find out from them?
What will I do with their answers to each question?
Create the Questions
Once you have established the specific information you are seeking, you can begin creating questions to help you obtain it in a structured way. This involves determining the question formats and the wording as discussed in section 9.2 (Common Types of Marketing Research Survey Questions).
The way in which questions are worded will affect how respondents answer them. Therefore, it is essential that questions be worded in a neutral way, free from any bias. It is also important that questions be kept simple and specific, which will improve their reliability.
Finally, questions must not contain spelling or grammar errors, which can instantly turn a respondent off from completing a survey.
A Note on Confidentiality versus Anonymity
A common mistake found in survey introductions is the phrase, “Your information will be kept anonymous and confidential.”
When responses are kept confidential, it means that they will be kept private among research team members and used only for purposes of the research.
confidential
Responses are kept private among research team members and used only for purposes of the research.
When responses are kept anonymous, however, it means that there is no possible way of knowing who completed the survey. Therefore, anonymity should not be guaranteed if:
Any member of the research team is able to trace an email address back to a respondent.
The survey asks for an email address at the end (say, for a prize draw).
IP addresses are being stored by the survey software provider.
Any unique identifiers (e.g., a unique survey ID) are being attached to respondents.
Any information that is being captured can link the respondent’s identity to the research.
anonymous
There is no possible way of knowing who completed the survey.
152
Order the Questions
After you have created your questions, it’s time to determine the right order for presenting them. In general, it is advised that:
The questionnaire begins with questions that are relevant to the main topic and are easy to answer.
To ensure a natural flow, questions with a similar topic should be grouped together (e.g., personal questions, shopping habits, customer service).
Broader questions should be asked before specific ones (e.g., Ask whether or not respondents purchase soft drinks before asking them what brand of soft drinks they purchase).
Questions that are most pertinent to the main topic and require more effort to answer are placed between the beginning and the middle.
Demographic and potentially sensitive questions are placed at the end.
Pre-Test, Revise, and Re-Test
When you believe that your questionnaire is ready, it is imperative that it be tested for flaws by asking co-workers or friends to complete it and provide their feedback. Below are some questions for your survey testers that will help identify potential issues with reliability and validity:
From the introduction, did you know what the survey was about?
Was the font style and font size readable?
Were any questions unclear?
Did each closed question offer enough answer choices?
Was the character count large enough when answering open-ended questions?
Was the survey engaging for you?
Were any parts of the survey uninteresting or tiresome to complete?
Was completing the survey a worthwhile use of your time?
Did the survey reflect well on the company doing the research?
Take the input of your survey testers seriously, apply it, and test the new version again with them and others. Repeat until you are satisfied that your questionnaire is “perfect.”
Thank Respondents for their Time
Finally, as discussed in Chapter 3—Conducting Marketing Research Ethically, it is essential that respondents are properly thanked for their participation at the conclusion of the survey. While a sincere thank you message is a minimum, additional ways of showing appreciation include holding a prize draw, giving low-value gift cards, or delivering on a promise to share some of the research results.
A common illustration of ensuring reliability and validity in questionnaire design is the image of multiple darts on a bulls-eye. The bulls-eye represents validity. (Am I measuring what I am supposed to be measuring? Is my question “on target”?) The multiple darts represent reliability. (Will my question be understood the same way by all respondents? Will their responses be “on target”?)
Below is an example of a thank you message:
You’re done! Thank you very much for your time and your input.
As a show of thanks, we would like to enter you in a draw to win 1 of 5 smart watches.
If you would like to enter the draw, please provide your email address below. Your email address will be kept confidential and never shared or used again.
If you do not wish to enter the draw, simply click “No thanks.”
(Enter email)
No thanks
9.2 Levels of Measurement Data
9.2 Distinguish among nominal, ordinal, and interval/ratio data
In marketing research, levels of measurement describe the nature of the data being researched. This is important because the nature of a data set determines which kinds of analyses and summary graphs are possible.
Each level of measurement assigns numbers differently. From least complex to most complex, the levels are nominal, ordinal, and interval/ratio.
Nominal Data
Nominal data are the least complex because there are no quantifiable differences among variables. It is sometimes referred to as categorical or qualitative data. While nominal data can be stored as words or text, it can also be assigned numeric codes or symbols.
nominal data
Data with no quantifiable differences among variables and that can only be summarized as frequencies or percentages.
Nominal data comes from answers to questions that are named categories. For example:
Are you . . .?
Male
Female
What brand of luxury car do you drive?
Mercedes
Lexus
BMW
Audi
Jaguar
Other (specify)
Since there are no measurable differences between the males and females or among luxury cars (they are simply labels), the order in which they are presented can be changed, too.
Since nominal data are non-numeric, an average score cannot be calculated from a nominal data set. Nominal data can only be summarized as frequencies, which are the number of times certain variables are selected, or as percentages. These are graphically displayed as pie charts, column or bar charts, or stacked column or bar charts (discussed in Chapter 13—Communicating the Results).
frequencies
The number of times certain variables were selected.
Ordinal Data
Ordinal data are data that are ordered. The variables are presented from lowest/least to highest/most or vice versa. This allows the researcher to determine that there are differences between variables. For example, first is ranked higher than second, and third is ranked higher than fourth.
ordinal data
Data that are ordered from lowest/least to highest/most or vice versa. However, the differences among variables (e.g., small, medium, large) may not be equal and cannot be compared mathematically. Ordinal data can be summarized as frequencies or percentages.
However, it is important to note that the differences between variables in an ordinal data set may not be equal and cannot be compared mathematically. For example, look at the variables in the following question:
What is your shirt size?
Extra Small
Small
Medium
Large
Extra Large
154
As you can see, there is no quantifiable difference between Small and Medium or between Large and Extra Large. They are not of equal distance apart numerically. We just know that Small is smaller than Medium and Large is smaller than Extra Large.
Here is another example:
Please indicate your level of satisfaction with the following aspects of our ride-sharing service:
Poor
Fair
Good
Very good
Excellent
App Usability
Friendliness of Driver
Value for Money
Here, there is no quantifiable difference between Very Good and Fair. We just know that Very good is higher than Fair, but we don’t know how much higher.
Like nominal data, ordinal data can be summarized as frequencies or percentages. Ordinal data are graphically displayed as a column or bar chart with their original logical order preserved.
Interval and Ratio Data
Interval data and ratio data are variables that are not only ordered, but are also of equal distance apart numerically. This allows the researcher to observe quantifiable differences among variables. Interval/ratio data are sometimes referred to as scale or quantitative data.
interval data
Data that are both ordered and of equal distance apart numerically, allowing the researcher to observe quantifiable differences among variables.
ratio data
Data that are both ordered and of equal distance apart numerically, allowing the researcher to observe quantifiable differences among variables. Unlike interval data, ratio data have a true zero point.
The main difference between interval and ratio data is that interval data do not have a true zero point and ratio data do. The most commonly discussed type of interval data can be found in temperature, since the degrees in Celsius or Fahrenheit are equally distanced apart and can also go below their respective “artificial” zero points.
Ratio data can be found in the quantitative data that are measured in the real world, such as height, weight, length, annual income, years of education—any variables that are ordered with equal, measurable distances among each other.
Take the following example:
How many children do you have?
None
1
2
3
4
5 or more
Because the variables follow an order and are equally distanced apart, we know that the difference between 2 kids and 3 kids is the same as the difference between 3 kids and 4 kids.
Hair colour is an example of nominal data. There is no quantifiable difference between brown, red, pink, or blue. There is also no established logical order for presenting them.
In marketing research, ratio data are also commonly derived from rating-scale questions on attitudes and opinions. For example:
Please indicate your level of satisfaction with each area of our dog-walking service, where 1 is lowest and 5 is highest.
My dog walker is
Level of Satisfaction
Punctual
1
2
3
4
5
Friendly
1
2
3
4
5
A good fit for my dog
1
2
3
4
5
155
Interval/ratio data are usually graphically displayed as a column or bar chart for categories of data or as a histogram, which uses rectangles to present differences among ranges of values.
histogram
A graphical representation of frequency distribution, in which the rectangles represent data intervals and are proportional to the corresponding frequencies.
9.3 Common Types of Marketing Research Survey Questions
9.3 Create different types of marketing research survey questions
In this section you will get an overview of types of questions that are commonly used in marketing research. While it is not required, most engaging surveys contain more than one type of survey question. When creating questions for your survey, consider all the ways in which your question can be worded and then use the question type that will provide you with the most meaningful data.
Open-Ended Question
An open-ended question provides a text box so respondents can answer in their own words. As mentioned throughout this text, it is essential that respondents are given a chance to speak. Open-ended questions allow the researcher to explore a respondent’s underlying emotions and drivers of behaviour.
open-ended question
Provides a text box so that respondents can answer in their own words.
An open-ended question can be a simple How, Why, or Please explain question, or it can be an enabling tool such as a word association or sentence completion. It can also be a projective tool like a brand obituary, cartoon test, or withdrawal (for more on projective tools, see Chapter 7—Qualitative Marketing Research and Analysis).
Open-ended questions are especially useful in getting detailed information about:
A respondent’s personal experiences
Why a respondent chose your product over that of your competitor
How you can improve your offering
How your customer service can be improved
It should be noted that open-ended questions require a lot of time and money to tabulate, especially if the sample size is large (there may be hundreds or thousands of answers per open-ended question). Therefore, the associated time and cost of analyzing open-ended questions should be anticipated before including them in a questionnaire.
Closed Question
A Closed question forces the respondent to select from a set of pre-determined answers. If the question and its answer choices are written clearly, the respondent will be able to answer it easily. This is important since—as we discussed earlier in this text— people are generally better at reacting than they are at articulating their thoughts and feelings.
closed question
Forces the respondent to select from a set of pre-determined answers.
The following question types are closed and allow the researcher to summarize and present responses efficiently: single choice, multi choice, ranking scale, rating scale, Likert scale, grid or matrix question, and constant-sum scale. Each will be discussed in more detail on the following pages.
Single Choice
A single-choice question forces the respondent to choose only one response. While the most common single-choice question is one that asks respondents to answer Yes or No, others ask them to indicate their gender, primary language, marital status, income range, education level, or other characteristics.
single-choice question
Forces the respondent to choose only one response.
156
Here is an example of a single-choice question:
In what type of home or dwelling do you live?
An apartment or condo
A semi-detached house
A detached home
A mobile home/trailer
Other (specify)
Other (Specify)
By providing an “Other (specify)” option at the end of a structured list of answers, respondents have the opportunity to type their own answer if they feel that the available choices do not represent them. Since “Other (specify)” is generally selected much less frequently in a carefully designed questionnaire, the costs related to timing and analysis of these answers may be manageable.
Multi Choice
A multi-choice question asks the respondent to select one or more answers. The question should include the instructions Select all that apply.
multi-choice question
Asks the respondent to select one or more answers and includes the instructions Select all that apply.
Multi-choice questions are created when the researcher understands that more than one answer is likely to apply. To illustrate, review the following question:
How do you normally get to campus?
I walk.
I ride a bike.
I drive.
I take public transit.
I use a ride-sharing service.
Other (specify)
After reflecting upon this question, it is reasonable to assume that some students may “normally” get to campus in more ways than one. This may depend on the weather, the time of day, one’s mood, or whether or not one is in a rush. It is therefore better to give the respondent the ability to choose multiple answers.
By adding one sentence, Select all that apply, the question becomes a multi-choice one:
How do you normally get to campus? Select all that apply.
I walk.
I ride a bike.
I drive.
I take public transit.
I use a ride-sharing service.
Other (specify)
Ranking Scale
A ranking scale (or rank-order scale) asks the respondent to rank a set of items by dragging them in order of highest to lowest or by assigning a numerical value to each option (e.g., 1, 2, 3, 4, and 5). A ranking scale is useful when the researcher wants to assess the favourability of different items.
ranking scale
A scale that asks the respondent to rank a set of items by dragging them in order of highest to lowest favourability or by assigning a numerical value to each option (e.g., 1, 2, 3, 4, and 5).
Given the amount of analysis that is involved, and in order for the differences among rankings to be meaningful, it is advised that respondents rank no more than five items. This can be challenging if there are more than five items to be tested. In such157cases, a ranking scale can be administered by showing a longer list of items and asking respondents to select their top three or top five.
An example of a ranking scale is as follows:
Please rate your favourite types of cuisine from the list below, with “1” being most favourite and “5” being least favourite.
Cuisine
Rank
Italian
French
Mexican
Chinese
Japanese
Rating Scale
A rating scale is a type of single-choice question that asks a respondent to rate an item from highly positive (e.g., excellent, outstanding, or very good) to highly negative (e.g., poor, awful, or bad). It is used when a researcher wants to measure a respondent’s opinions and feelings toward a topic.
rating scale
A type of single-choice question that asks a respondent to rate an item from highly positive to highly negative. It is used when measuring opinions and feelings toward a topic.
When asking respondents a rating-scale question, researchers usually use a balanced or “odd-number” scale so that the positives fall on one side of the middle item and the negatives fall on the other. Doing so offers the respondent a neutral zone in case they do not feel strongly in either direction.
An example of a three-point rating scale question is as follows:
To what extent is the following statement true for you: “My job is very stressful.”
Not true for me at all
Partly true for me
True for me
Other three-point rating-scale questions may have the following answer choices:
Disagree – Unsure – Agree
Poor – Fair – Good
Not at all – Moderately – Extremely
Would not consider – Might consider – Would consider
An even-numbered, four-point rating scale question is typically used when asking respondents to rate the quality of a product or service. For example:
Please rate the quality of our continental breakfast.
Poor
Fair
Good
Excellent
An example of a five-point rating-scale question is as follows:
Thinking about your personal expenses, how you do prioritize paying off your tuition debt?
Not a priority
Low priority
Medium priority
High priority
Top priority
There many more examples of five-point (as well as seven-point) rating-scale questions that are based on the Likert scale, which will be discussed in the next section.
Likert Scale
Pronounced “lick-ert” and named after its creator, psychologist Rensis Likert, the Likert scale is a five-point or seven-point rating scale that contains distinct labels of agreement and disagreement. It is useful for determining the intensity of one’s feelings toward a statement. The Likert scale is balanced, with the agree labels on one side of the centre and the disagree labels on the other.
Likert scale
A five-point or seven-point rating scale that contains distinct labels of agreement and disagreement. The scale is balanced, with the agree (or positive) labels on one side of the middle and the disagree (or negative) labels on the other.
158
An example of a question using a five-point Likert scale is as follows:
Please indicate your level of agreement or disagreement with the following statement:
Statement
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
The campus library has enough study space.
Another example of a five-point Likert scale question is as follows:
How favourable of a review would you give this book?
Strongly unfavourable
Generally unfavourable
Uncertain
Generally favourable
Strongly favourable
An example of a seven-point Likert scale question is as follows, with negatives on one side of the middle and positives on the other.
How satisfied are you with our service?
Extremely Dissatisfied
Dissatisfied
Slightly Dissatisfied
Neutral
Slightly Satisfied
Satisfied
Extremely Satisfied
Likert scales are also used when asking questions about purchasing intent:
Which of the following statements best describes the likelihood of you purchasing a season pass to Canada’s Wonderland in the next three months?
I will definitely not buy it.
I will probably not buy it.
I am uncertain if I will buy it.
I will probably buy it.
I definitely will buy it.
By reviewing the following scale extremes, you should be able create your own five-point or seven-point Likert scale questions based on your research goals.
Very undesirable to Very desirable
Extremely unlikely to Extremely likely
Not at all familiar to Extremely familiar
Very difficult to Very easy
Strongly oppose to Strongly support
Grid or Matrix Question
In marketing research, a question that rates multiple items using the same rating scale is called a grid or matrix question. It looks similar to a rating scale, but instead of one item there are several rows of items to be measured.
grid or matrix question
Similar to a rating scale, but instead of one statement there are several rows of statements to be measured.
How did you enjoy your meal? An efficient way of getting input about several aspects of a product, service, or experience is by providing a grid or matrix question that uses one scale for measuring several related items.
Below is an example:
Please rate the quality of the following parts of your meal:
Poor
Fair
Good
Excellent
Appetizer
Main Course
Dessert
Drinks
While a grid or matrix question saves the respondent from having to read the same scale question multiple times for each item, to avoid confusion it is important that the items are all related to the same topic. As well, measuring too many items at one time should be avoided. Some recommend that no more than seven or eight items be displayed at once.
159
Given the amount of time that a respondent will spend answering a grid or matrix question, it is essential that survey testers provide feedback on its content and structure before the survey is launched.
Constant-Sum Scale
In a constant-sum scale, respondents are asked to allocate points to individual items in a set, typically totalling to 100. It is useful when seeking to understand the level of importance of specific attributes. In other words, rather than ranking items as first, second, and third in terms of importance, the researcher can get a sense of how much more important the first-ranked item is than the second, and how much more important the second-ranked item is than the third.
constant-sum scale
A scale on which respondents are asked to allocate points to individual items in a set, typically totalling to 100. It is useful when seeking to understand the level of importance of specific attributes.
Here is an example of a question using a constant-sum scale:
The following table contains some common considerations when choosing a club to attend on a Saturday night. Please allocate a total of 100 points among these considerations. The more important a consideration is, the more points it should receive. If a consideration is not important, give it a 0.
Consideration
Number of Points
Type of music
Cover charge
Reputation
Cost of drinks
Wait time to enter
Distance from home
Total points
100
The following is an example of how the question might be answered by a respondent, with points totalling 100.
Consideration
Number of Points
Type of music
0
Cover charge
40
Reputation
10
Cost of drinks
30
Wait time to enter
15
Distance from home
5
Total points
100
For this respondent, you can see that a night club’s cover charge (40) is the most important consideration and the cost of drinks (30) is the second most important. As well, the cover charge (40) is four times more important than the night club’s reputation (10).
One obvious downside of using this scale is that respondents may incorrectly allocate points so that the total is over or under 100. While this is especially common when completing a paper survey, most online survey software will perform the calculations automatically.
9.4 Programming Instructions
9.4 Write programming instructions for a questionnaire
When designing a questionnaire, it is important to provide clear instructions for the respondent. An example mentioned earlier was the inclusion of the wording Select all that apply for a multi-choice question. This lets respondents know that they can choose more than one answer.
160
When designing a questionnaire for an online survey, it is also essential to provide clear instructions for another audience—the person who will be programming the survey. This individual could be a professional programmer who will use in-house survey software, or it could be you or a colleague who will program a survey in Qualtrics, SurveyMonkey, or Google Forms.
Writing Programming Instructions for a Questionnaire
This section provides an overview of common programming instructions when creating a questionnaire. These programming instructions are typically written in a different colour or font, or within square brackets, to denote that the respondent should not see them in the survey.
programming instructions
Instructions used in questionnaire design that are for survey programmers. These are typically written in a different colour or font, or within square brackets, to denote that the respondent should not see them when completing the survey.
Take the following example:
Q3. Which of the following fast-food restaurants have you seen advertised the most on social media in the two weeks? [SINGLE CHOICE] [RANDOMIZE]
3.1 McDonald’s
3.2 Burger King
3.3 Kentucky Fried Chicken
3.4 Subway
3.5 Wendy’s
3.6 Harvey’s
3.7 A&W
3.8 Tim Hortons
3.9 Other (specify) [LOCK]
3.10 None of these [LOCK]
As you can see, the programming instructions are in red font, capitalized, and within square brackets. Respondents will not see these instructions when completing the survey, but they will see the output. That is, instead of seeing the list of fast-food restaurants displayed in the above order, which might influence a respondent’s choice, each will see a randomized list. However, Other (specify) and None of these will remain “locked” where they are at the bottom.
As well, the respondent will only be able to select one item in the list since the programming instruction is “single choice.”
You also probably noticed that each answer choice is preceded by a number in red font. If needed, this allows the questionnaire writer to reference a specific item without writing the entire word or phrase. In the above example, which is question number 3 in a questionnaire, “Burger King” would simply be written as “3.2” and “Harvey’s” would be written as “3.6.” This is especially useful when creating “skip logic,” which we will discuss next.
Creating Skip Logic
Skip logic (also called skip patterns) is instructions for the programmer to either include respondents or exclude respondents from a particular question or set of questions. This is done efficiently by added numerical labels (seen in red below) to each answer choice. When correctly programmed, skip logic can be useful in getting more information from a particular group of respondents.
skip logic
Programming instructions to either include or exclude respondents from a particular question or set of questions. Also called skip patterns.
Look at the following example from an employee survey. Take your time and read the questions and the skip logic programming instructions carefully.161
Q1. In what department do you work? [SINGLE CHOICE]
1.1 Research
1.2 Operations
1.3 Finance
IF 1.2 OR 1.3, SKIP TO Q5
Q2. Which of the following best describes your position in the research department? [SINGLE CHOICE]
2.1 Analyst
2.2 Manager
2.3 Director
2.4 Vice President
IF 2.1 OR 2.4, SKIP TO Q5
In Q1 above, anyone who selects Operations (1.2) or Finance (1.3) will skip ahead to Q5 (not shown here). In other words, only those who work in Research (1.1) will be shown Q2.
In Q2 (answered only by those work in Research), anyone who selects Analyst (2.1) or Vice President (2.4) will skip ahead to Q5. That means only Research Managers and Research Directors will be shown Q3 and Q4. After completing those questions, they will continue with the survey at Q5 with everyone else.
While skip logic ensures that certain respondents will skip questions that do not apply to them, note that this programming feature is usually not free in most DIY survey software.
Piping
As a customer and an email user, you have likely received an email from a company that was addressed to you personally. In other words, if your name is Blair, you received an email that started off with “Hi, Blair.” Now, it is highly unlikely that someone at this company took the time to type in your name before clicking the Send button. Instead, your name was inserted or “piped” into an email template, just as other people’s names were “piped” into the same template. This is called—you guessed it—piping.
piping
A programming instruction that is used when the researcher wants to insert an answer from a previous question into a question that is asked later on.
In questionnaire design, piping means inserting an answer from a previous question into a question that is asked later on. It is especially useful for keeping a question focused on a specific topic.
Here is an example of the programming instructions for piping:
Q1. Which of the following search engines do you use most often? [SINGLE SELECT]
1.1 Ask
1.2 Bing
1.3 DuckDuckGo
1.4 Google
1.5 MSN
1.6 Yahoo
1.7 Other (specify)
Q2. Why do you use [insert Q1 response]? [OPEN-END]
Q3. Based on your recent experiences with [insert Q1 response], please explain how it can be improved. [OPEN-END]
Note that piping is another premium programming feature and is usually not free in most DIY survey software.
Screening Questions
When administering a survey, you may wish to exclude or “screen out” those who do not meet your sample criteria. They may not be the primary shopper in the household. They may be too young or too old. They may not work in a field that you’re interested in learning about. They may not be avid readers. In any case, the criteria for screening people out is yours to determine.
When designing screening questions, simply apply skip logic to your programming instructions as shown below. You can also label screening questions as S1, S2, … to distinguish them from questions in your main questionnaire (Q1, Q2, …).
screening questions
Questions that can appear at the beginning of a survey and are used to exclude or screen out people who do not meet the sample criteria.
Here is an example of a brief survey introduction and subsequent screening questions.
Our marketing research class is conducting a survey about your experiences with the Fitness Centre at Red Deer College. It should take about 5 minutes and your answers will be kept confidential. Let’s begin!
[SCREENER QUESTIONS]
S1) Are you currently enrolled as a student at Red Deer College? [SINGLE CHOICE]
1.1) Yes
1.2) No [TERMINATE STATEMENT]
S2) In the last 6 months, have you used the college’s Fitness Centre? [SINGLE CHOICE]
2.1) Yes
2.2) No [TERMINATE STATEMENT]
[TERMINATE STATEMENT] We appreciate your time, but unfortunately you do not meet our criteria for this survey. Thank you and have a great day.
[MAIN QUESTIONNAIRE]
Q1) Great! You’re perfect for this survey. Tell us, what year are you currently in? [SINGLE CHOICE]
1.1) 1st year
1.2) 2nd year
1.3) 3rd year
1.4) 4th year
1.5) 5th year or more
In the above example, you can see that the initial respondents who were not students at Red Deer College were screened out after the first question and shown the polite “terminate” statement. While the word “terminate” may sound harsh, it is still commonly used in programming instructions. Think of it as a technical way of saying “dismiss” or “let go.”
After the second question, Red Deer College students who did not use the fitness centre in the last six months were screened out and shown the “terminate” statement.
Thanks to these two screener questions, the researchers can ensure that survey respondents will comprise only Red Deer College students who have used the fitness centre in the last six months.
9.5 DOS and DON’TS of Questionnaire Design
9.5 Explain the dos and don’ts of questionnaire design
In your career as a marketer, you may be asked to design and launch a survey for your company. It could be a few quick poll questions or you may be asked to design a customer satisfaction survey and send it to members of your email database. As well, you may be asked to review a questionnaire draft from an MRP who is163working for you. In any of these scenarios, we encourage you to refer to the following dos and don’ts of questionnaire design.
DOs
Ensure that Your Questions can be Answered by Every Respondent
When a survey contains questions that do not apply to a respondent, the risk of a respondent dropping out increases. It is therefore important that all questions being asked can be answered by everyone in the sample. This requires careful consideration about the sample’s knowledge base with respect to the survey topic. If certain questions do not apply to some respondents, then use skip logic.
Ensure that Your Layout is Consistent so Respondents can Complete the Survey Efficiently
A consistent layout includes the following formatting considerations:
All questions are evenly spaced apart.
All scale questions are in one direction throughout the survey (i.e., positive to negative or negative to positive).
Instructions such as Select all that apply are included with every applicable question.
Answer choices like Don’t know, Not applicable, and Prefer not to say are locked at the end of a list or scale.
Open-ended questions provide enough space to type an answer.
Put Respondents at Ease
When designing a questionnaire, it is important to consider the general nature of your audience and to write in a “voice” that puts them at ease. The more respondents feel like they are providing help, the more likely they are to be forthright. Conversely, the more respondents feel like they are being interrogated, the more likely they are to embellish their answers just to please you or to “look good.”
Putting respondents at ease is done by explaining to them why the survey is being conducted and that their opinions matter. Communicate that you want them to be themselves and that their answers will remain confidential.
Be Concise
Write and then re-write questions to ensure that they are quickly getting to the point.
Be Conversational
While it is important to be concise, it also important to remember that you are having a conversation with your respondent. Therefore, remember to use transitional statements to indicate topic changes when applicable. As well, it is helpful to update the respondent on their progress (some survey software provide a progress bar).
Below are some examples of transitional statements:
Now, let’s move on to…
Next, let’s talk about…
Finally, we will end with some questions about you.
Below are some examples of progress statements:
Great. You’re more than halfway through…
You’re almost finished. Just a few closing questions…
Ensure that Answer Lists are Exhaustive
Answer choices must cover all possible answers. Ensuring this requires that you test your questionnaire with other people and pay attention to when they say things like What about ? or The answer164list doesn’t include . If your survey testers noticed an omission or confusing wording, there is a good chance that respondents will also notice it.
Include an “Other (Specify)”
When applicable, by providing a space for your respondents to type an answer that did not appear in your exhaustive list, you are giving them an opportunity to speak and showing that you respect their input.
Ensure that the Initial Questions are Topic-Focused
Since the email invitation gives a glimpse of what to expect, respondents are likely to be in a frame of mind to answer questions on that topic. Therefore, the most important questions should be asked earlier in the survey.
Place Sensitive Questions Toward the End of the Survey
Some respondents may be put off when questions of a personal nature are asked at the beginning of a survey. These include questions about one’s income level, marital status, ethnicity, or religion. In order to increase the chances of respondents answering sensitive questions, such questions should be positioned as “wrapping up” questions near the end.
Use Scales to Improve Accuracy
Scale questions are powerful tools for assessing complex topics. However, sometimes the wording of the question can lead to less meaningful results.
Consider the following example of patrons being asked about their enjoyment of their dining experience:
To what extent do you agree or disagree with the following statement?
Statement
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
I enjoyed the restaurant’s atmosphere.
This question could be asked in a different way so that the answer choices reflect the main topic—enjoyment:
How much do you enjoy the restaurant’s atmosphere?
Did not enjoy it at all
Enjoyed a little bit
Neutral
Enjoyed quite a bit
Enjoyed a lot
This second scale is a more focused way of asking the question. It will also provide more precise results related to five different degrees of enjoyment.
Label All Answer Options on a Numbered Scale
Finally, instead of asking “On a scale from 1 to 5,” assign a label to each number (e.g. 1=Poor, 2=Fair, 3=Good, 4=Very Good, 5=Excellent). In everyday life, people generally use words—not numbers—to describe how they’re feeling about a topic.
DON’Ts
Don’t Ask Double-Barrelled Questions
As discussed earlier, a double-barrelled question asks the respondent a single question about two different things.
Below is an example of a double-barrelled question:
Are you happy with the speed and friendliness of our service?
Instead, two separate questions should be asked:
Are you happy with the speed of our service?
Are you happy with the friendliness of our service?
Don’t Ask Leading Questions
A leading question steers respondents toward a certain answer, typically by citing an authority figure or the majority of a group. Here are some examples:
Most doctors agree that sugary drinks are bad for one’s health? Do you agree?”
Do you agree with the majority of students that classes have too many group assignments?
leading question
A question that subtly or overtly prompts a participant to answer in a desired way.
165
Instead, these questions might be presented using a Likert scale. Here is an example using a question about sugary drinks.
To what extent do you agree or disagree with the following statement?
Statement
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Sugary drinks are bad for one’s health.
Don’t Use Ambiguous Terms
Consider the impact on a question’s reliability when it includes terms that are ambiguous, that is, terms that might be interpreted differently by different people. Below are some suggestions.
Instead of saying:
Recently…
Say:
In the last two weeks…
Instead of asking:
Which brand of cereal do you eat?
Ask:
Which brand of cereal have you eaten most often in the last 3 months?
It is also important to ensure that your numerical ranges are unambiguous. For example, in the following question, the age ranges overlap:
What is your age range?
18–24
24–34
34–54
54+
Which option should respondents select if they are 24, 34, or 54?
Instead, the ranges should be presented without overlap, as follows:
What is your age range?
18–24
25–34
35–54
55+
Being unambiguous with numerical ranges applies to income, number of years working, years of education, and so on. In regard to income, it is also important to be specific about whether respondents should report their personal income, household income, gross income, net income, etc.
Avoid the Use of Negative Wording
The use of negative wording can be confusing. For example, the following question asks the respondent to answer Yes or No to a negative question.
Are you against the use of pesticides when growing fruit?
Yes
No
Instead, the following question is presented in a neutral way:
How do you feel about the use of pesticides when growing fruit?
I’m for it
I’m against it
I’m not sure
166
Avoid Wording that Might Offend
Consider your audience when asking a question and decide if it can be asked in a more respectful way. A common example is when you want to know a respondent’s age. For some people, the following question is considered impolite:
How old are you?
Instead of asking people how “old” someone is, the question can be phrased in a few different ways:
In what year were you born?
(Dropdown box of years of birth)
In what age range do you fall?
(List of age ranges)
Don’t Use Jargon or Acronyms
Finally, don’t assume that all respondents are familiar with colloquialisms or how specific acronyms are defined. If you do, then you risk that some respondents will try to guess the meaning. If they guess incorrectly, this will negatively affect the reliability of your data set.
Summary: Questionnaire Design
You’re In It Together
While this chapter provided with you with many tips for questionnaire design, when testing your survey it is critical that you put yourself in the shoes of a typical respondent. This is not always easy, especially if you are studying a group that you are not familiar with, but you must do your best to think about them. In other words, respondents need to feel that their time is being well spent, that they are helping you in your quest to get answers, and that you appreciate them for it.
In other words, while fulfilling your research objective is important, it is also essential that your respondents feel that your survey is useful for them, too.
Learning Objectives Review
LO 9.1 Apply the concepts of reliability and validity to questionnaire design
When designing a questionnaire, careful attention should be paid to ensuring reliability and validity.
Reliability means that a question has been worded in a way that leaves no room for an alternative interpretation and allows data to be collected in a consistent way. For example, asking people if they donated to charity in the last three months will result in responses that are based on each individual’s definition of “charity.” This data cannot be relied upon when making a business decision. Reliability also applies to the consistent administration of the survey itself.
Validity means that what is supposed to be measured is actually being measured. This involves ensuring that the right questions are being asked and the right answer choices are being provided. If obvious answers are missing from a list of choices, which means that respondents were not clicking on answers that they would likely have clicked on, then the resulting data set is incomplete. Marketers should avoid making decisions based on incomplete data.
Guidelines for ensuring that a survey is both reliable and valid include:
Ensure reliability by preparing an introduction that is succinct, unambiguous, and that informs respondents of the survey’s purpose, length, why their input matters, and the incentive for completion. It should also provide an assurance of confidentiality, which means that all answers will be kept private. Anonymity should only be guaranteed if there is no possible way of knowing who completed the survey.
Ensure validity by remembering the research objective when designing questions. Ask yourself Who will be answering my questions? What do I really want to find out from them? What will I do with their answers to each question?
Create questions that are simple and specific, which will improve their reliability. There should be no spelling or grammar errors.
Order your questions according to a natural flow, with similar topics grouped together, with main167topic questions near the beginning and middle, and with potentially sensitive questions near the end.
Pre-test your questionnaire by asking survey testers to consider the list of questions you have created. After applying their input, test the new version again with them and others.
Thank respondents for their time and provide an equal opportunity for them to claim the survey incentive.
LO 9.2 Distinguish among nominal, ordinal, and interval/ratio data
Levels of measurement describe the nature of the data being researched, which determines the kinds of analyses and summary graphs that are possible. In order of least complex to most complex, the levels are:
NOMINAL DATA – Sometimes referred to as categorical or qualitative data, there are no quantifiable differences among variables. Nominal data can only be summarized as frequencies or percentages and are graphically displayed as pie charts, column or bar charts, or stacked column or bar charts.
ORDINAL DATA – These data are ordered from lowest/least to highest/most or vice versa. While this allows the researcher to determine that there are differences among variables (e.g., first is ranked higher than second), these may not be equal and cannot be compared mathematically. Ordinal data can be summarized as frequencies or percentages and are graphically displayed as a column or bar chart with their original logical order preserved.
INTERVAL AND RATIO DATA – Sometimes referred to as scale or quantitative data, the data are both ordered and are of equal distance apart numerically. This allows the researcher to observe quantifiable differences among variables. The main difference between interval and ratio data is that interval data do not have a true zero point and ratio data do. Interval/ratio data are usually graphically displayed as a column or bar chart for categories of data or as a histogram.
LO 9.3 Create different types of marketing research survey questions
Common types of questions that are used in marketing research include:
OPEN-ENDED QUESTION – Provides a text box so that respondents can answer in their own words.
CLOSED QUESTION – Forces the respondent to select from a set of pre-determined answers. All of the question types that follow are closed.
SINGLE CHOICE – Forces the respondent to choose only one response.
MULTI CHOICE – Asks the respondent to select one or more answers and includes the instructions Select all that apply.
RATING SCALE – A type of single-choice question that asks a respondent to rate an item from highly positive to highly negative. It is used when measuring opinions and feelings toward a topic.
LIKERT SCALE – A five-point or seven-point rating scale that contains distinct labels of agreement and disagreement. The scale is balanced, with the agree (or positive) labels on one side of the middle and the disagree (or negative) labels on the other.
GRID/MATRIX QUESTION – Similar to a rating scale, but instead of one statement there are several rows of statements to be measured. Be careful not to display too many statements at once.
RANKING SCALE – A scale that asks the respondent to rank a set of items by dragging them in order of highest to lowest favourability or by assigning a numerical value to each option (e.g., 1, 2, 3, 4, and 5).
CONSTANT-SUM SCALE – Respondents are asked to allocate points to individual items in a set, typically totalling 100. It is useful when seeking to understand the level of importance of specific attributes.
LO 9.4 Write programming instructions for a questionnaire
When designing a questionnaire, it is important to provide clear instructions for the person who will be programming the survey. Programming instructions are typically written in a different colour or font, or within square brackets, to denote that the respondent should not see them when completing the survey. Some common programming instructions include:
[SINGLE CHOICE]— Indicates that a question type must be single choice
[MULTI CHOICE]— Indicates that a question type must be multi choice
[RANDOMIZE]— Indicates that a question’s answer list must be randomized
[LOCK]— Indicates that an answer choice should be remain “locked” where it is when other choices are randomized
[OPEN END]— Indicates that a text box should be provided for the respondent to type their own answer
Skip logic (or skip patterns) is instructions for the programmer to either include respondents or exclude respondents from a particular question or set of questions. This is168done by ensuring that each answer choice is given a specific label (e.g., 1.2) and that the programming instructions refer to those labels (e.g., IF 1.2, SKIP TO Q5).
Piping is used when the researcher wants to insert an answer from a previous question into a question that is asked later on. For example: Q2. Why do you use [insert Q1 response]? [OPEN-END]
Screening questions are used to exclude or screen out people who do not meet the sample criteria from completing your survey. When designing screening questions, simply apply skip logic to your programming instructions, including a “terminate statement” for those who are screened out. Below is an example:
S1) Are you currently enrolled as a student at Red Deer College? [SINGLE CHOICE]
1.1) Yes
1.2) No [TERMINATE STATEMENT]
[TERMINATE STATEMENT] We appreciate your time, but unfortunately you do not meet our criteria for this survey. Thank you and have a great day.
LO 9.5 Explain the dos and don’ts of questionnaire design
In your career as a marketer, keep in mind the following dos and don’ts of questionnaire design:
DOs
Ensure that your questions can be answered by every respondent
Ensure that your layout is consistent so respondents can complete the survey efficiently
Put respondents at ease by explaining to them why the survey is being conducted and that their opinions matter
Be concise by re-writing questions to ensure that they are getting to the point
Be conversational by using transitional statements and by updating respondents on their progress
Ensure that answer lists are exhaustive and include Other (specify)
Ensure that the initial questions are topic-focused
Place sensitive questions toward the end of the survey
Use scales to improve accuracy
Label all answer options on a numbered scale
DON’Ts
Don’t ask double-barrelled questions, which ask a respondent a single question about two different things
Don’t ask leading questions, which cite the opinion of an authority figure or the majority of a group
Don’t use ambiguous terms like “Recently…” (Instead use “In the last 2 weeks…”)
Avoid the use of negative wording
Avoid wording that might offend
Don’t use jargon or acronyms that even a few respondents may not understand
Short Answer
Think of a business-related topic that interests you. Imagine that your sampling frame is an email list of all adults age 18+ in your province. Design an 8- to 10-question questionnaire that contains clear programming instructions and includes:
A proper introduction
A few screening questions to focus your sample
Topic questions that follow the tips presented in this chapter
A proper thank-you statement
Key Terms
anonymous
closed question
confidential
constant-sum scale
double-barrelled question
frequencies
grid or matrix question
histogram
interval data
leading question
Likert scale
multi-choice question
nominal data
open-ended question
ordinal data
piping
programming instructions
ranking scale
rating scale
ratio data
reliability
response rate
screening questions
single-choice question
skip logic
validity