Every survey, no matter how well designed, is subject to influences beyond the questions themselves. One unavoidable influence is human nature. People may make assumptions, act on preexisting beliefs, fail to pay attention, and engage in other behaviors that affect their responses to a survey.1 For example, if a survey shows an image of an athletic shoe and asks which company makes the shoe, some consumers might say “Nike” simply because they know that Nike makes athletic shoes, regardless of any logo, shape, color, design, or other characteristics of the shoe in the survey. If ignored, this type of “noise” in the data makes it less reliable because irrelevant responses are included.
Survey controls exist to address this problem. A control in a survey is like a placebo in medical research: it uses a false image, phrase, question, or answer option to detect survey noise. Respondents may provide answers that appear confused or misleading, but if the disputed feature is not actually present, their responses cannot reflect a real reaction to that feature. Those responses can be used to determine the amount of noise in the data.2
Why Do Surveys Use Controls?
Most surveys contain some amount of noise among the useful data, which describes influences on answers other than respondents’ true opinions. This noise includes answers that reflect respondents’ preexisting views, guessing, inattention, a tendency to say yes to every question, or some other form of noise.3 Controls help remove this noise, ensuring that the survey measures are attributable to the issue of interest, such as consumer confusion.
Controls are used in many types of litigation surveys: likelihood of confusion, false advertising, secondary meaning, materiality, fame, likelihood of dilution, and more. A control is needed when the survey shows significantly high measures for the issue of interest; it may not be needed for a survey intended to show significantly low measures.4
Controls are so important in litigation surveys that courts may not admit surveys lacking controls, or may admit them, but give them little weight as evidence.5 One author goes so far as to say that a survey without a control is “fatally flawed,” because such a survey “cannot satisfy the reliability mandate.”6 A control may therefore be vital to the success of a litigation survey. But that control must be well designed to avoid misleading the court with data that seems more reliable than it really is.7
There are two categories of controls: external and internal controls.8 They are both valid types of controls, but there are important differences in how they work, and therefore, how they can be used. This article explains each in turn.
What Is an External Control?
External controls are the more commonly used type of survey control,9 and also the type that more closely resembles a placebo in medical research. Researchers split the respondents for the survey into two groups. One group, called the test group, sees the real stimulus—the ad, image, trademark, or other item that the survey is intended to gather information about. The other group, called the control group, sees a modified version of the ad, image, trademark, etc., with the features of interest modified or removed. In a false advertising survey involving disputed claims, for example, the control version of the ad modifies or removes those disputed statements.
Respondents assigned to see the control ad may still provide answers suggesting that the ad deceived them, but those responses cannot have been caused by the disputed statements. If the control was designed correctly, these responses must be due to sources of survey noise like guessing, biases, or inattention.
What Is an Internal Control?
Internal controls work on the same principle as external controls, but do not require a separate (“external”) control group. In an internal control, all of the respondents see the same stimulus and the same questions, but one or more of those questions contain false information.10 If an external control is like a placebo, an internal control might be like a plainclothes police officer included as a “ringer” in a police lineup of suspects. The plainclothes officer is not suspected of committing the crime but is included to detect eyewitness identifications caused by something other than the officer’s guilt.
Internal controls can be questions asking about fictional trademarks or products, but they can also be images or answer options.11 For example, a survey question might ask whether various real product attributes were among the respondent’s reasons for purchasing a product. A fictional product attribute, such as “the product comes in a cardboard box,” can be inserted into the real reasons for purchase and used as an internal control. Assuming packaging is not a reason for purchasing the product, any respondent who selects it cannot provide a real reason for purchase.
As with external controls, responses to internal controls cannot be genuine, so they serve as a measurement of inattention, preexisting biases, a tendency among some respondents to say yes to everything, or other forms of survey noise.12
How Does a Control Help Eliminate Survey Noise?
Both external and internal controls detect the types of survey noise described above. In both cases, the control measures function as measures of the noise in the survey.13
This is important because the measures for the test group in a survey with an external control and the measures for the responses of interest in a survey with an internal control also have noise. Because all other elements of the survey are the same, the noise rate is likely to be similar for the test and control measures. This permits researchers to remove the estimated noise from the measures of interest using simple arithmetic: [test measure] – [control measure] = net measure.
For example, if 30.8% of respondents in the test group for a false advertising survey provide responses that appear to show they were misled by the ad, but 1.6% of respondents in the control group provide this type of response, the math would be: 30.8% – 1.6% = 29.2%.
For an internal control, if 4% of respondents to the question about reasons for purchase choose “the product comes in a cardboard box,” the fictional answer option above, and 25% choose one of the real reasons, the math would be: 25% – 4% = 21%.
The 29.2% and the 21% are “net measures,” meaning that the effects of survey noise have been removed.
Researchers can calculate valid net measures only by using corresponding test and control measures from the survey. In a survey with an external control, the test measure and the control measure must come from the two different groups’ responses to the same question. In a survey with an internal control, the measure for the control response must be subtracted from the measures for the real responses to the same question.
Which Type of Control Is Best for Your Survey?
In many cases, the answer to this is determined by the legal issue the survey is exploring. External controls are routine in surveys measuring likelihood of confusion,14 secondary meaning,15 false advertising,16 and likelihood of dilution.17 Internal controls are more common in fame surveys and may also be used in false advertising surveys.18
Some surveys can use both types of control. This may be advisable in certain situations. Your survey expert will tell you what types of controls are best in your situation.
How Do You Create a Valid Control?
Creating valid controls can be important to the success of a survey.19 A bad control can actually make the survey less reliable by skewing its data in a way that is not easily detected. For example, when creating an external control for a likelihood of confusion survey, researchers must modify or remove all challenged elements of the trademark, product, or ad. Leaving some of the disputed elements in the control creates the possibility that survey respondents will react the same way to the test item and the control item,20 which will artificially shrink the net survey results. Similarly, controls can be problematic if they are themselves infringing or deceptive, if they are leading, or if they are unrealistic.21
An external control item should be as similar as possible to the test item, except for the aspects of the test item that are at issue in the case. If every other aspect of the test and control items is consistent, only the change in the at-issue aspects can be responsible for differences between the test measure and the control measure.22 Typically, this means modifying an existing image, trademark, package, or other item to remove the at-issue aspects. Courts have sometimes accepted a real ad, package, or trademark that does not contain the objectionable content.23 However, courts have sometimes criticized controls based on real products because of other problems they can introduce.
The aspect of the test image that should not appear in the control will vary depending on the legal topic under investigation. Depending on the survey type, your control may need to change the allegedly infringing trademark,24 the allegedly deceptive advertising claim,25 or the element that one party claims has secondary meaning.26 Your survey expert can explain your options.
When designing internal controls, researchers should use something similar in form to the genuine questions, response options, or stimuli; no more difficult (if applicable); and no less plausible.27 For example, in a materiality survey testing whether a certain advertising claim makes consumers more likely to purchase an item, a control statement should be realistic, but not make rational consumers more likely to purchase.28 Thus, a statement like “the product comes in a blue package” is likely to be an acceptable internal control, assuming the package color would not be a motivating factor in a purchase decision.
Creating controls is a complicated topic about which much has been written.29 As one expert observed, “the devil is, of course, in the details.”30 Because of a bad control’s potential to destroy a survey’s credibility, controls should be designed with care.
Reliable Expert Evidence in Trademark and False Advertising Cases
The Litigation Surveys and Consumer Science team at IMS Legal Strategies has extensive experience designing, conducting, analyzing, and testifying regarding litigation surveys, including secondary meaning surveys. Contact us to discuss how we can help with your matter.
