03 · What You Need to Know
Translating a Research Question Into a Defensible Comparison
Why Must the Comparison Be Defined Before Data Collection?
Research questions often use terms such as difference, effect, improvement, influence, or effectiveness without specifying the contrast those terms imply.
For example, asking whether an educational technology is effective raises an immediate question: effective compared with what?
The answer might involve conventional instruction, another technology, no additional intervention, or an alternative implementation strategy. Each reference condition defines a different research question.
Without a clear comparison, researchers may collect data that cannot support the conclusion they intend to draw.
Defining the contrast early helps align sampling, measurement, analytical procedures, and interpretation.
What Are the Essential Elements of a Research Comparison?
A useful comparison specification identifies the units, contrasting conditions, outcome, reference, and timing.
| Element |
Question to answer |
Illustration |
| Units |
Who or what is being compared? |
University students enrolled in writing courses. |
| Condition or characteristic |
What distinguishes the comparison conditions? |
Assignment to AI-supported feedback or conventional feedback. |
| Outcome |
What difference is being investigated? |
Writing performance measured using a specified assessment. |
| Reference |
Against what is the focal condition evaluated? |
The conventional feedback condition. |
| Timing |
When is the outcome assessed? |
At the end of the instructional period. |
| Target inference |
What conclusion should the comparison support? |
The average effect of assignment to one feedback condition rather than another. |
These elements do not automatically establish a valid causal design. They clarify the comparison that the design must support.
What Is the Difference Between a Comparison and a Research Question?
The research question identifies what the researcher wants to know. The comparison specifies the contrast needed to answer it.
For example, the question "Do students improve after receiving AI-supported feedback?" requires measuring change over time.
The question "Does AI-supported feedback produce greater improvement than conventional feedback?" requires comparing changes or outcomes under two feedback conditions.
The first can establish observed change, but without further design features it cannot establish that the intervention caused the change.
The second introduces a reference condition that may support a stronger causal interpretation when assignment, measurement, and relevant assumptions are appropriate.
Researchers should not assume that the presence of a comparison group alone establishes causation.
How Do Descriptive, Associational, and Causal Questions Require Different Comparisons?
The intended inference determines what the comparison must accomplish.
Descriptive comparison
Examines how observed characteristics or outcomes differ between defined units, groups, or periods without necessarily attributing the difference to a cause.
Associational comparison
Examines whether measured characteristics or exposures are related to outcomes, without automatically establishing that one causes the other.
Causal comparison
Seeks to estimate what would differ under alternative intervention or exposure conditions, requiring an appropriate counterfactual interpretation and supporting assumptions.
For example, comparing average writing scores across academic programs is descriptive unless the research question and design support a more specific inference.
Comparing scores between students who voluntarily use AI feedback and those who do not may establish an association, but usage decisions may be related to other determinants of performance.
A randomized assignment to feedback conditions can support a causal comparison of assignment effects when the trial is appropriately designed and analyzed.
What Is the Reference Condition?
The reference condition is the alternative against which the focal condition is evaluated.
In an intervention study, it might be usual instruction, another intervention, or a defined absence of the intervention.
In observational research, it may be a category of exposure or a specified value of a continuous characteristic.
The reference should be substantively meaningful. Comparing an AI-supported feedback system with no feedback addresses a different question from comparing it with high-quality teacher feedback.
Neither comparison is inherently superior. The appropriate reference depends on the decision the research is intended to inform.
Researchers should describe the reference condition precisely rather than relying on vague labels such as traditional, normal, or control.
Why Does the Exact Definition of the Comparison Matter?
Small changes in the definition of a comparison can change the estimand, meaning the quantity or relationship the researcher intends to estimate.
For example, the effect of being assigned access to AI-supported feedback differs conceptually from the effect of actually using the system.
Participants assigned access may not use it, while those who use it extensively may differ from others in motivation or prior skill.
The ICH E9(R1) addendum provides a formal framework for specifying estimands in clinical trials, including treatment conditions, populations, outcomes, and intercurrent events. Although developed for clinical research, the underlying principle of specifying the target effect before analysis is relevant to educational intervention studies.
Researchers should avoid changing the target comparison after examining results merely because a different contrast produces a more favorable finding.
Should You Compare Final Outcomes or Changes From Baseline?
The appropriate choice depends on the design, research question, and analytical assumptions.
Comparing final outcomes addresses differences at a specified follow-up point. Comparing changes addresses differences in observed movement from baseline.
These quantities are related but not identical.
In randomized studies with baseline measurements, appropriate baseline-adjusted analysis may improve precision under suitable conditions. In observational studies, baseline adjustment alone does not eliminate confounding.
Vickers and Altman (2001) discuss the distinction between analyzing change scores and using analysis of covariance in randomized trials with baseline and follow-up measurements.
The analysis should be chosen according to the intended comparison and statistical properties of the data, not simply because a pretest and posttest are available.
When Is a Within-Participant Comparison Appropriate?
A within-participant comparison examines different measurements or conditions involving the same individual.
For example, students may complete writing tasks under two feedback conditions, or their performance may be measured before and after an instructional intervention.
Such comparisons can reduce some variation attributable to stable individual characteristics.
However, repeated measurement may introduce practice effects, maturation, carryover, period effects, or other changes unrelated to the focal condition.
The appropriate choice between within-participant and between-participant comparisons depends on the intervention, outcome, and feasibility of observing comparable conditions.
When Is a Between-Participant Comparison Appropriate?
A between-participant comparison examines outcomes or characteristics across different individuals or groups.
For example, one group of students may receive AI-supported feedback while another receives conventional feedback.
This arrangement can be appropriate when participants cannot reasonably experience both conditions or when carryover would undermine repeated exposure.
However, differences between groups may reflect characteristics other than the focal condition.
Randomization can reduce systematic baseline differences in expectation, but chance imbalances and post-assignment problems may remain.
In nonrandomized research, researchers must evaluate selection processes, baseline characteristics, and alternative explanations.
How Do You Know Whether Two Groups Are Comparable?
Comparability concerns whether differences other than the factor of interest could account for the observed outcome contrast.
For example, students who voluntarily adopt AI-supported feedback may differ from nonusers in prior writing ability, motivation, digital literacy, or access to learning resources.
These characteristics may influence performance independently of feedback use.
Researchers should identify plausible sources of confounding using substantive knowledge and an appropriate causal framework.
Perfect similarity across all observed characteristics is not generally achievable or required. The relevant issue is whether the design and assumptions support the intended comparison.
The question of what makes research groups sufficiently comparable requires attention to the specific inference being made.
Can the Same Dataset Support Several Comparisons?
Yes. A dataset may support descriptive, associational, and other comparisons, provided each is appropriately defined and interpreted.
For example, an educational intervention dataset might permit comparisons of overall outcomes, subgroup patterns, and changes over time.
However, multiple comparisons can increase analytical complexity and the risk of selective interpretation.
Researchers should distinguish prespecified primary comparisons from exploratory analyses.
Subgroup findings may be especially uncertain when sample sizes are small or when many comparisons are examined.
The availability of additional comparisons does not mean that every one deserves equal emphasis.
What If the Research Question Has No Obvious Comparison Group?
Not every research question requires a conventional control group.
A descriptive study may investigate the prevalence of a characteristic within a defined population. A qualitative study may examine experiences without requiring a numerical comparison.
Even when comparison is relevant, the reference may involve time points, conditions, exposure levels, or theoretically meaningful categories rather than a separate control group.
Researchers should not manufacture a comparison simply because comparative designs appear more rigorous.
The question should determine whether a comparison is necessary and what form it should take.
How Can Researchers Avoid Choosing a Comparison After Seeing the Results?
Where feasible, the primary comparison should be specified before outcome analysis.
This includes defining the focal and reference conditions, population, outcome, timing, and intended analysis.
Exploratory comparisons may still be scientifically valuable, but they should be identified as exploratory.
Researchers should not present a post hoc contrast as though it had been the original research objective.
Transparent reporting helps readers distinguish planned inference from hypothesis generation.
How Do You Check Whether the Comparison Answers the Original Question?
A practical test is to write the intended conclusion before collecting data, without assuming what the result will be.
For example: "The study will estimate the difference in end-of-course writing performance between students assigned to AI-supported feedback and those assigned to conventional feedback."
Then ask whether the proposed sampling, assignment, measurement, and analysis can actually support that statement.
If the data only compare voluntary users and nonusers, the conclusion must be revised or the design strengthened.
The purpose is to align the comparison with the evidence rather than stretch the evidence to fit a preferred claim.
Watch Out
Do not choose comparison groups solely because they are available or produce large differences. Define the target contrast first, then determine whether the research design can estimate it credibly. A statistically clear difference may still answer the wrong research question.
