03 · What You Need to Know
Building a Measurement Plan From Your Research Objectives
Begin With the Quantity or Phenomenon You Want to Understand
Variable selection begins with the research question, not with the questionnaire or dataset.
Consider a study asking whether access to an AI-supported writing tool improves undergraduate students' academic writing performance.
The question immediately identifies two essential elements: the instructional condition or exposure being compared and an outcome representing writing performance.
However, further decisions remain. Does writing performance mean the quality of an argumentative essay, grammatical accuracy, organization, originality, or a composite score? Is the intended comparison between randomly assigned instructional conditions, or between students who independently choose whether to use AI?
These distinctions affect which variables are needed and how they should be measured.
For quantitative research, specifying the estimand can clarify the intended comparison, population, outcome, and time point. In qualitative research, the equivalent planning task is to identify which experiences, processes, or contextual information are needed to address the inquiry.
A measurement plan should follow these decisions rather than attempt to compensate for an imprecise question by collecting an unusually large dataset.
Distinguish Variables by Their Role in the Study
Variables can serve different purposes, and the same variable may occupy different roles in different research questions.
| Variable role |
Purpose |
Illustrative example |
| Outcome |
Represents the response, condition, or result being investigated |
Academic writing score |
| Exposure or intervention |
Represents the condition, experience, or treatment of interest |
Assignment to AI-assisted writing |
| Descriptive variable |
Characterizes the study population |
Academic program or year level |
| Potential confounder |
May create a noncausal association between exposure and outcome |
Prior writing proficiency in a suitable observational causal model |
| Mediator |
Represents a possible mechanism connecting exposure and outcome |
Writing revision behavior after tool exposure |
| Moderator or effect modifier |
Identifies variation in an association or effect across conditions |
Baseline digital literacy |
| Implementation variable |
Documents how an intervention or procedure was delivered |
Number of completed instructional sessions |
These categories are not permanent labels attached to particular variables.
For example, digital literacy might be an outcome in a digital skills intervention, an effect modifier in an AI learning study, or a descriptive characteristic in a survey.
The variable's role follows the research question and assumed relationships among the constructs.
Select Outcomes Before Adding Secondary Measurements
An outcome is appropriate when it represents something the research genuinely seeks to understand and can be measured in a way that supports the intended interpretation.
In an educational intervention study, a primary outcome might be writing performance at the end of the semester. Secondary outcomes might include writing self-efficacy, revision behavior, or perceived cognitive workload.
These outcomes are related but not interchangeable.
A student may report greater confidence without demonstrating improved writing performance. Similarly, improved writing scores do not necessarily establish improved long-term learning.
Researchers should identify which outcome most directly answers the primary question and distinguish it from supporting or exploratory measurements.
The selection should consider construct relevance, validity evidence, responsiveness to plausible change, timing, and the importance of the outcome to relevant stakeholders.
See what makes an outcome measure appropriate for a research question for a closer examination of these considerations.
Not Every Study Needs the Same Number of Primary Outcomes
A single clearly defined primary outcome can simplify interpretation and sample size planning, particularly in confirmatory studies.
However, some research questions genuinely require more than one primary outcome, or a prespecified composite or multivariate approach.
For example, an intervention might be evaluated jointly for effectiveness and safety, with both dimensions essential to the study's conclusions.
Multiple primary outcomes introduce questions about multiplicity, statistical power, and how success will be determined. Researchers should address these issues before examining the data.
Exploratory research may have different conventions and may not require a single primary outcome in the same sense as a confirmatory clinical trial.
The appropriate structure depends on the design and inferential objectives rather than a universal rule that every study must measure exactly one main result.
Use the Conceptual Framework to Identify Relationships Worth Measuring
A conceptual framework organizes the constructs and relationships considered relevant to a research problem.
It can help researchers identify candidate outcomes, explanatory variables, contextual factors, and mechanisms.
However, a framework is not automatically a data collection inventory.
Some constructs may provide theoretical background without requiring direct measurement. Others may be essential to testing the proposed explanation.
Suppose a framework links access to digital learning resources, learning engagement, self-regulation, and academic performance.
If the primary objective is to estimate the association between resource access and performance, measuring self-regulation may be important under some analytical models but unnecessary under others.
If the objective is to investigate whether self-regulation mediates the relationship, measuring it becomes central to the study.
Researchers should therefore distinguish the concepts included for theoretical completeness from the variables needed for the planned empirical claims.
This issue is explored further in whether every variable in a conceptual framework should be measured.
Operationalize Each Essential Construct Before Finalizing the Variable List
A construct such as engagement, satisfaction, learning quality, or institutional trust cannot be measured adequately merely by assigning it a column name.
Researchers must specify how it will be represented empirically.
For example, student engagement might be operationalized using a validated self-report instrument, observed participation behavior, learning platform records, or a combination of measures.
These approaches may capture different dimensions of engagement.
The appropriate choice depends on the conceptual definition and intended inference. Platform logins, for example, may indicate access or activity without necessarily demonstrating cognitive engagement.
Measurement decisions should consider available validity evidence, reliability or precision where relevant, accessibility, and the conditions under which the instrument was developed.
The processes described in operationalizing research constructs and aligning conceptual and operational definitions provide the necessary foundation.
Measure Confounders According to a Causal Rationale
In observational research, identifying potential confounders is important when the objective is causal inference.
A confounder may create a noncausal association between an exposure and an outcome because of the underlying causal relationships.
Researchers should use substantive knowledge, previous evidence, and explicit causal assumptions to identify variables that require measurement or adjustment.
Directed acyclic graphs can help represent these assumptions and identify potentially sufficient adjustment sets.
However, not every variable associated with an outcome is a confounder. Variables affected by the exposure, including some mediators or colliders, may introduce bias when adjusted for inappropriately.
For example, if AI tool use influences writing confidence, and writing confidence subsequently influences performance, controlling for post-intervention confidence may remove part of the effect the study intends to estimate.
Researchers should not select confounders solely because they produce statistically significant associations in the observed dataset.
The methodological literature on causal inference, including Hernán and Robins' Causal Inference: What If, provides a foundation for understanding why adjustment decisions require a causal model.
Measure Mechanisms Only When the Research Question Requires Them
Researchers may want to understand not only whether an intervention works but how it works.
In that case, variables representing plausible mechanisms may be necessary.
For example, a study of AI-assisted writing might examine whether the intervention changes revision frequency, feedback interpretation, or writing self-efficacy, and whether these changes contribute to writing performance.
However, observing an association between a proposed mediator and an outcome does not establish mediation.
Causal mediation analysis requires assumptions about temporal ordering, confounding, and the relationships among the intervention, mediator, and outcome.
Researchers should measure proposed mechanisms at theoretically appropriate times and distinguish exploratory mechanism analysis from confirmatory causal claims.
If mechanism testing is not part of the research objective, collecting extensive mediator data may add burden without contributing to the primary inference.
Consider Moderators and Effect Heterogeneity Before Data Collection
Some studies investigate whether effects or associations differ across participant characteristics or contexts.
For example, the effect of an instructional intervention might differ according to baseline writing proficiency or access to reliable technology.
Investigating such heterogeneity requires relevant moderator measurements and an analytical strategy capable of evaluating interactions or subgroup differences.
Researchers should ideally identify important moderators before data collection, particularly in confirmatory research.
Exploring many subgroup comparisons after observing the results can increase the risk of chance findings and unstable estimates.
Subgroup analyses may also require larger samples than analyses of the overall effect.
Measuring a moderator is therefore justified when the variable serves a credible theoretical or practical purpose and the study can support meaningful interpretation.
Descriptive Variables Should Serve a Reporting or Interpretive Purpose
Researchers often collect age, gender, educational attainment, socioeconomic indicators, institutional affiliation, and other background characteristics.
Such variables may help describe the sample, evaluate recruitment coverage, examine generalizability, or support planned analyses.
However, demographic information should not be collected automatically without considering its relevance and sensitivity.
For example, asking participants for precise household income may be unnecessary when a study concerns a narrowly defined laboratory task and income has no justified descriptive or analytical role.
In other studies, socioeconomic circumstances may be essential to understanding unequal educational access.
Researchers should collect information at a level of detail appropriate to the research question and avoid requesting identifiable or sensitive data merely because similar variables appear in previous questionnaires.
Distinguish Essential Variables From Optional and Exploratory Variables
A useful measurement plan separates variables according to their contribution to the study.
| Priority |
Meaning |
Planning implication |
| Essential |
Required to answer the primary research question or conduct the prespecified primary analysis |
Prioritize measurement quality and completeness |
| Important supporting |
Needed for justified secondary objectives, interpretation, or key design requirements |
Collect when feasible and proportionate |
| Exploratory |
Supports additional hypotheses or descriptive investigations |
Clearly distinguish from confirmatory measurements |
| Unnecessary |
Has no defensible role in the study's objectives, design, or interpretation |
Omit unless a new justification is established |
This classification should not be confused with statistical importance. An essential variable may have a weak association with an outcome, while an unnecessary variable may show a strong association by chance.
Priority should be determined before the data are examined whenever possible.
More Variables Can Increase Burden and Data Quality Problems
Every additional measurement has a cost.
Participants may face longer questionnaires, more intrusive questions, additional appointments, or greater cognitive demands.
Researchers may also need more staff time, data storage, instrument licensing, training, and quality assurance.
Extensive measurement can contribute to fatigue, incomplete responses, or reduced attention, although the extent depends on the instruments and population.
Collecting more variables may also increase the number of possible analyses, making selective reporting more tempting.
These concerns do not justify omitting essential information. They support deliberate prioritization and appropriate data minimization.
The question is not whether a variable might someday be interesting, but whether collecting it is justified by the current research objectives and foreseeable analytical needs.
Consider Timing and Measurement Frequency
Choosing a variable is only part of measurement planning. Researchers must also decide when and how often it should be measured.
Baseline characteristics may need measurement before an intervention or exposure occurs.
Potential mediators may need assessment after the exposure but before the outcome, depending on the proposed causal sequence.
Outcomes may require immediate, delayed, or repeated measurement to capture the phenomenon of interest.
For example, a writing intervention could produce short-term improvements that do not persist after several months.
Measuring only immediately after the intervention would not answer a question about long-term retention of writing skills.
Conversely, collecting weekly measures for a relatively stable characteristic may add burden without meaningful scientific value.
Timing should follow the conceptual model, anticipated change process, and analytical design.
Evaluate the Availability and Quality of Existing Data
Some variables may already be available through institutional records, administrative databases, or routinely collected assessments.
Using existing information can reduce participant burden, but availability does not guarantee suitability.
Administrative variables may have been created for operational rather than research purposes. Definitions, missingness, measurement timing, and data quality may differ from what the study requires.
For example, a learning management system may record assignment submissions but not the quality of the student's reasoning.
Researchers should evaluate whether existing data adequately represent the intended construct and whether their use is ethically and legally authorized.
Linkage to administrative records may also introduce privacy risks and selection mechanisms when records are incomplete or cannot be matched reliably.
Align the Variable List With the Statistical Analysis Plan
Before data collection, researchers should verify that the planned analysis can be conducted using the selected variables.
For example, an analysis of change requires appropriately timed measurements. A planned interaction analysis requires the moderator and sufficient information across its relevant levels.
A causal adjustment strategy requires the variables identified by the assumed causal model.
Sample size and precision considerations may also affect whether the planned model is realistic.
However, researchers should not collect variables simply because statistical software can include them in a regression equation.
The analytical plan should follow the research question, and the measurement plan should provide the information needed to implement that analysis.
Use a Variable Specification Matrix Before Finalizing Instruments
A practical way to evaluate the measurement plan is to create a variable specification matrix.
For each variable, record its conceptual definition, role, measurement method, timing, intended analysis, and justification.
The matrix can reveal missing measurements, redundant instruments, inconsistent definitions, and variables with no clear purpose.
It can also help coordinate work among researchers, instrument developers, data collectors, and analysts.
For studies involving multiple institutions or research teams, consistent definitions are particularly important because apparently identical variable names may represent different measurements.
The matrix should be reviewed before instruments are finalized and revised through appropriate procedures when substantive changes become necessary.
Watch Out
Do not select variables exclusively through statistical significance testing or collect every available measure as insurance against an unclear research question. A defensible measurement plan requires conceptual justification, appropriate operational definitions, and alignment with the intended analysis.
