03 · What You Need to Know
Understanding Whose Priorities Shape Research Outcomes
Outcome Selection Is Not a Neutral Administrative Decision
Every research study makes decisions about what counts as evidence of success, improvement, or harm.
Those decisions reflect the research questions, disciplinary traditions, theoretical assumptions, and interests of the people designing the study.
For example, an educational technology researcher may consider examination scores a central indicator of instructional effectiveness. A student may instead judge an intervention by whether it improves understanding, reduces confusion, or supports independent problem-solving.
Neither perspective is automatically illegitimate.
The methodological concern arises when researchers assume that an outcome representing one perspective adequately represents another without justification.
Outcome selection therefore influences not only what is measured but also which consequences of an intervention become visible in the findings.
What Makes an Outcome Important to Researchers?
Researchers may prioritize outcomes for several defensible reasons.
An outcome may test a theoretical prediction, permit comparison with previous studies, support a causal estimand, or provide evidence about an underlying mechanism.
For example, response latency in a cognitive experiment may be theoretically important even when participants do not consider milliseconds of response time personally meaningful.
Likewise, a laboratory measure of memory performance may help evaluate a psychological theory without directly representing a participant-valued educational benefit.
Such outcomes can be scientifically useful.
The problem is not that participants fail to value them. It is whether the researchers subsequently interpret the results as evidence of a benefit that the measure does not establish.
What Makes an Outcome Relevant to Participants?
A participant-relevant outcome reflects a result that matters to the people affected by the phenomenon or intervention.
In educational research, relevant outcomes may include the ability to apply knowledge, confidence in performing meaningful tasks, accessibility, workload, or sustained learning.
In health research, participants may prioritize symptoms, functioning, quality of life, treatment burden, or adverse effects.
However, participant relevance is not a fixed property of an outcome.
Different participants may value different results, and preferences can change according to circumstances.
Researchers should avoid assuming that a single outcome represents the priorities of an entire population.
Researcher Importance and Participant Relevance Are Different Dimensions
An outcome can be scientifically important, participant-relevant, both, or neither.
| Outcome relationship |
Illustrative example |
Implication |
| Scientifically important and participant-relevant |
Improved ability to perform a meaningful professional task |
Potentially strong alignment between scientific and practical objectives |
| Scientifically important but not directly participant-relevant |
Response latency used to test a cognitive theory |
May be appropriate for a theoretical question |
| Participant-relevant but not central to the researcher's original model |
Unexpected workload created by an instructional intervention |
May justify an additional outcome or revised research agenda |
| Neither clearly relevant nor justified |
Platform activity recorded without a defined analytical purpose |
May not warrant collection or emphasis |
The classifications are illustrative. An outcome's relevance depends on the research context and the intended use of the evidence.
Why Easily Measured Outcomes Can Become Overrepresented
Researchers often work within constraints involving funding, time, available instruments, and access to data.
These constraints can make convenient outcomes attractive.
For example, a learning management system may automatically record page views, logins, and assignment submissions. Measuring independent reasoning or sustained learning may require additional assessments and trained evaluators.
Researchers may consequently rely on the available platform indicators.
That decision can be reasonable when the research question concerns platform use.
However, platform activity should not automatically be interpreted as learning effectiveness.
Availability and precision do not establish that an outcome represents what participants consider meaningful.
Participant-Relevant Outcomes Are Not Necessarily Subjective
It is tempting to associate participant relevance exclusively with satisfaction surveys or self-reported experiences.
However, an objectively assessed outcome can be highly relevant to participants.
For example, the ability to complete a professional task independently may matter greatly to learners and can be assessed through a performance-based examination.
Conversely, a subjective measure may have little relevance if it concerns an experience unrelated to the study's practical objective.
Researchers should distinguish whose priorities an outcome reflects from whether it is measured objectively or subjectively.
The measurement approach should follow the construct, not the presumed superiority of one evidence type.
Surrogate Outcomes Can Create a Gap Between Evidence and Meaning
Some outcomes are used as substitutes for results that are more difficult to observe.
For example, immediate knowledge acquisition may be used as an indicator of an intervention's potential to improve long-term competence.
However, improvement in a short-term test does not necessarily establish sustained learning or practical application.
Similarly, increased engagement may be associated with beneficial educational experiences without being a validated surrogate for academic achievement.
Researchers should distinguish direct evidence of the measured outcome from inferences about a more consequential result.
When a surrogate is used, its limitations should be explicit, particularly when claims concern participant-important benefits.
Statistical Significance Does Not Establish Participant Importance
A statistically significant difference may be too small to matter to participants.
For example, a large study might detect a small change in a satisfaction score that does not meaningfully alter the learning experience.
Conversely, an estimated improvement that participants would consider valuable may be statistically uncertain because the study has limited precision.
Researchers should therefore examine effect magnitude, confidence intervals, measurement scales, and the practical meaning of the difference.
Where appropriate, stakeholder input can help identify what changes would be considered meaningful.
However, thresholds for meaningful change should be justified for the intended population and context rather than invented after results are known.
Participants May Value Harms That Researchers Do Not Initially Measure
Researchers often design interventions around expected benefits.
This focus can lead to insufficient attention to unintended consequences.
For example, an AI-assisted learning intervention might improve assignment completion while increasing cognitive workload, dependency concerns, or uncertainty about academic integrity.
These are hypothetical possibilities rather than established effects of AI-assisted instruction.
Nevertheless, they illustrate why an effectiveness study may benefit from considering outcomes beyond its intended benefit.
Participant consultation can help identify consequences that researchers have not anticipated.
Not every concern must become a primary outcome, but consequential outcomes should not be ignored simply because they fall outside the original theoretical model.
Different Participants May Prioritize Different Outcomes
Participant populations are rarely homogeneous in their preferences.
Students with limited connectivity may prioritize accessibility, while students with reliable access may emphasize feedback quality or flexibility.
Participants with substantial employment or caregiving responsibilities may place particular importance on workload and scheduling.
Researchers should avoid treating the views of a small, convenient group of participants as representing everyone affected by an intervention.
Engagement activities should consider diversity in experience, access, and circumstances.
Where important differences emerge, researchers may need to explain how competing priorities influenced outcome selection.
Participant Involvement Can Improve Outcome Selection
Participants or representatives of the population of interest can contribute to research planning before outcomes are finalized.
Possible approaches include consultation interviews, advisory groups, workshops, surveys, and structured priority-setting exercises.
These activities can reveal outcomes that researchers overlook or help clarify whether proposed measures reflect meaningful experiences.
However, consultation quality depends on who participates, how questions are framed, and whether contributors have genuine opportunities to influence decisions.
Inviting participants to endorse an already finalized outcome list is not equivalent to involving them in its development.
Researchers should explain how participant input was considered and where other scientific or practical considerations affected the final decision.
Core Outcome Sets Illustrate a Structured Approach
In health research, core outcome sets identify a minimum collection of outcomes recommended for measurement and reporting in studies of a particular condition or area.
The COMET Initiative supports the development and use of these sets.
Development processes may involve patients, clinicians, researchers, and other stakeholders.
COMET-related guidance emphasizes transparent methods for identifying and prioritizing outcomes.
Core outcome sets can improve consistency across studies while incorporating outcomes important to affected populations.
However, a core outcome set does not necessarily specify the instruments or measurement times for every outcome, and additional study-specific outcomes may still be appropriate.
Comparable stakeholder-informed approaches can be considered in education and other fields without assuming that clinical research conventions transfer unchanged.
Participant Priorities Do Not Automatically Determine Every Research Outcome
Participant involvement is important, but it does not mean that researchers must select outcomes solely through preference voting.
Some outcomes are necessary to test scientific theories, assess safety, evaluate mechanisms, or satisfy legitimate policy questions.
Participants may also disagree about what matters most.
Researchers should integrate participant priorities with construct validity, feasibility, study objectives, and ethical considerations.
The goal is not to replace scientific judgment with preference aggregation.
It is to ensure that scientific judgment is informed by the experiences and interests of those affected by the research.
Participant Relevance Should Influence Interpretation, Not Just Selection
Even when participant-important outcomes are measured, researchers may overemphasize outcomes that produce statistically favorable results.
For example, an intervention may improve a standardized performance score while showing uncertain effects on the ability to apply knowledge independently.
If practical application is important to participants, the conclusion should acknowledge that uncertainty.
Researchers should distinguish evidence about the measured outcome from broader claims about educational or personal benefit.
This is especially important when a study uses intermediate or surrogate measures.
The limitations of the outcome should remain visible in the discussion and conclusions.
Use an Outcome Relevance Matrix
A structured matrix can help researchers compare scientific objectives and participant priorities.
| Candidate outcome |
Research purpose |
Potential participant relevance |
Planning consideration |
| Writing performance |
Evaluate instructional effectiveness |
Ability to complete meaningful writing tasks |
Ensure authentic construct coverage |
| Writing self-efficacy |
Investigate confidence |
Perceived capability and willingness to write |
Do not substitute for demonstrated competence |
| Platform activity |
Describe intervention use |
May have limited direct importance |
Interpret as activity rather than learning |
| Perceived workload |
Assess intervention burden |
Time and effort required from learners |
Consider as a consequential outcome |
| Long-term writing retention |
Evaluate persistence of learning |
Ability to use skills beyond the course |
Plan appropriate follow-up |
The matrix is illustrative and should not be treated as evidence of actual student preferences.
Researchers should obtain relevant stakeholder input rather than infer priorities solely from their own assumptions.
What If Researcher and Participant Priorities Conflict?
Conflicts may arise when an outcome is theoretically important but participants regard it as burdensome or irrelevant.
Researchers should first clarify the purpose of the study.
A basic science investigation may legitimately prioritize a theoretical measure without claiming immediate participant benefit.
An intervention evaluation intended to inform practical adoption should give greater consideration to outcomes relevant to affected populations.
Where multiple priorities cannot all be accommodated, researchers should document the rationale for selection and acknowledge what the study will not establish.
Transparency is preferable to presenting researcher-selected outcomes as though they automatically represent participant interests.
Watch Out
Do not equate a statistically significant or easily measured outcome with a meaningful participant benefit. The measure may be scientifically useful, but claims about practical improvement require evidence that the outcome represents something consequential to the people affected.
