03 · What You Need to Know
How Stage, Severity, and Experience Affect Population Definition and Research Validity
What Does It Mean to Define a Population by Disease Stage?
Disease stage describes the extent, progression, or classification of a condition according to a specified staging system.
For example, some oncology studies define eligibility using established cancer staging criteria, while other clinical fields use different classifications of disease progression.
A study investigating treatment outcomes among patients with a particular disease stage may appropriately include that stage in its target population definition.
However, staging systems are condition-specific and may be revised over time.
Researchers should identify the relevant classification system, its version where applicable, and the time at which staging is assessed.
Using an undefined label such as advanced disease may be insufficient when different clinicians or institutions interpret the term differently.
How Is Disease Severity Different From Disease Stage?
Disease stage and disease severity are related concepts, but they are not necessarily interchangeable.
Stage may describe the extent or progression of a condition according to a formal classification.
Severity may describe symptom burden, functional impairment, physiological disturbance, clinical risk, or another measure of how seriously the condition affects the patient.
For example, two patients classified within the same disease stage may experience different symptom severity or functional limitations.
Likewise, a condition may be clinically severe without having a staging system comparable to those used in oncology.
Researchers should specify which construct is relevant to the research question rather than treating stage and severity as synonyms.
What Does Experience Mean in Population Definition?
Experience refers to prior involvement with, exposure to, or familiarity with a phenomenon relevant to the research.
It may include professional experience, educational experience, treatment experience, technology use, or participation in particular activities.
For example, a study of researchers' experiences using generative AI for systematic reviews may require participants to have actually used such tools in that context.
However, experience should be defined operationally.
Terms such as experienced user, expert practitioner, or frequent user can be ambiguous unless the study establishes meaningful criteria.
Experience may be assessed through duration, frequency, documented activities, demonstrated competence, or other indicators, depending on the research question.
These indicators should not be assumed equivalent.
When Should These Characteristics Define the Population?
The decision should follow the intended estimand and the scientific role of the characteristic.
| Research objective |
Possible population definition |
Rationale |
| Evaluate treatment for a specific disease stage |
Patients meeting the defined staging criteria |
The treatment question concerns that clinical subgroup. |
| Estimate severity distribution |
All eligible patients with the condition |
Different severity levels are needed to estimate the distribution. |
| Investigate experiences of advanced disease |
Patients meeting justified advanced-disease criteria |
The phenomenon specifically concerns advanced disease. |
| Estimate prevalence of generative AI use |
All eligible researchers, including nonusers |
Nonusers are necessary for the prevalence denominator. |
| Understand experiences of AI-assisted manuscript writing |
Researchers with relevant firsthand experience |
Direct experience is central to the qualitative question. |
| Compare novice and experienced users |
Eligible users across relevant experience levels |
Restricting to experts would eliminate the intended comparison. |
The same characteristic can appropriately define eligibility in one study and serve as a measured variable in another.
Should Disease Stage Be Defined Before Selecting Participants?
Generally, yes, when disease stage is essential to the research question.
Researchers should establish the intended staging criteria before recruitment to avoid inconsistent eligibility decisions.
For example, a clinical intervention intended for patients with a specified stage should use a recognized and appropriately applied staging system.
Where staging information is uncertain, the protocol should define how eligibility will be verified.
Changing staging requirements after examining outcomes may alter the target population and introduce concerns about selective analysis.
Researchers should distinguish scientifically justified eligibility changes from post hoc restrictions designed to produce more favorable results.
Should Severity Be Used to Create a More Homogeneous Population?
Severity-based restriction can sometimes improve alignment with an intervention or reduce heterogeneity that would otherwise complicate interpretation.
For example, a treatment designed for severe symptoms may not be clinically appropriate for people with mild symptoms.
However, greater homogeneity is not automatically preferable.
Restricting the population to one severity category may prevent researchers from examining whether the intervention works differently across severity levels.
It may also reduce the applicability of findings to patients encountered in routine practice.
The decision should reflect the treatment indication, research objective, safety considerations, and intended population of application.
Can Restricting Disease Severity Improve Internal Validity?
Restriction may reduce variation in certain prognostic characteristics, but it does not automatically improve internal validity.
For example, limiting a study to patients with moderate disease may reduce differences in baseline severity.
However, selection bias, confounding, measurement error, and other threats may remain.
In randomized studies, broader eligibility may preserve internal causal validity while allowing evaluation across clinically relevant subgroups.
Researchers should therefore distinguish reducing population heterogeneity from eliminating bias.
A narrowly defined population can be scientifically appropriate without necessarily being methodologically superior.
What Happens When Severity Is Related to Treatment Selection?
In observational clinical research, disease severity may influence which treatment a patient receives.
For example, patients with more severe disease may be more likely to receive intensive treatment.
If severity also predicts the outcome, comparisons between treatment groups may be confounded by indication.
Restricting eligibility to one severity category may reduce some variation, but it may not eliminate residual confounding.
Researchers should measure relevant baseline severity and consider an appropriate causal adjustment strategy.
Severity assessed after treatment begins requires additional care because it may be affected by the treatment itself.
Why Does the Timing of Severity Assessment Matter?
Severity can change over time, particularly in progressive or fluctuating conditions.
A patient classified as having moderate disease at diagnosis may have severe disease at treatment initiation.
Researchers should specify whether eligibility is determined using severity at diagnosis, enrollment, baseline, intervention initiation, or another clinically meaningful time point.
The chosen timing should correspond to the intended research question.
Using severity measured after treatment initiation to define a baseline treatment comparison may introduce selection or causal interpretation problems.
The protocol should align eligibility, exposure assessment, and outcome follow-up with a clearly defined temporal structure.
Can Disease Severity Be Measured Instead of Used as an Eligibility Criterion?
Yes. When the research question concerns a broad clinical population, measuring severity across participants may be preferable to excluding particular severity categories.
For example, a study examining quality of life among adults with a chronic condition might include patients across the clinically relevant severity spectrum.
Severity could then be examined as a predictor, stratification variable, or potential effect modifier.
This approach preserves variation that may be central to the scientific question.
However, it requires appropriate measurement, sufficient representation across severity categories, and analytical methods consistent with the design.
Researchers should avoid assuming that including more heterogeneous participants automatically produces valid subgroup estimates.
When Should Prior Experience Be Required?
Prior experience should be required when the phenomenon cannot meaningfully be examined without firsthand involvement.
For example, a qualitative investigation of researchers' experiences negotiating AI-generated text during manuscript preparation may reasonably require direct experience with that process.
Similarly, a study examining patients' experiences of a particular treatment may target people who have received it.
However, the eligibility requirement should correspond to the question.
If the objective concerns why some researchers adopt generative AI and others do not, excluding nonusers would remove a crucial comparison group.
The researcher should distinguish experience as a defining characteristic from experience as an outcome, exposure, or explanatory variable.
How Should Experience Be Operationalized?
Experience can be defined through several dimensions, but the selected measure should match the underlying construct.
Duration of experience, frequency of use, task complexity, and demonstrated competence are not interchangeable.
For example, someone who has used generative AI occasionally for three years may have less task-specific expertise than someone who has used it intensively for six months.
Researchers should avoid defining expertise solely through arbitrary time thresholds unless those thresholds are substantively justified.
Possible indicators include documented activities, frequency of relevant practice, completion of specific tasks, or validated measures of competence.
The operational definition should be transparent and feasible to apply consistently.
Can Self-Reported Experience Be Used for Eligibility?
Yes, when self-report is appropriate to the research question and the expected measurement error is acceptable.
For example, participants may report whether they have used a particular educational technology.
However, self-reported experience may be affected by recall limitations, differences in interpretation, or social desirability.
Researchers should define the relevant activity clearly and consider whether additional verification is necessary.
For some studies, documented usage records or task-based assessments may provide more suitable evidence.
The choice depends on the importance of eligibility classification, feasibility, privacy, and the validity of available measures.
What If Stage or Severity Classification Is Uncertain?
Clinical classifications may be uncertain because of incomplete records, measurement variability, or disagreement between assessors.
Researchers should specify the accepted assessment methods and how uncertain or borderline cases will be handled.
For example, the protocol might require a particular diagnostic assessment or define an adjudication procedure for inconsistent records.
However, excluding every uncertain case may systematically remove patients whose circumstances differ from those with complete assessments.
Researchers should evaluate whether uncertainty reflects measurement limitations, clinical ambiguity, or missing information.
Eligibility decisions should be consistent, documented, and aligned with the intended population.
Can Restricting to Experienced Participants Introduce Selection Bias?
Yes, depending on the research question and how experience is acquired.
Suppose a study examines whether an educational technology improves academic performance but includes only students who continued using the technology for an entire semester.
Continued use may depend on motivation, perceived usefulness, access, or early academic outcomes.
Restricting the analysis to persistent users could therefore select participants based on characteristics related to the outcome.
If the original objective concerns the effect of being offered the technology, the restriction may change the estimand and introduce selection bias.
Researchers should distinguish baseline experience from experience accumulated after exposure or intervention assignment.
What Is the Difference Between Baseline and Post-Exposure Eligibility?
Baseline eligibility criteria are assessed before the relevant exposure or intervention begins.
Post-exposure characteristics may be influenced by the exposure itself.
For example, requiring participants to achieve a particular level of treatment adherence after randomization changes the population analyzed.
Such restrictions may undermine the comparability created by random assignment.
Similarly, defining a population by survival or continued participation after exposure can create selection problems.
Researchers should avoid conditioning primary treatment comparisons on post-exposure characteristics without a clearly justified estimand and suitable causal methods.
How Do These Restrictions Affect Generalizability?
Findings from a stage-specific, severity-specific, or experience-specific population apply most directly to the population defined by those characteristics.
For example, treatment outcomes observed among patients with advanced disease may not apply identically to patients with early-stage disease.
Similarly, findings about experienced AI users may not describe the attitudes or challenges of novice users.
However, differences in population characteristics do not automatically prove that outcomes or effects differ.
Generalization requires considering whether relevant mechanisms and effect modifiers are comparable across populations.
Researchers should evaluate how population differences affect the intended inference.
Should Every Important Severity Group Be Represented?
Not necessarily. Representation should follow the intended research question.
If the objective concerns only patients with severe disease, including mild cases may not be necessary.
However, a study intended to estimate outcomes across the full disease spectrum should consider whether relevant severity groups are adequately represented.
Similarly, a study comparing novice and experienced researchers requires sufficient observations from both groups.
When some groups are rare, stratified sampling or deliberate oversampling may be useful, followed by appropriate weighting when population estimation is intended.
The number of participants required depends on the estimand, variability, and desired precision rather than an arbitrary requirement for equal group sizes.
Can Stage, Severity, or Experience Be an Effect Modifier?
Yes. These characteristics may influence the magnitude or direction of an intervention effect.
For example, a clinical treatment may have different benefits or risks across disease severity levels.
An educational AI intervention may also have different effects among novice and experienced users.
If such heterogeneity is scientifically important, restricting the population to one category may prevent the study from examining it.
Researchers should consider whether subgroup comparisons or interaction analyses are needed.
These analyses require adequate sample sizes and should preferably be planned before outcomes are examined.
How Does Population Restriction Affect Sample Size?
Restricting eligibility can reduce the number of available participants and affect recruitment feasibility.
It may also alter outcome variability, expected event rates, and treatment-effect assumptions.
For example, a clinical trial restricted to a rare disease stage may require recruitment across multiple hospitals.
Conversely, a more homogeneous population may sometimes reduce outcome variance, although this should not be assumed without supporting evidence.
Sample size calculations should use assumptions appropriate to the restricted population and intended estimand.
Researchers should not narrow eligibility solely to achieve a favorable power calculation.
What If Disease Classification Systems Change Over Time?
Clinical staging and severity classifications may be updated as scientific knowledge and diagnostic practices evolve.
When combining historical and contemporary records, researchers should examine whether classifications remain comparable.
For example, patients classified under different editions of a staging system may not be directly comparable without careful harmonization.
Researchers should document classification versions, assessment dates, and any recoding procedures.
Where reliable harmonization is not possible, sensitivity analyses or period-specific analyses may be necessary.
Uncritical combination of differently classified records can introduce misclassification and distort subgroup comparisons.
How Should These Population Characteristics Be Reported?
Methods sections should specify whether stage, severity, or experience defines the target population, determines eligibility, or serves as a measured analytical variable.
Describe the operational definitions, classification systems, measurement procedures, and relevant assessment dates.
Where thresholds are used, explain their substantive or clinical justification.
Report how uncertain classifications, missing information, and borderline cases were handled.
For intervention studies, distinguish baseline eligibility from post-assignment characteristics and explain any restrictions affecting the analysis population.
Transparent reporting helps readers determine whether the findings apply to the intended population and whether important categories were excluded.
