03 · What You Need to Know
Why Methodological Rigor and Population Relevance Can Diverge
What Does Methodological Rigor Actually Establish?
Methodological rigor concerns whether research procedures and reasoning are appropriate for producing defensible evidence about a specified question. Its requirements depend on the research approach and intended inference.
In a randomized controlled trial, rigor may involve appropriate randomization, allocation concealment, prespecified outcomes, suitable statistical analysis, and careful handling of missing data.
In observational research, important considerations may include measurement quality, confounding, selection mechanisms, temporal ordering, and the assumptions underlying statistical models.
In qualitative research, rigor may concern the coherence between research questions and methods, the adequacy of participant selection, reflexivity, the depth of analysis, and transparency about how interpretations were developed.
These practices can support strong conclusions. However, the conclusions remain tied to the population, setting, procedures, and assumptions under which the evidence was generated.
A rigorous method cannot automatically compensate for a mismatch between the people studied and the people about whom the researcher intends to make claims.
How Can Internal Validity Coexist With Poor Population Coverage?
Internal validity and population coverage address different methodological questions.
Internal validity, particularly in causal research, concerns whether the study supports an unbiased causal inference for the population and conditions to which the analysis pertains. Population coverage concerns whether the sampling or recruitment process adequately includes the intended population.
Internal validity
Whether the design and analysis support the intended within-study inference without consequential bias.
Population coverage
Whether the sampling or recruitment mechanism includes the people who belong to the defined target population.
External validity
Whether findings can reasonably apply to other populations, settings, interventions, or conditions.
Transportability
Whether an effect estimated in one population can be inferred for another population under explicit assumptions and relevant information.
For example, a randomized trial involving only participants who can attend frequent research appointments may estimate an intervention effect without substantial confounding within the randomized population.
Nevertheless, the trial may provide limited information about people unable to attend because of transportation, employment, or health-related constraints.
Whether the estimated effect applies to those excluded individuals depends partly on whether relevant effect modifiers differ between the studied and target populations.
The trial's internal validity does not, by itself, resolve that question.
How Can the Population Most Affected by a Problem Become the Least Represented?
Research participation often requires resources or capabilities that are unevenly distributed across the population. These requirements may be particularly consequential when the research problem itself concerns access, disadvantage, vulnerability, or unequal opportunities.
A study of digital exclusion may recruit only through online platforms. Research on transportation difficulties may require attendance at a distant facility. An investigation of language barriers may use recruitment and consent materials in only one language.
In each case, the condition being investigated may also influence whether someone can participate.
This creates a potential mismatch between the research problem and the achieved sample. The participants most accessible to researchers may be those experiencing fewer of the difficulties under investigation.
Researchers should therefore identify which population groups may be systematically excluded by the design and whether those exclusions are related to the phenomenon being studied.
What Is the Difference Between a Sampling Frame Problem and an Eligibility Problem?
A sampling frame problem occurs when relevant population members are absent from the mechanism used to identify or select participants.
An eligibility problem occurs when individuals are excluded because they do not satisfy the study's formal inclusion or exclusion criteria.
These mechanisms can overlap, but they require different responses.
| Exclusion mechanism |
Illustrative situation |
Possible methodological response |
| Sampling frame omission |
A survey of all residents uses only a list of registered online service users |
Improve the frame or use an appropriate supplementary sampling approach |
| Restrictive eligibility |
A study excludes older adults without a relevant scientific justification |
Review whether the restriction is necessary and proportionate |
| Recruitment channel limitation |
Invitations are distributed only through a platform used by a subset of the population |
Consider additional channels and evaluate coverage |
| Participation burden |
Participants must attend several lengthy appointments during working hours |
Assess scheduling, visit frequency, and feasible alternatives |
| Measurement accessibility |
A questionnaire cannot be used with relevant assistive technologies |
Improve accessibility while evaluating measurement equivalence |
| Differential nonresponse |
People experiencing the greatest difficulties are less likely to complete follow-up |
Investigate response mechanisms and apply appropriate retention or analytical strategies |
Identifying the mechanism matters because simply increasing the sample size will not necessarily address it.
Can Random Sampling Still Exclude the Most Affected Population?
Yes. Random sampling provides known selection probabilities within an appropriately defined sampling frame. It does not automatically include people who are absent from that frame.
Suppose researchers randomly select 1,000 users from an online service database to estimate satisfaction among all residents eligible for the service. The selection may be genuinely random within the database, yet residents who never registered online have no chance of selection.
The design may support inference about registered users while providing an incomplete basis for inference about all eligible residents.
Random selection and adequate population coverage are separate requirements.
Researchers should also examine nonresponse after selection. Equal selection probabilities do not guarantee equal probabilities of providing usable data.
Can a Large Sample Compensate for Systematic Population Exclusion?
No, not merely by becoming larger.
A large sample can reduce sampling variability under appropriate conditions. It does not automatically remove systematic errors associated with coverage, recruitment, nonresponse, or measurement.
For example, collecting 10,000 online survey responses may provide precise estimates among reachable respondents while still omitting people without internet access.
If the omitted population differs systematically on the outcome, increasing the number of online respondents may make the estimate more precise without making it more accurate for the intended population.
Researchers should distinguish precision from bias and evaluate both when assessing methodological quality.
When Does Exclusion Threaten External Validity?
Exclusion can threaten external validity when the characteristics of omitted participants are relevant to how findings would apply in the target population.
For intervention studies, a particularly important consideration is effect modification. An intervention may produce different effects depending on participant characteristics, implementation conditions, or available resources.
If the studied population excludes important effect modifiers or contains substantially different distributions of them, applying the observed average effect to the broader population may be misleading.
However, demographic differences alone do not prove that an intervention effect differs. The relevant question is whether the characteristics associated with exclusion influence the effect or other aspects of the intended inference.
Researchers should examine when inclusive research design strengthens scientific validity through improved population coverage, measurement, or applicability.
Can Systematic Exclusion Also Threaten Internal Validity?
Under some conditions, yes. Selection mechanisms can introduce bias when participation or continued observation is related to variables relevant to the estimated association or causal effect.
For example, conditioning analysis on a variable influenced by both exposure and outcome-related factors may create a noncausal association under particular causal structures.
In randomized trials, eligibility restrictions applied before randomization generally define the randomized population. They do not necessarily undermine the internal validity of treatment comparisons within that population.
By contrast, excluding participants after randomization based on post-assignment characteristics can threaten the comparability established through random assignment.
Researchers should therefore examine the timing and causal structure of selection rather than assuming that every exclusion produces the same kind of bias.
What Happens When the Most Affected Population Is Missing From Qualitative Research?
Qualitative research can be methodologically rigorous without statistically representative sampling. Nevertheless, participant selection must support the research question and intended interpretation.
Suppose a study investigates why adults struggle to access continuing education but interviews only people who successfully enrolled. Participants may provide valuable accounts of enrollment challenges, yet the experiences of people who attempted to enroll and failed may remain absent.
The problem is not that the sample fails to reproduce population percentages. It is that the selection process may omit experiences essential to understanding the phenomenon.
Purposive sampling, theoretical sampling, and other qualitative approaches can address this concern when their logic is appropriate to the research aims.
Researchers should explain why particular participants were selected and how the resulting evidence supports the claims being made.
Can Statistical Adjustment Recover Evidence About Excluded Groups?
Statistical adjustment may help under appropriate assumptions, but it cannot automatically replace evidence from people who were never observed.
Survey weighting can account for known selection probabilities and certain observed differences between respondents and the target population. Transportability methods may support inference about a target population when relevant effect modifiers and suitable population information are available.
These methods require assumptions about selection, measurement, and the relationship between observed and unobserved populations.
For example, if a subgroup has no representation in the study and its intervention response may differ in unknown ways, statistical adjustment cannot reliably establish its effect without additional information or assumptions.
Researchers should consider whether oversampling relevant underrepresented groups or improving recruitment coverage would provide more defensible evidence than relying exclusively on post hoc adjustment.
How Can Eligibility Criteria Produce an Artificially Narrow Study Population?
Eligibility restrictions may be introduced to simplify procedures, reduce variability, or avoid anticipated complications. Some restrictions are necessary, but others can produce a population that differs substantially from those affected by the research problem.
For example, a trial of an intervention intended for people with chronic health conditions may exclude participants with common comorbidities, limited mobility, or additional support needs.
The resulting evidence may remain informative for the selected population, but its applicability to routine practice may be limited.
Researchers should evaluate whether eligibility criteria unnecessarily exclude relevant participants and whether narrower, study-specific restrictions could address the same scientific or safety concerns.
Where exclusions are justified, the target population and interpretation should reflect them explicitly.
How Can Participation Requirements Create a Contradiction in the Research Design?
A particularly consequential problem occurs when participation requires the very resource, capability, or opportunity whose absence defines the research problem.
Examples include requiring reliable internet access to study digital exclusion, requiring extensive travel to investigate transportation disadvantage, or requiring high literacy to evaluate access to information among people with limited reading proficiency.
These requirements may systematically filter out participants experiencing the greatest barriers.
Researchers should examine whether internet access requirements exclude relevant participants and whether alternative procedures could address the underlying barrier.
Where participation requirements cannot be changed, the study may need to redefine its target population or narrow its claims.
How Should Researchers Evaluate the Population Most Affected by the Research Problem?
Researchers should first define what it means to be affected by the phenomenon. This may involve exposure to a condition, experience of a barrier, likelihood of receiving an intervention, or vulnerability to a particular outcome.
The population most affected is not always a single demographic group. It may consist of people whose circumstances intersect in ways that increase exposure or limit access.
For example, transportation disadvantage may depend on geographic location, income, mobility needs, and service availability rather than any one characteristic alone.
Relevant population information may come from administrative records, population surveys, previous research, or consultation with affected communities.
Researchers should distinguish evidence-supported population characteristics from assumptions based on stereotypes.
The purpose is to determine whether the study design provides an adequate opportunity for the relevant experiences and characteristics to be represented.
What Can Researchers Do When the Original Design Excludes Key Populations?
The appropriate response depends on the source of exclusion and the intended inference.
Researchers may revise eligibility criteria, expand recruitment channels, provide accessible materials, introduce alternative participation modes, or modify unnecessary scheduling and travel requirements.
In probability-based research, improvements to the sampling frame or allocation may be necessary. In qualitative research, additional purposive recruitment may help capture experiences absent from the original sample.
These changes should preserve participant safety and the scientific integrity of the procedures. Researchers must also consider whether modifications alter the intervention or measurement.
Where broader inclusion is operationally difficult, balancing inclusive recruitment with safety and feasibility requires explicit consideration rather than treating convenience as a sufficient scientific justification.
Material changes to approved protocols may require additional ethics or institutional review.
How Should Researchers Report Findings When Important Groups Were Excluded?
Transparent reporting should identify the intended population, eligibility criteria, recruitment sources, achieved sample, and relevant participation barriers.
Researchers should explain whether the exclusions primarily affect population coverage, applicability, measurement, or other aspects of inference.
For randomized trials, the CONSORT 2025 statement and its explanation and elaboration provide reporting recommendations relevant to eligibility, participant flow, trial settings, and interpretation.
For observational research, STROBE provides guidance on reporting participant selection, eligibility, study size, bias, and generalizability. These reporting frameworks support transparency but do not establish that a study is free from bias.
Researchers should avoid presenting findings from a narrowly selected sample as though they necessarily apply to all people affected by the research problem.
When the evidence supports conclusions only for a restricted population, that restriction should be visible in the abstract, discussion, and practical implications.
Watch Out
A study can be internally valid, statistically precise, and transparently reported while still providing limited evidence for the population most affected by the research problem. Methodological quality should be evaluated in relation to the claims being made, not solely through procedural compliance or the sophistication of the analysis.
