03 · What You Need to Know
A Design-Stage Audit of the Selection Processes That Matter for Inference
Why Should Selection Bias Be Evaluated Before Statistical Analysis?
Statistical methods operate on the information available in the dataset. They cannot automatically reconstruct individuals who were never eligible, never contacted, or never observed.
For example, regression adjustment cannot directly estimate outcomes for a subgroup entirely absent from the study without additional assumptions or external evidence.
Similarly, matching cannot correct an inappropriate comparison merely by making observed characteristics look balanced.
Selection mechanisms should therefore be examined while defining the population, treatment strategies, recruitment procedures, and outcome collection.
The analysis plan should follow the design and causal question rather than determine them retrospectively.
What Is the First Question in a Selection Bias Audit?
Identify the target population and the quantity the study intends to estimate.
For descriptive research, the estimand may be the prevalence of a behavior or the mean value of an outcome in a defined population.
For causal research, it may be an average treatment effect among eligible participants or another explicitly defined group.
The same selection procedure may be problematic for one estimand but appropriate for another.
For example, recruiting members of an AI learning community may be suitable for describing that community but insufficient for estimating AI adoption across all university students.
Without a defined target, researchers cannot determine whether sample selection has distorted the intended inference.
What Is the Difference Between a Selection Process and Selection Bias?
A selection process determines which units enter a study or analysis. Selection bias is a systematic distortion resulting from that process under the relevant inferential target.
Selection process
A rule or mechanism determining eligibility, recruitment, participation, observation, or analytical inclusion.
Selection bias
A systematic difference between the intended estimate and the estimate produced through a consequential selection mechanism.
Limited generalizability
Insufficient evidence that an estimate obtained in one population applies to another population.
Every study involves selection. The relevant question is whether the mechanism distorts the intended relationship or limits the population to which conclusions can be applied.
What Sampling-Frame Risks Should Be Examined?
The sampling frame determines which members of the intended population can be identified for selection.
Researchers should ask whether the frame excludes important groups, contains outdated records, or depends on characteristics related to the outcome.
For example, an educational study based on active platform accounts may omit students who withdrew, lost access, or stopped using the system.
If the objective is a population estimate, incomplete coverage can distort the result.
If inclusion depends on exposure and outcome-related causes, it may also distort associations.
Researchers should investigate selection mechanisms established before recruitment begins.
How Should Research Site Selection Be Evaluated?
Site selection determines which institutional environments and participants become available for study.
Institutions volunteering for educational technology research may differ from nonparticipating institutions in resources, leadership, infrastructure, or teaching practices.
These differences can limit the range of contexts represented.
However, selecting technologically advanced institutions does not automatically bias a randomized effect estimated within those institutions.
Researchers should distinguish site-level generalizability concerns from selection mechanisms that distort the within-study comparison.
When treatment is assigned at the institutional level, site characteristics may also create confounding that requires explicit attention.
Which Eligibility Criteria Deserve Particular Scrutiny?
Eligibility rules should be evaluated according to their purpose, timing, and causal role.
Baseline restrictions may define a meaningful study population or reduce variation in a known confounder.
Restrictions based on postexposure variables can be more problematic.
For example, including only students who completed an intervention may condition on a variable influenced by treatment and academic preparation.
This can distort the comparison if preparation also affects the outcome.
Researchers should examine whether eligibility criteria create selection bias rather than assuming that narrower criteria always improve validity.
How Can Recruitment Channels Affect Selection?
Recruitment channels determine which eligible individuals encounter an invitation.
Social media groups, institutional mailing lists, classroom announcements, and professional networks have different coverage properties.
For example, recruiting students through AI-focused communities may overrepresent those already interested in the technology.
Researchers should identify the population each channel can reach and whether participation probabilities relate to the study variables.
Using multiple channels may improve coverage but does not automatically establish representative sampling.
The implications of recruitment-channel selection should be assessed before invitations are distributed.
How Does Informed Consent Affect the Analytical Sample?
Consent is an ethical requirement under applicable research frameworks, but participation decisions can affect sample composition.
Individuals may decline because of privacy concerns, limited time, discomfort with the topic, or other reasons related to the study variables.
Consent materials that are inaccessible or difficult to understand may also prevent willing individuals from participating.
Researchers should improve clarity and accessibility while respecting voluntary refusal.
They should examine how consent procedures influence participation without weakening ethical safeguards.
Information about nonparticipants may be limited by privacy and consent requirements, which should be respected during selection assessment.
What Comparison-Group Selection Risks Matter?
In causal research, comparison groups should represent meaningful alternative exposure or treatment strategies.
Researchers should evaluate whether groups share compatible eligibility criteria, a common time zero, and comparable outcome measurement.
For example, comparing students who voluntarily used a platform with students who never had access to it may combine treatment effects with differences in resources and institutional conditions.
These differences may involve confounding as well as selection.
Researchers should distinguish the problems rather than applying one statistical adjustment indiscriminately.
A defensible comparison-group design begins with the intended causal contrast.
Why Is Common Time Zero Important?
Time zero is the point at which eligibility, treatment classification or assignment, and follow-up begin under the intended design.
If treatment groups are defined using information obtained after follow-up starts, selection can become entangled with survival, persistence, or outcome availability.
For example, classifying students as sustained platform users only after four weeks of use requires them to remain observable for those four weeks.
Comparing those students with nonusers followed from the first day can introduce time-related bias.
Researchers should align treatment strategies and follow-up according to a target trial or another defensible design framework.
Common time zero is a design safeguard, not a statistical correction applied after the fact.
What Is the Role of Collider Bias in Selection?
A collider is a variable influenced by two or more other variables in a causal structure.
Conditioning on a collider can create a noncausal association between its causes.
For example, participation in an academic support program may be influenced by both motivation and academic difficulties.
Restricting a study to program participants may induce an association between those characteristics even if they were independent in the broader population.
Researchers should examine whether conditioning on study participation creates misleading associations.
Not every selective sample produces collider bias. The relevant causal structure must be justified.
What Risks Arise From Matching and Restriction?
Matching and restriction can reduce measured differences between exposure groups, but they can also change the analytical population.
Matching on posttreatment variables or colliders may introduce bias.
Excluding unmatched participants may change the estimand to a narrower overlap population.
Restriction may remove variation in a confounder but limit direct evidence about excluded groups.
Researchers should distinguish matching-induced selection problems from legitimate changes in the target population.
The analytical plan should document why particular observations are retained or discarded.
How Does Loss to Follow-Up Affect Selection?
Loss to follow-up determines which initially enrolled participants remain observable at later assessments.
If outcome availability depends on treatment and outcome-related factors, analyzing only retained participants may distort the intended estimate.
Equal attrition rates across groups do not guarantee unbiased results because the characteristics of missing participants may differ.
Researchers should plan follow-up procedures and distinguish intervention discontinuation from withdrawal of outcome data.
They should examine how follow-up loss introduces selection bias before deciding on complete-case analysis or other missing-data methods.
Can Complete-Case Requirements Create a New Analytical Population?
Yes. Requiring complete information on every variable may exclude participants who otherwise satisfied the study's eligibility criteria.
For example, students with intermittent connectivity may have incomplete platform logs, while students who withdraw may lack final assessment scores.
Excluding them can change the distribution of characteristics in the analyzed sample.
The consequences depend on the missingness mechanism and estimand.
Researchers should distinguish missingness introduced by data collection from exclusions introduced by analysis.
Complete-case analysis can be valid under certain assumptions, but the completeness of retained records does not establish those assumptions.
Can Selection Bias Be Identified From a Participant Flow Diagram?
A participant flow diagram can reveal where individuals were excluded, lost, or omitted from analysis.
It may show the number screened, eligible, enrolled, assigned, followed, and analyzed.
However, flow counts alone do not establish whether selection biased the intended estimate.
Researchers also need information or assumptions about why selection occurred and how it relates to exposure, outcomes, and relevant causes.
Flow diagrams support transparency but are not substitutes for causal reasoning.
How Can Directed Acyclic Graphs Support a Selection Audit?
A directed acyclic graph, or DAG, represents hypothesized causal relationships among variables.
Researchers can use a DAG to examine whether participation, eligibility, or follow-up depends on exposure-related and outcome-related causes.
For example, a diagram may show that course completion is influenced by both intervention assignment and prior academic preparation.
Conditioning on completion could then open a noncausal path between assignment and preparation.
DAGs do not prove the underlying relationships. They make assumptions explicit and help researchers identify variables that should not be conditioned on without justification.
The diagram should be informed by substantive knowledge rather than constructed solely from observed correlations.
Which Selection Problems Can Statistical Adjustment Address?
Some selection problems can be addressed through weighting, standardization, imputation, likelihood-based methods, or other approaches under appropriate assumptions.
For example, inverse probability weighting may address selective follow-up when relevant predictors of observation are measured, positivity holds, and the required conditional independence assumptions are defensible.
However, unmeasured causes of selection and outcomes can undermine adjustment.
Groups with no probability of inclusion may require external information or extrapolation.
Researchers should not assume that including more covariates in a regression model automatically corrects selection bias.
The appropriate method follows the selection mechanism and estimand.
What Is Positivity, and Why Does It Matter?
Positivity requires that relevant treatment or selection probabilities be nonzero within the covariate profiles needed for identification.
For example, if students without internet access have no chance of entering an online-only study, their outcomes cannot be directly represented by the enrolled sample.
Similarly, if students with particular characteristics never receive the intervention, an observational treatment effect for those profiles may require extrapolation.
Weighting cannot recover information where the relevant probability is truly zero.
Researchers should examine whether the available data provide sufficient overlap for the intended estimate.
How Should Researchers Prioritize Selection Risks?
Not every potential selection mechanism deserves equal attention.
Prioritization should consider the estimand, plausibility of the mechanism, strength of the relevant causal relationships, proportion of the population affected, and availability of information needed to assess or address the problem.
For example, excluding all students without reliable connectivity may be especially consequential when connectivity is central to both intervention access and effectiveness.
A small amount of unrelated administrative missingness may be less consequential under defensible assumptions.
Researchers should avoid ranking threats solely by exclusion percentages.
A relatively small but highly selective loss can sometimes produce greater bias than a larger loss that is unrelated to the relevant outcomes.
What Should Be Prespecified Before Analysis?
Researchers should prespecify the target population, estimand, eligibility criteria, treatment definitions, time zero, primary outcome, follow-up period, and planned analytical inclusion rules.
The protocol should identify important anticipated selection mechanisms and how they will be assessed.
Where missing data or selective observation is expected, specify the primary assumptions, analytical strategy, and relevant sensitivity analyses.
Prespecification does not prevent researchers from investigating unexpected problems.
However, exploratory changes should be distinguished from decisions established before examining the outcomes.
This separation helps reduce the risk that analytical choices are shaped by preferred results.
How Should Selection Bias Limitations Be Reported?
Reports should describe the selection pathway from the intended population to the analytical sample.
Identify sampling-frame limitations, eligibility restrictions, recruitment procedures, consent, follow-up, and analytical exclusions.
Where feasible, report observed differences between included and excluded individuals using ethically and lawfully available information.
Explain the causal assumptions supporting any selection-adjustment method.
Distinguish demonstrated bias from plausible but unverified selection mechanisms.
Conclusions should reflect the population and comparison supported by the actual design and available evidence.
