Can Different Recruitment Channels Produce Participant Subgroups That Require Analytical Adjustment?

Should You Adjust Your Analysis When Recruitment Channels Attract Different Participants?

Suppose you recruit participants for a study through university email, social media advertisements, and community organizations. After data collection, you discover that participants recruited through social media are younger, those recruited through community organizations report different experiences, and those responding to university email have higher average educational attainment.

Should you include recruitment channel as a covariate in your statistical model? Should you weight participants differently, analyze each channel separately, or combine everyone without adjustment?

These decisions matter because recruitment channels may shape who enters the sample. Yet statistical adjustment is not automatically beneficial. An inappropriate adjustment can change the research question being answered, reduce precision, or introduce bias.

The central issue is whether recruitment-related differences threaten the particular estimate or relationship the study is designed to investigate.

Recruitment Differences May Require Adjustment, but Only With a Clear Rationale

Understanding Recruitment-Related Differences Before Choosing an Adjustment

Why Recruitment Channels Produce Different Participant Subgroups

Recruitment channels provide access to different segments of a population and may attract individuals with different motivations or circumstances.

For example, clinic recruitment may disproportionately reach people who use healthcare services frequently. Social media recruitment may reach people who are active on particular platforms. Community organizations may reach individuals who have established relationships with those organizations.

These differences can affect demographic characteristics, health status, educational experiences, institutional trust, or the outcomes being investigated.

The mechanisms may involve unequal access to invitations, different probabilities of responding, or eligibility differences across recruitment settings.

Researchers should distinguish the consequences of recruitment methods that influence participant selection from differences that would exist even if all eligible individuals participated.

Importantly, recruitment source is usually an observed characteristic of the enrollment process, not a randomly assigned treatment. An association between recruitment channel and an outcome therefore does not establish that the channel caused the outcome difference.

First Identify the Estimand

An estimand is the quantity the study intends to estimate. It might be the prevalence of a characteristic in a defined population, the mean outcome among eligible participants, or the effect of an intervention under specified conditions.

The appropriate treatment of recruitment channel depends on this objective.

Consider two studies recruiting through the same three channels.

The first aims to estimate the proportion of all university students experiencing difficulties with online learning. Recruitment differences may threaten population representation, especially if participation probabilities are related to those difficulties.

The second aims to investigate whether a particular educational experience is associated with academic engagement among recruited participants. Its primary concern may be whether selection mechanisms distort that association, rather than whether the sample reproduces the university's demographic distribution.

The same recruitment imbalance can therefore have different analytical implications.

Clarifying the estimand before selecting covariates helps prevent adjustment decisions from being driven solely by whichever subgroup differences appear in the dataset.

Recruitment Channel Can Be a Marker of Selection Rather Than Its Cause

Participants recruited through a community organization may report higher levels of social support than participants recruited through social media.

One explanation is that the recruitment channel itself influences responses, perhaps through differences in study framing or data-collection procedures.

Another explanation is that people connected to community organizations already differ in social support before recruitment.

Both mechanisms may operate simultaneously.

Including recruitment channel in a regression model cannot, by itself, distinguish these explanations.

Researchers should therefore develop a plausible account of the selection process, including how participant characteristics, exposure variables, outcomes, and recruitment opportunities may be related.

Conceptual diagrams, including directed acyclic graphs when appropriate, can help clarify assumptions about these relationships.

When Does Recruitment Source Become a Confounding Variable?

Recruitment source is sometimes described informally as a confounder, but that label requires care.

For a causal analysis, confounding concerns alternative pathways that create a noncausal association between an exposure and an outcome. A variable should not be treated as a confounder solely because it is associated with both variables in the observed sample.

Recruitment channel may be a proxy for pre-existing characteristics, a consequence of participant characteristics, or part of the mechanism determining inclusion in the study.

In some designs, adjustment for channel can help account for relevant differences. In others, conditioning on channel can create or amplify bias, particularly when channel membership is influenced by variables related to both the exposure and outcome.

This possibility follows from the broader literature on selection bias and collider stratification.

Researchers should therefore ask what recruitment channel represents in the proposed causal structure before including it as a routine covariate.

Descriptive Comparisons Should Usually Come Before Complex Models

A useful first step is to describe participant characteristics separately by recruitment channel.

Relevant variables may include age, gender where appropriate, geographic location, socioeconomic circumstances, baseline outcomes, exposure characteristics, and other factors central to the research question.

The comparison should emphasize substantively meaningful differences rather than relying exclusively on significance tests.

With a sufficiently large sample, small differences may produce low p-values while having little practical importance. Conversely, substantial differences may remain statistically uncertain in small recruitment subgroups.

Descriptive summaries can reveal patterns requiring investigation without establishing that adjustment is necessary.

The STROBE reporting guidance emphasizes transparent descriptions of participant selection, relevant characteristics, statistical methods, and efforts to address bias. These principles are particularly useful when studies combine recruitment sources. STROBE guidance should be interpreted as reporting guidance rather than a guarantee that any particular adjustment is valid.

When Is Regression Adjustment Appropriate?

Regression adjustment may be appropriate when recruitment source or associated variables are relevant to the estimand and the assumed data-generating process.

For example, a researcher may want to describe an association conditional on recruitment setting because the settings correspond to substantively meaningful differences in the populations studied.

Alternatively, adjustment for measured characteristics may be part of a prespecified strategy to reduce confounding or improve precision.

However, including a recruitment-channel indicator does not necessarily correct unequal selection probabilities.

Regression models fitted to enrolled participants generally describe relationships in the observed sample unless additional assumptions justify broader inference.

Researchers should also consider whether recruitment source is strongly associated with other covariates. Sparse categories, highly correlated predictors, and limited overlap can make adjusted estimates unstable.

The decision should follow the research question and defensible assumptions rather than an automatic rule to include every variable that differs across groups.

When Is Weighting More Appropriate Than Regression Adjustment?

Weighting can be useful when the objective is to estimate characteristics of a target population and researchers have a defensible basis for assigning different contributions to sampled individuals.

In probability sampling, design weights commonly reflect unequal inclusion probabilities, with further adjustments potentially addressing nonresponse or calibration to known population totals.

For nonprobability recruitment, researchers may consider calibration, propensity-based methods, or other model-assisted approaches when suitable external data and assumptions are available.

These methods are not interchangeable, and their validity depends on the selection mechanism, available auxiliary variables, and target population.

Simply assigning equal total weight to every recruitment channel is not generally justified. It would impose a particular channel distribution without establishing that the distribution corresponds to the population or estimand.

Similarly, using the inverse of the number of participants recruited through each channel is not automatically an appropriate sampling weight.

Weighting also requires attention to extreme weights, effective sample size, variance estimation, and whether important population groups are absent from the data.

Can Statistical Adjustment Correct Selection Bias?

Sometimes, but only under appropriate assumptions.

Selection bias may arise when participation depends on variables related to the exposure, outcome, or other relevant factors. Hernán, Hernández-Díaz, and Robins (2004) describe how conditioning on selection can create noncausal associations in epidemiological research.

Adjustment may reduce some selection-related distortions when the relevant determinants are measured and the required assumptions are plausible.

However, recruitment source alone rarely captures the complete selection mechanism.

For instance, if participation depends on unmeasured dissatisfaction with healthcare services, controlling for whether a participant came from a clinic or social media may leave substantial bias unresolved.

When some groups have effectively no opportunity to enter the sample, the absence of coverage creates an additional limitation. Statistical models cannot recover their experiences from observed data without external information and assumptions.

These limitations are especially relevant when social media recruitment excludes particular population groups.

When Should Researchers Analyze Recruitment Channels Separately?

Stratified analysis may be useful when recruitment channels correspond to substantively different populations or when the relationship of interest plausibly varies across recruitment contexts.

For example, an association between digital access and learning engagement might differ between participants recruited through online communities and those recruited through campus-based support services.

However, separate estimates can become imprecise when channel-specific sample sizes are small.

Researchers should distinguish descriptive channel comparisons from formal tests of effect modification. Observing a statistically significant association in one channel and a nonsignificant association in another does not establish that the associations differ.

When effect heterogeneity is a genuine research question, appropriate interaction analyses and uncertainty estimates may be more informative.

These analyses should ideally be prespecified when they are central to the study's objectives.

Recruitment Channel and Data-Collection Mode Must Not Be Confused

Recruitment channel describes how participants were identified or invited. Data-collection mode describes how their responses or measurements were obtained.

A participant recruited through Facebook may complete the same online questionnaire as someone recruited through institutional email.

Alternatively, clinic-recruited participants may complete interviewer-administered questionnaires while social-media participants complete self-administered surveys.

In the second situation, differences in outcomes may reflect participant composition, measurement mode, or both.

If recruitment channel and data-collection mode are perfectly aligned, their separate contributions may not be identifiable from the observed data alone.

Researchers should avoid attributing observed differences exclusively to recruitment source when measurement procedures also differ.

When Is No Adjustment the Better Decision?

Adjustment is not always necessary.

For a qualitative study, recruitment differences may be addressed through sampling rationale, reflexive interpretation, and transparent reporting rather than statistical modeling.

For a randomized trial, recruitment source may be relevant to external validity or precision, but it does not automatically confound the randomized treatment assignment. The appropriate analysis depends on the randomization procedure, prespecified model, and trial estimand.

For some descriptive analyses, reporting channel-specific characteristics and clearly limiting inference may be more defensible than applying unsupported weights.

Researchers should also avoid adding channel indicators merely because an exploratory statistical test found differences among groups.

A model with more covariates is not necessarily less biased, and adjustment can alter the meaning of the reported estimate.

Sensitivity Analyses Can Assess the Robustness of Conclusions

When recruitment differences could affect interpretation, sensitivity analyses may help researchers evaluate how conclusions change under alternative defensible analytical assumptions.

Examples include comparing prespecified adjusted and unadjusted estimates, examining results across recruitment sources, or evaluating the influence of a channel that contributes a distinctive participant subgroup.

Such analyses should be chosen according to the suspected source of bias.

Excluding a recruitment channel may reveal whether results are sensitive to its inclusion, but it does not establish which analysis is unbiased. Likewise, agreement between adjusted and unadjusted estimates does not prove that unmeasured selection bias is absent.

The value of sensitivity analysis lies in clarifying the dependence of conclusions on assumptions and analytical choices.

Deciding Whether to Adjust for Recruitment Source in an Educational Study

Assume the engagement scores are measured on the same scale and the displayed channel means are exact for this hypothetical calculation.

The correctly calculated combined mean is 71.8, not 72.8. The calculation illustrates why channel counts and summary statistics should be checked before interpreting the results.

More importantly, neither the combined mean nor the differences between channel means establish the average engagement score for all students at the university.

The research team must first clarify its objective.

Suppose participants recruited through student support offices report more connectivity difficulties. This observation may help explain their lower engagement scores, but it does not establish that recruitment through support offices caused lower engagement.

Including channel in a regression model could change the estimated association between internet access and engagement. Whether that change represents an improvement depends on the relationships among selection, internet access, engagement, and other relevant variables.

The researchers should not select whichever adjusted model produces the most favorable or statistically significant result.

Instead, they should justify the primary analysis, report relevant descriptive differences, and use sensitivity analyses to evaluate consequential assumptions.

Common Mistakes When Analyzing Recruitment-Related Subgroups

Let the Research Question Determine Whether and How You Adjust

Before analyzing recruitment-related subgroups, write down the quantity or relationship your study intends to estimate. Then examine how recruitment could influence that estimate.

Separate the descriptive observation that groups differ from the methodological claim that those differences introduce bias.

Where feasible, record recruitment source during enrollment and specify relevant analyses in the protocol. Avoid introducing numerous post hoc comparisons merely because channel information is available.

For studies planning multiple recruitment channels to improve sample diversity, the analytical implications should be considered before data collection begins.

The most defensible approach may be a relatively simple analysis supported by a clear sampling rationale. Complexity is useful only when it addresses a genuine methodological problem.

Before Adjusting for Recruitment Channel

Frequently Asked Questions About Recruitment-Channel Adjustment

Adjustment Should Address a Specific Bias, Not Simply a Different Recruitment Source

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.