Should Data From Remote and In-Person Participants Be Combined in One Analysis?

Can You Analyze Remote and In-Person Participants Together Without Distorting the Results?

You are conducting a study in which some participants complete questionnaires, assessments, or interviews remotely while others participate in person.

Perhaps the study originally used classroom-based data collection but later introduced an online option. Alternatively, participants may choose their preferred mode because of distance, scheduling, accessibility, or connectivity constraints.

Combining the observations into one dataset seems practical. It may increase sample size, reduce missing representation, and simplify the analysis.

However, participants using different modes may differ systematically. The measurement procedures may also influence scores, response behavior, or data completeness.

Should all observations be pooled? Would including administration mode as a covariate solve the problem? Or should the researcher analyze remote and in-person participants separately?

The decision depends on whether the measurements, populations, and intended statistical inference are sufficiently compatible to support a combined analysis.

Combine Remote and In-Person Data When Pooling Supports the Intended Inference

How to Evaluate Whether Mixed-Mode Participant Data Can Be Pooled

What Is Mixed-Mode Research Data Collection?

Mixed-mode data collection involves obtaining research information through more than one administration or communication mode.

For example, participants may complete the same questionnaire online, on paper, or through an interviewer.

In another study, some participants may complete assessments at home while others attend a research facility.

The term mixed-mode should not be confused with mixed methods research, which combines qualitative and quantitative approaches.

A mixed-mode study may be entirely quantitative or qualitative. The defining feature is variation in how data are collected.

Distinguish Measurement Mode, Recruitment Mode, and Participant Selection

These dimensions can operate independently.

For example, participants recruited by email may complete assessments in person, while participants recruited through institutional visits may subsequently answer questionnaires online.

Researchers should document these processes separately because they can influence sample composition and measurement in different ways.

Why Might Remote and In-Person Participants Differ?

Participants may select a mode according to geographical distance, work schedules, mobility, digital access, privacy preferences, or familiarity with technology.

These characteristics may also be associated with the research outcome.

For example, students choosing remote assessments may differ in commuting distance, employment commitments, or educational circumstances from students attending classroom sessions.

If the groups have different average scores, the difference may reflect participant characteristics, measurement conditions, or both.

Researchers should not interpret an unadjusted difference between modes as evidence that the mode itself caused the difference.

What Conditions Support Combining Data?

These conditions do not imply that every mode difference must be zero. The question is whether the differences are sufficiently understood and compatible with the intended inference.

Does Using the Same Questionnaire Justify Pooling?

Not automatically. Identical questions may produce different response processes under different administration conditions.

For example, participants may answer sensitive questions differently when an interviewer is present than when responding privately online.

Screen size, question presentation, navigation, and accessibility can also affect responses.

Researchers should examine whether the instrument's measurement properties remain sufficiently comparable across modes.

Where the instrument has established mode-equivalence evidence relevant to the population and implementation, pooling may be easier to justify.

What Is the Difference Between Pooling and Statistical Adjustment?

Simple pooling combines observations without explicitly modeling administration mode.

Adjusted pooling combines observations while including mode or relevant associated characteristics in the analytical model.

For example, a regression model may include an indicator identifying whether an assessment was completed remotely or in person.

However, adjustment changes the interpretation of the estimated coefficients and relies on model assumptions.

Researchers should identify the intended estimand before deciding whether mode adjustment is appropriate.

Can Including Mode as a Covariate Solve the Problem?

Sometimes it can address certain observed differences, but it is not a universal solution.

For example, if mode introduces an approximately constant additive shift in a continuous outcome and the model assumptions are suitable, a mode indicator may help account for that shift.

However, measurement differences may vary across participants, outcome levels, or other conditions.

More importantly, if remote and in-person procedures measure different constructs, adding a covariate does not restore conceptual equivalence.

Researchers should distinguish statistical adjustment for an observed association from validation of the underlying measurement.

What If Mode Effects Differ Across Participant Groups?

Mode effects may interact with participant characteristics.

For example, a digital assessment may be more difficult for participants with limited experience using computers, while producing similar results across modes for participants with high digital familiarity.

In such circumstances, a single mode coefficient may not adequately describe the measurement difference.

Researchers may need to investigate interactions or subgroup-specific measurement properties.

However, subgroup comparisons require sufficient data and should be guided by plausible mechanisms rather than indiscriminate exploratory testing.

How Does Measurement Invariance Affect Pooling?

For questionnaires measuring latent constructs, measurement invariance analysis may help determine whether the instrument operates comparably across administration modes.

Researchers may examine factor structures, loadings, intercepts, or thresholds according to the intended comparison.

For example, comparing latent means may require stronger measurement assumptions than examining certain associations.

However, formal factor-analytic invariance testing is not appropriate for every outcome.

Physical measurements, event counts, and other variables may require different validation procedures.

The method should reflect the measurement model and the research question.

Should Researchers Compare Mean Scores Before Pooling?

Comparing means can identify differences worth investigating, but it cannot independently establish whether pooling is valid.

For example, equal mean scores may conceal substantial differences in variance, individual agreement, or item functioning.

Conversely, different means may reflect genuine differences between participants rather than measurement bias.

Researchers should examine the mode assignment process and relevant measurement evidence.

A nonsignificant mean difference is not proof of equivalence.

What If Participants Were Randomly Assigned to Modes?

Random assignment can strengthen the ability to estimate the causal effect of administration mode under appropriate conditions.

For example, participants may be randomly allocated to complete the same assessment online or in person.

Differences in scores can then be interpreted in relation to mode assignment, subject to adherence, missingness, and implementation assumptions.

However, randomization does not automatically establish that the instrument measures the intended construct validly in either mode.

Researchers should distinguish estimating a mode effect from demonstrating measurement validity.

What If Participants Choose Their Preferred Mode?

Self-selection into administration mode creates additional interpretive challenges.

For example, participants with limited mobility may prefer remote assessments, while those with poor connectivity may choose in-person participation.

These characteristics may be related to the outcomes.

Researchers may adjust for observed differences where scientifically justified, but unmeasured selection may remain.

Mode-specific comparisons should therefore not be interpreted causally without an appropriate identification strategy.

Can Pooling Increase Statistical Power?

Combining compatible observations can increase the amount of information available for analysis and may improve precision.

However, the benefit depends on the statistical model, outcome variability, and degree of measurement comparability.

Adding observations collected through a systematically different measurement process may introduce bias that outweighs gains in precision.

A larger pooled dataset is not automatically preferable to a smaller, methodologically coherent dataset.

Researchers should evaluate both precision and validity rather than optimize sample size alone.

What Is a Mode-by-Exposure Interaction?

A mode-by-exposure interaction occurs when the relationship between an exposure or predictor and an outcome differs according to administration mode.

For example, an association between digital literacy and assessment scores may differ between online and in-person testing conditions.

Such differences may reflect measurement effects, participant selection, genuine effect heterogeneity, or a combination of mechanisms.

Researchers should investigate interactions when they are theoretically relevant and supported by adequate data.

However, detecting a statistical interaction does not by itself identify the mechanism producing it.

When Are Stratified Analyses Useful?

Stratified analysis estimates results separately within remote and in-person groups.

This may reveal whether associations or descriptive patterns differ across modes.

For example, researchers may estimate the relationship between study habits and assessment scores separately for remotely and classroom-assessed students.

However, differences between stratum-specific estimates may reflect sampling variation or differences in participant composition.

Stratification can improve transparency but does not automatically resolve confounding or measurement non-equivalence.

When Should Data Be Analyzed Separately?

Separate analyses may be preferable when measurement procedures differ substantially and comparability cannot be established.

For example, a remotely administered performance assessment may lack observations available during an in-person assessment.

If the two scores represent materially different evidence, pooling them under one outcome label may be misleading.

Researchers may instead report mode-specific findings and explain why direct comparison is limited.

However, separate analysis is not automatically superior when modes are sufficiently compatible and pooling would answer the research question more efficiently.

What Is a Sensitivity Analysis for Mode Effects?

Sensitivity analysis examines whether conclusions change under alternative defensible analytical assumptions.

For example, a researcher may compare an unadjusted pooled model, a model including administration mode, and mode-specific analyses.

Agreement among results may increase confidence that the main conclusion is not highly sensitive to the examined modeling choices.

However, similar results do not establish that unmeasured selection or measurement differences are absent.

Sensitivity analyses should address plausible sources of uncertainty rather than simply produce multiple models without a clear rationale.

How Can Missing Data Differ Between Modes?

Remote participants may experience connection failures, incomplete uploads, or interrupted assessments.

In-person participants may face missed appointments, travel difficulties, or other barriers.

The amount and mechanism of missingness may therefore differ across modes.

For example, students with poor connectivity may be more likely to miss online follow-up assessments.

If connectivity is also related to academic outcomes, analyzing only complete cases may distort the results.

Researchers should examine missingness by mode and apply methods consistent with defensible assumptions.

Can Mixed-Mode Collection Improve Population Coverage?

Yes. Offering remote and in-person participation may reduce barriers associated with geographical distance, mobility, connectivity, or scheduling.

However, improved coverage does not automatically establish representativeness.

Participants who choose each mode may still differ systematically, and some groups may remain inaccessible.

Researchers should evaluate whether the combined sample better represents the intended population and whether the measurement modes support comparable outcomes.

The benefits of inclusion should be considered alongside the methodological consequences of mixed-mode measurement.

Can Data Be Pooled When Only Some Variables Are Equivalent?

Potentially. Measurement comparability may differ across outcomes.

For example, demographic information may be collected comparably through remote and in-person questionnaires, while a timed performance measure may be sensitive to device and supervision differences.

Researchers should evaluate each primary variable according to its measurement requirements.

It may be defensible to pool some outcomes while analyzing others separately.

A single decision to combine or separate the entire dataset may be unnecessarily restrictive or insufficiently cautious.

How Should Mixed-Mode Data Be Organized?

The dataset should identify relevant administration modes and important procedural differences.

Researchers may record recruitment mode, assessment mode, date, device type, and significant interruptions where these variables are relevant and appropriately authorized.

Consistent variable definitions, coding, missing-value conventions, and processing procedures should be established.

For longitudinal studies, administration mode may change within participants across measurement occasions.

Researchers should preserve this information rather than assign one permanent mode label when the actual procedure varies over time.

How Should Mode Be Represented in Longitudinal Analysis?

When participants complete assessments through different modes over time, mode may be a time-varying characteristic.

For example, a participant may complete baseline assessment in person and follow-up remotely.

If administration mode affects measurement, an apparent change in the outcome may partly reflect the mode transition.

Researchers should consider whether the design supports separating genuine temporal change from measurement differences.

Repeated-measures models, appropriate validation evidence, or bridging observations may be useful depending on the outcome and available data.

However, statistical modeling cannot automatically recover information lost through incompatible measurement procedures.

What If the Modes Were Introduced at Different Times?

Mode may be confounded with calendar time when a study changes from in-person to remote collection.

For example, all baseline observations may be collected in classrooms, while all follow-up observations are collected online.

If scores change, the researcher may be unable to distinguish temporal or intervention effects from mode-related differences without additional evidence.

Including mode and time in a model may not solve the problem when they are perfectly aligned.

Researchers should recognize this identification limitation and consider whether bridging measurements or a revised design are possible.

Can Researchers Harmonize Scores Across Modes?

Harmonization may be possible when an empirically supported relationship between measurements exists.

For example, a validation study may identify a stable difference between two assessment modes under relevant conditions.

However, a simple conversion may be inadequate when differences vary across score ranges or participant characteristics.

Harmonization procedures should be developed and evaluated using suitable data rather than selected merely to make results appear consistent.

Researchers should preserve the original measurements and document any transformations.

How Should Mixed-Mode Findings Be Reported?

Reports should identify the modes used, the number of participants or observations in each mode, and how mode was assigned or selected.

Researchers should describe relevant instrument adaptations, administration conditions, and evidence supporting comparability.

Analytical decisions concerning pooling, adjustment, stratification, missing data, and sensitivity analysis should be explained.

Where mode-related uncertainty remains, conclusions should acknowledge its possible implications.

Reporting should distinguish evidence of measurement equivalence from the mere absence of statistically significant differences.

When Is Pooling Methodologically Indefensible?

Pooling may be indefensible when the modes produce materially different constructs or measurement properties and no suitable method can establish comparability.

It may also be problematic when mode is inseparable from a key exposure or time period and the resulting ambiguity undermines the intended inference.

For example, a study estimating small intervention effects may be compromised if baseline and follow-up scores are collected through non-equivalent assessment procedures.

Researchers should consider whether separate reporting, a different outcome, additional validation, or a revised research design would be more appropriate.

Deciding Whether to Combine Online and Classroom Assessment Scores

The numbers are hypothetical. The five-point mean difference does not establish whether remote administration changes scores because the groups may differ in participant characteristics.

The larger combined sample may improve precision, but only if the observations support a coherent analytical interpretation.

If the online assessment permits resources unavailable during classroom testing, the scores may partly reflect different performance conditions rather than one common measurement process.

The example illustrates why pooling should be justified by measurement and design evidence, not simply by the availability of compatible spreadsheet columns.

Common Misconceptions About Combining Remote and In-Person Data

How to Choose Between Pooled, Adjusted, and Mode-Specific Analyses

Begin by identifying the intended outcome and the degree of measurement comparability required.

Then examine how participants entered each mode and whether the available data support the assumptions of a combined analysis.

Before pooling, examine whether remote and in-person measurements are equivalent.

For differences in participant composition, review how remote recruitment affects who participates.

When access barriers influence mode choice, consider how digital exclusion introduces sampling bias.

For implementation consistency, examine how to standardize remote data collection across participants.

The decision should preserve a coherent interpretation of the outcome and the intended population rather than maximize the number of records in one analysis.

Before Combining Remote and In-Person Participant Data

Frequently Asked Questions About Combining Remote and In-Person Research Data

Combine Remote and In-Person Data Only When the Measurements Support a Common Interpretation

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.