03 · What You Need to Know
How Analysis Planning Can Diagnose Research Design Problems
Why Can an Analysis Plan Reveal Design Weaknesses?
Research questions are often expressed in broad conceptual language, while statistical analyses require precise definitions of populations, variables, comparisons, and assumptions.
For example, asking whether AI-assisted feedback improves learning appears straightforward.
However, an analysis plan must clarify what improvement means, which learners are included, what comparison represents the absence of the intervention, and how the effect will be estimated.
These decisions expose whether the proposed design actually contains the information required to answer the question.
The analysis plan therefore functions as a diagnostic tool for research design rather than merely a document describing calculations to be performed later.
What Is the Difference Between a Design Problem and an Analysis Problem?
Design problem
The study lacks the observations, comparisons, measurements, timing, or identification conditions necessary to support the intended inference.
Analysis problem
The available information may support the intended inference, but the proposed statistical method is inappropriate, incomplete, or insufficiently specified.
The distinction is not always absolute. An analytical difficulty may reveal a design weakness, and a design weakness may limit the available analytical options.
For example, failure to account for classroom clustering may initially appear to be a statistical modeling problem. If the study includes too few classrooms, however, the limitation may be fundamentally related to design.
What Research Design Problems Can Analytical Planning Expose?
| Problem |
How it becomes visible |
Possible implication |
| Missing comparison group |
The intended causal contrast cannot be identified from the planned observations. |
Revise the comparison strategy or limit causal claims. |
| Unmeasured confounders |
Essential variables for the identification strategy are absent. |
Collect additional information or reconsider causal inference. |
| Inadequate measurement |
The recorded variable does not adequately represent the intended construct. |
Revise instruments or narrow the question. |
| Insufficient independent units |
Clustering or allocation limits effective information. |
Increase clusters, simplify the model, or revise objectives. |
| Incorrect timing |
Measurements do not establish the required temporal sequence. |
Revise follow-up or limit temporal claims. |
| Missing outcome information |
The primary estimand cannot be estimated without strong assumptions. |
Improve measurement and retention planning. |
| Incompatible analytical objectives |
The proposed model answers a different question from the stated objective. |
Align the estimand, analysis, and research question. |
These problems should be evaluated according to their consequences for the intended inference rather than treated as interchangeable technical deficiencies.
Can an Analysis Plan Reveal That a Causal Question Is Unsupported?
Yes. Causal questions require an identification strategy that connects the observed data to the intended causal effect under defensible assumptions.
For example, a study may ask whether using an AI tutoring system improves academic achievement while comparing students who voluntarily use the system with those who do not.
Users and nonusers may differ in prior achievement, motivation, access, or other characteristics that also influence outcomes.
A regression model can adjust for measured variables, but it cannot automatically eliminate unmeasured confounding.
The analysis plan should identify the causal estimand and the assumptions required to estimate it.
If those assumptions are not defensible, the study may need a stronger design or a more limited associational question.
What Is Statistical Identifiability?
Identifiability concerns whether the target quantity can be uniquely determined from the observed-data distribution under the stated model and assumptions.
For example, estimating a causal effect from observational data may require assumptions concerning confounding, consistency, and positivity.
When relevant treatment or exposure groups do not overlap sufficiently across important covariate patterns, some causal comparisons may not be supported by the available data.
Likewise, certain complex models may contain parameters that cannot be separately estimated from the planned observations.
Increasing computational sophistication does not necessarily resolve nonidentifiability.
Researchers should distinguish identifiability from precision: a quantity may be identifiable but estimated imprecisely, or it may remain unidentified regardless of sample size.
Can a Larger Sample Fix an Inadequate Design?
Sometimes, but not always.
A larger sample may improve precision when the design and identification assumptions are otherwise appropriate.
However, increasing the number of observations does not automatically correct confounding, systematic measurement error, or the absence of an appropriate comparison.
For example, collecting data from thousands of voluntary AI-system users may yield precise descriptive estimates without establishing what would have happened had those same students not used the system.
Researchers should identify whether the limitation concerns information quantity, information quality, or the logical basis of inference.
Can Analysis Planning Reveal Missing Variables?
Yes. Specifying the model and identification strategy can identify variables necessary for the intended analysis.
For example, an observational study of digital access and academic performance may require information about prior achievement, socioeconomic circumstances, or institutional conditions.
However, selecting variables solely because they are statistically associated with the outcome is not equivalent to identifying confounders.
Researchers should use substantive knowledge and an appropriate causal framework to determine which variables require measurement.
Some variables may be mediators or colliders, and inappropriate adjustment can introduce rather than remove bias.
Analytical planning should therefore inform the data collection plan before essential information becomes unavailable.
Can an Analysis Plan Reveal That the Outcome Is Poorly Defined?
Yes. An analysis requires an operationally defined outcome, not merely a broad conceptual label.
For example, student engagement might be represented by platform logins, participation in discussions, assignment completion, or self-reported engagement.
These measures may capture different aspects of the construct.
If the research question concerns meaningful learning engagement but the data contain only login counts, the intended interpretation may exceed the measurement evidence.
Researchers should review construct validity and determine whether additional or alternative measurements are necessary.
A statistical model cannot establish construct validity simply by producing a significant coefficient.
Can an Analysis Plan Reveal Problems With Measurement Timing?
Yes. Some research questions require information collected in a particular temporal sequence.
For example, investigating whether an intervention changes academic confidence requires appropriate measurement before and after exposure, or another design capable of supporting the intended comparison.
Measuring exposure and outcome simultaneously may be insufficient to establish their temporal ordering.
Likewise, a study of long-term retention cannot answer its question if outcomes are collected only immediately after instruction.
The analysis plan should specify when variables are measured and how that timing supports the intended inference.
Where the schedule is inadequate, revising the measurement plan may be necessary.
Can an Analysis Plan Reveal That Sample Size Is Inadequate?
Yes. Once the primary estimand and model are specified, researchers can evaluate the sample size needed for acceptable power or precision.
For example, a study may have enough students for an overall comparison but too few classrooms to support a cluster randomized analysis.
Similarly, a rare binary outcome may provide insufficient events for a complex regression model.
Researchers should consider the number of independent units, outcome variability, event frequency, and model complexity.
However, sample size adequacy cannot be determined solely by a universal participant-to-variable ratio.
The appropriate calculation depends on the intended analytical objective and assumptions.
Can Clustering Make a Proposed Analysis Infeasible?
Yes. Clustering may reduce the independent information available and create challenges for statistical inference.
For example, a study with many students but very few schools may have limited ability to estimate school-level effects or support complex multilevel models.
Researchers should identify the allocation, sampling, and measurement units before calculating sample size.
Ignoring clustering can produce misleading standard errors and exaggerated precision.
The consequences should be evaluated during sample size planning for clustering and repeated measures.
Can an Analysis Plan Reveal Problems With Repeated Measures?
Yes. Longitudinal analyses require attention to measurement occasions, within-person dependence, attrition, and the intended change or trajectory.
For example, a study intending to estimate nonlinear learning trajectories may collect too few measurement occasions to support the proposed model.
Alternatively, follow-up intervals may not align with the process being investigated.
Researchers should examine whether the planned data structure can identify the intended longitudinal parameters.
Repeated observations alone do not guarantee that complex temporal models are feasible.
Can Missing Data Make the Research Question Unanswerable?
Potentially. Missingness may be so extensive or consequential that the intended estimand cannot be estimated credibly without strong assumptions.
For example, substantial differential attrition in an intervention study may undermine confidence in a treatment comparison.
Missing-data methods can support inference under specified assumptions, but they do not automatically recover unavailable information.
Researchers should anticipate missingness during design and collect relevant predictors where appropriate.
When plausible missing-data assumptions produce substantially different conclusions, the study may need to acknowledge that its main question cannot be answered with the desired certainty.
Can a Statistical Test Be Correct but the Research Design Still Be Wrong?
Yes. A statistical procedure can be implemented correctly while addressing a question that the study was not designed to answer.
For example, a paired t-test may correctly evaluate the mean difference between pretest and posttest scores among observed participants.
However, the resulting difference does not automatically establish that an intervention caused the change.
History, maturation, testing effects, regression to the mean, and other explanations may remain plausible.
The statistical test estimates a change; the design determines whether that change can be attributed to the intervention.
Researchers should distinguish mathematical correctness from the validity of the intended scientific inference.
What If Several Statistical Models Could Be Used?
Multiple plausible models do not necessarily indicate that the research design is inadequate.
Different methods may estimate the same target quantity under different assumptions or provide complementary perspectives.
However, if the models answer different scientific questions, researchers should clarify which estimand is primary.
For example, an analysis of final scores adjusted for baseline performance and an analysis of change scores may differ in their assumptions and interpretation.
The choice should follow the research question and design rather than the statistical significance of the resulting estimates.
Important alternatives may be evaluated through prespecified sensitivity analyses.
Can Analysis Planning Reveal an Inappropriate Research Design Label?
Yes. Researchers sometimes describe a study using terminology that does not match its actual procedures.
For example, a study may be described as experimental despite lacking researcher-controlled allocation or an appropriate experimental comparison.
Similarly, a cross-sectional survey may be described as longitudinal because participants report experiences from several past periods.
Specifying the analytical units, timing, comparisons, and exposure assignment can reveal these inconsistencies.
Researchers should classify the design according to what the study actually does rather than the statistical test or terminology they prefer.
How Can Researchers Test Whether the Design Supports the Analysis?
A practical approach is to trace the intended inference backward from the statistical result to the data collection procedures.
First, define the research question and target quantity. Then identify the variables, comparisons, timing, and assumptions required to estimate it.
Next, determine whether the proposed design will generate those observations and whether the assumptions are defensible.
Finally, assess whether the planned sample and analysis provide sufficient precision and appropriate uncertainty estimates.
This process can expose weaknesses that remain hidden when researchers begin by choosing a familiar statistical test.
What Is an Analysis-to-Design Alignment Matrix?
An analysis-to-design alignment matrix maps each research question to the information and methodological conditions required to answer it.
| Element |
Question to evaluate |
| Research question |
What scientific claim is intended? |
| Estimand or target quantity |
What precisely must be estimated? |
| Outcome |
Does the proposed measurement represent the intended construct? |
| Comparison |
Does the design provide the contrast required for inference? |
| Covariates |
Are essential variables available and appropriately timed? |
| Sampling |
Does the sample support the intended population inference? |
| Dependence |
Are clustering and repeated measurements addressed? |
| Assumptions |
Are the identification and analytical assumptions defensible? |
| Precision |
Can the planned data support sufficiently informative estimates? |
The matrix is a planning aid rather than a substitute for substantive methodological review.
Should Researchers Redesign the Study When Analysis Planning Reveals Problems?
Redesign may be necessary when the identified problem prevents the study from supporting its central inference.
For example, a causal question may require a different comparison strategy, additional baseline information, or a more defensible allocation procedure.
However, not every analytical difficulty requires abandoning the original design.
Some problems can be addressed through better model specification, additional measurement, or a narrower inferential objective.
The appropriate decision depends on whether the revised approach remains scientifically valid and feasible.
Researchers should consider when to redesign a study because the required analysis is infeasible before collecting data that cannot answer the intended question.
What If Data Collection Has Already Begun?
Researchers should assess which design limitations can still be corrected without introducing unacceptable bias or violating approved procedures.
For example, additional measurements may be possible for future participants, but inconsistent measurement across recruitment periods can create new comparability problems.
Changes may require ethics approval, protocol amendments, or registration updates.
If the intended inference cannot be supported, the research question or claims should be revised transparently.
Researchers should not conceal a design limitation by selecting a statistical method that produces a numerical result.
Can Exploratory Research Proceed Without a Fully Specified Analysis?
Exploratory research may legitimately retain analytical flexibility because its purpose includes identifying patterns and generating hypotheses.
However, the design must still collect information appropriate to its exploratory objectives.
For example, an exploratory survey cannot meaningfully investigate experiences that its instruments do not measure.
Researchers should describe the exploratory purpose and distinguish later hypotheses from prospectively specified confirmatory claims.
Flexibility does not eliminate the need for coherent research questions, valid measurement, or transparent analytical decisions.
How Should Design Limitations Revealed by Analysis Planning Be Documented?
Researchers should record the identified limitation, its implications for the intended inference, alternatives considered, and the resulting decision.
Where the design is revised, the protocol and analysis plan should be updated consistently.
Consequential changes should be dated and justified, particularly when outcome information has already been examined.
If limitations remain, the final report should explain how they affect interpretation rather than present them as minor technical qualifications.
Transparent documentation allows readers to distinguish design constraints from analytical decisions made after observing the results.
Watch Out
A statistical model cannot create a valid comparison, recover unmeasured essential variables, or establish causal identification merely through computational complexity. When the design does not support the intended inference, changing the statistical test may change the calculation without solving the scientific problem.
