03 · What You Need to Know
Judge the Study Against the Claim It Needs to Support
Not Every Research Question Is a Causal Explanation Question
A common mistake is to evaluate every study as though its purpose were to identify a unique causal explanation. Research serves other legitimate purposes.
A study might estimate how common a phenomenon is, characterize an emerging behavior, develop or evaluate a measure, forecast an outcome, document variation across settings, assess feasibility, estimate an association, replicate a previous result, or generate evidence needed for a later explanatory study.
Consider a survey estimating how university students use generative AI for coursework. The survey may not determine whether AI use improves learning, reduces independent thinking, or is caused by prior academic difficulties. If its actual purpose is to estimate patterns of use, those unresolved causal explanations do not invalidate the descriptive contribution.
Watch Out
A limited design becomes a serious problem when the research question or conclusions silently demand more than the design provides. A cross-sectional association is not defective because it fails to establish causation if causation was never the claim. It becomes problematic when the association is interpreted as though causal direction had been established.
The Importance of an Alternative Depends on the Claim
Suppose students who frequently use an AI tutor have higher examination scores. Several explanations are possible: the tutor improves learning, stronger students use it more effectively, motivated students both use it and study more, or some combination of these processes occurs.
If your claim is simply that AI-tutor use and examination performance are associated in the sampled population, those alternatives do not necessarily invalidate the association itself.
If your claim is that using the AI tutor causes higher examination scores, the same alternatives become central threats to the inference.
If your claim is that the tutor improves scores specifically by increasing retrieval practice, the evidential burden becomes narrower still. You now need evidence capable of distinguishing that mechanism from other ways the intervention might work.
| Intended claim |
What the study needs to establish |
Role of unresolved explanations |
| “X is common in this population.” |
Credible measurement and population estimation |
Causal explanations may be largely irrelevant to the primary claim. |
| “X predicts Y.” |
Reliable predictive performance for the intended setting |
Causal ambiguity may remain acceptable if prediction rather than intervention is the goal. |
| “X is associated with Y.” |
A defensible estimate of the relationship |
Alternative causal explanations limit causal interpretation but need not erase the association. |
| “X causes Y.” |
A defensible causal contrast under explicit assumptions |
Alternatives capable of producing the effect become central threats. |
| “X causes Y through mechanism M.” |
Evidence about the causal effect and the proposed mechanism |
Competing mechanisms become especially important. |
No Single Study Is Expected to Resolve Every Threat
Research designs involve trade-offs. Increasing control over one source of uncertainty may reduce realism or generalizability. Measuring more variables may increase burden and attrition. Adding comparison conditions may improve explanatory discrimination while making recruitment substantially harder. Longer follow-up may improve temporal information while increasing cost and missing data.
Frameworks for evaluating validity have long treated threats as context-dependent problems to prioritize rather than a checklist that every individual study must eliminate simultaneously. The practical task is to identify which threats are most plausible and consequential for the inference being made.
This means that identifying alternative explanations for an expected finding should help you prioritize design decisions. It should not lead to the impossible requirement that every conceivable explanation disappear before research can begin.
Ask Whether the Study Changes What We Know
A useful study should move uncertainty somewhere meaningful.
Suppose researchers already know that X and Y are associated across dozens of similar cross-sectional studies. Conducting another nearly identical study in a conveniently available sample may add little, particularly if the important unresolved question concerns causal direction.
By contrast, if the phenomenon has never been documented in the relevant population, if previous estimates are highly imprecise, if measurement has been poor, or if the new study introduces informative temporal or contextual evidence, a study that still leaves some causal ambiguity may make a worthwhile contribution.
The appropriate comparison is therefore not between your proposed study and a theoretically perfect study. It is between what is currently known and what will be known after your study.
Incremental Evidence Can Be Scientifically Useful
Research often advances cumulatively. One study establishes that a phenomenon occurs. Another tests whether it replicates. A later study improves measurement. Another exploits a stronger design for causal identification. Subsequent work examines mechanisms or boundary conditions.
No individual contribution necessarily answers the entire question.
Replication illustrates this cumulative logic. Repeating research with new data can increase confidence in whether a result recurs, although successful replication does not by itself prove that the original interpretation was correct. A recurring association may still share unresolved biases or theoretical ambiguities if studies repeatedly rely on similar designs.
Different Designs Can Contribute Through Different Weaknesses
Sometimes the most persuasive evidence comes not from one supposedly definitive method but from several approaches with different assumptions and vulnerabilities.
For example, an observational study with rich covariate measurement may address some confounding concerns but remain vulnerable to unmeasured confounding. A quasi-experimental design may reduce dependence on those same measured confounders while introducing different assumptions. An experiment may strengthen causal identification in a constrained setting but provide limited evidence about implementation under ordinary conditions.
If results converge across approaches whose major sources of bias differ, the combined evidence may be more informative than repeated use of one method. This logic is often described as triangulation.
It does not mean that several weak studies automatically become strong when placed together. The value comes from understanding whether their assumptions and potential biases are sufficiently different for convergence to be informative.
Some Unresolved Explanations Matter More Than Others
Imagine a study designed to estimate whether a new instructional program improves examination performance. One alternative explanation is that the treatment group received considerably more instructional time. Another is that an unknown characteristic of the classroom environment might have had a tiny effect.
These alternatives should not necessarily receive equal attention.
Prioritize an alternative when it is plausible, capable of accounting for a substantial portion of the expected result, and consequential for the study's main conclusion. An alternative that could completely reverse the interpretation deserves more design attention than a remote possibility with little substantive consequence.
An Unresolved Explanation Can Become the Next Research Question
Sometimes a study's most useful contribution is to narrow the field of possibilities rather than identify one final explanation.
Suppose an intervention reliably improves performance but the study cannot determine whether the benefit occurs through increased practice, feedback, motivation, or another mechanism. Establishing the intervention effect may still matter. The mechanism question can then become the focus of subsequent research.
What would be inappropriate is claiming that the study established a particular mechanism simply because that mechanism motivated the intervention.
Feasibility Is Part of Research Design, Not an Embarrassing Afterthought
The strongest conceivable design may be financially, ethically, logistically, or practically impossible. Random assignment may be inappropriate. Long-term follow-up may exceed available resources. Rare outcomes may require datasets that one research team cannot assemble. Some variables cannot ethically be manipulated.
A feasible study with clearly bounded claims can be more useful than an ideal study that will never be conducted.
That does not justify choosing whatever design is easiest. The question is whether the feasible design answers something consequential enough to warrant the resources and participation it requires.
Sometimes the Unresolved Alternative Really Is Fatal
There are cases where the study should be redesigned or reconsidered.
If two explanations make identical predictions for everything you plan to observe, yet the entire contribution is supposed to establish that one explanation is correct, the design cannot deliver its promised contribution.
If a known selection process could completely generate the apparent effect, if the key construct is measured in a way that cannot distinguish it from something else, or if the causal direction could plausibly run the other way while the study's sole purpose is directional causal inference, merely listing the issue under “limitations” may not be enough.
In those cases, the unresolved alternative strikes at the central inferential target rather than merely limiting its scope.
04 · A Practical Example
A Useful Study That Cannot Establish Why the Pattern Exists
Hypothetical Example
Generative AI use and assessment performance
A university has little reliable information about how students use generative AI across different academic tasks. A researcher proposes a semester-long study measuring patterns of AI use, assessment performance, prior achievement, and several student characteristics. Because AI use is self-selected, important unmeasured differences between users may remain.
What the study can do Estimate patterns of AI use, describe how use differs across academic activities, and estimate associations between reported use and subsequent performance while accounting for measured characteristics.
What remains unresolved Students are not randomly assigned to use AI. Unmeasured characteristics may influence both AI use and performance, and some forms of reciprocal influence may remain plausible.
When the study is still useful The university currently lacks basic evidence about how AI is being used, the study improves substantially on one-time convenience surveys, and its descriptive and associational findings can inform later hypotheses and stronger designs.
When the same study becomes inadequate If its primary objective is advertised as establishing whether AI use causes improvements or declines in learning, unresolved confounding and self-selection become central obstacles to that claim.
Appropriate conclusion The study reports what patterns were observed, distinguishes prospective from exploratory analyses, and identifies which causal explanations remain unresolved rather than selecting one after seeing the results.
The dataset has not become stronger or weaker simply because the wording changed. What changed is the match between evidence and inference. A study becomes defensible when the question it promises to answer is one the design can actually address.
07 · A Quick Checklist
Check Whether the Study Is Still Worth Doing
Before accepting unresolved explanations, check:
State the primary contribution without relying on claims the design cannot support.
Identify the strongest plausible explanations the study will leave unresolved.
Ask whether any unresolved explanation could overturn the study's central conclusion rather than merely qualify it.
Determine whether a feasible change in design could address the most consequential alternative substantially better.
Compare what is currently known with what researchers will know if the study succeeds.
Check whether the proposed contribution duplicates evidence already available from studies with essentially the same limitations.
Consider whether complementary methods or later studies could address uncertainties that one feasible design cannot resolve.
Plan conclusions that distinguish observed findings from causal or mechanistic interpretations that remain uncertain.
Proceed only if the answerable part of the question is consequential enough to justify the study's time, cost, and participant burden.
