03 · What You Need to Know
How Researchers Should Evaluate AI Recommendations About Which Findings to Present
What Makes a Research Result Important?
The importance of a research result depends on its relationship to the study's objectives and its contribution to scientific understanding.
A finding may be important because it directly answers the primary research question, challenges an established assumption, identifies a meaningful limitation, or provides evidence relevant to practice or policy.
Statistical significance may contribute to interpretation, but it is not a complete measure of importance.
A small effect estimated precisely may have limited practical relevance. A substantial but uncertain effect may deserve discussion because it identifies a question requiring further investigation.
Similarly, a null finding may be central when the study was designed to evaluate whether an intervention produces a meaningful benefit.
Researchers should therefore distinguish the scientific role of a result from how dramatic it appears.
Why Should Primary Outcomes Usually Receive Priority?
Primary outcomes are ordinarily those designated as central to answering the study's main research question.
In confirmatory research, they may be specified before data collection or analysis through a protocol or preregistration.
These designations matter because they establish which outcomes the study was principally intended to evaluate.
Suppose a study examines whether an online learning intervention improves assessment performance. The primary outcome shows little evidence of improvement, but an exploratory analysis identifies an association with student satisfaction.
An AI assistant might select the satisfaction finding because it appears more positive or statistically striking.
However, presenting it as the study's central result would misrepresent the original research objective.
Researchers should ensure that AI recommendations preserve the priority of prespecified outcomes where relevant.
Can Statistical Significance Determine Which Findings Matter Most?
No. Statistical significance addresses a particular inferential question under specified assumptions. It does not directly measure practical importance, theoretical contribution, or the value of a result for decision-making.
The American Statistical Association's statement on p-values cautions against treating a threshold such as 0.05 as a definitive boundary between important and unimportant findings.
A result with a small p-value may reflect a trivial effect estimated from a large sample. Conversely, a result with a larger p-value may be scientifically informative when considered alongside its effect estimate and uncertainty.
AI systems may nevertheless prioritize findings based on familiar statistical markers because these provide easily recognizable signals.
Researchers should examine the magnitude, precision, context, and relevance of results rather than rank them solely by p-values.
Why Is Effect Size Important?
Effect sizes describe the magnitude of a relationship, difference, or other estimated quantity.
They can help researchers assess whether a finding has substantive importance beyond its statistical detectability.
However, a large effect estimate is not automatically important or reliable.
Its interpretation depends on the measurement scale, research design, uncertainty, and relevant disciplinary context.
For example, a five-point difference in assessment scores may be consequential in one educational setting but relatively minor in another.
AI may identify the largest numerical difference without recognizing the meaning of the scale or the precision of the estimate.
Researchers should therefore evaluate effect magnitude in relation to the research question and the decisions the evidence might inform.
How Should Researchers Compare Different Types of Results?
Different results contribute different forms of evidence.
| Result type |
Potential scientific importance |
Selection concern |
| Primary outcome |
Directly addresses the study's principal objective |
Should not be displaced merely by a more dramatic secondary result |
| Secondary outcome |
Provides additional evidence relevant to the research question |
Must be distinguished from primary analyses |
| Exploratory finding |
May generate hypotheses or identify unexpected patterns |
Should not be presented as prespecified confirmatory evidence |
| Null or inconclusive finding |
May challenge expectations or reveal uncertainty |
Should not be omitted solely because it lacks statistical significance |
| Subgroup result |
May reveal heterogeneity or contextual differences |
Requires attention to sample size, multiplicity, and interaction evidence |
| Sensitivity analysis |
Shows whether conclusions depend on analytical choices |
May be essential when robustness is uncertain |
| Unexpected finding |
May challenge theory or motivate further research |
Requires careful distinction between observation and explanation |
AI can help organize these categories, but the researcher must determine which findings are central to the scientific argument.
Why Can Null Findings Be Important?
A nonsignificant result does not automatically demonstrate that an effect is absent.
It may reflect a small effect, substantial uncertainty, inadequate precision, or other features of the analysis.
Nevertheless, a null or inconclusive finding can be scientifically important when it concerns the study's primary objective.
For example, if a well-designed evaluation finds little evidence that an educational intervention improves learning outcomes, that result may be highly relevant to institutional decision-making.
AI may overlook such findings when instructed to identify the most impressive or compelling results.
Researchers should instead ask whether the finding materially changes what can reasonably be concluded.
Can AI Overprioritize Unexpected Findings?
Yes. Unexpected findings can attract attention because they appear novel or surprising.
However, surprise does not establish reliability.
An unexpected association may arise from chance variation, multiple comparisons, measurement problems, or analytical decisions made after examining the data.
Researchers should distinguish exploratory observations from findings supported by prespecified analyses.
AI may help explain why a result is interesting, but it should not automatically elevate an unexpected pattern into the presentation's central conclusion.
Where the result is exploratory, the presentation should communicate that status explicitly.
How Can Multiple Testing Affect Result Selection?
Studies that examine numerous outcomes, predictors, or subgroups may produce apparently significant results by chance.
When AI ranks findings by their p-values, it may disproportionately select results emerging from extensive exploratory testing.
The interpretation of such findings depends on the analytical design and any relevant procedures for addressing multiplicity.
Researchers should examine whether comparisons were prespecified, whether appropriate adjustments were made, and whether the selected findings remain credible in the broader analytical context.
AI should not treat the smallest p-value as automatically identifying the strongest scientific contribution.
Why Is Selective Reporting a Concern?
Selective reporting occurs when the presentation of research findings is influenced by their direction, magnitude, statistical significance, or other characteristics in ways that misrepresent the overall evidence.
A conference presentation necessarily includes fewer results than a full manuscript. Selection itself is not misconduct.
The concern arises when omitted findings materially change the interpretation of what was investigated and discovered.
For example, presenting only favorable subgroup results while omitting an inconclusive primary outcome may create an inaccurate impression of intervention effectiveness.
Researchers should therefore evaluate the overall pattern of evidence, not merely whether each selected slide contains a technically correct statistic.
Can AI Identify Results With Practical or Policy Relevance?
AI can suggest potential practical implications, but these require contextual judgment.
A statistically detectable difference may not justify changing educational practice if the effect is too small, the intervention is costly, or the evidence is not generalizable to the intended population.
Conversely, a modest improvement may be important when the intervention is inexpensive, scalable, or relevant to an underserved population.
These judgments depend on implementation conditions, opportunity costs, stakeholder priorities, and the strength of the evidence.
AI may help formulate questions about practical relevance, but researchers should not allow it to invent policy implications unsupported by the study.
Should Researchers Present Sensitivity Analyses?
Sensitivity analyses examine how conclusions change under alternative assumptions, specifications, or analytical decisions.
They can be particularly important when the primary result depends on contested methodological choices.
For example, an estimated association may weaken substantially when additional covariates are included or when a different missing-data procedure is applied.
Omitting such information could make the result appear more robust than it is.
Not every sensitivity analysis requires a dedicated slide, but consequential robustness findings should be communicated when they materially affect interpretation.
AI should not automatically exclude sensitivity analyses merely because they appear technical or less visually engaging.
Can AI Select Findings From Qualitative Research?
Yes, but qualitative findings require a different approach from ranking numerical estimates.
In qualitative research, importance may depend on thematic relevance, explanatory depth, variation across participants, and the relationship between findings and the research questions.
A frequently mentioned theme is not necessarily the most analytically significant.
Likewise, an unusual or contradictory account may be important because it challenges an emerging interpretation.
AI may overprioritize themes that appear frequently in summaries or contain easily recognizable terminology.
Researchers should evaluate qualitative findings in relation to the analytical framework, supporting evidence, contextual variation, and the credibility of the interpretation.
Participant quotations should be selected carefully and reproduced accurately, subject to confidentiality and consent requirements.
How Can AI Misrepresent Mixed-Methods Findings?
Mixed-methods studies integrate quantitative and qualitative evidence to address research questions that may not be adequately answered by either approach alone.
AI may select the most prominent quantitative result while treating qualitative findings as decorative examples.
Alternatively, it may emphasize a compelling participant quotation while overlooking quantitative evidence that complicates the interpretation.
The importance of a mixed-methods finding often depends on how the strands of evidence converge, diverge, or complement one another.
Researchers should preserve the intended integration rather than present disconnected findings selected for their individual appeal.
Can AI Help Build a Results Selection Matrix?
Yes. A structured matrix can make the selection process more transparent.
Researchers can list each candidate finding alongside its relationship to the research objectives, analytical status, effect magnitude, uncertainty, and substantive implications.
AI can help organize this information and identify possible omissions or inconsistencies.
However, the model should not assign arbitrary numerical importance scores that create a false impression of objective ranking.
Researchers should use the matrix as a decision aid rather than a substitute for methodological judgment.
How Should Researchers Evaluate AI-Recommended Findings?
The first question is whether each proposed finding directly addresses a research objective or contributes materially to the study's scientific argument.
The second is whether the finding is supported by an appropriate analysis and interpreted within its uncertainty.
Researchers should also consider whether the proposed selection omits contradictory, null, or qualifying evidence.
For presentation purposes, it is useful to distinguish what must appear on the slides from what can be explained orally or included in supplementary materials.
The aim is a coherent presentation that accurately represents the overall evidence rather than a collection of isolated impressive results.
Can AI Help Communicate Findings Without Exaggerating Them?
Yes. AI can help simplify technical descriptions and suggest clearer visual representations.
However, it may also strengthen language in ways that exceed the evidence.
For example, a modest association may become a "major impact," or a statistically uncertain result may be described as a demonstrated improvement.
Researchers should verify that slide titles, annotations, and speaker notes preserve the original statistical interpretation.
The broader challenge of preventing oversimplification in AI-assisted research presentations is closely connected to selecting findings responsibly.
How Does Result Selection Relate to Research Integrity?
Research integrity concerns not only whether reported findings are numerically accurate but also whether the overall presentation represents the evidence fairly.
A presentation can contain individually correct results while creating a misleading impression through selective omission or disproportionate emphasis.
Researchers should therefore evaluate both the accuracy of selected findings and the representativeness of the selection.
AI can assist by comparing proposed slides with the full results section and identifying outcomes that have been omitted.
However, the researcher must determine whether those omissions are justified by the presentation's purpose and time constraints.
Responsible selection involves communicating what the study found, including uncertainty and findings that complicate the preferred narrative.
