What Should Researchers Do When AI Advice Contradicts a Supervisor or Human Expert?

What Should You Do When AI and Your Research Supervisor Give Conflicting Advice?

You ask a generative AI system whether your proposed statistical analysis is appropriate. It recommends a different method and explains why. Your research supervisor, however, has already approved your original approach.

Should you follow the AI recommendation because its explanation sounds technically convincing? Should you disregard it because your supervisor has greater experience and formal responsibility for the project? Or should you raise the disagreement, even if doing so might complicate the supervisory relationship?

This situation is not limited to statistical analysis. Conflicting recommendations may concern research design, theoretical frameworks, data interpretation, literature selection, ethical procedures, or manuscript preparation. The difficulty is that both AI systems and human experts can provide useful insights, but neither is necessarily correct in every instance.

The challenge is to resolve the substantive research question without confusing scientific evidence with professional authority.

Investigate the Disagreement Rather Than Automatically Choosing a Side

How to Evaluate Conflicting AI and Human Research Recommendations

First, Identify Exactly What the Disagreement Concerns

Two recommendations can appear contradictory without addressing precisely the same question.

Suppose your supervisor recommends a conventional statistical test because it is familiar, interpretable, and appropriate for the planned research design. An AI system recommends a more sophisticated model because it detects a possible violation of an assumption.

The apparent conflict may concern several different matters: whether the assumption is actually violated, whether that violation affects the validity of the planned analysis, whether the alternative model fits the study design, or whether the additional complexity is justified.

Before deciding which recommendation to accept, translate the disagreement into a specific, answerable question.

For example, replace "The AI says my supervisor is wrong" with "Does the proposed analysis adequately account for the clustering of observations within participants?"

The second formulation can be investigated through methodological literature, examination of the dataset, and discussion with an appropriate statistical expert.

Distinguish Scientific Claims From Professional and Institutional Decisions

Not every disagreement can be resolved by consulting scientific literature alone.

A supervisor may be offering methodological advice, communicating an institutional requirement, exercising delegated project authority, or expressing a preference based on experience. These are different kinds of recommendations.

An AI system might accurately describe a general research principle while misunderstanding the institutional rules governing a particular study. Conversely, a supervisor may accurately describe local procedures while offering a methodological recommendation that warrants further examination.

These distinctions prevent researchers from treating every disagreement as a competition over who possesses superior knowledge.

Evaluate the Evidence Behind Both Recommendations

The most productive question is not simply who made the recommendation, but what supports it.

Ask the AI system to explain its assumptions, identify possible counterarguments, and suggest authoritative sources. Treat these responses as leads for investigation, not verified evidence. AI-generated references and quotations must be checked against their original sources.

With your supervisor or human expert, ask which methodological principle, study characteristic, empirical finding, or institutional requirement supports the recommendation.

These requests need not imply distrust. Academic supervision routinely involves explaining and defending methodological choices.

The aim is to examine the evidence supporting competing recommendations rather than relying on the confidence with which they are expressed.

Recognize That AI Explanations Can Sound More Certain Than Their Evidence Warrants

Generative AI systems can produce coherent explanations even when their underlying claims are inaccurate or insufficiently supported. Their conversational fluency should not be mistaken for independent validation.

The National Institute of Standards and Technology's AI Risk Management Framework identifies validity, reliability, transparency, and accountability among important considerations in managing AI-related risks.

For researchers, this means that a recommendation should be evaluated in relation to the actual research task. A plausible explanation of a statistical technique is not evidence that the technique is appropriate for your particular data.

An AI system may also overlook details that were absent from the prompt, including sampling procedures, measurement limitations, missing-data mechanisms, or decisions already approved in a research protocol.

When these details matter, the recommendation may be incomplete even if its general methodological description is correct.

Recognize That Human Expertise Is Relevant but Not Infallible

Human experts have advantages that general-purpose AI systems may lack. They can understand the history of a research project, recognize disciplinary conventions, interpret institutional expectations, and accept professional responsibility for decisions.

Nevertheless, expertise is domain-specific. A researcher with extensive experience in qualitative methodology may not have equivalent expertise in advanced longitudinal modeling. An experienced supervisor may also rely on a familiar technique when a newer or less familiar alternative deserves consideration.

Research on expert judgment has demonstrated that confidence and accuracy are not interchangeable. Kahneman and Klein's discussion of conditions for intuitive expertise is particularly relevant: dependable expert intuition requires environments that provide sufficiently regular patterns and opportunities to learn from meaningful feedback.

Consequently, the appropriate weight given to human expertise should reflect its relevance to the specific decision, not merely the expert's seniority or reputation.

Use Independent Evidence to Resolve the Scientific Question

Independent evidence is especially important when the disagreement could materially affect a study's conclusions.

Depending on the question, this may involve consulting original methodological publications, official reporting guidelines, validated software documentation, the research protocol, or a statistician with relevant expertise.

For a statistical disagreement, independent verification might involve inspecting the data structure, testing assumptions, reproducing calculations, or comparing defensible alternative analyses.

For a theoretical disagreement, it may require reviewing the foundational literature and determining whether the competing interpretations are compatible with the study's conceptual framework.

The central principle is that independent evidence should carry substantial weight when evaluating competing scientific claims.

However, evidence does not always produce a single unambiguous answer. Several methods may be defensible, with the choice depending on research objectives, assumptions, and practical constraints.

Discuss the Disagreement Without Turning It Into a Contest

Presenting the issue as "AI says you are wrong" is unlikely to encourage a productive academic discussion. It also places unwarranted authority on an AI-generated response.

A more constructive approach is to describe the specific methodological concern and the evidence that prompted it.

For example:

"I reviewed our planned analysis and found a possible issue with the independence assumption because participants provide repeated observations. Would it be appropriate to examine whether a mixed-effects model would better account for that structure?"

This formulation does not conceal the use of AI where disclosure is relevant. It simply keeps the scientific question at the center of the discussion.

If AI contributed to identifying the concern, explain that contribution when appropriate and follow applicable institutional disclosure rules.

Know When a Disagreement Requires Formal Approval

A scientifically reasonable alternative is not necessarily one that an individual researcher can implement without authorization.

Changes to an approved protocol, participant recruitment procedures, consent arrangements, data handling, or certain analyses may require consultation or formal approval, depending on institutional requirements and the nature of the change.

For example, the World Medical Association's Declaration of Helsinki establishes ethical principles for medical research involving human participants, including responsibilities concerning research protocols and ethics committee review.

Do not use AI advice as a justification for bypassing supervisory, ethics, contractual, or regulatory requirements.

If a disagreement remains unresolved and raises a serious concern about research validity or participant welfare, consider seeking a second qualified opinion or using the institution's established consultation and escalation procedures.

Document Consequential Decisions and Their Rationale

When a disagreement leads to a material change in research design, analysis, interpretation, or reporting, retain a concise record of the issue, evidence examined, people consulted, decision reached, and reasons for that decision.

If AI materially contributed, record the relevant tool information and interaction where institutional policy, reproducibility, or research transparency requires it. Protect confidential information when preserving these records.

This creates an audit trail that can help explain why a research decision changed after AI consultation.

Documentation is particularly valuable when a reviewer later asks why a particular analysis was selected or why an alternative was rejected.

When AI Questions a Supervisor's Recommended Statistical Test

The AI recommendation helped bring attention to a genuine methodological question. It did not establish that a particular alternative was automatically correct.

Equally, the supervisor's initial recommendation did not make the underlying assumption irrelevant. The research decision improved because the disagreement prompted examination of the study design and applicable statistical principles.

Had the supervisor originally meant a repeated-measures ANOVA, the apparent contradiction might have resulted partly from ambiguous terminology rather than an actual methodological error.

Common Mistakes When AI Disagrees With a Human Expert

How to Make a Defensible Decision When Recommendations Conflict

Begin by assessing the potential consequences of the disagreement. A minor wording preference does not warrant the same verification effort as a proposed change that could alter statistical conclusions or affect participant welfare.

For consequential decisions, establish what is disputed, what evidence would resolve the dispute, and who must participate in or authorize the decision.

It is also useful to ask whether the AI has exposed a genuine weakness or merely produced an appealing alternative. That distinction requires checking whether the apparent reasoning problem survives independent scrutiny.

Your objective is not to demonstrate that AI is superior to your supervisor, or that your supervisor is superior to AI. It is to reach a research decision that remains defensible when the source of the recommendation is no longer part of the argument.

What to Check Before Acting on Conflicting Research Advice

Frequently Asked Questions About AI and Supervisor Disagreements

Let Evidence Guide the Science and Appropriate Authority Govern the Decision

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.