03 · What You Need to Know
The Competencies That Constitute Research AI Literacy
AI Literacy Is Broader Than Technical Proficiency
Long and Magerko (2020) conceptualize AI literacy as a set of competencies enabling people to critically evaluate AI technologies, communicate and collaborate effectively with AI, and use AI as a tool.
Their work provides a useful foundation, although research-specific AI literacy requires additional attention to scholarly methods, evidence, ethics, and accountability.
For example, a researcher may understand how a language model generates text but lack the statistical knowledge needed to evaluate its analytical recommendations.
Another may be an accomplished statistician but unfamiliar with the confidentiality implications of uploading participant records to an external AI service.
Neither technical familiarity nor disciplinary expertise alone guarantees responsible AI-assisted research.
AI literacy should therefore be understood as a combination of general AI knowledge and competencies applied to specific research responsibilities.
How Does Research AI Literacy Relate to Existing Frameworks?
Several established frameworks provide complementary perspectives.
The UNESCO AI Competency Framework for Students (2024) addresses human-centered understanding, ethics, AI techniques and applications, and AI system design.
The UNESCO AI Competency Framework for Teachers (2024) considers competencies relevant to educational practice and professional development.
The NIST AI Risk Management Framework (2023) provides a governance-oriented approach to identifying and managing AI risks.
The Association of College and Research Libraries' Framework for Information Literacy for Higher Education emphasizes critical engagement with information creation, authority, searching, and scholarly communication.
These frameworks were developed for different purposes and should not be treated as interchangeable.
For research education, they can inform a competency model that integrates AI knowledge with established scholarly responsibilities.
1. Understanding What AI Systems Can and Cannot Do
Researchers should possess a foundational understanding of the systems they use.
This includes recognizing that generative AI can produce plausible information without independently establishing its factual accuracy.
Researchers should also distinguish among systems that generate responses from learned patterns, retrieve information from external sources, execute code, or combine these capabilities.
For example, a model that produces a plausible citation without consulting a database is performing a different operation from a system that retrieves bibliographic records.
However, retrieval does not guarantee accurate interpretation.
AI literacy requires understanding which capabilities were actually used and what limitations remain.
2. Selecting Appropriate AI Tools for Research Tasks
Researchers should be able to evaluate whether AI is suitable for a particular task.
This includes considering the intended output, data sensitivity, reliability requirements, consequences of error, and availability of alternative methods.
For example, a verified statistical function may be preferable to a language model for routine numerical computation.
AI may be useful for suggesting alternative search terms but inappropriate for independently determining participant eligibility without suitable authorization and safeguards.
Researchers should therefore learn when not to use AI, not merely how to use it efficiently.
Tool selection is itself an exercise in methodological and professional judgment.
3. Communicating Effectively With AI Systems
Researchers should understand how to formulate requests that provide relevant context, specify tasks, and clarify desired outputs.
Effective prompting can reduce ambiguity and make AI assistance more useful.
For example, a request for statistical guidance should identify the research question, outcome, predictor variables, data structure, and relevant constraints.
However, detailed prompts do not guarantee correct recommendations.
A system may still misunderstand the task, omit assumptions, or generate unsupported information.
Prompting competence should therefore be evaluated alongside the researcher's ability to verify the resulting output.
The quality of the question does not establish the validity of the answer.
4. Evaluating AI-Generated Scientific Information
Researchers should be able to distinguish supported claims from unsupported assertions.
AI-generated references should be verified using authoritative bibliographic records.
Substantive findings should be checked against original publications.
Methodological explanations should be evaluated using appropriate disciplinary sources.
Researchers should also recognize that a genuine citation may be attached to an inaccurate interpretation.
These competencies support recognition of AI hallucinations and other generated errors.
Verification should be proportionate to the importance of the claim and the potential consequences of error.
5. Maintaining Methodological Judgment
AI may recommend research designs, sampling strategies, measurement instruments, statistical models, and qualitative analytical procedures.
Researchers should be able to determine whether these recommendations fit the research question and methodological assumptions.
For example, a system may recommend an independent-samples statistical test for repeated measurements from the same participants.
The recommendation may sound reasonable while failing to address dependence among observations.
Recognizing the problem requires methodological understanding, not merely AI familiarity.
Research AI literacy therefore depends partly on the researcher's disciplinary competence.
It does not replace the need for methodological education or qualified expert consultation.
6. Understanding Research Data and AI Processing Risks
Researchers should understand the information they provide to AI systems and the implications of processing it.
Research material may include participant records, unpublished manuscripts, institutional documents, or confidential analytical results.
Different systems may have different data-retention, access, and processing arrangements.
Researchers should evaluate these conditions before sharing restricted information.
They should also understand that de-identification may reduce but not necessarily eliminate privacy risks.
AI literacy includes recognizing when data-processing authorization is required and when a proposed workflow should be avoided.
7. Verifying AI-Generated Code and Analytical Outputs
Generative AI can assist with programming, statistical analysis, data cleaning, and visualization.
However, generated code may contain logical errors even when it executes successfully.
For example, an AI-generated script may incorrectly merge datasets using a nonunique identifier, producing duplicated observations.
Researchers should understand how to inspect relevant code, test consequential operations, and evaluate analytical results.
Wilson and colleagues (2017) emphasize documentation, testing, and other good practices in scientific computing.
These principles remain relevant when AI generates the implementation.
Researchers who lack the necessary technical expertise should obtain qualified review rather than assume that a successful execution establishes correctness.
8. Recognizing Automation Bias and Inappropriate Reliance
Researchers should understand how automated recommendations can influence human judgment.
Parasuraman and Manzey (2010) examine automation complacency and bias, including circumstances in which users may monitor automated systems inadequately.
Lee and See (2004) emphasize appropriate reliance on automation.
These perspectives are relevant because researchers may accept AI recommendations more readily when outputs appear confident or have previously been useful.
However, the presence of automation does not necessarily produce bias in every interaction.
AI literacy involves recognizing circumstances that warrant independent judgment and verification.
The broader concern about automation bias in AI-assisted research should therefore be included in research training.
9. Preserving Independent Research Competence
Researchers should retain the ability to understand and defend consequential scientific decisions.
AI may support literature searching, analytical implementation, or manuscript editing, but the researcher should remain able to explain why the selected methods and interpretations are appropriate.
For students, this also involves preserving opportunities to develop foundational skills.
For experienced researchers, it involves recognizing the limits of task-specific expertise and obtaining independent review where necessary.
The objective is independently demonstrable research competence, not the elimination of every technological tool.
10. Understanding Research Integrity and AI Disclosure
Researchers should understand how AI assistance relates to accurate reporting, intellectual contribution, confidentiality, and scholarly accountability.
The European Code of Conduct for Research Integrity identifies reliability, honesty, respect, and accountability as fundamental principles.
These principles remain applicable when AI contributes to research.
Researchers should also know that institutional and publisher disclosure requirements may differ.
The International Committee of Medical Journal Editors provides recommendations concerning AI-assisted technologies and human author responsibility.
Researchers should consult the requirements applicable to their own work rather than assume that a single disclosure statement is universally sufficient.
11. Documenting Consequential AI Contributions
AI literacy includes understanding when research records are necessary to explain automated assistance.
For example, a researcher using AI-generated analytical code may need to preserve the final verified scripts, relevant tests, and methodological decisions.
A researcher using AI to suggest interpretations may need to explain which recommendations were accepted or rejected.
However, complete prompt histories are not universally necessary.
Documentation should reflect research significance, reproducibility needs, applicable requirements, and confidentiality considerations.
The purpose is to make consequential research decisions understandable and accountable.
12. Recognizing Ethical and Social Implications
AI systems may reflect biases associated with training data, design choices, evaluation procedures, and deployment contexts.
Researchers should consider whether these characteristics could affect the populations, concepts, or outcomes involved in their work.
For example, an AI system developed primarily using English-language materials may perform differently when interpreting multilingual research data.
However, the magnitude and direction of such differences depend on the system and task.
Researchers should avoid assuming either universal bias or universal neutrality.
Ethical evaluation should be grounded in the actual research context and relevant evidence.
13. Understanding the Limits of AI-Based Evaluation
Researchers may use AI to review manuscripts, evaluate arguments, detect anomalies, or provide methodological feedback.
These applications can support scholarly work, but automated judgments may contain false positives, false negatives, or unsupported interpretations.
For example, an AI-generated criticism may identify a methodological limitation that does not apply to the study.
Researchers should evaluate such feedback against the original evidence and relevant standards.
AI literacy includes recognizing that automated evaluation is itself an output requiring scrutiny.
It should not automatically become the final authority in scientific decisions.
14. Collaborating Responsibly With AI and Human Experts
Research increasingly involves combinations of human expertise, computational tools, and automated assistance.
Researchers should understand how to allocate tasks according to competence and accountability.
For example, AI may help generate preliminary code, while a qualified analyst verifies its implementation and the research team evaluates its scientific implications.
AI assistance should not obscure the intellectual contributions of human collaborators.
Researchers should also recognize when independent expert advice is necessary.
Competent AI use involves collaboration without surrendering responsibility for the resulting research.
15. Updating AI Knowledge Without Becoming Dependent on Particular Products
AI tools change rapidly. Models, interfaces, retrieval capabilities, privacy arrangements, and institutional policies may evolve.
Researchers should therefore develop transferable competencies rather than rely exclusively on knowledge of a particular platform.
For example, the ability to verify a bibliographic claim remains relevant whether the information comes from a chatbot, a retrieval-enabled assistant, or an AI-powered literature platform.
Similarly, principles of data confidentiality and methodological justification remain applicable across different tools.
AI literacy should support adaptation to technological change while preserving stable scholarly responsibilities.
A Research-Specific AI Literacy Competency Framework
The following framework organizes the preceding competencies into domains relevant to research practice.
| Competency domain |
Core capability |
Evidence of competence |
| Technical understanding |
Recognize AI capabilities and limitations |
Explain what the system did and what remains uncertain |
| Tool selection |
Evaluate whether AI is appropriate for a task |
Justify use, restriction, or non-use |
| AI interaction |
Formulate clear and contextually appropriate requests |
Produce relevant requests without assuming output correctness |
| Information verification |
Evaluate generated claims and references |
Locate independent evidence and identify unsupported information |
| Methodological judgment |
Assess research design and analytical recommendations |
Explain assumptions and justify final decisions |
| Data governance |
Protect confidential and restricted information |
Apply authorization and handling requirements |
| Computational verification |
Evaluate generated code and analytical results |
Inspect, test, or obtain qualified verification |
| Human oversight |
Recognize inappropriate automation reliance |
Challenge consequential recommendations using evidence |
| Research integrity |
Maintain honesty, transparency, and accountability |
Apply applicable disclosure and reporting requirements |
| Independent competence |
Preserve scholarly understanding and judgment |
Explain and defend research decisions without AI as the sole authority |
This is a proposed synthesis for research education, not a formally validated psychometric instrument.
Its domains should be adapted to disciplinary responsibilities and evaluated before being used as the basis for consequential certification or assessment.
Why AI Literacy Cannot Be Measured by Prompting Skill Alone
A researcher may produce sophisticated prompts while accepting inaccurate outputs.
Another may use relatively simple prompts but demonstrate strong verification and methodological judgment.
Prompting proficiency is therefore an incomplete indicator of AI literacy.
Research-specific assessment should examine whether the researcher can select appropriate tools, identify errors, evaluate consequences, and justify decisions.
For example, a researcher who correctly rejects an unsuitable AI recommendation may demonstrate stronger judgment than one who produces an elaborate prompt but accepts the resulting error.
The relevant outcome is responsible and defensible research practice.
How Should Research AI Literacy Be Assessed?
Assessment should align with the competencies being evaluated.
Technical understanding may be assessed through explanations of system capabilities and limitations.
Verification competence may be assessed through exercises involving generated references, methodological claims, or analytical outputs.
Data governance competence may be assessed through scenarios requiring researchers to identify confidentiality restrictions and appropriate processing arrangements.
Human oversight competence may be assessed through tasks requiring researchers to accept, modify, or reject automated recommendations with justification.
No single assessment is likely to capture every domain adequately.
Where scores are used for consequential decisions, the assessment requires evidence supporting its validity and fairness.
Should AI Literacy Expectations Differ Across Research Career Stages?
Beginning researchers may need explicit instruction in foundational AI concepts, research verification, and methodological reasoning.
Advanced doctoral students may be expected to evaluate more complex AI-assisted workflows and explain consequential research decisions.
Experienced researchers may require specialized training in data governance, computational verification, or supervisory responsibilities.
However, academic seniority should not be treated as a universal measure of AI competence.
A senior researcher may be unfamiliar with a particular AI system, while a graduate student may possess substantial technical expertise.
Expectations should therefore reflect task-specific responsibilities and demonstrated competence.
Can AI Literacy Be Developed Without Extensive AI Use?
Some competencies can be developed through conceptual instruction, case analysis, and examination of documented AI outputs.
For example, researchers can learn to evaluate fabricated citations or inappropriate methodological recommendations without continuously interacting with a live AI service.
However, practical experience may be useful for understanding system behavior, limitations, and interaction strategies.
The appropriate balance depends on educational objectives and access to authorized tools.
AI literacy should not be equated with the number of hours spent using AI.
How Does AI Literacy Connect to Research Governance?
Individual competence operates within institutional systems.
Even a highly AI-literate researcher may lack authority to process confidential data through a particular service.
Similarly, a technically competent analyst may be required to obtain ethics approval before introducing AI into participant-facing procedures.
The NIST AI Risk Management Framework emphasizes governance alongside risk identification and management.
Researchers should therefore understand both their own responsibilities and the institutional structures governing AI use.
A research laboratory AI policy can translate these expectations into practical procedures.
Why AI Literacy Is Ultimately About Scientific Judgment
AI can accelerate research activities, but speed does not establish validity.
A generated literature review may be fluent but incomplete. A statistical recommendation may be plausible but unsuitable. An analytical script may execute successfully while producing incorrect transformations.
Researchers need the knowledge and judgment to identify these possibilities.
AI literacy therefore involves understanding how automated assistance interacts with established research responsibilities.
The objective is not technological mastery for its own sake, but the capacity to use AI without surrendering evidential standards, methodological reasoning, or scientific accountability.
