03 · What You Need to Know
Why Generative AI Performance Is Uneven Across Languages
Multilingual Capability Does Not Mean Multilingual Equivalence
A generative AI model may be described as multilingual because it can process or generate text in several languages.
However, supporting a language is different from performing equally well in that language.
A model might produce grammatical responses in Filipino while making more factual errors than it does in English. It may translate Korean text competently but struggle with specialized terminology. It may answer routine questions in Arabic while performing inconsistently on complex analytical tasks.
These are possible performance differences rather than universal characteristics of the named languages.
Multilingual capability
The model can process or generate text in more than one language.
Cross-language performance equivalence
The model demonstrates sufficiently comparable performance across languages under defined tasks, metrics, and evaluation conditions.
Researchers should not infer the second from the first.
Training Data Availability Can Influence Language Performance
Large language models learn from substantial collections of text and other training material. The amount, quality, diversity, and relevance of material available for each language can influence model capabilities.
English has historically occupied a prominent position in many publicly documented language-model training resources and evaluation datasets.
Languages with less available digital text may be represented through smaller or less diverse collections. Certain regional varieties, technical registers, and culturally specific forms of communication may also receive limited coverage.
However, the amount of training data is not the only relevant factor. Data quality, model architecture, tokenization, multilingual transfer, instruction tuning, and post-training procedures can also affect performance.
Researchers should therefore avoid assuming that a language with fewer speakers or fewer digital resources will always produce worse results in every model.
The relationship between available language resources and model reliability deserves particular attention when examining AI performance in low-resource languages.
Languages Differ in Linguistic Structure
Languages vary in morphology, syntax, writing systems, word order, and how grammatical relationships are expressed.
These differences can affect how models process language and how researchers should evaluate their outputs.
For example, some languages express grammatical relationships through extensive word inflection, while others rely more heavily on word order or particles.
Languages may also encode politeness, social relationships, evidentiality, or grammatical distinctions differently.
A model may generate grammatically plausible sentences while misunderstanding which distinctions are important for a particular task.
Researchers should be cautious about attributing performance differences solely to linguistic complexity. Languages are not inherently deficient or unsuitable for AI. Uneven performance reflects the interaction between language characteristics, model design, training resources, and evaluation practices.
Tokenization Can Affect How Languages Are Processed
Language models commonly process text through tokens, which may represent whole words, word fragments, characters, or other units.
The number of tokens required to represent similar amounts of information can differ across languages and tokenizers.
For some models, expressing an equivalent message in one language may require more tokens than expressing it in another.
This can influence context-window consumption, computational costs, and the amount of material that fits within a model's input limits.
Tokenization differences do not automatically establish lower reasoning ability or poorer linguistic understanding. Nevertheless, they can create practical disadvantages in multilingual research workflows, particularly when processing long transcripts or documents.
Translation Quality and Reasoning Performance Are Different Capabilities
A model may translate between two languages accurately while performing less reliably when asked to reason directly in one of them.
Conversely, a model may answer familiar factual questions competently while struggling to translate specialized or culturally situated expressions.
Researchers should distinguish among different tasks rather than treating multilingual performance as a single property.
| AI task |
What should be evaluated |
Potential research consequence |
| Translation |
Semantic fidelity, terminology, omissions, cultural meaning |
Distorted research materials |
| Question answering |
Factual accuracy and completeness |
Incorrect research advice |
| Summarization |
Coverage, faithfulness, and preservation of uncertainty |
Misrepresentation of source evidence |
| Qualitative coding |
Conceptual consistency and sensitivity to context |
Biased or incomplete themes |
| Instruction following |
Compliance with task constraints |
Inconsistent workflow execution |
| Technical explanation |
Terminological and disciplinary accuracy |
Misunderstanding of scientific concepts |
A general multilingual benchmark cannot establish reliable performance across all these activities.
English-Language Reasoning Results May Not Generalize
Researchers frequently encounter model evaluations conducted primarily in English.
These evaluations may provide useful evidence about performance on particular tasks, but they do not necessarily predict how the same model will perform when questions, instructions, and supporting documents are presented in another language.
Cross-lingual benchmark studies have identified performance differences across languages and tasks. Their findings also demonstrate why evaluation design matters.
A translated benchmark may introduce its own linguistic ambiguities, and some questions may depend on cultural knowledge rather than language comprehension alone.
Researchers should therefore examine how benchmark items were constructed, whether translations were validated, and whether the evaluation reflects their intended use.
Code-Switching Creates Additional Interpretation Challenges
Multilingual communication does not always occur in neatly separated languages.
In the Philippines, for example, participants may combine Filipino and English within a sentence or conversation. Other multilingual communities use comparable patterns of code-switching.
These language choices may carry conversational, social, or disciplinary meaning.
A model may process a code-switched sentence successfully but misinterpret the relationship between its components or normalize the text into one language while losing important distinctions.
Researchers should examine AI interpretation of code-switching separately from performance on monolingual material.
Success on standard Filipino and English benchmarks does not necessarily establish reliable performance on naturally occurring Taglish interview responses.
Regional Varieties and Dialects May Be Underrepresented
Performance can vary within a language, not only between languages.
A model may handle standardized written language competently while struggling with regional vocabulary, colloquial expressions, or locally specific grammatical patterns.
These differences matter when research involves participants whose everyday language differs from the standardized variety represented in published materials.
Researchers should avoid treating one language label as evidence that all varieties have been adequately evaluated.
Where possible, validation should include authentic examples of the language varieties used by the intended population.
Cultural Knowledge and Language Fluency Are Not Equivalent
A model may generate fluent text in a language without accurately representing the social and cultural context associated with that text.
For example, it may translate a locally specific educational practice into a familiar international category that only partially overlaps with the original concept.
This can affect qualitative interpretation, instrument adaptation, and research communication.
Researchers should distinguish linguistic competence from contextual understanding.
The risk of English-language conceptual assumptions influencing non-English research is especially relevant when AI-generated interpretations are used to construct themes or explain participants' experiences.
Multilingual Benchmarks Provide Useful but Limited Evidence
Multilingual benchmarks can compare model performance across languages using common evaluation tasks.
Examples include multilingual question-answering and language-understanding datasets designed to examine cross-lingual capabilities.
However, benchmark scores depend on task selection, translation quality, scoring procedures, and the populations or language varieties represented.
A model that performs well on a multilingual multiple-choice benchmark may still make consequential errors when interpreting open-ended interview responses.
Benchmark performance should therefore be treated as task-specific evidence rather than a general certificate of multilingual reliability.
Prompt Language Can Influence the Response
Researchers may receive different answers when asking equivalent questions in different languages.
Differences can arise from translation ambiguity, learned language-specific associations, instruction-following behavior, or the model's familiarity with the topic in each language.
For example, a researcher might ask for a methodological recommendation in English and then submit a carefully translated equivalent in Filipino.
If the model recommends different methods, the researcher should investigate whether the prompts are genuinely equivalent and whether the recommendations reflect defensible contextual differences or inconsistent model behavior.
Cross-language consistency is not automatically desirable when the context legitimately differs. The concern arises when equivalent research questions receive materially different treatment without a justified reason.
Multilingual AI Performance Can Affect Research Fairness
Uneven language performance may produce unequal access to reliable AI-assisted research support.
Researchers using languages that receive less accurate explanations, weaker literature recommendations, or poorer translation support may face additional verification burdens.
Within a multilingual study, inconsistent AI performance can also influence how participant responses are classified or interpreted.
If one language group receives more accurate processing than another, observed analytical differences may partly reflect the tool rather than the participants.
This makes multilingual performance evaluation relevant to research validity and fairness, not merely software convenience.
Researchers Should Evaluate the Model for Their Actual Task
Before using generative AI in a multilingual research workflow, researchers should identify the relevant languages, varieties, tasks, and potential consequences of errors.
Evaluation should use representative materials and appropriate reference judgments.
For translation, this may involve bilingual assessment of semantic fidelity and terminology. For qualitative coding, researchers may compare AI-generated classifications with independently developed human coding and examine disagreements.
For factual question answering, verified answers and error analysis may be appropriate.
Where feasible, researchers should evaluate performance separately for each language rather than reporting only an aggregate score that conceals disparities.
These procedures form part of a broader approach to validating AI-assisted multilingual research analysis.
