03 · What You Need to Know
How to Evaluate AI Performance Across Relevant Population Subgroups
Why Overall AI Performance May Be Misleading
Overall performance metrics summarize results across the complete evaluation dataset.
When one population represents most of the observations, its performance contributes more heavily to aggregate measures such as accuracy.
For example, a classifier may perform extremely well for English-language responses while making substantially more errors on responses written in a less represented language.
The combined accuracy may remain high because the larger language group dominates the evaluation sample.
Researchers should therefore distinguish average model performance from performance for the populations relevant to their research.
The problem is illustrated by AI systems that achieve high overall accuracy while performing poorly for minority subgroups.
What Counts as a Relevant Population Subgroup?
A subgroup is a defined subset of the population distinguished by a characteristic relevant to the evaluation.
Depending on the research question, subgroups may involve age, gender, language, disability-related needs, socioeconomic circumstances, institutional setting, or educational program.
For example, an AI model predicting student performance might warrant evaluation across synchronous and asynchronous course delivery modes if those environments produce different learning activity patterns.
However, not every available characteristic is scientifically relevant to every AI task.
Researchers should identify subgroup comparisons based on the intended use, plausible sources of error, and potential consequences.
Collecting sensitive demographic information solely because it is available or analytically convenient may introduce unnecessary ethical and privacy risks.
Subgroup Selection Should Be Justified Before Evaluation
Researchers should establish which subgroup comparisons matter before examining the final model results whenever feasible.
Predefining comparisons can reduce the temptation to emphasize only statistically significant or favorable differences.
For example, a multilingual qualitative classification study may reasonably prioritize language-specific performance because the model processes responses in different languages.
A student-risk prediction model may prioritize course delivery mode if its predictors rely on learning platform activity.
Exploratory subgroup analyses can also be valuable, but researchers should distinguish them from planned evaluations.
Unplanned findings may identify important concerns while requiring further validation before broad conclusions are drawn.
Subgroup Evaluation Begins With Valid Measurement
Performance metrics depend on the outcome or reference classification used to evaluate the AI system.
If the reference outcome is conceptually inappropriate, accurate prediction may not establish meaningful reliability.
For example, defining engagement solely through login frequency may disadvantage asynchronous learners who study offline.
A model might predict this operational definition accurately while failing to represent actual educational engagement.
Researchers should evaluate whether the target construct has comparable meaning across relevant populations.
Where measurement equivalence is uncertain, subgroup performance differences may be difficult to interpret.
Different AI Tasks Require Different Performance Metrics
There is no universal metric suitable for every AI-assisted research task.
For classification, researchers may examine accuracy, sensitivity, specificity, precision, recall, and F1 score.
For probabilistic prediction, calibration and discrimination may be important.
For regression, researchers may examine measures such as mean absolute error, root mean squared error, or systematic prediction error.
For translation, semantic fidelity and preservation of consequential meaning may require qualified linguistic review.
For generative research advice, researchers may evaluate methodological quality, unsupported assumptions, and consistency across matched scenarios.
The metric should reflect the actual function performed by AI and the consequences of errors.
Classification Error Rates Should Be Examined Separately
For binary classification, a confusion matrix distinguishes true positives, true negatives, false positives, and false negatives.
These quantities allow researchers to calculate performance measures that answer different questions.
For example, sensitivity measures the proportion of actual positive cases correctly identified.
Specificity measures the proportion of actual negative cases correctly identified.
Two groups may achieve similar accuracy while exhibiting different sensitivity and specificity.
Researchers should examine the error types most relevant to the intended scientific or practical use.
Subgroup Calibration Matters for Predicted Probabilities
Some AI models produce probabilities rather than only categorical predictions.
Calibration examines whether predicted probabilities correspond to observed outcome frequencies.
For example, among students assigned a predicted failure probability of approximately 30%, a well-calibrated model would be expected to observe failure in roughly 30% of comparable cases.
A model may be calibrated overall while systematically overestimating risk for one subgroup and underestimating it for another.
Researchers should examine subgroup calibration when predicted probabilities influence analyses or interventions.
Small subgroup samples can make calibration estimates unstable, so uncertainty should be considered.
Outcome Prevalence Can Affect Subgroup Comparisons
The proportion of participants experiencing a target outcome may differ across groups.
For example, the prevalence of reported technological barriers may vary between institutions with different infrastructure.
These differences can influence metrics such as precision and overall accuracy.
Researchers should therefore avoid interpreting every performance difference as evidence of model discrimination.
Relevant evaluation may require examining sensitivity, specificity, calibration, or other measures alongside outcome prevalence.
The appropriate interpretation depends on the research objective and the validity of the reference outcome.
Small Subgroup Samples Produce Imprecise Estimates
Subgroup analysis often involves fewer observations than aggregate evaluation.
For example, a model may appear to achieve 80% accuracy in a subgroup containing only 20 cases.
However, changing the classification of two cases would alter the estimate by 10 percentage points.
Researchers should report subgroup sample sizes and suitable measures of uncertainty.
Where relevant, confidence intervals for performance differences can help communicate the precision of comparisons.
A nonsignificant difference does not establish equivalence, particularly when the evaluation has limited statistical power.
Researchers Should Consider Statistical Power and Precision
Subgroup evaluations may be underpowered to detect meaningful differences.
Researchers should consider the expected number of cases and outcome events when planning validation.
For example, estimating a false-negative rate requires sufficient actual positive cases within the subgroup.
A subgroup containing many participants but very few positive outcomes may provide limited information about sensitivity.
Sample size planning should reflect the metric of interest and desired precision rather than relying on one universal minimum.
When adequate evaluation is not feasible, researchers should acknowledge the limitation and avoid claiming that performance is equivalent.
Intersectional Evaluation Can Reveal Hidden Disparities
Performance differences may emerge at the intersection of multiple characteristics.
For example, an AI classifier might perform adequately for working students overall and for Filipino-language respondents overall while performing poorly for working students who respond in Filipino.
Examining only one characteristic at a time could conceal this pattern.
Intersectional evaluation may therefore be relevant when combinations of characteristics plausibly affect model performance.
However, dividing the sample into many small groups increases uncertainty and may create confidentiality concerns.
Researchers should prioritize meaningful comparisons and interpret estimates cautiously.
Subgroup Evaluation Can Reveal Measurement Nonequivalence
A construct may not have the same meaning or measurement properties across populations.
For example, learning platform activity may represent engagement differently in synchronous and asynchronous courses.
Similarly, translated questionnaire items may not function equivalently across language groups.
If the reference measurements differ in meaning, apparent AI performance disparities may partly reflect measurement nonequivalence.
Researchers should investigate whether the measurement process is appropriate before attributing differences solely to the AI model.
This is particularly important when automated classifications are used to compare populations.
Subgroup Performance Differences Do Not Automatically Establish Unfairness
Observed performance disparities are empirical findings that require interpretation.
They may reflect data representation, measurement differences, task difficulty, or other factors.
Whether a disparity constitutes unfairness depends on the intended use, relevant ethical criteria, and consequences of errors.
For example, a small increase in false negatives may be consequential when it prevents students from receiving essential support.
In another low-stakes exploratory task, the same numerical difference may have more limited implications.
The relationship between AI fairness and subgroup performance should therefore be evaluated contextually.
Equal Subgroup Performance Does Not Guarantee Fairness
A model may achieve comparable performance across groups while relying on an inappropriate target variable or supporting inequitable decisions.
For example, a student-risk model may predict an historically unequal outcome equally accurately for every group.
That does not establish that using the predictions to restrict opportunities is justified.
Researchers should distinguish statistical performance from the fairness of the broader decision process.
The limitations of equal AI performance as evidence of fairness should be considered when interpreting subgroup results.
Subgroup Testing Should Include Relevant Deployment Conditions
A model may perform well across groups in its development dataset but behave differently when applied in a new institution or population.
For example, a student-risk model trained on conventional university courses may not generalize adequately to asynchronous-first programs.
Differences in assessment practices, platform use, and student circumstances may affect predictions.
Researchers should evaluate whether the validation data resemble the intended deployment setting.
External validation may be necessary when models are transferred to substantially different contexts.
Performance monitoring may also be appropriate when the underlying conditions change over time.
Human Review Should Complement Subgroup Evaluation
Human reviewers can examine why particular AI outputs appear inappropriate.
However, case-by-case review may not reveal systematic error patterns across groups.
Quantitative subgroup evaluation can identify recurring disparities, while human review can help interpret the underlying cases.
For example, bilingual reviewers may identify why an AI classifier misses indirect criticism in Filipino responses.
Subgroup metrics can then estimate how frequently the problem occurs in an evaluation sample.
Neither procedure independently establishes every relevant aspect of validity or fairness.
Subgroup Results Should Influence Model Use
Identifying a disparity is only the beginning of evaluation.
Researchers should determine whether the observed difference affects the scientific conclusions or decisions supported by the model.
Possible responses include additional validation, revised measurement procedures, improved training data, recalibration, or restrictions on particular uses.
However, corrective interventions can introduce trade-offs and should be evaluated using suitable validation data.
Researchers should avoid modifying the model merely to produce numerically equal subgroup metrics without considering their meaning.
Researchers Should Report Subgroup Findings Transparently
Research reports should identify the subgroups evaluated, the rationale for their selection, and the performance measures used.
Relevant results should include subgroup sample sizes, outcome counts, estimates, and uncertainty.
Researchers should also describe whether comparisons were planned or exploratory.
Where performance limitations affect the intended use, their consequences should be discussed.
Reporting should distinguish observed disparities from hypotheses about their causes.
Transparent subgroup reporting supports more defensible conclusions without implying that every possible population has been evaluated.
