03 · What You Need to Know
Why Aggregate AI Accuracy Can Conceal Subgroup Performance Problems
Overall Accuracy Is a Weighted Average of Group Performance
Classification accuracy measures the proportion of predictions that match reference classifications.
When a dataset contains groups of different sizes, the overall accuracy is influenced more heavily by the larger groups.
Suppose a model is evaluated using 900 responses from one group and 100 responses from another.
If the model performs very well for the larger group, its aggregate accuracy may remain high even when performance for the smaller group is substantially worse.
Mathematically, overall accuracy can be expressed as:
Overall accuracy = (Total correct predictions) / (Total evaluated predictions)
For mutually exclusive and exhaustive subgroups, it is also the weighted average of subgroup accuracies, with weights determined by their sample proportions.
Researchers should therefore report subgroup performance when group differences could affect the validity or consequences of the model's use.
Minority Subgroups Can Be Statistically Invisible in Aggregate Results
A minority subgroup may represent a relatively small proportion of the evaluation dataset.
Even a substantial deterioration in its performance may produce only a modest change in overall accuracy.
For example, reducing accuracy from 95% to 65% among a subgroup representing 5% of the evaluation sample would reduce overall accuracy by only 1.5 percentage points if performance for everyone else remained unchanged.
The aggregate change may appear small, but the subgroup experiences a 30-percentage-point decline in accuracy.
Researchers should distinguish the statistical influence of a subgroup on an aggregate metric from the scientific or ethical importance of its errors.
Underrepresentation Can Contribute to Unequal Performance
Some AI systems are developed using datasets that contain fewer examples from particular populations, languages, or institutional contexts.
When relevant variation is inadequately represented, the model may learn patterns that generalize poorly to those groups.
For example, an AI classifier trained predominantly on standardized English-language responses may struggle with regional expressions or code-switching.
However, underrepresentation does not necessarily cause every observed disparity.
Differences may also arise from measurement procedures, label quality, deployment conditions, or other factors.
Researchers should examine training-data bias without assuming that data representation alone explains the results.
Minority Status and Model Underrepresentation Are Not Identical
A minority subgroup may be defined by its smaller size within the study population, a socially marginalized position, or another contextually relevant characteristic.
These meanings should not be conflated.
A group may represent a numerical minority in one dataset while being extensively represented in the model's development data.
Conversely, a numerically large population may be poorly represented in the resources used to develop an AI system.
Researchers should specify whether the concern involves sample size, development-data coverage, social disadvantage, or a combination of these factors.
Clear definitions help prevent unsupported assumptions about the causes of performance differences.
Accuracy Alone Does Not Reveal Which Errors Occur
Two subgroups may have similar accuracy while experiencing different false-positive and false-negative rates.
For example, an AI classifier might overlook expressions of dissatisfaction in one language while incorrectly identifying neutral comments as dissatisfaction in another.
Both errors affect research, but they can distort results differently.
Researchers should examine confusion matrices and task-appropriate measures such as sensitivity, specificity, precision, and recall.
These measures provide information that overall accuracy cannot supply.
False-Negative Rates May Be Particularly Consequential
In some applications, failing to identify a genuine positive case may have substantial consequences.
For example, an educational support model may overlook students who need assistance.
If the model has a higher false-negative rate for a minority subgroup, students in that group may be less likely to receive support through the automated process.
However, the significance of false negatives depends on the intended use.
False positives may also create serious consequences when classifications lead to stigma, restrictions, or inappropriate interventions.
Researchers should evaluate both error types in relation to the decisions they influence.
Outcome Prevalence Can Affect Performance Metrics
The prevalence of the target outcome may differ across groups.
For example, the proportion of students reporting technological difficulties may vary across institutional contexts.
These differences can influence performance measures such as precision and accuracy.
Researchers should avoid attributing every metric disparity to discriminatory model behavior.
They should examine whether the reference outcomes are valid, whether the evaluation samples are comparable, and which metrics are appropriate.
Performance disparities require investigation rather than automatic interpretation.
Calibration May Be Poor for a Minority Subgroup
Some AI models generate predicted probabilities rather than only categorical labels.
Calibration concerns whether predicted probabilities correspond to observed outcome frequencies.
A model may appear well calibrated overall while systematically overestimating or underestimating probabilities for a smaller subgroup.
For example, students assigned a predicted failure risk of 30% might experience substantially different observed failure rates across groups.
Researchers should examine subgroup calibration when probability estimates influence analysis or decisions.
Small samples may limit the precision of these assessments.
Small Subgroup Samples Create Statistical Uncertainty
Minority subgroup evaluation often involves fewer observations.
This can make performance estimates unstable.
For example, an accuracy estimate based on 20 cases may change substantially when only a few predictions differ.
Researchers should report sample sizes and suitable uncertainty estimates rather than presenting subgroup percentages without context.
A small observed disparity may reflect sampling variation, while a large disparity may still require additional evidence to determine its magnitude and cause.
Researchers should avoid treating statistically imprecise estimates as definitive proof of model fairness or unfairness.
Confidence Intervals Can Clarify the Precision of Subgroup Estimates
Confidence intervals provide information about uncertainty under the assumptions of the statistical procedure used.
For example, a subgroup accuracy estimate of 70% based on a small evaluation sample may have a relatively wide confidence interval.
Researchers should select interval estimation methods appropriate to the performance metric and sample structure.
When comparing groups, uncertainty in the difference between their performance measures may be more informative than separate point estimates alone.
Confidence intervals do not correct unrepresentative sampling or invalid reference labels.
They should be interpreted alongside the evaluation design.
Statistical Significance Is Not the Same as Practical Importance
A performance disparity may be statistically detectable but have limited practical consequences in a particular application.
Conversely, a disparity may be consequential even when the available sample is too small to establish conventional statistical significance.
Researchers should examine the magnitude of the difference, uncertainty, affected population, and consequences of errors.
For example, a modest increase in missed cases may matter greatly when the model determines access to essential support.
Bias assessment should not be reduced to whether a statistical test produces a p-value below a conventional threshold.
Intersectional Groups May Experience Hidden Performance Disparities
Evaluating one demographic characteristic at a time may conceal differences affecting combinations of characteristics.
For example, a model may perform adequately for working students overall and for Filipino-language respondents overall while performing poorly for working students who respond in Filipino.
Such patterns may warrant intersectional evaluation.
However, dividing a dataset into many small groups can produce unstable estimates and increase confidentiality concerns.
Researchers should select meaningful comparisons and consider whether sufficient data are available.
Intersectional analysis should be guided by the research question and potential consequences rather than performed mechanically.
Subgroup Evaluation Requires Appropriate Reference Labels
Performance metrics depend on the quality of the reference classifications used for evaluation.
If reference labels are systematically inaccurate for a minority subgroup, the resulting accuracy estimates may be misleading.
For example, reviewers unfamiliar with a language variety may incorrectly label indirect expressions of dissatisfaction.
An AI model evaluated against those labels could appear accurate while reproducing the same misunderstanding.
Researchers should examine reference-label validity and use appropriately qualified reviewers.
Human agreement alone does not establish that the labels represent the intended construct.
Dataset Shift Can Affect Minority Subgroup Performance
A model may perform well during development but become less reliable when applied to a new population or setting.
For example, an educational prediction model developed using conventional classroom data may perform differently for students enrolled in asynchronous-first programs.
Changes in course structure, assessment practices, and platform behavior may alter relevant relationships.
Researchers should evaluate whether the deployment context matches the conditions represented in the development data.
External validation may be necessary when models are transferred to new institutions or populations.
Unequal Performance Does Not Automatically Establish Unfairness
A model may perform differently across groups for several reasons.
Some differences may reflect measurement problems or inadequate representation. Others may arise from differences in outcome prevalence, task difficulty, or evaluation procedures.
Researchers should investigate whether the disparity produces unjustified disadvantages or undermines the intended scientific use.
The broader question of AI fairness in research requires both empirical evaluation and normative judgment.
A numerical difference is evidence to examine, not a complete ethical conclusion.
Equal Performance Is Not Always Sufficient Either
Even when subgroup performance is similar, the model may rely on invalid measurements or support inappropriate decisions.
For example, an AI system may classify student engagement consistently across groups while defining engagement through login frequency alone.
This could disadvantage learners whose study practices are not adequately represented by the measure.
Researchers should therefore consider whether equal AI performance establishes fairness before drawing broad conclusions.
Performance parity is one part of evaluation, not a substitute for construct validity and contextual analysis.
How Can Researchers Evaluate Minority Subgroup Performance?
Researchers should identify relevant groups and establish the scientific rationale for comparing them.
They should then construct suitable evaluation samples, define valid reference outcomes, and calculate task-appropriate performance measures.
Relevant analyses may include subgroup accuracy, sensitivity, specificity, calibration, and error-rate differences.
Researchers should report sample sizes, uncertainty, and any limitations associated with the evaluation data.
Where appropriate, they may investigate whether observed disparities remain after considering relevant contextual or measurement differences.
The procedures for testing AI-assisted research workflows for bias should be adapted to the task rather than applied as a universal checklist of metrics.
Mitigation Should Address the Identified Problem
Possible responses include improving evaluation coverage, revising measurement procedures, correcting reference labels, recalibrating predictions, or developing alternative models.
However, different interventions address different sources of error.
Adding more subgroup examples may not resolve an invalid outcome definition.
Changing classification thresholds may reduce one error type while increasing another.
Researchers should evaluate the effects and trade-offs of mitigation using suitable validation data.
Improvement in one subgroup metric does not establish that every relevant fairness concern has been resolved.
Researchers Should Report Subgroup Limitations Transparently
Scientific reports should distinguish aggregate model performance from subgroup-specific evidence.
Where relevant, researchers should provide subgroup sample sizes, outcome counts, performance measures, uncertainty, and identified limitations.
If the evaluation sample is too small to support reliable conclusions for a particular group, that limitation should be acknowledged.
Researchers should avoid claiming that a model performs equally well for everyone when relevant populations were not adequately evaluated.
Transparency supports appropriate interpretation and helps prevent broad reliability claims from exceeding the available evidence.
