Can Generative AI Calculate Effect Sizes Correctly?

Can You Ask AI to Calculate the Effect Size of Your Research Findings?

You have completed a statistical analysis and found a significant difference in students' examination scores. Your research adviser asks you to report an effect size, perhaps Cohen's d, alongside the p-value.

You provide the means, standard deviations, and sample sizes to a generative AI tool. It calculates Cohen's d = 0.65 and describes the result as a medium effect.

But which version of Cohen's d did it calculate? Did it account for paired observations? Was the denominator based on a pooled standard deviation, the standard deviation of paired differences, or another reference distribution?

Effect sizes are often presented as simple numerical summaries, yet different effect-size measures describe different quantities. An AI system can perform the arithmetic correctly while calculating a measure that does not match the research design or intended interpretation.

AI Can Calculate Effect Sizes, but the Exact Measure Must Be Specified

How Generative AI Calculates Effect Sizes and Where Errors Arise

What Is an Effect Size?

An effect size is a numerical measure describing the magnitude of a difference, association, or other statistical quantity.

For example, a researcher might report an eight-point difference in examination scores, a standardized mean difference between instructional groups, or a correlation between study time and academic performance.

Effect sizes help communicate information that a p-value alone does not provide.

A p-value concerns the compatibility of observed evidence with a specified null model. An effect-size estimate describes the magnitude of a particular quantity, although its interpretation depends on the chosen measure and research context.

The American Statistical Association cautions that statistical significance does not measure the size or importance of an effect. Reporting an appropriate effect-size estimate can therefore improve the interpretability of research findings.

Which Effect Sizes Can Generative AI Calculate?

AI systems with computational tools can calculate many common effect-size measures. The appropriate choice depends on the analysis, outcome, and intended comparison.

These measures are not interchangeable. A correlation coefficient of 0.40, Cohen's d of 0.40, and odds ratio of 1.40 describe different statistical quantities.

AI should not select an effect size solely because a familiar reporting convention uses it.

How Does AI Calculate Cohen's d for Two Independent Groups?

One common version of Cohen's d standardizes the difference between two independent-group means using their pooled within-group standard deviation.

The value 0.80 indicates that the observed mean difference is eight-tenths of the pooled within-group standard deviation. It does not mean an 80% improvement or that 80% of participants benefited.

The pooled-standard-deviation formulation is commonly associated with independent groups and a shared within-group variance scale. Other standardization choices may be preferable when the groups have substantially different variances or when a particular reference population is scientifically meaningful.

Why Can AI Calculate Different Values of Cohen's d for the Same Study?

The name Cohen's d is sometimes used without specifying the standardization method. This can be particularly problematic for paired or repeated-measures research.

Consider a study in which the same students complete a pretest and posttest. One standardized effect may divide the mean paired difference by the standard deviation of the paired differences. Another may use a standard deviation derived from the two measurement occasions.

These approaches describe related but different standardized quantities.

For a paired study with a mean difference of 8 and a standard deviation of differences of 10, d_z = 0.80.

However, the same study might produce a different standardized value using another defensible denominator.

Researchers should explicitly identify the effect-size definition and confirm whether AI has correctly recognized paired or independent observations.

Two reported effect sizes should not be treated as directly comparable merely because both are labeled Cohen's d.

What Is the Difference Between Cohen's d and Hedges' g?

Hedges' g commonly refers to a standardized mean difference adjusted to reduce small-sample bias.

For the conventional independent-groups pooled-standard-deviation formulation, the adjustment multiplies Cohen's d by a correction factor that depends on the degrees of freedom.

A commonly used approximation is:

This estimate remains a standardized mean difference, not a percentage. The correction reduces small-sample bias under the relevant model but does not eliminate sampling uncertainty or design-related bias.

Other formulations of Hedges' g may use different standardizers. AI should identify the exact estimator rather than treat every quantity labeled g as identical.

Can AI Calculate Eta-Squared and Partial Eta-Squared Correctly?

AI can calculate variance-explained effect-size measures from appropriate ANOVA output, but it may confuse eta-squared with partial eta-squared.

In a conventional ANOVA decomposition, eta-squared for a term can be expressed as the sum of squares attributable to that term divided by the total sum of squares.

Partial eta-squared instead compares the term's sum of squares with the sum of that term and its associated error sum of squares.

These hypothetical values describe different proportions under the specified decomposition. They do not establish that the factor caused 20% or 30% of the outcome.

In simple one-factor designs, the two measures can coincide. In more complex designs, they may differ substantially.

AI should therefore inspect the ANOVA structure and identify the exact sum-of-squares terms rather than apply a formula mechanically.

Can AI Calculate Effect Sizes From t, F, or Other Test Statistics?

Sometimes. Certain effect sizes can be reconstructed from a test statistic and additional information about the design.

For example, in a conventional paired t-test, d_z can be calculated as t divided by the square root of the number of complete pairs.

In a conventional independent-groups pooled-variance t-test, Cohen's d can be reconstructed using the t statistic and the two group sample sizes.

However, these conversions depend on the exact statistical procedure. A formula appropriate for Student's independent-samples t-test may not apply unchanged to Welch's test or a clustered analysis.

Likewise, an F statistic can sometimes be converted into partial eta-squared when the relevant numerator and denominator degrees of freedom are available.

AI should not assume that a test statistic uniquely determines every effect-size measure.

Researchers should provide the design and test formulation or request that AI identify which additional information is necessary.

Can AI Interpret Cohen's Small, Medium, and Large Effect Sizes?

AI frequently interprets Cohen's d using conventional reference values of approximately 0.20, 0.50, and 0.80 for small, medium, and large standardized mean differences.

These values are familiar conventions, not universal standards of scientific importance.

An effect considered modest in one educational context may be substantial in another. The interpretation depends on the outcome, intervention cost, measurement reliability, baseline conditions, and consequences of the change.

For example, a small improvement in a high-stakes learning outcome may be practically important when an intervention is inexpensive and scalable. A larger standardized difference may be less consequential when the outcome has limited substantive relevance.

Researchers should avoid allowing AI to convert an effect-size threshold directly into a judgment that an intervention was successful.

Effect magnitude should be interpreted using substantive evidence and, where available, meaningful benchmarks from comparable research.

Can AI Calculate Effect Sizes for Regression Results?

Yes, but the term effect size can refer to several different quantities in regression.

An unstandardized regression coefficient describes the modeled change in the outcome associated with a change in a predictor, conditional on the model.

A standardized coefficient expresses a relationship on a standardized scale, but its interpretation depends on the model and standardization method.

For logistic regression, exponentiating a coefficient commonly produces an odds ratio. An odds ratio of 2 indicates twice the odds, not necessarily twice the probability.

Measures such as R-squared, partial R-squared, and other model-fit quantities may also describe aspects of explanatory or predictive performance, but they are not interchangeable with individual coefficient effects.

AI must therefore distinguish the substantive parameter of interest from general model-fit measures.

Researchers should verify that the selected regression model and its coefficient interpretation match the research objective.

Can AI Calculate Confidence Intervals for Effect Sizes?

AI systems with suitable computational tools can calculate confidence intervals for many effect-size measures.

However, interval construction depends on the estimator. Standardized mean differences may require methods based on noncentral distributions, suitable approximations, or bootstrap procedures.

Correlation intervals may use Fisher's z transformation under appropriate conditions, while odds-ratio intervals may be constructed on the log-odds scale and exponentiated.

Simply adding and subtracting 1.96 times an arbitrary standard error is not a universally valid approach.

Researchers should verify the confidence-interval method used by AI and confirm that it corresponds to the reported effect-size estimator.

Reporting uncertainty is particularly important because an apparently large effect may be estimated imprecisely.

Why Can an Accurate Effect Size Still Be Misleading?

A correctly calculated effect size may describe an association that is confounded, a comparison affected by selection bias, or a result based on unreliable measurement.

For example, a standardized difference between students who voluntarily used an educational application and those who did not may reflect differences in motivation or prior achievement.

The effect-size calculation does not establish that the application caused the observed difference.

Likewise, an effect size derived from an inappropriate statistical model may be difficult to interpret even if the numerical conversion is correct.

Researchers should distinguish the accuracy of the calculation from the validity of the scientific interpretation.

How Should Researchers Verify AI-Generated Effect Sizes?

First, identify the effect-size measure and its exact mathematical definition. Confirm the research design, group coding, direction of subtraction, and standardization method.

Next, reproduce the calculation using independently reviewed statistical code or software. Check whether small-sample corrections, missing-data rules, or model-specific settings were applied.

For results that support research conclusions, independent verification of AI-generated statistics should include the effect-size estimate and its uncertainty.

The final interpretation should remain grounded in the research question, outcome scale, study design, and substantive context.

When AI Calculates the Wrong Version of Cohen's d

The example demonstrates why asking AI to "calculate Cohen's d" can be insufficient for repeated-measures research.

The appropriate request identifies the design, the intended effect-size definition, and the quantity used for standardization.

Misconceptions About AI-Generated Effect Sizes

How to Ask AI for Effect Sizes That Match Your Research Design

Before requesting a calculation, identify what you want the effect size to communicate. A raw mean difference may be more interpretable than a standardized measure when the original outcome scale is meaningful.

If a standardized effect is required, specify the definition and denominator rather than asking for a generic Cohen's d.

AI can support the calculations, but the researcher must establish which effect size represents the intended scientific quantity.

When effect sizes are used to compare studies or support a meta-analysis, consistent definitions become especially important. Combining superficially similar but differently standardized estimates can produce misleading comparisons.

Before Reporting Effect Sizes Calculated by AI

Frequently Asked Questions About AI Effect-Size Calculations

AI Can Calculate Effect Sizes, but Their Definitions Determine Their Meaning

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.