Can Poor Reporting Make a Good Study Impossible to Evaluate?

What if the study may be good, but the paper does not give you enough information to tell?

Critical appraisal depends on information. To judge whether a study used appropriate sampling, measurement, procedures, analysis, and safeguards against bias, you need to know what the researchers actually did.

Sometimes a paper does not tell you enough. Randomization is mentioned but not explained. Eligibility criteria are incomplete. Missing data appear in the results without a clear account of how they were handled. An analytical procedure is named, but important implementation details are absent.

This creates a difficult distinction: the research may have been conducted properly, yet the published report may not contain enough evidence for you to establish that.

Poor reporting can make methodological quality impossible to establish

Why reporting quality determines what readers can appraise

Study conduct and study reporting are different things

A useful starting point is to separate what researchers did from what they reported doing. Methodological quality concerns the design and conduct of the research. Reporting quality concerns whether those methods and findings are described completely and transparently enough for others to understand and scrutinize them.

The distinction is not merely theoretical. Research comparing published trial reports with their underlying protocols has found that poorly reported methodological quality did not necessarily reflect poor actual methodological quality. In other words, a publication can make a study look methodologically uncertain because important procedures were inadequately described even when those procedures were appropriately conducted.

Those conclusions should not be treated as equivalent. If a paper does not explain allocation concealment, for example, you cannot safely conclude from silence alone that allocation was not concealed. But neither can you assume that it was.

This is one reason writing and reporting quality should be distinguished from methodological quality. The manuscript is the evidence available to the reader about the underlying study, not a perfect window into everything that occurred.

What information do you need to appraise a study?

The exact requirements depend on the study design. In general, you need enough information to reconstruct the important methodological decisions and determine how those decisions affect the inference being made.

For randomized trials, CONSORT explicitly treats transparent reporting as necessary for readers to evaluate design, conduct, analysis, and findings. Other study designs have their own reporting guidelines. The EQUATOR Network maintains a searchable collection covering many forms of health research.

Missing information creates uncertainty, not automatic proof of poor conduct

This distinction is central to fair critical appraisal. If a paper does not provide enough information to evaluate an important methodological feature, there are at least two possibilities. The researchers may have handled the issue properly but failed to report it, or the underlying conduct may genuinely have been problematic.

The published article alone may not tell you which explanation is correct.

Some formal risk-of-bias frameworks therefore distinguish between evidence of a methodological problem and insufficient information for making a judgment. The Agency for Healthcare Research and Quality's methods guidance, for example, explicitly notes that risk-of-bias assessment depends on sufficiently clear reporting and that poor reporting does not always reflect poor study conduct.

Some missing details matter much more than others

Incomplete reporting is not equally consequential across every piece of information. A missing minor descriptive detail may have little effect on your interpretation. Missing information about a procedure that protects against bias may prevent a confident judgment about the validity of a central result.

Ask what appraisal question the missing information prevents you from answering. If you cannot determine how participants entered the study, you may struggle to evaluate selection processes. If outcome measurement is inadequately described, you may be unable to judge whether the measures were suitable. If the analytical model is poorly documented, you may not be able to determine whether the analysis was appropriate for the design.

The importance of the omission therefore depends partly on the claim you are trying to evaluate.

Reporting guidelines help identify what should be available

Reporting guidelines provide structured recommendations about information that should appear in reports of particular study types. They can therefore be useful during appraisal as a way of noticing consequential omissions.

They should not, however, be confused with methodological-quality instruments. A study can report an inappropriate procedure perfectly. Conversely, researchers may conduct a sound procedure but describe it inadequately. Reporting completeness and methodological rigor are related to appraisal, but they answer different questions.

The journal article may not be the only source of methodological information

Before concluding that an important question is unanswerable, check whether additional documentation exists. Depending on the research design, relevant information may appear in supplementary files, study protocols, preregistrations, trial registries, statistical analysis plans, data or code repositories, methodological appendices, or linked publications.

These sources can sometimes resolve ambiguities that the main article leaves open. They can also reveal discrepancies between planned and reported methods, so they should be treated as evidence to compare rather than merely as missing paragraphs of the journal article.

When the paper does not tell you enough about group assignment

Common mistakes when methods are incompletely reported

Treat unreported methods as unresolved questions

When information is missing, first identify exactly what you cannot determine. Avoid jumping immediately from “unclear” to “wrong.” Then ask whether the uncertainty matters for the claim you intend to use.

This becomes particularly important when deciding whether to rely on a study whose key methodological information remains unavailable. The question is no longer merely whether the reporting is poor. It is what the unresolved uncertainty permits you to conclude.

What to check when reporting is incomplete

Questions about incomplete research reporting

You cannot give a study credit for methods you cannot verify

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.