How Do You Evaluate Local Evidence That Has Weaker Reporting Than International Journal Articles?

What should you do when locally important evidence is poorly reported?

You find a study conducted in exactly the population or setting your research concerns. The topic is highly relevant. Unfortunately, the report barely explains how participants were selected, gives only a few sentences about the instrument, and provides limited information about the analysis.

An international journal article on the same topic may be much easier to evaluate because its methods and results are reported in greater detail. Does that automatically make the international study better evidence? Not necessarily. But neither should local relevance excuse information that you need in order to judge credibility.

The methodological problem is to distinguish poor reporting from poor research while remaining explicit about what the available report does and does not allow you to conclude.

Treat missing information as uncertainty, not as proof of either quality or failure

The first task is to separate research quality from reporting quality

A report is your window into the study, not the study itself

Researchers usually appraise a study through documents describing it. Those documents may be journal articles, reports, theses, dissertations, conference papers, or other records.

A study can be competently conducted but poorly reported. It can also be elegantly reported and methodologically weak. Reporting quality and methodological quality overlap because transparent reporting makes appraisal possible, but they are not identical concepts.

The distinction prevents two opposite errors. You should not declare a study methodologically poor merely because one detail is absent from the report. But you also should not assume that an unreported procedure was carried out correctly.

Start with the appraisal criteria appropriate to the study design

Do not create a special, easier appraisal standard for local research. Use criteria appropriate to the design and question.

A randomized trial raises different validity questions from a cross-sectional survey. A qualitative interview study cannot be evaluated using the same criteria as a cohort study. A diagnostic-accuracy study, programme evaluation, case-control study, and economic analysis similarly require design-appropriate appraisal.

The central questions usually concern whether the methods could have introduced systematic error, whether measurements and analyses were appropriate, and whether the evidence supports the interpretation being made. Exactly how those questions are operationalized depends on the appraisal framework used.

Use three judgments when the report permits them

For each important appraisal domain, distinguish among a problem you can identify, adequate methods you can verify, and insufficient information to make a judgment.

The second category should not be quietly converted into the first, but neither should it be converted into reassurance. Depending on the appraisal tool, the formal labels may differ. Preserve the terminology and decision rules of the framework you are actually using.

Ask whether the missing information matters to the claim you want to use

Not every omitted detail has equal consequences. A missing description of a minor administrative procedure may have little bearing on your use of the findings. Failure to explain participant selection, measurement validity, allocation, attrition, confounding control, or analytic procedures can be much more consequential, depending on the study design.

Focus on information that could materially change your confidence in the claim you intend to make.

For example, if you want to cite a survey as evidence that 70% of a local population holds a particular view, knowing how the sample was obtained matters greatly. If the sampling process is not reported, the percentage may still describe the respondents accurately, but its representativeness cannot simply be assumed.

Look for other reports before accepting the information gap

One study can have multiple reports. A short local journal article may originate from a thesis containing a much fuller methodology. A conference abstract may later become a journal article. A government evaluation may have a technical appendix. A report may be accompanied by a protocol, instrument, data dictionary, or supplementary document.

Search the title, authors, project name, institution, sample, and distinctive study terms. If you find multiple reports, collate them. Cochrane guidance explicitly recommends using multiple reports of a study because secondary reports can contain valuable information about its design and conduct.

This is particularly useful when evidence was discovered through local theses and dissertations, conference materials, or institutional repositories.

Contact authors when the missing information could change an important judgment

Author contact can be useful when an otherwise relevant study lacks a small number of consequential details. Ask focused questions rather than requesting an entire reconstruction of the study.

For example, you might ask how participants were recruited, whether a reported scale had been validated for the study population, how missing observations were handled, or whether the conference abstract has a full report.

Cochrane guidance recommends contacting investigators where appropriate to clarify study eligibility or obtain missing methodological information or results. If the information remains unobtainable, the uncertainty should remain visible in the review.

Do not repair the methods using plausible assumptions

A common temptation is to infer what researchers “probably” did. Perhaps the authors were based at a reputable university. Perhaps the instrument is widely used. Perhaps a thesis committee would have required ethical approval. None of these assumptions tells you what actually happened in the study.

Appraise the evidence available. If the report does not state how sampling was performed, write that the sampling procedure is insufficiently reported. Do not silently supply the most reasonable procedure yourself.

Local relevance does not cancel risk of bias

A study from the exact setting you care about may have unusually high contextual relevance. That is valuable, but it addresses a different dimension of evidence from internal validity.

If a local survey uses a seriously biased sampling procedure, geographic proximity does not repair the sampling problem. If an intervention study has substantial unexplained attrition, local relevance does not restore the missing participants. If a qualitative analysis is inadequately described, being conducted locally does not reveal how the themes were derived.

The appropriate response is to preserve both judgments: this evidence may be highly relevant to the setting and still provide limited confidence for a particular claim.

International publication does not automatically mean better evidence

The reverse shortcut is equally problematic. A study in an internationally indexed journal may be more completely reported, but indexing, journal prestige, and geographic reach are not direct measures of the validity of an individual study.

Compare studies on the methodological features relevant to the question. Better reporting may allow a more confident appraisal, which is genuinely useful. It should not be converted into a blanket assumption that internationally published evidence is methodologically superior.

Use the local study for claims it can actually support

Evidence does not have to be either fully accepted or completely discarded. Sometimes a weakly reported study can support a narrower claim than the authors themselves make.

A descriptive report with unclear sampling may establish that a phenomenon was observed among the participants studied without supporting a prevalence estimate for the entire locality. An inadequately controlled evaluation may document how a programme was implemented without establishing that the programme caused the observed outcome.

This claim-by-claim approach is often more defensible than assigning an informal overall label such as “good study” or “bad study.”

Do not average away uncertainty when synthesizing several weakly reported studies

Several local studies with similar findings may appear reassuring. But repetition does not automatically eliminate shared methodological or reporting limitations.

If all studies use convenience samples, poorly described instruments, or unclear analyses, their numerical agreement does not establish that those limitations have disappeared. Examine whether the studies provide genuinely independent and methodologically credible support for the conclusion.

Where formal evidence-certainty frameworks are appropriate, use their specified criteria rather than inventing an informal score based on the number of local studies.

How to use a locally relevant study without pretending it reports more than it does

Notice what changed. Better reporting did not automatically make the study stronger. It made the methodological judgment more defensible and allowed the researcher to calibrate the claim to what the study could support.

Common mistakes when appraising weakly reported local research

Preserve local relevance while calibrating confidence to what can actually be verified

This approach avoids an unfortunate choice between discarding locally important evidence and lowering methodological standards to keep it. You can acknowledge that a study is uniquely relevant while also being precise about the uncertainty surrounding its findings.

The distinction becomes particularly important when asking whether strong international evidence can compensate for limited local evidence. A small amount of local evidence should not be inflated simply because it is local, but neither should contextual information disappear because the international literature is easier to appraise.

Before drawing conclusions from weakly reported local evidence

Questions about evaluating incompletely reported local studies

Do not punish missing information with assumptions, and do not reward locality with assumptions either

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.