03 · What You Need to Know
Choosing Between a Documentary Census and a Document Sample
What Is Complete Document Analysis?
Complete document analysis involves examining every document that meets the study's eligibility criteria within a defined documentary population or corpus.
It is sometimes described as a census of eligible documents.
For example, a researcher might analyze every official generative AI policy issued by a defined group of universities and effective on a particular reference date.
However, complete analysis does not mean analyzing every document that exists anywhere on the topic.
The scope is established through the research question and eligibility criteria.
Researchers should distinguish complete coverage of a defined corpus from complete coverage of all potentially relevant documents in the world.
What Is Document Sampling?
Document sampling involves selecting a subset of eligible materials for analysis.
The selection may be probability-based, purposive, theoretical, stratified, or otherwise justified by the research design.
For example, a researcher investigating public institutional communication about AI may select documents from different university types, periods, or communication channels.
The resulting sample should provide evidence appropriate to the intended analysis.
However, a sample is not automatically representative simply because documents were selected systematically.
Researchers must distinguish sampling for statistical inference from sampling for contextual or theoretical understanding.
Why the Decision Begins with the Documentary Population
Before deciding whether to sample, researchers must identify what constitutes the population of eligible documents.
For example, a study might concern all official university AI policies within a particular jurisdiction during one academic year.
Alternatively, it might concern all public online discussions about AI in higher education over several years.
The first population may be relatively bounded and manageable. The second may be enormous, dynamic, and difficult to enumerate.
The procedure for defining a document corpus therefore precedes the sampling decision.
Without a clear population, researchers cannot establish what complete analysis would mean or what a sample represents.
When Is Complete Analysis Preferable?
Complete analysis may be appropriate when the number of eligible documents is manageable and each document is relevant to the research question.
For example, a study comparing all officially authorized AI policies across a defined set of universities may benefit from including every eligible policy.
This can avoid excluding documents that contain unusual or contradictory provisions.
It may also support comprehensive description of the defined documentary population.
However, complete analysis remains dependent on the quality of document retrieval, eligibility criteria, and analytical procedures.
Including every available document does not guarantee that the corpus captures every relevant institutional practice.
When Is Sampling Preferable?
Sampling may be preferable when the documentary population is too large for the intended depth of analysis or when a strategically selected subset can address the research question adequately.
For example, a qualitative study examining how institutional communications construct responsibility for AI use may require close reading of documents from contrasting institutional contexts.
Analyzing every public communication may add substantial workload without proportionately improving the interpretation.
However, the sampling strategy should ensure that analytically important differences are not excluded merely for convenience.
Sampling should be justified by the research purpose rather than treated as a shortcut around systematic selection.
How Do Complete Analysis and Sampling Compare?
| Consideration |
Complete analysis |
Document sampling |
| Coverage |
All eligible documents within the defined frame |
Selected subset of eligible documents |
| Feasibility |
Best suited to manageable populations |
Useful for large or open-ended populations |
| Sampling uncertainty |
No document-selection sampling uncertainty within a genuinely complete frame |
Depends on selection procedure and inference |
| Analytical depth |
May be difficult with very large corpora |
Can support intensive analysis of selected materials |
| Selection risk |
Still affected by eligibility and retrieval limitations |
Additionally affected by sample selection |
| Quantitative inference |
Describes the defined complete population |
May support population inference with appropriate probability sampling |
| Qualitative inference |
May support comprehensive contextual interpretation |
May support information-rich and contrasting case analysis |
The appropriate choice depends on the intended claims and the relationship between the documentary frame and the phenomenon.
Does Complete Analysis Eliminate Sampling Bias?
Complete analysis eliminates the need to select a subset from the defined corpus, provided every eligible document is actually included.
However, the corpus itself may be selective.
For example, a study analyzing every publicly accessible university AI policy may exclude institutions whose guidance is available only internally.
The resulting analysis may be complete for publicly accessible documents but incomplete for all institutional guidance.
Researchers should distinguish completeness within the accessible frame from representativeness of the broader documentary population.
A census of an incomplete frame can still support biased conclusions about a wider population.
Can Complete Analysis Still Produce Measurement Bias?
Yes. Including every document does not guarantee accurate coding or interpretation.
For example, a researcher may code whether university policies contain an explicit requirement to disclose AI use.
If the coding definition overlooks equivalent language, the analysis may underestimate how many institutions impose such requirements.
Likewise, documents may differ in structure, terminology, and authority.
Complete coverage does not correct poorly defined analytical categories.
Researchers should distinguish errors arising from document selection from errors arising during measurement or interpretation.
What Is Probability-Based Document Sampling?
Probability sampling selects documents through a procedure in which eligible units have known, nonzero selection probabilities.
Depending on the design, methods may include simple random, systematic, stratified, or cluster sampling.
For example, a researcher might randomly select documents from a complete register of eligible institutional reports.
Probability sampling can support statistical inference to the defined population when the sampling frame, measurement procedures, and analytical assumptions are appropriate.
However, random selection does not correct incomplete document frames or systematic errors in coding.
Researchers should also account for unequal selection probabilities and clustering where relevant.
What Is Purposive Document Sampling?
Purposive sampling selects documents because they are expected to provide relevant information about the phenomenon.
For example, a researcher may select university AI policies representing different governance arrangements, institutional types, or policy-development stages.
This approach can support detailed comparison and contextual interpretation.
However, purposive sampling does not ordinarily justify estimating the prevalence of a policy feature across all universities.
Researchers should explain why the selected documents are information-rich and how they contribute to the research question.
The purpose is analytical relevance rather than statistical representativeness.
What Is Stratified Document Sampling?
Stratified sampling divides the documentary population into relevant categories before selecting documents within them.
For example, a researcher may distinguish public and private universities, different geographical regions, or documents issued during different policy periods.
Stratification can improve coverage of important subgroups and support planned comparisons.
However, the implications depend on how documents are selected within each stratum.
Random selection within strata may support probability-based inference, while purposive selection supports a different analytical purpose.
Researchers should not assume that using strata automatically makes a sample statistically representative.
What Is Theoretical Sampling of Documents?
In some qualitative methodologies, document selection may evolve in response to emerging analytical concepts.
For example, initial analysis of AI governance policies may suggest that institutional responsibility is represented differently in assessment regulations and professional-development guidance.
The researcher may then seek additional documents to examine those emerging distinctions.
This can be appropriate when the methodology supports iterative data collection and conceptual development.
However, theoretical sampling should not be used as a label for arbitrary additions to the corpus.
The selection of additional materials should be linked to explicit analytical questions.
How Should Rare or Unusual Documents Be Treated?
Some documents may contain unusual provisions or represent institutional arrangements that differ substantially from the majority.
For example, one university may explicitly assign responsibility for AI-generated assessment errors to a designated governance committee, while most policies assign responsibility to individual instructors.
Such a document may be particularly informative for qualitative comparison.
However, purposively selecting unusual cases may overrepresent rare arrangements if the researcher later makes prevalence claims.
Researchers should distinguish studying exceptional cases from estimating how common those cases are.
The sampling strategy should reflect that distinction.
How Does Document Length Affect Sampling?
Documents vary substantially in length and complexity.
A short policy statement may contain several important provisions, while a lengthy institutional report may devote only a small section to the research topic.
Researchers should therefore avoid assuming that document count adequately represents the amount of analyzable information.
Sampling may occur at the document level or at the level of sections, provisions, passages, or other defined units.
However, selecting excerpts can remove important context.
The researcher should distinguish the unit of selection from the unit of analysis and explain how contextual meaning is preserved.
Should Every Institution Contribute the Same Number of Documents?
Not necessarily.
Institutions differ in how they organize and publish information.
One university may issue one comprehensive AI policy, while another uses several separate documents.
Artificially restricting each institution to one document may omit relevant requirements.
Conversely, treating every document as an independent institutional case may give greater analytical weight to institutions that publish more materials.
Researchers should decide whether the analytical unit is the institution, document, policy provision, or another entity.
Document counts and institutional counts should not be conflated.
How Does Sampling Affect Quantitative Content Analysis?
Quantitative content analysis may estimate the frequency or prevalence of defined features across documents or institutions.
For example, a researcher might estimate the proportion of university policies containing explicit AI disclosure requirements.
If the study uses a probability sample, estimates should account for the sampling design and relevant uncertainty.
If the study analyzes a complete defined population, the observed proportion describes that population, subject to coverage and measurement limitations.
If documents were purposively selected, prevalence estimates generally should not be generalized statistically to a wider population.
The decision between quantitative, qualitative, and mixed-methods content analysis should therefore be coordinated with document selection.
How Does Sampling Affect Qualitative Document Analysis?
Qualitative document analysis often emphasizes meaning, context, variation, and relationships among documentary materials.
For example, a researcher may examine how different university policies construct student responsibility for AI-generated content.
Purposively selected documents may support close analysis of contrasting institutional approaches.
However, qualitative sampling still requires justification.
Researchers should explain why the selected documents are relevant, what variation they capture, and which perspectives or contexts may be missing.
A small sample is not automatically inadequate, but neither is it automatically sufficient because the analysis is qualitative.
Can Sampling Be Based on Document Availability?
Availability may constrain documentary research, but it should not be confused with a complete sampling rationale.
For example, a researcher may collect policies from universities whose websites are accessible and searchable.
Those institutions may differ systematically from universities with limited public documentation.
Convenience-based selection can be acceptable for narrowly exploratory work when appropriately acknowledged.
However, researchers should avoid presenting the resulting sample as representative of all institutions without supporting evidence.
Access limitations should be documented rather than concealed within broad claims of systematic selection.
What If Documents Are Missing from the Sampling Frame?
Missing documents can affect both complete analysis and sampling.
For example, a university may have internal AI guidance that is not publicly accessible.
A researcher cannot include that document in a public-document census or sampling frame without obtaining access.
However, the document's absence from the frame does not establish that it does not exist.
Researchers should consider how to account for missing or unavailable documents when defining the scope of inference.
Sampling procedures cannot correct unknown coverage gaps automatically.
Can a Sample Be Expanded During Analysis?
Potentially, depending on the research design.
In iterative qualitative research, emerging findings may justify seeking additional documents to examine contrasting cases or refine interpretations.
However, in a probability-based design intended to estimate population characteristics, changing selection procedures after examining results can complicate inference.
Researchers should distinguish planned adaptive sampling from post hoc selection driven by preferred findings.
Changes should be documented and their analytical consequences assessed.
How Do You Determine an Adequate Document Sample Size?
There is no universal minimum number of documents for all documentary studies.
For quantitative analysis, sample size may depend on population size, expected variability, precision requirements, confidence level, and sampling design.
For qualitative analysis, adequacy may depend on the research question, conceptual diversity, contextual richness, and analytical approach.
Researchers should avoid using arbitrary document counts as evidence of rigor.
For example, analyzing 50 policies is not inherently more defensible than analyzing 20 if the larger set was selected inconsistently or lacks relevance.
The justification should explain what the sample is expected to support.
Does Complete Analysis Guarantee Exhaustive Interpretation?
No. Including every eligible document does not mean that every possible meaning, feature, or relationship has been examined.
Analysis remains selective because researchers choose questions, concepts, coding categories, and interpretive procedures.
For example, a complete corpus of AI policies may be analyzed for disclosure requirements without examining every aspect of privacy, accountability, or institutional governance.
The researcher should distinguish complete document inclusion from exhaustive analysis of all conceivable documentary features.
Claims of completeness should specify what is complete.
How Do You Report a Document Sampling Strategy?
The methods section should define the documentary population, eligibility criteria, sampling frame, selection procedure, and final number of documents.
For complete analysis, explain how completeness was established and identify any inaccessible materials.
For probability sampling, report the selection method and relevant implications for estimation.
For purposive sampling, explain the analytical rationale and the variation represented by the selected documents.
Researchers should also describe how duplicates, versions, and documents from the same institution were handled.
Transparent reporting allows readers to evaluate the relationship between document selection and the conclusions drawn.
