What Should You Know About a Dataset Before Building a Study Around It?

Is the Dataset You Found Actually Suitable for the Study You Want to Conduct?

You have identified an existing dataset containing variables that appear relevant to your research question. Perhaps it includes students' academic performance, learning management system activity, demographic characteristics, or participation in technology-supported instruction.

The dataset looks promising. It contains thousands of observations, several potentially useful variables, and enough information to begin statistical analysis.

But what do those variables actually measure? Who was included in the dataset, and who was excluded? Were the records collected consistently? Are important values missing? Can you legally and ethically use the information for your proposed study?

More importantly, does the dataset contain the evidence needed to answer your research question, or does it merely contain variables with familiar names?

These questions should be resolved before the study is built around the dataset. Otherwise, researchers may invest substantial effort in an analysis whose central limitations were already present in the original data.

Evaluate the Dataset's Origin, Measurements, Coverage, Structure, and Fitness for Purpose

The Essential Questions to Ask Before Using an Existing Dataset

Why Was the Dataset Originally Created?

The original purpose of a dataset influences what information was collected, which observations were retained, and how variables were defined.

For example, a university may collect learning management system logs to support instructional operations rather than to measure student engagement for research.

Those logs may accurately record selected platform events without adequately representing students' cognitive or behavioral engagement.

Similarly, administrative records may be designed for reporting, billing, compliance, or institutional planning.

Researchers should identify the original collection objectives and determine whether they align sufficiently with the new research question.

The more detailed issue of assessing alignment between a dataset's original purpose and a new research question should be considered before selecting variables or analytical models.

Who Created and Maintains the Dataset?

Dataset provenance concerns the source, history, and circumstances under which the data were generated and maintained.

Researchers should identify the collecting organization, responsible investigators or administrators, data custodians, and relevant documentation.

For example, a national statistical agency may provide detailed sampling documentation, while a locally maintained administrative dataset may require consultation with institutional personnel to understand recording practices.

Neither source should be judged solely by institutional prestige.

Credibility depends on the collection procedures, documentation, quality controls, and suitability of the evidence.

Researchers should also establish whether the dataset is an original release, a revised version, or a transformed extract.

What Population Does the Dataset Represent?

The dataset's population determines which individuals, institutions, events, or other units could have contributed observations.

For example, a dataset containing students enrolled in online courses at one university does not automatically represent all university students.

Researchers should examine the target population, sampling frame, inclusion criteria, exclusion criteria, recruitment procedures, and participation patterns.

Where probability sampling was used, the sampling design and available weights may be important for estimation.

Where administrative records are used, coverage may depend on institutional systems and operational rules.

A large number of records does not establish that the dataset represents the population the researcher wants to study.

What Is the Unit of Observation?

The unit of observation is the entity represented by an individual record or measurement.

It may be a student, course, institution, transaction, assessment, platform event, or another defined unit.

For example, one learning analytics dataset may contain one row per student, while another contains one row per platform event.

These structures support different analyses.

Researchers should determine whether records are independent, repeated within individuals, or nested within courses or institutions.

Confusing event counts with student counts can substantially distort descriptive results and statistical uncertainty.

The unit of observation should be distinguished from the unit about which the research ultimately intends to make conclusions.

What Does Each Variable Actually Mean?

Variable names do not provide sufficient evidence of measurement meaning.

For example, a variable labeled participation might refer to attendance, login frequency, assignment submission, discussion activity, or a composite institutional score.

These operational definitions are not interchangeable.

Researchers should examine the data dictionary, codebook, collection instruments, and technical documentation to determine what each variable records.

They should identify measurement scales, reference periods, coding rules, units, and relevant transformations.

The methodological question of whether variables in secondary data were measured appropriately should be answered before interpreting statistical relationships.

What Is a Data Dictionary?

A data dictionary describes the variables and data structures within a dataset.

It may include variable names, definitions, data types, permissible values, units of measurement, and coding conventions.

For example, a variable named grade_final may contain a numerical course grade, a categorical classification, or a standardized score.

The data dictionary should clarify its intended meaning.

However, dictionaries vary in completeness and may not explain how the underlying measurements were originally obtained.

Researchers may need to consult additional instruments or collection protocols.

What Is a Codebook, and Why Is It Important?

A codebook documents how variables and values are defined and represented.

For survey datasets, it may contain question wording, response categories, skip patterns, and codes for missing responses.

For administrative data, it may describe operational classifications and institutional recording rules.

For example, the value 9 might represent a legitimate response category in one variable but indicate missing information in another.

Without documentation, researchers may mistakenly treat missing-value codes as substantive observations.

Codebooks are therefore essential for interpreting data accurately and constructing defensible analytical variables.

What Time Period Does the Dataset Cover?

Researchers should establish when observations were collected and which periods the recorded information represents.

A dataset released recently may contain measurements collected several years earlier.

Conversely, a dataset originally created long ago may contain updated observations.

For example, records collected before widespread institutional adoption of generative AI may be unsuitable for estimating contemporary AI-assisted learning practices.

However, those same records may be valuable for historical comparison.

Temporal suitability depends on the research question rather than dataset age alone.

Is the Dataset Cross-Sectional or Longitudinal?

A cross-sectional dataset generally represents observations associated with a defined point or period, while longitudinal data contain repeated measurements across time.

Researchers should inspect the actual record structure rather than rely solely on a dataset description.

For example, several years of course records do not necessarily constitute a student-level longitudinal dataset if students cannot be linked reliably across periods.

Longitudinal analysis may require stable identifiers, comparable measurements, appropriate timing, and information about attrition.

Researchers should verify whether the data structure supports the intended examination of change or temporal relationships.

How Complete Are the Data?

Missing values may arise from nonresponse, technical failure, skipped questions, incomplete records, or other processes.

Researchers should determine the extent and distribution of missingness for variables central to the study.

For example, prior academic performance may be unavailable for transfer students, while platform activity logs may be incomplete during system outages.

These patterns may affect which observations can be analyzed and whether the resulting sample differs from the intended population.

Missingness should be examined by variable, subgroup, time period, and relevant data source where appropriate.

The existence of a large dataset does not guarantee that sufficient complete information is available for the intended model.

Are Missing Values Different from Missing Variables?

Yes. A missing value occurs when a variable exists but has no recorded value for a particular observation.

A missing variable is information that was not collected or is not available in the dataset at all.

For example, some students may lack recorded prior achievement scores. That is a missing-value problem.

If prior achievement was never measured for anyone, the dataset lacks the variable.

Statistical missing-data methods may address certain missing-value problems under appropriate assumptions, but they cannot automatically recover an entirely unmeasured construct.

This distinction is especially important when unavailable variables could affect confounding control or measurement validity.

Are There Duplicate or Inconsistent Records?

Researchers should examine whether records are unique according to the intended observation unit.

For example, repeated student identifiers may represent legitimate course enrollments, multiple platform events, or accidental duplicates.

Removing all repeated identifiers without understanding the data structure could destroy valid observations.

Researchers should establish expected uniqueness rules and investigate inconsistencies in dates, identifiers, categories, and recorded values.

Data cleaning should be documented and should preserve the distinction between genuine repeated measurements and erroneous duplication.

How Were the Data Processed Before Release?

Datasets may undergo cleaning, recoding, anonymization, aggregation, linkage, or other transformations before researchers receive them.

These procedures can influence the meaning and completeness of the available information.

For example, individual assessment scores may be converted into categories to reduce disclosure risks.

Such aggregation may prevent analyses requiring the original numerical variation.

Researchers should determine whether variables are raw, derived, standardized, imputed, or otherwise transformed.

Where processing procedures are insufficiently documented, uncertainty about measurement should be acknowledged.

Does the Dataset Contain the Variables Needed for Confounding Control?

Observational analyses may require information about variables that influence both the exposure and outcome.

For example, a study examining the association between AI-supported learning and academic performance may need to consider prior achievement, course characteristics, and other plausible confounders.

Researchers should identify relevant confounding structures using substantive knowledge rather than simply adjusting for every available variable.

If important confounders were never collected, the intended causal interpretation may be unsupported.

Researchers should also avoid controlling indiscriminately for mediators or colliders, which can introduce additional bias.

Variable availability must therefore be evaluated in relation to the intended causal or associational question.

Can the Dataset Support Subgroup Analysis?

A dataset may contain many observations overall but relatively few within particular subgroups.

For example, an institutional dataset may include thousands of students but only a small number enrolled in a particular academic program.

Subgroup analysis may be limited by sparse observations, missing information, sampling design, or disclosure restrictions.

Researchers should inspect subgroup counts and the information available for the proposed comparisons.

Where complex sampling is involved, effective sample sizes and appropriate variance estimation may be relevant.

The question of whether secondary data support intended subgroup analyses

How Do You Assess Dataset Documentation?

Useful documentation may include collection protocols, questionnaires, data dictionaries, codebooks, technical reports, sampling information, and records of processing decisions.

The FAIR principles described by Wilkinson and colleagues (2016) emphasize making data findable, accessible, interoperable, and reusable.

However, FAIR characteristics do not automatically establish that a dataset is scientifically valid for every research purpose.

Researchers should evaluate whether the documentation explains how the data were generated and what their limitations are.

A well-documented dataset can still be unsuitable for a particular question, while incomplete documentation may prevent otherwise promising data from being interpreted reliably.

What If the Dataset Has Already Been Used in Published Research?

Previous publications may provide useful information about the dataset's structure, limitations, and analytical possibilities.

However, prior publication does not guarantee that the data are appropriate for a new research question.

Researchers should examine whether earlier studies used the same variables, population restrictions, data releases, and measurement definitions.

They should also consider whether relevant methodological criticisms or corrections have been published.

Using a dataset that appears frequently in the literature is not a substitute for independent suitability assessment.

How Do You Evaluate Dataset Access and Permissions?

Researchers should verify whether the dataset is openly accessible, restricted, licensed, or subject to a data-use agreement.

Some datasets require institutional approval, secure computing arrangements, or restrictions on sharing derived outputs.

For example, identifiable student records may require institutional authorization and privacy safeguards.

Researchers should distinguish the ability to download a dataset from permission to use it for a particular research purpose.

Applicable consent conditions, ethical review requirements, and data-protection obligations should be evaluated before analysis.

Can Dataset Quality Be Reduced to a Single Score?

A single quality score may conceal differences that matter to particular research questions.

For example, a dataset may have excellent population coverage but weak measurement of the construct central to the study.

Another dataset may contain precise measurements from a narrowly selected population.

Neither is universally superior.

Researchers should assess quality across dimensions relevant to their intended inference.

Where scoring systems are used, their criteria and weighting should be justified rather than treated as objective measures of overall scientific value.

What Should a Dataset Feasibility Assessment Include?

The purpose of this assessment is to identify whether limitations are manageable, require changes to the question, or make the proposed study infeasible.

When Should You Reject a Dataset?

A dataset may be unsuitable when it lacks essential measurements, represents an incompatible population, has insufficient documentation, or cannot support the required temporal or analytical structure.

For example, a study seeking to examine how students critically evaluate AI-generated explanations cannot rely solely on login counts if those records contain no evidence of evaluation behavior.

Similarly, a proposed causal analysis may be infeasible when essential confounders or exposure timing are unavailable.

Researchers should not assume that sophisticated statistical methods can overcome every limitation of data generation.

Rejecting an unsuitable dataset before analysis is a methodological decision, not a research failure.

How Should the Dataset Evaluation Be Reported?

The methods section should identify the dataset's source, original purpose, population, relevant period, variables, and data structure.

Researchers should explain eligibility restrictions, cleaning procedures, missingness, and analytical transformations.

Important limitations affecting interpretation should be reported explicitly.

For observational studies, relevant reporting guidance such as STROBE may support transparent descriptions of data sources and measurements.

However, reporting checklists do not replace substantive evaluation of whether the dataset supports the intended research question.

Evaluating a University Learning Analytics Dataset Before Research

The example illustrates why dataset evaluation must precede model selection. A large dataset can support useful research while remaining unsuitable for broader constructs that its variables were never designed to measure.

Common Mistakes When Evaluating Existing Datasets

How to Decide Whether a Dataset Is Ready for Research

Begin with a dataset feasibility review before committing to a final hypothesis, statistical model, or research proposal.

Prioritize limitations that could invalidate the intended interpretation rather than attempting to correct every minor data imperfection.

The practical objective is to establish whether the dataset can support a meaningful, defensible answer to the research question. Analytical sophistication should follow that determination, not substitute for it.

Evaluate Your Dataset Before Building the Study

Frequently Asked Questions About Evaluating Datasets Before Research

Evaluate the Evidence Before Designing the Analysis

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.