Should Researchers Collect Every Available Variable?

If More Data Are Available, Should You Collect All of Them?

You are preparing a research instrument and realize that you could collect substantially more information than your original research questions require. Your institution already maintains demographic records, academic performance data, attendance information, learning management system logs, and student feedback. Adding these variables to your dataset might be relatively straightforward.

Perhaps collecting everything would make the study more comprehensive. You could examine unexpected relationships, address questions raised by reviewers, or reuse the dataset for future investigations.

Yet the apparent convenience of collecting additional information raises several concerns. Some variables may have little relevance to the research objectives. Others may be measured poorly, contain sensitive information, or require assumptions that are difficult to justify.

There is also a distinction between information that already exists and information that researchers must actively collect from participants. The first may create relatively little additional response burden, while the second can lengthen assessments and increase the possibility of missing or unreliable answers.

Should researchers therefore collect as much information as possible, or should every variable have a defensible purpose before it enters the dataset?

Collect Variables for a Justified Purpose, Not Merely Because They Are Available

Why Collecting More Variables Is Not Automatically Better Research

Data Availability and Scientific Relevance Are Different

Modern research environments often provide access to large quantities of information. Universities maintain educational records, hospitals generate clinical data, and digital platforms continuously record user activity.

These data may offer valuable research opportunities, but the fact that a variable exists does not mean it contributes to the study's intended inference.

For example, a university may have information about students' library transactions, campus entry records, online activity, and academic performance. A study investigating the effectiveness of an instructional intervention does not necessarily need all these records.

The relevant question is what each variable contributes to the research objective.

A variable may be essential for estimating an outcome, necessary for an appropriate statistical adjustment, useful for describing the sample, or important for a planned secondary investigation.

If none of these purposes applies, collecting the variable may add little scientific value.

This distinction is central to deciding which variables and outcomes to measure in a research study.

More Variables Do Not Necessarily Mean More Information

The number of variables and the amount of useful information are not equivalent.

Several variables may measure essentially the same characteristic, while one poorly selected indicator may fail to capture an important construct.

Suppose researchers want to measure student engagement. They collect the number of platform logins, total clicks, time spent online, page views, and downloaded files.

These indicators may provide useful evidence about platform activity, but they do not necessarily measure cognitive engagement, emotional engagement, or the quality of learning.

Adding more activity indicators does not automatically resolve the conceptual mismatch.

Researchers should evaluate whether each measurement captures a distinct and relevant aspect of the intended construct.

Where several indicators are necessary, their combined use should follow a defensible measurement model rather than an assumption that more indicators always improve validity.

Distinguish Essential, Supporting, and Exploratory Variables

A practical measurement plan can classify proposed variables according to their intended contribution.

These categories are not determined by whether a variable eventually produces a statistically significant finding.

An essential variable may show little association with the outcome, while an exploratory variable may produce an apparently strong relationship by chance.

Collection priorities should therefore be established through research objectives and methodological reasoning rather than retrospective statistical results.

Unnecessary Measurement Can Increase Participant Burden

When additional variables require new questions, interviews, tests, or assessments, they impose demands on participants.

Longer questionnaires may require more time and concentration. Repeated sensitive questions may cause discomfort. Additional appointments may create travel costs or scheduling difficulties.

The consequences depend on the population, assessment format, question complexity, and research setting.

A ten-minute questionnaire may be manageable for one group but difficult for participants experiencing illness, limited connectivity, or substantial competing responsibilities.

Unnecessary measurements may also contribute to incomplete responses or reduced attention, although their effects should not be assumed to be identical across studies.

Researchers should assess the cumulative burden of the instrument rather than evaluating each additional question in isolation.

The relevant question is whether the expected scientific contribution justifies asking participants to provide the information.

Existing Administrative Data Still Require Justification

Using previously collected information may reduce the need for new participant assessments.

However, existing data were often generated for administrative or operational purposes rather than research.

For example, a university may record attendance to administer institutional policies, not to measure student motivation. Similarly, learning platform activity logs may be designed for system monitoring rather than educational assessment.

Researchers should examine how variables were defined, collected, coded, and updated.

They should also assess missingness, data provenance, measurement timing, and whether the recorded information adequately represents the intended construct.

Administrative convenience does not establish construct validity.

Furthermore, institutional possession of information does not automatically authorize its use for every research purpose. Applicable privacy requirements, institutional permissions, and ethics arrangements remain relevant.

Data Minimization Is a Research Design Consideration

Data minimization concerns limiting personal information to what is appropriate and necessary for specified purposes.

Under Article 5(1)(c) of the European Union's General Data Protection Regulation, personal data must be adequate, relevant, and limited to what is necessary in relation to the purposes for which they are processed.

The Philippines' Data Privacy Act of 2012 also establishes principles governing personal information processing, including proportionality.

These frameworks do not mean that every research project must use the smallest imaginable dataset. The appropriate scope depends on the research purpose, applicable legal basis, safeguards, and relevant research provisions.

A longitudinal research resource may legitimately require broader collection than a narrowly focused experiment.

Nevertheless, researchers should be able to explain why particular categories of personal information are needed and how their collection remains proportionate.

Removing unnecessary sensitive variables can also reduce the consequences of unauthorized disclosure.

More Variables Create More Opportunities for Multiple Testing

Collecting many variables increases the number of relationships researchers could examine.

This can be useful for exploratory research, but it also increases opportunities to identify apparently noteworthy findings that arise through random variation.

Suppose researchers test twenty independent null hypotheses using a significance threshold of 0.05. If all null hypotheses are true and the tests are independent, the probability of at least one false positive is:

This is a mathematical illustration under specific assumptions, not a prediction that 64.2% of studies with twenty variables will contain false findings.

Tests may be correlated, hypotheses may not all be null, and research designs differ substantially.

The broader concern is that unrestricted searching across many relationships can increase false-positive findings and selective interpretation.

Researchers should distinguish prespecified confirmatory analyses from exploratory investigations and consider appropriate multiplicity procedures when required.

Additional Variables Can Encourage Selective Reporting

A large dataset permits many analytical choices.

Researchers may examine alternative outcomes, different covariate combinations, numerous subgroups, and several definitions of the same construct.

Such flexibility can support legitimate investigation, but it can also create opportunities to emphasize favorable findings while overlooking analyses that produce different results.

Simmons, Nelson, and Simonsohn (2011) demonstrated how undisclosed flexibility in data collection and analysis can increase false-positive findings under the conditions they examined.

The appropriate response is not to prohibit exploratory research.

Rather, researchers should document their analytical objectives, distinguish exploratory from confirmatory findings, and report relevant decisions transparently.

Preregistration or a prespecified statistical analysis plan may be useful when the research is intended to test confirmatory hypotheses.

Collecting More Covariates Does Not Automatically Improve Causal Estimates

Researchers sometimes collect extensive background information because they expect to control for every available variable during analysis.

However, statistical adjustment should follow the intended estimand and a defensible causal model.

A variable associated with the outcome is not necessarily a confounder.

Adjusting for certain mediators can remove part of the effect under investigation. Conditioning on a collider can introduce a noncausal association.

Consequently, adding more covariates may sometimes increase rather than reduce bias.

Researchers should distinguish variables worth measuring from variables appropriate for inclusion in a particular model.

Hernán and Robins' Causal Inference: What If provides a detailed explanation of why causal adjustment requires explicit assumptions about relationships among variables.

High-Dimensional Data Create Analytical Demands

Datasets containing many variables may require additional analytical safeguards.

When the number of candidate predictors is large relative to the sample size, conventional regression estimates may become unstable or poorly generalizable.

Highly correlated predictors can complicate interpretation, while sparse categories may produce imprecise estimates.

Predictive modeling can sometimes accommodate large variable sets through regularization, dimension reduction, or other methods.

However, such approaches require appropriate validation and do not eliminate the need to evaluate data quality and leakage.

For example, a predictive model should not use information recorded after the prediction time point when the intended application requires predictions beforehand.

Researchers should select analytical methods according to the research objective rather than assuming that a large dataset automatically supports sophisticated modeling.

Broad Data Collection May Be Appropriate for Exploratory Research

Not every study begins with a narrow set of hypotheses.

Exploratory investigations may intentionally collect a broad range of variables to characterize a phenomenon, identify patterns, or generate hypotheses for subsequent research.

Similarly, registries, longitudinal cohorts, and research infrastructures may be designed to support multiple related questions over time.

Such designs can justify broader measurement when the intended purposes, governance arrangements, participant protections, and analytical strategy are sufficiently clear.

However, exploratory purpose should not become a blanket justification for unrestricted collection of identifiable or sensitive information.

Researchers should establish the scope of anticipated uses, evaluate proportionality, and distinguish exploratory findings from confirmatory evidence.

Future Research Value Is a Legitimate but Conditional Consideration

A variable may not be essential to the current primary analysis but may have substantial value for a defined future investigation.

For example, a longitudinal educational cohort may collect baseline information about technological access because future analyses will examine changes in digital inequality.

This can be defensible when the additional measurement is consistent with the project's objectives, consent or other authorization, and governance arrangements.

However, a vague possibility that information might someday be useful is a weaker justification than a defined secondary or future research purpose.

Researchers should consider whether the information could instead be collected later, obtained from an appropriate existing source, or represented by a less intrusive measure.

Missing Data Can Increase With Unnecessary Measurements

Additional variables may introduce more opportunities for item nonresponse, measurement errors, and inconsistent coding.

However, missingness should not be assumed to increase uniformly with every added variable.

Some measurements may be obtained automatically or from existing records without additional participant effort.

Others may require sensitive disclosures that participants prefer not to provide.

Researchers should examine likely missingness mechanisms and the implications for the planned analysis.

For example, a sensitive income question may have different response patterns from a routine question about academic year level.

Adding a variable with extensive missingness may contribute little useful information while complicating analysis, although its scientific importance may still justify collection.

Evaluate the Incremental Value of Each Additional Variable

Before adding a measurement, researchers can ask what new information it contributes beyond variables already included.

The answer may involve improving construct coverage, addressing a plausible confounding pathway, evaluating an important subgroup, or supporting a prespecified secondary objective.

Redundancy is not always undesirable. Multiple indicators may be necessary for reliable measurement of a latent construct or for assessing agreement between methods.

However, redundant variables should be collected intentionally rather than accidentally.

A measurement plan should distinguish purposeful replication or triangulation from unnecessary duplication.

Use a Variable Justification Matrix

A variable justification matrix can make collection decisions transparent.

Variables without a sufficiently clear justification should be reconsidered before the instrument or data extraction plan is finalized.

The matrix can also help researchers identify variables that are essential but have not yet been included.

Deciding Which University Data to Include in an Educational Technology Study

Reviewing the Proposed Dataset

Refining the Collection Plan

The resulting dataset contains fewer variables than the institution could technically provide.

However, the reduction may improve interpretability and avoid unnecessary processing of personal information.

The researchers do not claim that the smaller dataset eliminates confounding or guarantees unbiased findings. They explain which variables were selected, why they were necessary, and what limitations remain.

Common Misconceptions About Collecting More Research Data

Make Variable Collection a Deliberate Scientific Decision

Start with the primary research question and the measurements needed to answer it.

Then consider additional variables according to their contribution to interpretation, secondary objectives, exploratory analyses, or legitimate study operations.

For confirmatory research, distinguish planned analyses from additional investigations developed after examining the data.

For exploratory research, establish a sufficiently clear scope and report analytical flexibility transparently.

The goal is not to minimize the dataset at any cost. It is to ensure that the information collected is proportionate to the scientific questions and responsibilities of the study.

Before Adding Another Variable to Your Dataset

Frequently Asked Questions About Excessive Variable Collection

Availability Is Not a Sufficient Reason to Collect Data

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.