03 · What You Need to Know
How Digital Information Systems Can Distort Longitudinal Research
What Is an Artificial Data Trend?
An artificial data trend is an apparent pattern in recorded observations that arises wholly or partly from changes in measurement, recording, processing, or population coverage rather than corresponding changes in the phenomenon of interest.
For example, a hospital may record more cases of a condition after introducing mandatory diagnostic fields. The increase could reflect more complete documentation rather than a comparable increase in disease occurrence.
Similarly, a university may report increased learning platform activity after a software update begins counting interactions that were previously excluded.
Artificial trends can appear as abrupt jumps, gradual increases, declines, seasonal patterns, or changes in variability. Their form depends on the recording process and the timing of the system modification.
Distinguish Real Change From Measurement Change
Substantive change
A genuine change in the behavior, condition, event, or outcome the research intends to investigate.
Measurement change
A change in how the underlying phenomenon is observed, classified, recorded, or quantified.
Mixed change
A pattern influenced by both genuine developments and changes in the measurement or recording process.
These possibilities are not always easy to separate. A new digital system may improve documentation while also changing organizational behavior.
For example, a platform update that makes learning activities easier to access could genuinely increase participation while simultaneously changing how participation is counted.
The researcher must therefore examine both the system's measurement rules and its potential influence on the underlying activity.
Which System Changes Commonly Produce Artificial Trends?
| System change |
Illustrative effect |
Potential research artifact |
| New event tracking |
Additional interactions are recorded |
Apparent increase in activity |
| Mandatory data fields |
Previously optional information becomes required |
Apparent increase in documented conditions |
| Revised classification rules |
Existing cases are assigned to new categories |
Apparent change in category prevalence |
| Software migration |
Historical records are imported or duplicated |
Unexpected changes in counts |
| Expanded system coverage |
Additional institutions or users enter the database |
Apparent population growth or increased utilization |
| Changed default values |
Blank fields become zeros or preset categories |
Artificial changes in distributions |
| Revised timestamps |
Upload time replaces event time |
Misleading temporal clustering |
| Algorithm updates |
Derived indicators are recalculated differently |
Discontinuities in scores or classifications |
The same technical change may affect several variables simultaneously. Researchers should evaluate the entire data-generation process rather than inspect only the outcome showing an unusual trend.
How Can Software Updates Change What Is Counted?
Digital systems frequently generate research variables through automated rules.
A learning platform may count page views, clicks, sessions, resource downloads, or completed activities. A software update may alter which events qualify for each indicator.
For example, an earlier version might count only deliberate resource openings, while a later version includes automated previews or background loading.
Both datasets may contain a field named resource views, but the underlying events are no longer equivalent.
Researchers should review event definitions, system documentation, release notes, and relevant technical specifications before combining observations across versions.
How Do Data Migrations Create Discontinuities?
Data migration involves transferring information between systems or database structures.
During migration, records may be transformed, recoded, merged, omitted, or duplicated. Historical dates may be replaced by import timestamps, and categories may be mapped into a new classification system.
For example, an institution may migrate student records to a new platform and inadvertently assign some historical transactions the migration date rather than their original event dates.
A time-series analysis could then show an artificial concentration of activity during the migration period.
Researchers should distinguish original event dates from extraction, modification, and migration dates whenever these fields are available.
Migration quality should be evaluated using reconciliation checks, source documentation, and appropriate validation procedures.
Can Improved Documentation Look Like an Increase in Disease or Behavior?
Yes. A change in documentation completeness can alter recorded prevalence without an equivalent change in actual prevalence.
For example, introducing a structured clinical assessment may increase the proportion of encounters containing a documented diagnosis.
Similarly, a university may introduce mandatory attendance recording, resulting in a substantial increase in the number of attendance observations.
The increase may indicate better ascertainment rather than greater occurrence of the underlying event.
Researchers should examine whether documentation requirements, staff training, incentives, or reporting policies changed during the study period.
Why Do Denominators Matter?
Apparent trends can arise when the number or characteristics of individuals represented in the system change.
Suppose a digital administrative system records 2,000 service encounters in one year and 3,000 the following year.
The increase may reflect greater service use, but it may also result from adding new facilities or expanding the population covered by the system.
Researchers should distinguish raw counts from rates based on an appropriate population at risk or eligible denominator.
However, denominator adjustment does not automatically solve the problem if the definition of an encounter also changed.
Both numerator and denominator comparability must be evaluated.
How Can Researchers Detect a Possible Artificial Trend?
Begin by plotting the outcome across the full observation period and examining changes in level, slope, variability, missingness, and category distributions.
Mark known software updates, migrations, policy revisions, and changes in institutional coverage.
Unexpected discontinuities near these dates warrant investigation.
Researchers can also compare related variables. If several unrelated outcomes change abruptly on the same date, a common recording change may be plausible.
However, shared discontinuities are not conclusive evidence of a system artifact. A genuine institutional event could also affect multiple outcomes.
Descriptive diagnostics should therefore be combined with documentation and knowledge of the operational setting.
Can Interrupted Time-Series Analysis Identify System Artifacts?
Interrupted time-series methods can estimate changes in the level or slope of an outcome around a defined intervention or event.
They may be useful for examining whether recorded values changed around a software transition.
However, identifying a discontinuity does not independently establish why it occurred.
If an institutional policy and a software update occur simultaneously, their effects may be difficult or impossible to distinguish using the same outcome series alone.
Interrupted time-series analysis also requires attention to pre-existing trends, autocorrelation, seasonality, concurrent events, and the number and spacing of observations.
The method can help characterize a discontinuity, but substantive interpretation requires evidence about the data-generation process.
What Is a Structural Break?
A structural break is a change in the statistical relationship or distribution governing a time series.
It may involve a shift in mean level, trend, variance, or other model parameters.
Structural-break methods can help identify periods when the observed data behave differently.
However, a statistical break is not necessarily a measurement artifact. It may reflect a genuine event, an institutional intervention, a population change, or a combination of influences.
Researchers should avoid interpreting a detected break as proof of a software effect without supporting evidence.
Can Control Variables Help Distinguish Real and Artificial Changes?
Comparison outcomes or data sources unaffected by the suspected system change may provide useful context.
For example, an institution may compare platform activity records with independently maintained attendance or assignment records.
If the platform indicator changes abruptly while independent measures remain relatively stable, the difference may support further investigation of the platform's recording rules.
However, the comparison variable must genuinely be unaffected by the relevant system change and sufficiently related to the phenomenon.
Negative-control approaches can sometimes strengthen causal reasoning, but their assumptions must be justified rather than inferred from convenient data availability.
How Can Researchers Reconstruct Comparable Indicators?
When detailed event-level data are available, researchers may be able to create a common indicator using event types recorded consistently across system versions.
For example, if both software versions reliably record completed assignment submissions, that indicator may be more comparable than a broad activity count whose definition changed.
Researchers should document the mapping rules and verify that the selected events have equivalent meanings across periods.
Reconstruction is not always possible. If earlier systems did not record particular distinctions, later transformations cannot recover information that was never collected.
In such cases, separate period-specific analyses may be more defensible than a harmonized series based on unsupported assumptions.
Should Researchers Exclude Data Collected During a System Transition?
Exclusion may be appropriate when transition-period records are demonstrably incomplete, duplicated, or unsuitable for the intended measurement.
However, excluding observations merely because they do not fit the expected trend may introduce selective analysis.
Researchers should establish defensible exclusion criteria based on documentation and data-quality evidence.
Where feasible, sensitivity analyses can compare results with and without affected periods.
The consequences for sample size, temporal coverage, and the research question should be reported.
What If the Software Change Is Itself the Intervention?
Sometimes the research question concerns the effect of implementing a new digital system.
In this situation, changes in recording behavior may be part of the intervention's operation, but they still complicate outcome interpretation.
For example, a new electronic health record system may improve clinical documentation while also changing healthcare workflows.
If the study aims to estimate whether clinical care improved, increased documentation alone may be insufficient evidence.
Researchers should distinguish outcomes reflecting system adoption, documentation quality, actual behavior, and downstream consequences.
Independent outcome measures may be especially valuable when the intervention changes the system used to measure its own effects.
How Should Artificial Trend Risks Be Reported?
Researchers should describe relevant system versions, implementation dates, changes in recording definitions, migration procedures, and known data-quality problems.
Analyses should explain how comparable observations were identified and whether alternative assumptions materially changed the findings.
When the evidence cannot distinguish a genuine change from a recording artifact, the uncertainty should be stated explicitly.
Reporting guidelines for routinely collected data, including the RECORD statement where applicable, emphasize transparent descriptions of data sources and processing procedures.
These details are necessary for readers to evaluate whether observed trends represent meaningful changes.
When Is a Longitudinal Analysis No Longer Defensible?
A longitudinal analysis may become unsuitable when the outcome definition changes substantially, no comparable indicator can be reconstructed, and the research question depends on interpreting observations as one consistent series.
For example, combining two fundamentally different engagement measures into one continuous outcome may produce a trend without coherent scientific meaning.
The researcher may instead restrict the analysis to a stable period, investigate the system transition itself, or select another outcome.
Watch Out
Do not interpret a statistically significant change immediately after a software update as proof of a real-world effect. A digital system may change both the phenomenon being studied and the mechanism used to observe it. These processes must be distinguished wherever the available evidence permits.
