03 · What You Need to Know
How Generative AI Can Support Responsible Research Web Scraping
What Is Web Scraping?
Web scraping is the automated retrieval and extraction of information from websites.
A scraping workflow typically requests webpage content, identifies relevant elements, extracts specified information, and stores the results in a structured format.
Researchers may use web scraping to study institutional policies, publication metadata, online discussions, product information, public announcements, or other internet-based material.
However, websites differ in their technical structures, access conditions, and permitted uses.
The fact that information can be viewed in a browser does not necessarily mean that automated collection, redistribution, or reuse is unrestricted.
What Can Generative AI Contribute?
Generative AI can help researchers understand webpage structures and write code for retrieving and parsing information.
It may also assist with debugging extraction errors, designing output schemas, and preparing validation checks.
| Scraping Task |
Potential AI Assistance |
Important Verification |
| Page retrieval |
Generate HTTP request code |
Authorized access and response handling |
| HTML parsing |
Suggest selectors for relevant page elements |
Correct association of extracted values |
| Pagination |
Generate navigation across multiple result pages |
Complete coverage without duplicate collection |
| Dynamic pages |
Suggest browser automation when appropriate |
Website permissions and actual rendering behavior |
| Data normalization |
Convert extracted values into structured fields |
Preservation of original meaning and formats |
| Error handling |
Generate retry and exception logging |
Respect for access limits and server responses |
| Data validation |
Suggest checks for missing or inconsistent values |
Agreement with original webpages |
| Documentation |
Help record collection parameters and procedures |
Accuracy of the final methodological record |
AI can assist with implementation, but researchers remain responsible for establishing that the collection method is permitted, accurate, and appropriate for the study.
Web Scraping Is Different From Using an API
Some websites provide application programming interfaces, commonly called APIs, that allow authorized software access to specified information.
APIs may offer structured responses, documented fields, authentication arrangements, and usage limits.
Web scraping generally extracts information from webpages intended for presentation to users.
Web scraping
Automated extraction of information from website content, often by parsing HTML or rendered pages.
API-based collection
Retrieval of information through a documented programmatic interface under its applicable access conditions.
Bulk dataset download
Acquisition of a prepared dataset made available through an authorized distribution mechanism.
When an appropriate API or downloadable dataset is available, it may provide a more stable and transparent collection method than parsing webpages.
However, APIs also have access conditions, coverage limitations, and data quality issues that researchers must evaluate.
Public Accessibility Does Not Automatically Mean Unrestricted Use
Researchers sometimes assume that publicly visible information can be collected and reused without limitation.
However, automated access may be governed by website terms, applicable laws, intellectual property rights, and data protection requirements.
The legal position can vary by jurisdiction, the nature of the information, the method of access, and the intended use.
Researchers should not treat a general statement that web scraping is legal or illegal as universally applicable.
Where collection raises consequential legal or contractual questions, institutional legal or research governance advice may be necessary.
What Is robots.txt?
A robots.txt file communicates crawling preferences or restrictions to automated agents using the Robots Exclusion Protocol.
It may identify paths that compliant crawlers should avoid.
Researchers should inspect and respect relevant crawling instructions as part of responsible automated access.
However, robots.txt is not a complete statement of legal permission, copyright status, or data protection obligations.
A path being technically accessible or not disallowed by robots.txt does not independently authorize every form of collection or reuse.
Conversely, the legal implications of particular access restrictions depend on the applicable circumstances.
The Internet Engineering Task Force standardized the Robots Exclusion Protocol in RFC 9309.
AI Should Not Be Used to Circumvent Access Restrictions
Some websites require authentication, restrict automated access, or use technical controls to manage requests.
Researchers should not use AI-generated code to bypass authentication, defeat access controls, or evade restrictions imposed by a website.
If necessary information is unavailable through permitted methods, the researcher may request access, use an authorized API, or identify another lawful data source.
Research value does not automatically override the rights or access conditions associated with a data source.
Responsible Scraping Must Consider Server Load
Automated scripts can generate requests much faster than manual browsing.
Excessive requests may burden a website or interfere with normal access.
Researchers should follow documented rate limits and use conservative request frequencies appropriate to the source.
They should also handle server responses responsibly, including temporary failures and requests to slow down.
For example, HTTP status code 429 commonly indicates that too many requests have been made.
A scraping script should not respond by aggressively increasing request volume or attempting to evade rate controls.
Retries should be bounded, logged, and consistent with the site's access conditions.
Website Structures Can Change Without Notice
Web scraping often depends on the structure of webpage elements.
For example, a script may locate a publication date using a particular HTML class or element.
If the website changes its layout, the selector may stop working or begin retrieving unrelated content.
Some errors are obvious because the script returns no data. Others are more difficult to detect because the script continues returning plausible but incorrect values.
Researchers should validate extracted fields against original pages and monitor unexpected changes in missing-value rates or record counts.
For long-running collection projects, periodic validation is particularly important.
Pagination Can Create Incomplete Datasets
Websites frequently divide search results or listings across multiple pages.
An AI-generated script may retrieve only the first page or incorrectly construct links to subsequent pages.
It may also collect duplicate records when pagination overlaps or results change during collection.
Researchers should verify the expected coverage and record which pages were successfully processed.
When the source does not provide a reliable total count, completeness may remain uncertain.
That limitation should be documented rather than concealed.
Dynamic Websites Require Additional Caution
Some websites load content through JavaScript after the initial HTML response.
A basic HTTP request may therefore retrieve an incomplete representation of the page.
Browser automation tools can sometimes access rendered content when their use is permitted.
However, automation introduces additional complexity, resource consumption, and potential compatibility problems.
Researchers should first determine whether an authorized API or structured data source is available.
They should also verify that browser automation does not violate applicable access restrictions.
Web-Scraped Data May Contain Personal Information
Public webpages can contain names, contact details, user-generated content, and information about identifiable individuals.
The public availability of personal information does not necessarily remove data protection or research ethics obligations.
Researchers should consider whether the proposed collection is necessary, proportionate, and consistent with the approved research purpose.
For studies involving individuals, contextual privacy expectations and potential harms may be relevant even when the information is publicly accessible.
Researchers should avoid collecting unnecessary personal information and apply appropriate storage, access, and retention safeguards.
In the Philippines, the Data Privacy Act of 2012 and applicable guidance may be relevant when processing personal information.
Copyright and Database Rights May Affect Reuse
Website content may be protected by copyright or other intellectual property rights.
Some jurisdictions also recognize specific protections for databases or collections of information.
Researchers should distinguish the technical act of retrieving information from the legal and ethical conditions governing its reproduction, distribution, or reuse.
For example, collecting publication metadata for analysis raises different considerations from republishing substantial portions of copyrighted articles.
Applicable exceptions, licenses, and contractual terms may affect what is permitted.
Researchers should verify the relevant conditions rather than assume that academic purposes automatically authorize unrestricted reuse.
How Should Researchers Prompt AI to Write Web-Scraping Code?
A useful prompt should identify the intended website, permitted collection scope, required fields, and safeguards.
It should also prohibit unauthorized access and require appropriate error handling and validation.
Example Prompt
Generating a Responsible Research Web-Scraping Script
Help me write a Python script to collect the specified information from a website for which automated access has been confirmed as permitted. Use the documented page structure and collect only the fields required for the research. Do not bypass authentication, access controls, robots.txt restrictions, or rate limits. Use conservative request handling, bounded retries, and appropriate error logging. Preserve source URLs and collection timestamps. Identify missing or ambiguous fields rather than inventing values. Include validation checks for pagination, duplicate records, and extraction accuracy. Do not claim that the script has been executed or that the collection is authorized unless those facts have been independently established.
The prompt may encourage responsible implementation, but it does not independently establish legal permission or guarantee correct extraction.
How Can Researchers Validate Scraped Data?
Validation should compare extracted records with the original webpages.
Researchers can inspect representative pages, verify field associations, and check whether the script captures the intended information.
They should also examine missing values, duplicate records, unexpected formats, and differences between the expected and observed number of pages.
Where feasible, extracted values can be compared with an authoritative API or downloadable dataset.
However, source websites may contain errors or inconsistencies themselves.
Accurate extraction does not automatically establish that the underlying information is factually correct.
Preserve Provenance and Collection Context
Web content changes over time. A page may be updated, removed, or reorganized after data collection.
Researchers should document source URLs, collection dates, relevant query parameters, and the extraction method.
Where permitted and appropriate, preserving source snapshots or archival references can support later verification.
However, archiving may itself be subject to copyright, privacy, or contractual restrictions.
Researchers should retain sufficient provenance information to explain where the data came from and under what conditions they were collected.
Can Web Scraping Be Reproducible?
Web scraping can support reproducible research when the collection code, parameters, source definitions, and relevant data state are documented.
However, rerunning the same script may produce different results because websites change.
Reproducibility may therefore require preserving an authorized dataset snapshot or documenting the source state at the time of collection.
Researchers should distinguish reproducing the extraction procedure from reproducing the exact historical dataset.
This connects to the broader requirements for AI-generated research data-extraction scripts.
What Is the Safest Role for AI in Web Scraping?
Generative AI can help researchers understand HTML structures, write parsing code, explain errors, and develop validation procedures.
However, it should not independently determine whether access is authorized or whether collected content may be reused.
Researchers should establish permissions and research requirements before generating the collection workflow.
Where an official API or downloadable dataset is available, that source may be preferable to scraping webpages.
The final code and collected data should be tested and documented according to the study's methodological and ethical requirements.
