This guide explains the first step of scraper development: reverse-engineering a website to identify where its data is exposed and how it can be collected efficiently.
It is written for developers, system administrators, and other technical readers. I have developed and maintained many scrapers. The methods described here are well known among scraper developers; my goal is to explain them from the defender's perspective. I also use these techniques as one part of my broader bot traffic audit.
This article only briefly covers defeating the challenges used by Cloudflare and similar reverse proxies. During reconnaissance, the cheaper option is usually to find a route around the reverse proxy and access the origin or an unprotected endpoint directly.
Why websites are scraped
Scraping converts data published on websites into structured datasets. These datasets are used for price comparison, market research, lead generation, search, AI training, and competing services.
Scraper output is commonly delivered as CSV files or through an API. Dataset structures are reasonably standardized within categories; data marketplaces even publish schemas for common datasets. The source data could take any shape, but the final output is always a matrix.
Commercial scraping often has a narrow profit margin. Development, maintenance, infrastructure, and proxy traffic must cost less than the resulting dataset is worth. Protection does not have to make scraping impossible. It has to make collection too expensive or unreliable to remain profitable.
Legal and ethical limits
Scraping is not automatically legal or illegal in the EU. The answer depends on the data, the access method, the site's terms, the intended use, and national law. The EU's Database Directive can protect substantial investment in collecting and maintaining a database. The GDPR applies when personal data is processed, including personal data available publicly. Terms of service may impose additional restrictions.
robots.txt does not technically prevent access, but it can express a machine-readable reservation of rights. Scrapers can choose to ignore it. A 2025 study of 130 self-declared bots found that compliance decreased as restrictions became stricter.
Scraping has legitimate uses, including research, accessibility, and archiving. It can also copy commercially valuable databases, collect personal information at scale, and create infrastructure costs for the source website. The fact that information is publicly accessible does not resolve these issues.
Regardless of the legality or morality, valuable exposed data will be scraped. Alongside the legal response, we also need a technical solution or, where possible, an agreement that replaces scraping with direct access.
Define the output first
The process starts by defining the fields to collect. Once the expected output is clear, reconnaissance looks for two things:
- An index: a list containing a reference to every required record.
- A data source: a way to retrieve the required fields for each record.
An accurate index makes it possible to detect additions and removals. Modification times, version numbers, or reliably ordered identifiers also allow incremental updates, which can reduce scraping costs substantially.
Many scrapers are quite primitive. They download the entire dataset again and again, every few days. It works, but wastes time, proxy traffic, and computation. Knowing which records changed makes updates both cheaper and faster.
Finding an index and data source
The required data is often exposed through several channels. The main website is only the first one to examine.
sitemap.xml can provide a complete index of public URLs. It may also contain a lastmod value for each page. RSS and Atom feeds, category pages, search results, export functions, print views, and structured metadata can provide other indexes.
robots.txt can identify relevant resources or application sections. Listing a path there does not secure it and may highlight it as something worth investigating.
Modern web applications frequently request structured data after the initial page has loaded. The browser's developer tools show these requests. A listing API with pagination and stable identifiers can provide both the index and the source data. If its per_page parameter is not limited, it may return the complete dataset in one response. At that point, the cost of scraping is effectively zero.
Mobile apps, partner portals, affiliates, and embedded widgets may use different interfaces for the same dataset. Even if the main site lacks an API and only serves rendered HTML, these other interfaces may still expose the structured data. Mobile traffic can also be inspected with an intercepting proxy. PortSwigger provides an Android setup guide for Burp Suite.
Subdomains are also in the crosshairs of scraper developers. Staging, demo, development, old application versions, and API documentation are commonly left accessible. A staging environment may expose an OpenAPI document that explains an API also used in production.
Finding the origin server
Cloudflare and other reverse proxies can be bypassed if the origin IP address is known and the server accepts direct traffic.
Historical DNS records may contain addresses used before the proxy was enabled. SecurityTrails provides historical A and AAAA records for this purpose. DNS-only subdomains can also point to the same server.
Mail hosted on the origin can expose its address too. Trigger a system email to an address you control, then inspect the message's Received headers. Bounce messages can expose the mail server in the same way. Cloudflare's documentation gives a more specific warning:
Evaluate mail infrastructure: If possible, do not host a mail service on the same server as the web resource you want to protect, since emails sent to non-existent addresses get bounced back to the attacker and reveal the mail server IP.
Shodan and Censys scan public internet services. Their indexes can associate an IP address with a TLS certificate, HTTP response, or other identifying property even if DNS never exposes the current address directly.
Once an origin IP has been exposed, changing public DNS is not enough. The address should be rotated. DNS records and mail infrastructure should be reviewed, and the origin should only accept traffic from the reverse proxy or an authenticated connection.
The cost of avoiding blocks
A scraper has three main cost categories:
- initial reconnaissance and development;
- maintenance when the source changes;
- a supply of IP addresses.
The defender has the most control over the last cost. Forcing a scraper to change IP addresses frequently is one of the most effective ways to increase its operating costs.
Most IP-based protection assumes that an address associated with abusive traffic is likely to remain abusive. Scrapers therefore need addresses that have not already acquired a poor reputation.
Commercial proxies are the most obvious source. Many services charge by transferred volume, which gives scraper developers an incentive to cache assets and avoid unnecessary requests. Not requesting images, fonts, or scripts reduces cost but can also make the traffic easier to identify. Fixed-price pools such as Storm Proxies charge for threads or ports and include unlimited bandwidth. They are cheaper to use heavily, but their addresses are more likely to have been used and blocked before.
VPNs may work against blocklists maintained by a single website. However, centralized anti-bot providers can identify more of them because they observe the same exit addresses across many customers. Cloudflare, for example, maintains lists of known VPNs, proxies, and anonymizers.
Tor is rarely useful because its exit nodes are public and commonly blocked. It should still be tested. Some websites do not block it, and when that happens Tor provides free IP rotation without managing proxy accounts or paying for traffic.
Mobile data provides another source of IP addresses. When a phone or modem connects to a 4G or 5G network, the carrier creates a data session and assigns an IP address to it. The device will often receive a private address, while carrier-grade NAT maps traffic from many subscribers onto a smaller pool of public IPv4 addresses. Disconnecting and reconnecting mobile data creates a new session. Depending on the carrier, this may also produce a different public IP address.
A scraper can automate reconnections of one or more mobile modems and repeatedly receive addresses from the carrier's pool. These addresses belong to networks used by ordinary phones, and one public address may represent many unrelated subscribers. This makes them useful for scraping and risky to block. RFC 6598 defines the shared address space used for carrier-grade NAT.
Modern anti-bot systems combine IP reputation with cookies, request order, JavaScript results, and TLS fingerprints such as JA3 and JA4. This increases the scraper's work, but it also makes protection more complicated to configure. One exposed origin, unprotected API, incorrect bypass rule, or trusted user-agent can make the other controls irrelevant. A scraper can claim to be Googlebot, for example, if the defender does not use Google's published IP ranges and DNS verification methods.
Increasing the cost of scraping
The proven starting point is a large centralized reverse proxy with bot management and rate limiting. These providers observe traffic across many websites, which gives them better information about proxy networks, VPNs, browser fingerprints, and abusive behaviour than a single website can collect.
The reverse proxy and the web server must both be configured correctly:
- all public web traffic should pass through the reverse proxy;
- the origin should reject direct connections;
- alternate hostnames, staging systems, APIs, and mail infrastructure must not expose a route around it;
- verified bots should be identified using published IP ranges, DNS verification, or cryptographic authentication;
- the application must still enforce authentication, pagination limits, and query limits.
A reverse proxy does not protect an endpoint that bypasses it, and it does not fix an API that intentionally returns the complete dataset without authentication. The effectiveness of the whole setup depends on closing these simpler routes. Developing custom bot detection inside the application should come after this foundation, not before it.
Offering direct access instead
If active scraping creates significant infrastructure cost or commercial harm, contacting the operator may be cheaper than continuing to expand technical protection. A business that depends on scraped data may be willing to pay for a reliable feed because maintaining scrapers, proxies, and data validation is expensive and risky.
A documented API can limit the available fields and request volume. An agreement can define permitted uses, retention, attribution, and resale. Some platforms now use structured access and data agreements instead of attempting to block every collector.