How to Scrape Websites with n8n (Legally and Reliably)

This page may contain affiliate links.

Scraping websites sounds like something a hacker in a hoodie does, but it’s actually a perfectly legitimate way to gather data for price comparisons, lead generation, or market research. The catch? Doing it badly can get your IP banned, waste hours of your time, or land you in legal hot water. The good news is that n8n — the brilliant open-source automation tool — makes scraping accessible to anyone, even if you’ve never written a line of code. In this guide, I’ll show you how to scrape ethically and reliably using n8n, with concrete examples you can adapt today.

Is Scraping Legal? The UK Rules You Need to Know

Before you scrape anything, understand the law. In the UK, scraping isn’t illegal per se, but it becomes problematic if you breach a website’s terms of service (ToS) or the UK GDPR. The key principles are:

• Check robots.txt — This file (usually at example.com/robots.txt) tells you which paths are off-limits. Respect it. If it says Disallow: /prices, don’t scrape that folder.
• Don’t scrape personal data — Under UK GDPR, collecting names, emails, or any identifiable info without consent is a big no-no. Stick to non-personal data like product prices, stock levels, or public business details.
• Check the ToS — Many sites explicitly forbid scraping. If they do, walk away. There are usually alternative APIs or data sources.
• Be polite — Even if scraping is allowed, hammering a server with hundreds of requests per second isn’t. You’ll get blocked, and you might cause downtime for others.

For a practical example, let’s say you want to monitor prices on a UK retailer’s website for a side hustle selling on eBay. That’s fine — as long as the retailer’s robots.txt allows it and you’re not collecting customer data.

Setting Up n8n for Scraping: The Basics

n8n runs either on their cloud (n8n Cloud) or self-hosted on your own server. For scraping, I’d recommend self-hosting on a cheap VPS (around £5–£10 per month from providers like DigitalOcean or Hetzner) because you control the IP and can run long workflows without hitting cloud limits. If you prefer zero maintenance, n8n Cloud starts at roughly £20 per month, but you’ll be sharing IPs with other users — which can get you blocked faster.

Once n8n is running, your basic scraping workflow looks like this:

1. HTTP Request node — Fetches the page HTML. Set the method to GET and the URL to your target.
2. HTML Extract node — Pulls out specific elements using CSS selectors or XPath.
3. Code node (optional) — For cleaning or transforming the data.
4. Google Sheets or Postgres node — Stores the results.

Here’s a concrete example. Suppose you’re scraping product prices from a site like Currys. Your HTTP Request node URL is https://www.currys.co.uk/products/search?q=laptop. In the HTML Extract node, you’d use a CSS selector like .product-price to grab each price. n8n returns an array of all matching elements. Run it once to test, and you’ll see the data flowing into your output.

Reliability: Handling Pagination, Dynamic Content, and Errors

The biggest frustration with scraping is when the site changes its layout or uses JavaScript to load content. Here’s how to handle the three main culprits:

1. Pagination

Most sites have multiple pages of results. In n8n, use a Loop Over Items node. First, extract the total number of pages from the first page (often in a rel="last" link). Then loop from page 1 to N, appending ?page=N to your URL. For example, if the URL is /search?q=laptop&page=1, your loop variable replaces the number each iteration. Add a Wait node of 2–5 seconds between requests to avoid hammering the server.

2. Dynamic Content (JavaScript)

If the data you need is loaded via AJAX after the page loads, plain HTTP won’t see it. You have two options: use the HTTP Request node to hit the underlying API directly (right-click in your browser’s DevTools, go to Network tab, find the XHR request that returns JSON — that’s your goldmine), or use a headless browser like Puppeteer in n8n. The Puppeteer node can render the full page, wait for elements, and extract data. It’s slower and heavier, so only use it when necessary. For most UK retail sites, the API trick works brilliantly — just check the request URL and headers.

3. Error Handling

Websites break. Your scraper will fail. Build in a Try/Catch pattern using n8n’s Error Workflow. Create a separate workflow that sends you an email or a Telegram message when your main workflow fails. Also, add a Retry option in the HTTP Request node — set it to retry twice with a 10-second delay. And always log your output to a file or database so you can see what happened.

Rate Limiting and Anti-Bot Measures

UK sites like Amazon, Argos, and John Lewis use sophisticated anti-bot systems. You can still scrape them, but you need to be smart:

• Set a realistic delay — 5–10 seconds between requests is polite. Use the Wait node with a random duration (e.g., 5 + Math.random() * 5 seconds) to mimic human behaviour.
• Rotate user agents — In the HTTP Request node, set a different User-Agent header each run. You can store a list in a Set node and randomly pick one.
• Use a proxy — If you’re scraping heavily, a residential proxy from providers like Bright Data costs around £3–£5 per GB. This hides your IP and reduces blocks. For light scraping, you can skip this.
• Cache results — Don’t re-scrape the same page every hour if the data changes daily. Store the last scrape time and only refresh if needed.

A real-world tip: if you’re scraping prices for a price comparison side hustle, run your workflow once a day at off-peak times (e.g., 2am) rather than continuously. This keeps you under the radar and saves bandwidth.

Putting It All Together: A Complete Workflow Example

Let’s build a simple but robust scraper for a UK electronics store (hypothetical, but the structure applies anywhere). Your workflow would look like:

1. Schedule Trigger — Run daily at 2am.
2. HTTP Request — Fetch https://www.example-store.co.uk/deals with a random User-Agent.
3. HTML Extract — Select all product cards using .product, then extract name (.product-name), price (.price), and link (a.product-link[@href]).
4. Code node — Clean the price string (remove £, commas) and convert to a number.
5. Wait — 5 seconds.
6. Loop — If there’s a next page, repeat from step 2.
7. Google Sheets — Append all rows to a spreadsheet.
8. Error Workflow — If any step fails, send you an email with the error message.

Test it on one product first, then scale up. You’ll soon have a reliable data pipeline that runs while you sleep.

If you’d rather not build all this from scratch, I’ve put together a set of n8n Starter Workflows — plug-and-play n8n workflow templates you can import in minutes (from £9). They cover common scraping patterns like pagination, error handling, and API extraction, so you can skip the trial-and-error and get straight to collecting data. It’s a genuine shortcut for anyone who wants the done-for-you version of what I’ve just described.

Storing and Using Your Scraped Data

Once you have the data, don’t just leave it in a spreadsheet. Think about what you’ll do with it. For a price-monitoring side hustle, you could set up an n8n workflow that checks if a price drops below your target, then sends you a Telegram notification. For a lead generation business, you might scrape public business listings (e.g., Companies House data) and enrich them with social profiles — but again, avoid personal data.

For storage, I recommend PostgreSQL (self-hosted or a cheap managed instance like Neon, which has a free tier) or Google Sheets if you’re only handling a few hundred rows. If you need to serve the data to a website or app, add a Webhook node at the end of your workflow so other systems can pull the latest data on demand.

One more reliability tip: version your selectors. When a site changes its HTML, your scraper will break. In n8n, you can store your CSS selectors in a Set node at the top of the workflow, so when you need to update them, you only change one place. It’s saved me hours of debugging.

Final Thoughts

Scraping with n8n is a superpower for any UK side hustler — whether you’re tracking competitor prices, building a product database, or monitoring stock levels. The key is to stay legal (respect robots.txt and GDPR), be polite (rate limit), and design for resilience (handle errors and pagination). Start small, test thoroughly, and you’ll have a reliable data pipeline that runs on autopilot. And if you want a head start, grab my n8n Starter Workflows to skip the boring setup. Happy scraping!