How to Scrape Websites with n8n (Legally and Reliably)

This page may contain affiliate links.

Ever found yourself manually copying data from websites, wishing there was a magic wand to just suck it all into a spreadsheet? If you’re running a digital side hustle, or looking to start one, the ability to collect specific information from the web automatically can be a game-changer. Whether it’s tracking competitor prices, monitoring job listings, gathering market research, or simply compiling data for your next big idea, web scraping is an invaluable skill. But here’s the catch: it needs to be done legally, ethically, and reliably. That’s where n8n comes in. This powerful, open-source automation tool makes web scraping accessible even without deep coding knowledge, allowing you to build robust, automated data collection workflows. Let’s dive in and see how you can harness n8n for your UK-based ventures.

Understanding Web Scraping Legally and Ethically

Before we even think about building a workflow, it’s absolutely crucial to understand the legal and ethical boundaries of web scraping. Doing it wrong can lead to legal issues, blocked IPs, or even a damaged reputation. For our UK audience, here’s what you need to keep in mind:

• Check robots.txt: This is your first port of call. Most websites have a robots.txt file (e.g., https://example.com/robots.txt). This file tells web crawlers and scrapers which parts of the site they are allowed or disallowed to access. Respect it – it’s the site owner’s explicit request.
• Review Terms of Service: Always glance at a website’s Terms of Service (ToS) or Legal section. Many explicitly forbid scraping. If they do, don’t scrape. Simple as that.
• Data Privacy (GDPR): The General Data Protection Regulation (GDPR) is incredibly important for UK businesses. Do NOT scrape personal data (names, email addresses, phone numbers, etc.) unless you have explicit consent or a legitimate legal basis. Focus on publicly available, non-personal data.
• Rate Limiting and Server Load: Don’t overwhelm a website’s server. Send requests at a reasonable pace. Too many requests too quickly can crash a site, get your IP blocked, and is just plain bad etiquette. Think of it like politely asking for information, not storming the library.
• Respect the Data: If you scrape data, use it responsibly. Don’t misrepresent it or use it for malicious purposes.

Our focus here will be on scraping publicly available data from websites that explicitly allow it (like quotes.toscrape.com, designed for this purpose) or for demonstrating concepts that would apply to *your own* content or content you have explicit permission to scrape.

Setting Up Your n8n Environment

To start scraping, you’ll need n8n up and running. You have a couple of excellent options:

1. n8n Cloud (Easiest for Beginners): This is the quickest way to get started. n8n hosts everything for you, so you don’t worry about server maintenance. Pricing starts from around £18/month for their Starter plan (at the time of writing), which is perfectly sufficient for many side hustles. You just sign up, log in, and you’re ready to build workflows.

Self-Hosted with Docker (Free/Cheaper for Scale): If you’re comfortable with a bit of technical setup, running n8n yourself via Docker is cost-effective and gives you full control. You’ll need a server (a cheap VPS from providers like DigitalOcean or Hetzner will do, starting from around £5/month) or even your local machine for development.Here’s a simple Docker command to get n8n running with persistent data:

docker run -it --rm --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n n8nio/n8n

This command pulls the n8n image, maps port 5678 from the container to your host, and uses a Docker volume (`n8n_data`) to ensure your workflows aren’t lost if the container stops. Once it’s running, open your browser to http://localhost:5678 (or your server’s IP:5678) to access the n8n interface.

Your First Scraping Workflow: Quotes from a Demo Site

Let’s build a practical example. We’ll scrape quotes, authors, and tags from http://quotes.toscrape.com/ – a site specifically designed for learning web scraping.

1. Start a New Workflow: In your n8n interface, click "New" to create a fresh workflow.

2. Add a Start Node: Search for and add the "Start" node. Set its trigger to "Manual" for now. This lets us run the workflow whenever we like.

3. Fetch the Web Page with HTTP Request:

• Click the ‘+’ button and search for "HTTP Request". Add it.
• In the HTTP Request node settings:
  • URL: http://quotes.toscrape.com/
  • Method: GET
  • Response Format: Make sure this is set to Text (not JSON) as we’re getting raw HTML.
• Execute the node (click the play button on the node) to see the raw HTML output.

4. Extract Data with HTML Extract: This is where the magic happens. We’ll use CSS selectors to pinpoint the data we want.

• Click the ‘+’ button after the HTTP Request node and add an "HTML Extract" node.
• In the HTML Extract node, you’ll define your selectors. To find these, open http://quotes.toscrape.com/ in your browser, right-click on an element you want to scrape (e.g., a quote), and select "Inspect" (or "Inspect Element"). Look for unique class names or IDs.
• Add the following properties:
  • Property Name: quote_text
    CSS Selector: span.text
    Attribute: textContent
  • Property Name: author
    CSS Selector: small.author
    Attribute: textContent
  • Property Name: tags
    CSS Selector: div.tags a.tag
    Attribute: textContent
    (Note: For multiple tags, n8n will often automatically handle this as an array of strings, which is perfect.)
• Execute this node. You should now see a clean list of objects, each containing a quote, its author, and associated tags.

Feeling overwhelmed? Our n8n Starter Workflows offer plug-and-play templates for common scraping and automation tasks, letting you import pre-built solutions in minutes, starting from just £9. It's a real shortcut if you want to jump straight to results without building from scratch.

5. Save Your Data (Example: Write to CSV):

• Add a "Write to CSV" node after the HTML Extract node.
• Configure it:
  • File Name: quotes.csv (or any name you prefer)
  • File Content: Select "Output of previous nodes"
  • Output Type: File
• Execute the node. n8n will generate a CSV file which you can then download from the output of the node. You could also connect this to a Google Sheets node, a database node, or send it as an email attachment, depending on your needs!

Advanced Scraping Techniques and Best Practises

Once you’ve mastered the basics, you can expand your n8n scraping capabilities significantly.

Handling Pagination

Most websites split content across multiple pages. To scrape all of it:

1. Identify Pagination Pattern: Look at the URLs as you click through pages (e.g., ?page=1, ?page=2, etc.).
2. Looping: Use a "Loop Over Items" node. You can initialise a page number variable (e.g., in a "Set" node) and increment it within the loop.
3. Dynamic URL: In your HTTP Request node, use an expression for the URL, like http://quotes.toscrape.com/page/{{$json.page_number}}/.
4. Stop Condition: Implement a conditional (e.g., using an "IF" node) to check if there are no more items or if a "Next" button is missing, then stop the loop.

Error Handling and Reliability

Websites change. Your scraper might encounter errors.

• Retry Mechanism: In the HTTP Request node, enable "Retry on Error" and set a suitable number of retries and delay.
• Try/Catch Nodes: For more complex scenarios, wrap parts of your workflow in "Try" and "Catch" nodes. This allows your workflow to continue gracefully or send you a notification even if an individual scrape fails.
• Delays: Add "Wait" nodes between requests to avoid overloading the server and reduce the chance of getting blocked. A few seconds between requests is often good practice.

Scraping Dynamic Content (JavaScript-rendered pages)

Some websites load content using JavaScript *after* the initial HTML loads. The standard HTTP Request node won’t see this. For these, you’ll need:

• Browserless Node: n8n has a dedicated "Browserless" node which uses a headless browser (like Chrome without the UI) to render the page fully before extracting data. This is more resource-intensive but necessary for JavaScript-heavy sites.
• Puppeteer/Playwright (Self-Hosted): If self-hosting, you could integrate custom code using these libraries within an n8n “Execute Command” or “Execute Code” node, though this requires coding knowledge.

Scheduling Your Workflows

To automate data collection without manual intervention, use the "Schedule" trigger in your Start node. Set it to run daily, hourly, or weekly – whatever your side hustle requires. You can even set it to only run during off-peak hours to be even kinder to the target server.

Conclusion

Web scraping with n8n opens up a world of possibilities for your side hustle, allowing you to legally and reliably gather valuable data with minimal coding. From monitoring trends to powering your next big digital product, the ability to automate data extraction is a powerful asset. Remember to always prioritise ethical and legal considerations, be a good internet citizen, and respect website terms. Start with simple workflows, experiment with different nodes, and gradually build up to more complex, robust data pipelines. The data you need to fuel your success is out there – now you know how to get it with n8n!