Overview

The client needed a daily feed of new residential property listings across several districts, enriched with public valuation data, landing directly in a spreadsheet they already used. The source is a large online marketplace with strict anti-bot protections, so this had to be reliable enough to run unattended, every day, without silently losing rows or duplicating them on overlap.

What It Does

• Pulls every new listing posted since the last run across multiple districts, with a safety overlap window on the first run so nothing near the boundary gets missed.
• Enriches each listing with detail-page data and public council valuation fields (capital value, decade built, last sale date).
• Writes to two places: a durable dataset store as the permanent raw record, and an append-only Google Sheet via a service-account credential, so a spreadsheet failure never costs the raw data.
• De-duplicates twice: once within each run, and again against everything already in the sheet, so re-runs and overlapping windows never create duplicate rows.

How It Stays Reliable

The source site fingerprints and blocks naive HTTP clients, so requests are made through a client that matches a real browser's network fingerprint rather than routing through rotating proxies. On top of that: a concurrency cap, randomized delays between requests, and exponential backoff with jitter on rate-limit responses, so the scraper backs off automatically instead of getting blocked outright. Progress is tracked with a persisted watermark, the timestamp of the last successful run, so each day's run only fetches what's genuinely new rather than re-scraping the full catalog.

Infrastructure

Built and deployed as an Apify actor: containerized, scheduled to run once a day, with its output schema frozen and versioned so a downstream change to the spreadsheet format is a deliberate decision, not an accident.

Outcome

A hands-off daily pipeline that has been running in production, keeping the client's spreadsheet current without manual exports or a single duplicate row.