The challenge
The client needed comprehensive datasets about food products, ingredients and nutrition, but the information was spread across many websites and libraries, each with a different structure that kept changing. Traditional hand-written scrapers were costly to build and broke whenever a site changed.
Our approach
- Mapped the target web sources and libraries with the client and defined one target data model for the datasets.
- Built AI-agent scrapers that navigate each source, understand page content and extract the required fields, adapting to layout changes without a rewrite.
- Normalised, deduplicated and matched records from all sources into a single unified schema, with validation and quality checks.
- Delivered the datasets in the client's preferred format, with repeatable runs to keep them up to date.