Portfolio / Data engineering
No interface, no brand, not one screen.
A logistics and procurement business needed people to talk to: niche industrial manufacturers and distributors, the kind that never surface in a general directory. That list exists, scattered across an industrial blue book, but assembling it by hand means opening results, entering each profile, copying the details, visiting the company's site and hunting for an email. We built the pipeline that does those five steps on its own and hands over a CSV: 720 unique companies, 12 fields each, 0 duplicates.
01The brief
The list existed. Assembling it did not.
Selling logistics and procurement services to industry starts with knowing who to call, and there the problem is not missing information: it is that the information is scattered. An industrial blue book holds thousands of manufacturers classified by what they make, but navigating it means opening a search, entering a category, paging through results, opening every profile and copying out what is useful by hand.
The brief was to automate that whole walk for a sales team that needed to prospect by product category, not by company name. They did not want software: they wanted the list, with phone, website and — where possible — email, in a file they could open and hand around on Monday.
What the pipeline had to solve
- 01Search by product category, not by company
- 02Walk results spread across many pages
- 03Drop the profile that shows up in two categories
- 04Extract data living in the directory listing
- 05Fill in the email, which the listing rarely carries
- 06Run again without redoing what is already done
Our role
Staged pipeline design, development in Python, integration with the extraction service, parsing and normalisation of the fields, duplicate control, incremental batch processing, concurrency with safe writes, and delivery of the dataset as CSV.
02The starting point
Five steps by hand, for every company.
The process being replaced was not complicated. It was long. Every company cost a few minutes, and a few minutes across seven hundred companies is one person tied up for two weeks copying and pasting.
A search returns categories, not companies.
The directory returns product pages, and behind each one sits its own paginated listing of manufacturers. That is two levels of navigation before the first useful profile appears.
Twelve fields, one at a time.
Legal name, full and broken-out address, activity, phone, fax and website. Copying that by hand is where the silent errors appear: the shifted cell, the state that ended up in the postcode column.
The email is rarely in the listing.
The directory publishes the website, not the email. Getting it means opening every site and searching — the step that eats the most time and the first one abandoned once the list gets long.
03The rule
What separates a script from a pipeline.
A scraper is judged the second time.
The first run always works. The problem shows up when the list has to be extended with another query, or picked up after the process died halfway: if the script starts from scratch, every run costs what the first one cost and the dataset fills with repeats.
So the design starts the other way round. Before requesting anything, the pipeline reads what it already has: it compares the profiles in the listing against those already in the CSV and processes only the difference, in batches of configurable size. Links are deduplicated with sets at every stage. In the delivered dataset that is measurable: 720 records, 720 unique links, 0 duplicates.
ReadCompareProcess the difference
04The system
From one typed word to a file you can hand around.
Four stages, each with its own output file. Intermediate state living on disk is not an implementation detail: it is what makes it possible to stop, inspect what was found, and resume without requesting anything again.
query.txt
The product queryZyte API
Remote extraction layerPattern parsing
Fields and linksParallel workers
Profiles and sites at onceLocked writes
One CSV, many threadsbusinessDetails.csv
The deliverable
- 01
Read the query and count
The pipeline reads the search from query.txt, normalises the whitespace, encodes it and requests the first page. From there it pulls the total result count and works out how many pages to walk: it is not tied to a fixed number.
- 02
Discover the categories
It walks the result pages and extracts the product-category links, dropping repeats with a set. They are stored in searchLinks.txt so they can be inspected before going further.
- 03
Discover the companies
Each category has its own paginated listing of manufacturers. The pipeline detects how many there are, walks them all, and accumulates the profiles in businessLinks.txt. A company appearing in three categories enters once.
- 04
Extract the profile
Every profile yields 12 fields: legal name, the full address and its street, city, state and postcode parts, activity, phone, fax and website. Profiles already in the CSV are skipped.
- 05
Look for the email on the site
The second pass visits the websites found, adds the protocol where it is missing, and searches the content for email addresses. They are lowercased, third-party domains and file extensions that look like emails are discarded, and several per company are allowed.
- 06
Update the file
Enrichment does not produce a new file: it updates the existing one. Separating extraction from enrichment means only the failing part can be re-run, which in scraping is almost always the part that depends on someone else's site.
4 stages · 12 fields per company · configurable batch size
05The deliverable
What came out of the first brief.
Figures from the delivered dataset, counted on the file itself: a single product query produced this volume. Coverage is published field by field, including email, which is the lowest and the most honest one to show.
- 720
- Unique companies in the dataset
- 667
- Records carrying a website
- 158
- Records with an email after enrichment
- 0
- Duplicate profile links
06Engineering
Three decisions you feel at seven hundred.
At twenty companies any implementation works. The decisions below change nothing in a small test and are the difference between finishing and not finishing once the list grows.
Other people's servers are waited on in parallel.
Both the profiles and the company websites are processed by several workers at once. A scraper's bottleneck is not the processor: it is how long someone else's server takes to answer, and that is solved by waiting on several at a time.
One file, many threads.
Every worker writes to the same CSV, so writing goes through a lock. Without it, two threads saving at once produce broken rows — and a corrupted file at row four hundred is found late and costs the whole run.
What fails is reported, not swallowed.
A failed request, empty HTML, a profile with no code, an unreachable site, a missing email: each one is reported to the console rather than counted as a success. A dataset that will not say what it missed cannot be audited.
07The pieces
The process that cannot be photographed.
| Company | Activity | Contact | Status |
|---|---|---|---|
| Ridgeline Fabrication Co. | Metal stamping | sales@***.example | Captured |
| Cobalt Valve & Fitting | Industrial valves | info@***.example | Enriched |
| Harrow Industrial Supply | MRO supply | no site published | No email |
| Cobalt Valve & Fitting | Industrial valves | profile already processed | Discarded |
| Westbrook Polymer Works | Injection moulding | contact@***.example | Enriched |
A reconstruction of one run: the stages are walked, companies drop in with their status, the repeated profile is discarded, and the counter reaches the batch before the file is written. The companies and emails are invented — the real ones are third parties' contact details and are not published.
| Company | Record status | Fields | |
|---|---|---|---|
| Ridgeline Fabrication Co. | Complete | Address · phone · fax · site | 1 email |
| Cobalt Valve & Fitting | Complete | Address · phone · site | 3 emails |
| Harrow Industrial Supply | No site | Address · phone · fax | — |
| Westbrook Polymer Works | Complete | Address · phone · site | 1 email |
| Pinehurst Gear & Drive | Site, no email | Address · phone · fax · site | — |
The companies and emails are invented; the structure and the coverage are the real deliverable's. 35 records in the dataset carry more than one email, and the field is stored as a list rather than loose text.
The deliverable's real shape: coverage by field on top and one row per company below, with the enrichment status. This is what the sales team opens on Monday — a CSV, not a dashboard.
08The limit
How far a scraper is allowed to go.
An extraction project is defined as much by what it collects as by what it decides not to. These three rules were agreed before the first line was written, not after the first problem.
Nothing behind a login.
The pipeline reads what the directory and the company sites publish openly: the business email a company puts on its own website is there so it can be contacted. There are no credentials, no sessions, and no access to anything that asks you to identify yourself.
Extraction goes through a service layer.
Directory pages are requested through an extraction service rather than hammering the origin from one address. The point is to complete the walk without degrading the site being read.
Not even in this case study.
The companies and emails on this page are invented. The real dataset is third parties' contact information: it is delivered to the client who commissioned the search and it is not reproduced in a portfolio.
09The status
Delivered, with the list of what comes next.
The pipeline is delivered and working: it runs in stages, it can be stopped and resumed, and it produces the CSV with 12 fields per company. Enrichment coverage is what it is — 158 of 720 records carry an email — and that is how it gets reported, because a site that does not publish its address will not publish it however many times you ask.
What comes next is identified: swap pattern-based parsing for a real HTML parser, add retries with growing backoff, write runs to a structured log rather than the console, move the dataset into a database once volume demands it, and add domain-level email validation. None of those changes disturbs the staged design: they fit inside it.
How it is built
- 01Python 3.11
- 02Requests through requests
- 03Extraction via Zyte API
- 04Regular-expression parsing
- 05ThreadPoolExecutor · locks
- 06Runtime on Replit
- 07Intermediate state in plain text files
- 08Final delivery as CSV
What this case does not claim
There are no reply, meeting or closed-contract figures. The pipeline delivers the list; the outreach is done by the client's sales team, and that result is not ours to publish. It is also not a product with an interface: it is a command-line tool, and it is presented as one.
More of the work
The software nobody sees is the one that decides how many hours Monday costs.
Prospecting, migrations, catalogs, reconciliations: some work does not need a screen, it needs to stop being done by hand. And those projects are judged on whether they survive the second run.
Tell us about your project