/Insolvency/
No API. No export button. Just 250,000+ public insolvency records buried inside a government search interface — and a client who needed all of them.
The UK Government's Individual Insolvency Register holds every active IVA, Debt Relief Order, and Bankruptcy case in England and Wales. The client needed the complete dataset — not a sample, not a third-party export — the real records, clean and structured. I developed a fully automated two-phase Python extraction system from scratch, delivered within seven days with zero manual input.
/Project Goals/
Extract every live insolvency record from a government register with no API access, no data export, and a 7-day hard deadline.
The challenge was scale and structure. Phase 1 required systematically generating 676 alphabetical search combinations (AA–ZZ), base64-encoding each as a search parameter, and crawling every paginated result to collect all case URLs. Phase 2 required visiting 22,301 individual case pages, parsing structured HTML, and extracting 11 fields per record with consistent accuracy across every entry.
/Results/
22,307 records extracted, 11 fields per record, delivered as a clean CSV — exactly on deadline, zero records missed.
The full extraction executed with randomised 1.5–3 second delays per request to remain undetected across 22,000+ requests. A custom UK address parser handled unstructured raw address strings — stripping counties, detecting postcodes via regex, and reconstructing the correct address hierarchy. Built-in checkpointing allowed mid-run resumption without data loss. Delivered seven days from project start, no extensions requested.
PythonWeb ScrapingBeautifulSoupPandasData ExtractionAutomationUK Government Data