When your scraper “works” but important fields go missing, treat it as a soft block until proven otherwise. The fastest fix is to locate the layer where the degradation starts (network, HTML, JSON, or rendering), then apply a controlled sequence: validate completeness, slow down, stabilize sessions, and only then rotate exits more aggressively.
Find the layer where the failure starts
Begin with raw response checks: is the HTML shorter than normal, is the JSON missing keys, or is the server returning a different template variant?
If the payload size drops while status codes stay 200, you are likely seeing throttling behavior that suppresses modules or hides data.
Separate status errors from missing fields
Hard blocks are noisy (403/429), but soft blocks are quiet: empty arrays, placeholder content, or a sudden switch to “consent” pages.
Build a small validator that asserts required fields. If the validator fails, stop and retry with a safer policy instead of continuing to collect bad data.
Start with low-risk checks before rotating everything
Lower concurrency and add jitter to request pacing. Then keep sessions pinned so the target sees coherent behavior.
Only after pacing and session stability are controlled should you rotate exits more frequently. Otherwise you will not know which change actually fixed the issue.
Prevent the issue from returning
Promote “valid page rate” as a first-class metric. Alert when completeness drops, not only when errors rise.
Keep a small set of canary URLs that you scrape continuously to detect early drift in page structure and defenses.
FAQ
Why do I get 200 responses but missing fields?
Many targets degrade responses under load without returning an explicit error, which is why completeness checks are essential.
Should I retry immediately when fields are missing?
Not with the same policy. First reduce pacing pressure and stabilize sessions, then retry a small sample and re-check completeness.
What is a good metric to detect soft blocks?
Track valid page rate and payload size distribution per target and per session, and alert on sudden shifts.