Fix guide
Polygon RPC for Scrapers: How to Avoid Silent Throttling
High-volume scrapers, indexing pipelines, and analytics crawlers often suffer from unannounced latency throttling on shared RPC providers. Here is how to diagnose and bypass it.
How Silent Throttling Happens
- Aggressive rate-limiting algorithms that silently delay responses (tarpitting) instead of returning clear 429 HTTP codes.
- TCP socket queueing and high connection latency when scrapers burst concurrent eth_getBlockByNumber or eth_getTransactionByHash queries.
- Shared infrastructure throttling large JSON payloads (such as receipt batches) during peak hours.
- Overlapping crawler workers saturating available outbound proxy connections.
How to Maintain Max Ingestion Speed
- Track p50, p95, and p99 response latencies on batch requests to detect artificial provider delays before failures occur.
- Use chunked batch RPC arrays (`batch: [req1, req2]`) with maximum sizes of 25-50 requests to avoid gateway payload caps.
- Route heavy historical data ingestion through an authenticated Polygon RPC plan with published throughput ceilings and transparent limits.
Shared endpoints work for scrapers when:
- scraping infrequent or small transaction sets
- data pipeline delays of 5-15 seconds are harmless
- concurrency is low and non-bursty
- backfills are run during off-peak hours
Dedicated scraping infrastructure is needed when:
- real-time ingestion pipelines require sub-50ms data availability
- silent throttling delays downstream analytics or trading triggers
- you need to ingest millions of Polygon blocks/receipts daily
- transparent, predictable request limits are mandatory for SLA guarantees
High-Throughput Scraping on Polygon Mainnet
BlazingNode gives your scrapers high RPS allocations, fast NVMe caching, and transparent request ceilings with zero hidden latency throttling.