Executive Summary
The `eth_getLogs` JSON-RPC method forms the backbone of blockchain data pipelines, smart contract event indexing, and analytics aggregators. However, executing broad historical event queries on Polygon PoS presents distinct operational challenges. Because Polygon processes transactions rapidly with a two-second block time, unconstrained log queries across thousands of blocks often trigger HTTP 504 gateway timeouts, response size limits, or rate limit throttling on shared cloud endpoints.
This guide provides a technical reference for indexing engineers. We review the internal mechanics of EVM log filtering, address common failure modes encountered during high-volume backfills, and present production architectures featuring bounded block chunking, bloom filter topic matching, and deterministic checkpoint recovery.
1. Internal Mechanics of Event Logging on Polygon
When an indexer transmits an `eth_getLogs` payload, the execution engine evaluates the query across several discrete stages:
1. Parameter Validation: The client parses `fromBlock`, `toBlock`, `address`, and up to four `topics` criteria. 2. Bloom Filter Evaluation: Before loading full receipts from disk, the node scans block header bloom filters. A 2048-bit bloom filter in each block header records probabilistic presence of emitting contract addresses and indexed event topics. If the bloom filter yields zero matching bits, the node skips reading that block entirely. 3. Receipt Deserialization: For blocks where the bloom filter indicates a potential match, the execution engine loads full transaction receipts from storage to examine individual log entries. 4. Serialization and Egress: Matching logs are collected, formatted as JSON-RPC results, and serialized for network transmission.
Queries that specify neither a target contract address nor specific topic hashes force the node to deserialize every block receipt across the requested range. On active chains, wide unfiltered queries create severe disk I/O and payload memory pressure, culminating in gateway timeouts.
2. Structural Failure Modes in Event Ingestion
Gateway Timeouts and Payload Truncation
Most public and multi-tenant RPC layers enforce strict request timeouts between 10 and 30 seconds alongside maximum response size caps between 5 MB and 15 MB. When an indexer requests a block range yielding thousands of dense token transfer logs, payload construction exceeds gateway limits, generating 504 errors or dropped connections.
Multiplier Penalties on Usage-Unit Providers
Shared cloud RPC architectures frequently impose heavy compute unit multipliers on log queries. While a basic read call like `eth_blockNumber` may consume 10 units, `eth_getLogs` queries routinely cost 75 units or more on platforms like Alchemy. Running an aggressive historical backfill across millions of blocks rapidly burns through monthly usage quotas, triggering surprise invoice spikes or 429 throttling.
3. Production Indexer Architecture: Bounded Chunking and Concurrency
To ensure continuous, uninterrupted indexing, production pipelines split broad historical ranges into bounded chunks and ingest them concurrently.
Dynamic Chunk Sizing Heuristics
Rather than applying a fixed block range across the entire chain history, adapt chunk sizes to the density of the target event:
- High-Density Events such as Uniswap V3 Swaps: 500 to 1,000 blocks per chunk to avoid hitting response size limits.
- Moderate-Density Events such as Governance Proposals: 2,000 to 5,000 blocks per chunk.
- Narrow Specific Contracts: 5,000 to 10,000 blocks per chunk when filtering by both address and topic.
Executable Concurrent Python Indexer Example
The following script implements bounded concurrency, header authentication with `x-api-key`, and exponential backoff retry handling:
import asyncio
import aiohttp
import os
BLAZINGNODE_ENDPOINT = "https://rpc.blazingnode.com"
API_KEY = os.getenv("BLAZINGNODE_API_KEY", "YOUR_API_KEY")
HEADERS = {
"Content-Type": "application/json",
"x-api-key": API_KEY,
}
async def fetch_log_chunk(session, from_block, to_block, address, topics, retries=3):
payload = {
"jsonrpc": "2.0",
"method": "eth_getLogs",
"params": [{
"fromBlock": hex(from_block),
"toBlock": hex(to_block),
"address": address,
"topics": topics,
}],
"id": from_block,
}
for attempt in range(retries):
try:
async with session.post(BLAZINGNODE_ENDPOINT, json=payload, headers=HEADERS, timeout=15) as resp:
if resp.status == 200:
data = await resp.json()
if "result" in data:
return data["result"]
raise RuntimeError(f"RPC error payload: {data.get('error')}")
elif resp.status in (429, 502, 503, 504):
if attempt == retries - 1:
raise RuntimeError(f"HTTP {resp.status} after {retries} retries for blocks {from_block} to {to_block}")
retry_after = resp.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 0.5 * (2 ** attempt)
await asyncio.sleep(delay)
else:
text = await resp.text()
raise RuntimeError(f"HTTP {resp.status}: {text}")
except (aiohttp.ClientError, asyncio.TimeoutError):
if attempt == retries - 1:
raise
await asyncio.sleep(0.5 * (2 ** attempt))
raise RuntimeError(f"Exhausted {retries} retries fetching logs for blocks {from_block} to {to_block}")
async def run_indexer_backfill(start_block, end_block, chunk_size, address, topics, max_concurrency=10):
semaphore = asyncio.Semaphore(max_concurrency)
all_logs = []
async with aiohttp.ClientSession() as session:
async def worker(from_b, to_b):
async with semaphore:
logs = await fetch_log_chunk(session, from_b, to_b, address, topics)
print(f"Ingested blocks {from_b} to {to_b}: {len(logs)} logs found")
return logs
tasks = []
for current in range(start_block, end_block + 1, chunk_size):
chunk_end = min(current + chunk_size - 1, end_block)
tasks.append(worker(current, chunk_end))
results = await asyncio.gather(*tasks)
for chunk_result in results:
all_logs.extend(chunk_result)
return all_logs4. Reorg-Safe Checkpointing and Backfill Reconciliation
Because Polygon PoS experiences short reorganizations during validator sprint boundaries, event indexing must account for block hash integrity:
1. Store Block Hash with Event Data: Always link ingested logs to their specific `blockHash`. If a reorganization occurs, records tied to orphaned hashes can be removed deterministically. 2. Maintain a Safety Margin at the Tip: For real-time streaming, query up to `latest - 16` blocks for standard pipeline persistence. Reserve immediate tip reads for real-time memory caches only. 3. Persistent Checkpointing: Store the highest contiguous confirmed block number in durable storage after each batch commits, enabling instant recovery without re-scanning historical data.
5. Architectural Advantages with BlazingNode
High-throughput indexing demands consistent throughput without arbitrary rate throttling. When indexers backfill historical data or ingest bursts of contract events, shared RPC providers frequently choke on request volume.
BlazingNode provides dedicated capacity engineered specifically for high-throughput indexers:
- Predictable Request Metering: Queries targeting recent state within 128 blocks count strictly 1:1 toward your monthly plan volume with zero Compute Unit multipliers.
- Deep Calls Allowances: Paid plans include generous monthly allowances for deep historical queries older than 128 blocks, with affordable expansion packs available on demand.
- High Burst Headroom: 300 RPS burst headroom on Builder, Operator, and Pro plans, and up to 500 RPS on Business, ensuring rapid backfill execution without unexpected 429 throttling.
- Sub-50ms Response Speeds: Sub-50ms query response speeds for USA NorthEast for rapid event backfilling and tip tracking.
To evaluate real-time multi-provider responsiveness, visit [https://www.blazingnode.com/rpc-checker](https://www.blazingnode.com/rpc-checker). For targeted troubleshooting on log timeouts and connection limits, consult [https://www.blazingnode.com/fix/polygon-eth-getlogs-timeout](https://www.blazingnode.com/fix/polygon-eth-getlogs-timeout).