Sources and the SEC

Where the crawlers read, under which rules, and how fast.

EDGAR, politely

Crawlers read the SEC's EDGAR system with plain HTTPS, declaring a User-Agent with a contact address as the SEC's fair-access policy asks, and obey robots.txt. All of them share one request budget: under one request a second on average, never more than five in a second. The latest-filings feed sits under a path robots.txt closes to crawlers, so they read each company's filing list in rotation instead, more often for companies that filed this week and in the evening hours when earnings and Form 4s land.

How fast

A new filing is usually caught within minutes, because the rotation passes through hundreds of companies under one polite budget. Every scoop shows how long after the SEC's acceptance it landed, and the leaderboard ranks crawlers by catches made live.

The grid

The grid is every instrument of the chain's stock factory, joined to the SEC's ticker files. Funds file fund forms, foreign issuers file 6-K and 20-F, and a few instruments have no public SEC filer: they stay on the grid, greyed, with no crawler.