edgar-toolkit

SEC EDGAR extractors, with an XBRL mapping that gets banks and REITs right

The problem it solves

Ask a general purpose helper for JP Morgan's revenue and you get $9.6B. The real number is $182B.

Nothing raises. No warning appears. The helper looked up one XBRL concept, found a smaller line item that happened to exist, and returned it. You get a number that is wrong by a factor of nineteen and looks perfectly reasonable in a table.

The cause is that companies do not file the same line item under the same concept. A software company reports revenue as RevenueFromContractWithCustomerExcludingAssessedTax. A bank reports Revenues. A REIT reports net income as NetIncomeLossAvailableToCommonStockholdersBasic because of preferred stock.

field_mapping.py is an ordered chain of concepts per field. First match wins, and when nothing matches it returns None rather than guessing. 25 fields, 60 concepts, including the bank and insurer specific lines that general mappings leave out.

from edgar_toolkit.field_mapping import REVENUE_CONCEPTS, find_value

value, concept = find_value(flat_list, REVENUE_CONCEPTS, "FY 2024")
# (177558000000.0, "Revenues")   for a bank
# (391035000000.0, "RevenueFromContract...")  for a standard filer
# (None, None)                   when the filing genuinely does not report it

That module imports nothing. If it is the only file you take, it still works.

The SEC client

SEC publishes a ceiling of 10 requests per second per IP and enforces it. Exceed it and you get 429s, then a block, and the block is per IP rather than per key.

edgar/client.py is a token bucket at 5 per second, half the ceiling, with exponential backoff on 429. It is the only module allowed to make SEC requests, which is the rule that keeps the rate limit honest when several jobs run at once.

Identity is required, on purpose

edgar_user_agent()   # RuntimeError if EDGAR_USER_AGENT is unset

SEC requires automated clients to identify themselves with an application name and a contact address. This library refuses to send a request without one and will not substitute a default, because a shared default would put somebody else's name on your traffic and their block would become yours.

export EDGAR_USER_AGENT="Acme Research acme.example someone@acme.example"

Sinks

Extractors return records. Storing them is optional.

extract_company(ticker, cik) is pure: it returns a list of dicts and touches nothing else. run() adds orchestration, and everything that persists goes through a Sink, which is two methods.

from edgar_toolkit import MemorySink, NullSink, SqliteSink

EdgarFinancials().run(companies)                  # NullSink, keeps nothing
EdgarFinancials(MemorySink()).run(companies)      # rows in a dict
EdgarFinancials(SqliteSink("f.db")).run(companies)

Write your own for Postgres, DuckDB, parquet or a queue.

What is in the box

Module	Does
field_mapping.py	XBRL concept chains. The headline. Zero imports.
edgar/client.py	Rate limited SEC client, 429 backoff, one place for all requests.
edgar/resume.py	Cursor store so a multi day backfill resumes instead of restarting.
base.py	Extractor base: concurrency, timeouts, per company isolation.
sinks.py	Null, memory and SQLite sinks.
extractors/	Financials, 13F, SC 13D, Form 4 insider trades, dividends, prices, short interest, FRED.
examples/	Backfill scripts, including the rate limit safe Form 4 pull.

The error handling is load bearing

base.py carries rules that each exist because something went wrong:

Both the extract and the write sit inside the timeout. Wrapping only the extract let a hung write hold a concurrency slot forever.

Companies are processed in chunks. Creating 1,800 coroutines up front wasted memory and serialised the staleness probes before the semaphore could throttle anything.

The no op upsert uses ON CONFLICT DO NOTHING. INSERT OR IGNORE is SQLite only and silently failed to translate on the other backend.

CancelledError is always re raised. One bad company never stops a run.

Tests

pip install -e ".[dev]"
pytest

29 tests. The interesting ones are in test_field_mapping.py: disable the fallback so only the first concept is tried, and 4 fail immediately, all of them the bank and REIT cases. That is the check that the tests exercise the behaviour rather than restate the constants.

Honest limitations

field_mapping expects the flat list shape that edgartools produces from a statement. If you parse XBRL another way you will want the concept chains, not the helper functions.

The extractors were written for one application and lifted out. The pure parts, field_mapping, the client, the sinks and the base class, are tested. The individual extractors are not, beyond importing cleanly, and some still expect edgartools or yfinance to be installed.

Rate limiting is per process. Two processes on one IP share the SEC ceiling but not the bucket, so lower the limit rather than raising it.

Unmaintained by design. No PyPI package, no CI, no issue triage. Copy what you need. MIT.

Licence

MIT. See LICENSE.