[IMP] common: halve the time odev takes to start and exit by brinkflew · Pull Request #176 · odoo-odev/odev

brinkflew · 2026-07-26T18:15:53Z

Why

Every odev invocation paid ~1.6s of framework overhead before the command it was asked to run even began, and kept the user waiting a further ~0.39s after that command had already printed its result. For a tool run dozens of times a day, that overhead is how fast odev feels — and almost none of it was the command: on odev version, 0.02s of the 1.59s was the command itself.

This PR removes that overhead. Nothing about what odev does changes, only what it loads and when.

Results

odev version, best of 10 alternating runs of this branch and beta on the same machine (interleaved so that background load affects both equally):

Phase	`beta`	This PR	Δ
`import odev.common`	0.832s	0.304s	−0.528s
`init_framework()` + `start()`	0.261s	0.215s	−0.046s
dispatch (the command itself)	0.109s	0.021s	−0.088s
exit tail (after the last output)	0.386s	0.105s	−0.281s
Total wall clock	1.593s	0.647s	−0.946s (59% faster)
Python modules imported	1890	1022	−868

Absolute numbers are machine-dependent; the ratio held at 54–59% across every run. tools/benchmark_startup.py reproduces this table.

One caveat worth stating plainly: the first run after an update or a plugin change rebuilds the command index and costs roughly what beta costs today (measured 1.014s cold vs 0.558s warm on the same machine). Every run after it is the fast path, and the index rebuilds itself transparently.

What changed

Five causes, each paid unconditionally on every run:

1. Telemetry blocked the main thread. run_command's finally joined a thread doing an HTTPS round-trip to the telemetry endpoint, i.e. the user watched a finished command for a full RTT. Records are now spooled to ~/.config/odev/odev-telemetry.jsonl when a command ends and submitted in the background by a later run, which has its own command's runtime to do so. No invocation ever waits on the endpoint.

2. The debugger scan spawned two grep processes at import time. debug.py shelled out over the odev tree and every plugin repo on every run (~0.15s) to decide whether to disable spinners. It now runs on demand and caches its result, keyed on a walk of the source trees that costs ~9ms.

3. The framework imported dependencies only a few commands need. PyGithub, paramiko, InquirerPy/prompt_toolkit and black were pulled in by odev.common because the connector and mixin packages re-exported everything they contained. Those imports moved to their point of use, and the packages resolve names lazily through PEP 562 __getattr__ (new odev/common/lazy.py). Plugins doing from odev.common.connectors import GitConnector keep working unchanged, and TYPE_CHECKING declarations keep type checkers and IDE navigation intact.

networkx was only used to topologically sort the plugin dependency graph — replaced by a local Kahn sort. Cycle reporting is byte-identical, including the a → b → a and "depends on itself" phrasings.

4. Command discovery executed all 46 command modules just to read their names. So every run paid for every command, plugins included — that is where copier (~0.14s) and networkx came from. Names, aliases and help texts now live in an index cached in ~/.config/odev/odev-commands.json; a command module is imported only when that command actually runs. Plugin patching is replayed from the recorded registration order.

The index is invalidated by the odev version, each plugin's version, and a hash over the command files' paths and mtimes — so adding, removing, renaming or editing a command rebuilds it transparently.

Two bugs found on the way

Telemetry.update() never ran. Its len(_command_stack) != 1 guard was evaluated after run_command had popped the stack, so it always returned early: exit codes and execution times have never reached the endpoint. Now reported (and the exit code is the real one instead of a hardcoded 0).
console.py compared a class to the string "checkbox", always false, so bypass_prompt never honoured defaults for checkbox prompts. Fixed as a side effect of passing prompt names as strings.

Notes for reviewers

tests/fixtures/capture.py needed a fix. CaptureOutput attached its handlers to the loggers that existed when it was entered — which no longer covers command modules imported while a command runs. Four database tests failed until it was switched to capture on the root logger. Production was never affected: logging.basicConfig installs root handlers and child loggers propagate to them.
.ruff.toml gained a tools/* per-file-ignore for T201, since a standalone developer script reporting on stdout is the point. The alternative was nine # noqa comments.
odev help output is unchanged. Verified by capturing odev help, odev help -1 and odev help <cmd> for 11 commands (including plugin-patched run, test, pre-commit) before the change and diffing after — byte-identical.

Verification

pytest tests — 150 passed (3 new).
pre-commit run --all-files — clean.
basedpyright — back to exactly the 3 pre-existing errors, zero new.
Manual smoke over the lazily-imported paths, since those only fail at call time: version, version -p, help, help -1, help <cmd>, list, config, a git-backed command, and every plugin command including the patched ones.
New tests/tests/common/test_startup_performance.py guards the structural causes rather than timings, which are too noisy to assert on: the framework must import none of the heavy modules, starting it must import zero command modules, and resolving one command must import only that one. They warm the index up first, since building it is the one run that legitimately imports everything.

Follow-up (not in this PR)

odev-plugin-ai-upgrade's __init__.py does from . import commands, which still drags networkx into startup (~80ms). Deleting that line in that repo removes the last heavy import; odev discovers commands on its own and does not need it.

🤖 Generated with Claude Code

https://claude.ai/code/session_012tWyDuUYenE92nsBtG19Du

Every invocation paid ~1.6s of framework overhead before the requested command even began, and kept the user waiting ~0.35s after it had already printed its result. For a tool run dozens of times a day that overhead is the dominant part of how slow odev feels, and none of it was the command itself: `odev version` measured 1.59s median, of which 0.02s was the command. Five causes, all of them paid unconditionally: - Telemetry blocked the main thread on an HTTPS round-trip in `run_command`'s `finally`. Records are now spooled locally when a command ends and submitted in the background by a later run, so no invocation ever waits on the endpoint. This also fixes `Telemetry.update()`, which never ran: its `len(_command_stack) != 1` guard was evaluated after the stack had been popped, so exit codes and execution times were silently dropped. The employee check, which ran a vault lookup, an SSH-agent decryption and a call to git on every command, is now cached in the configuration. - Scanning the sources for interactive debuggers spawned two `grep` processes at import time. It now runs on demand and its result is cached on disk, keyed on a walk of the command trees that costs a fraction of the scan itself. - The GitHub API client, paramiko, InquirerPy and black were imported by the framework although only a handful of commands need them; the connector and mixin packages re-exported everything they contained. Those imports moved to their point of use and the packages resolve their names lazily. Networkx was only used to sort the plugin dependency graph and is replaced by a topological sort, reporting the same cycles. - Discovering commands executed all 46 command modules only to read their names, so every run paid for every command, plugins included. Names, aliases and help texts are now cached in an index and a command module is only imported once that command runs. Plugin patching is replayed from the recorded registration order, leaving the output of `odev help` unchanged. - `CaptureOutput` attached its handlers to the loggers that existed when it was entered, which no longer covers command modules imported while a command runs. It now captures on the root logger, like odev does in production. `odev version` goes from 1.59s to 0.73s median, measured by interleaving runs against a clean checkout, and imports 1022 modules instead of 1503. `tools/benchmark_startup.py` reports the breakdown, and the new tests guard the structural causes rather than timings, which are too noisy to assert on. Claude-Session: https://claude.ai/code/session_012tWyDuUYenE92nsBtG19Du

The first run of a new version has no index yet and legitimately imports every command to build one, so the startup tests only passed on a machine where a previous run had already warmed it up. On a clean checkout they measured the cold path and failed, and test_03 only passed because test_02 happened to run first and leave an index behind. Claude-Session: https://claude.ai/code/session_012tWyDuUYenE92nsBtG19Du

The script carries a shebang and is meant to be run directly, which ruff enforces through EXE001. The mode bit went unnoticed locally because the working tree lives on a filesystem that does not report it faithfully. Claude-Session: https://claude.ai/code/session_012tWyDuUYenE92nsBtG19Du

brinkflew · 2026-07-26T18:49:16Z

The follow-up mentioned at the end of the description is now open: odoo-odev/odev-plugin-ai-upgrade#9.

It removes the from . import commands in that plugin's __init__.py, which was the last thing pulling a heavy import (networkx, ~300 modules) into odev's startup through load_plugins().

With both changes applied, odev version on this machine:

	`beta`	This PR (#176)	+ plugin PR #9
Total wall clock (median)	1.593s	0.688s	0.503s
Modules imported	1890	1022	714
Heavy modules at startup	several	`networkx`	none

This PR stands on its own — #9 is not required for it to merge, and the reverse is also true.

Merging beta brings in the deferred imports of #176, which moved the GitHub client out of the module scope. The GithubConnector extracted here keeps them inside the methods that need them, and the connectors package registers it with the lazy exports instead of importing it eagerly. Also bumps the version to 4.31.0, one increment above beta.

odev.common.logging configures logging on import, but logging.basicConfig is a no-op once the root logger has handlers: whether odev's handler gets installed depends on whether that import happens before or after pytest sets its own up, and importing the sandbox from conftest tips it. Combined with the capture added by #176 every record then reached the output twice, and a plain logger.info became a console.print that #177's run_hook test asserts on. The handler is dropped in pytest_configure so the suite no longer depends on import order. Also resolves the version command test, which this branch made namespace-agnostic while #175 rewrote it, and bumps the version to 4.31.2, one increment above the base branch.

Odev imports every plugin package on startup so that plugins can patch the framework before commands are registered. This one additionally pulled in its own `commands` subpackage, which imports `upgrade`, which imports networkx: every odev invocation paid for it, including the ones that have nothing to do with this plugin. Nothing needed those imports. Odev discovers commands itself, loading `commands/*.py` directly rather than through the package, and this plugin patches nothing on load: its `common/__init__.py` is empty. An empty `__init__.py` matches the four sibling plugins that do not patch either. Together with odoo-odev/odev#176 this removes the last expensive import from odev's startup path: 1022 modules down to 714, and `odev version` from 0.688s to 0.503s. Claude-Session: https://claude.ai/code/session_012tWyDuUYenE92nsBtG19Du

`commands/upgrade.py` imports networkx to sort modules by dependency, but only copier and jinja2 were declared: the package happened to be installed because odev core depended on it. odoo-odev/odev#176 removed core's last use of networkx, so this plugin is now its only consumer and cannot keep relying on someone else pulling it in. Claude-Session: https://claude.ai/code/session_012tWyDuUYenE92nsBtG19Du

…st suite (#178) ## Why This started as filling coverage gaps, and each step surfaced the next: 1. `.coveragerc` gates at 60% and the suite sat at **64%**, with the gap widest on pure logic the rest of the framework leans on. Writing those tests surfaced **six defects in the helpers** — without fixing them the tests would have pinned broken behaviour. 2. Verifying the fixes meant running the suite repeatedly, which is when it became clear that **two suites cannot run at once**: identical invocations produced anywhere from 0 to 68 failures, and the machine had accumulated 16 orphaned `/tmp/odev-test-*` directories. 3. Isolating the runs revealed that three concurrent suites exhaust PostgreSQL's connection slots, which turned out to be a **connection leak in odev itself**, not in the suite. The four sections below are independent and the commits are ordered to be reviewed in sequence. ## 1. Helper defects — `42569c4` | Location | Defect | |---|---| | `connectors/postgres.py` `columns_exist` | Returned `[]` when **none** of the requested columns existed — indistinguishable from all being present. `CREATE TABLE IF NOT EXISTS` leaves an existing table alone, so the missing-columns pass is the only thing that can migrate a table created from an older definition; it silently added nothing. | | `postgres.py` `PostgresDatabase.tables` | A class attribute, so every instance shared one registry and tables from different databases collided on their name alone. | | `string.py` `quote` | Chose its delimiter with `max()` over both quote offsets, picking the **last** rather than the first, mis-quoting any string mixing them. | | `version.py` `OdooVersion.__bool__` | Always `True` — `module` is padded to `MIN_VERSION_LENGTH` and is never an empty tuple. | | `string.py` `min_indent` | Raised `ValueError` on a text without any non-blank line, reachable from `odev help` through `dedent`. | | `float_to_hours`, `strip_styles` | Broken, but called nowhere in odev nor in the plugins. **Left alone**, documented in the tests with the correction spelled out. | `columns_exist` has exactly one caller, and it runs after `CREATE TABLE IF NOT EXISTS`, so the fix cannot make it issue `ALTER TABLE` against a missing table. ## 2. Coverage for the untested helpers — `be95197` - **`test_string.py`** (new) — `string.py` had no test module at all, despite backing `odev help`, `odev history` and the local database listing query. Sizes and their round-trip, indentation, joining, the `dirty_only` × `force_single` quoting matrix, Rich markup helpers, and the `help` column alignment contract. - **`test_git_worktree.py`** (new) — `connectors/git.py` was the least-covered large module (34%), and its `GitWorktree` parser turns `git worktree list --porcelain` into the objects the whole `fetch` / `pull` / `worktree` family works with. Porcelain parsing (branch, detached, bare, locked, prunable with reasons), the `-odev-` local-branch split that `create_worktree` writes and `fetch` / `pull` read back, identity by path, and `pending_changes` including the two swallowed `GitCommandError` messages. No network, no real repository. - **`test_postgres_table.py`** (new) — `PostgresTable.__add_missing_column`, the datastore's migration path, was entirely unreached; this covers it including the `InvalidTableDefinition` primary-key branch. - **`test_version.py`** — ordering (`15.0 < 16.0 < saas-16.4 < 17.0 < master`) is what actually picks a revision at runtime and nothing compared two versions. Corrections to existing tests, in the same commit: - **`test_bash.py` shelled out to a real `sudo cat >> /etc/shadow`.** The premise that the command fails only holds for an unprivileged user whose shell cannot open the redirection — a machine granting passwordless sudo runs it for real, and as root it appends to the file or hangs on stdin. The subprocess and the effective user are now simulated, which also lets the elevation path be asserted rather than inferred. - `test_odev.py` left a command line behind in `sys.argv` for whichever test ran next. - `tests/fixtures/case.py` — `_patches` was a list defined on `OdevTestCase` and mutated through `cls._patches.append`, so every subclass shared it and each class tore down the patches of all the classes before it. ## 3. An isolated, self-cleaning test suite — `87dc0e4`, `86630d3`, `716c8e1` `Odev.name` was the constant `"odev-test"` and **every** shared resource derived from it, so two suites shared one namespace and actively destroyed each other: - `test_99_delete_expression` ran `odev delete --expression "^odev-test-[a-z0-9]{8}" --include-whitelisted` against the real PostgreSQL, deleting a concurrent run's databases. - `PostgresDatabase.drop()` terminates every backend on `datname`, so each class teardown killed a concurrent run's cursors. - `CREATE TABLE IF NOT EXISTS` is not atomic, and `Config.save()` truncate-writes a fixed path — hence `UniqueViolation` on `pg_type_typname_nsp_index` and `DuplicateOptionError` from a torn config. A run now claims a sandbox named after itself and holds an exclusive `flock` on it for its whole life. Everything — datastore, test databases, config, temp directories — is named after it or nested under it. Cleanup runs at `pytest_sessionfinish`, which pytest calls from a `finally`, so `Ctrl+C` is covered; `SIGTERM` becomes the same orderly exit; and the next run's sweep collects whatever a `SIGKILL` left, because the kernel releases the lock when the owner dies whatever the cause. **The suite was also writing outside its sandbox**, which is worth a look on its own: - `TestSetup` ran the install scripts against their real destinations, so running the suite **repointed the developer's `~/.local/bin/odev` and bash-completion symlinks at whichever checkout it ran from**. `symlink.py` computed the destination halfway through creating it, leaving no way to redirect it; that decision moves to `link_path`. - Tests cloned into the real `~/odoo/repositories`. The repositories, dumps and upgrade paths now point inside the sandbox, as does `CONFIG_DIR` — which also means the suite no longer picks up whichever plugins the developer happens to have installed, so a local run and CI exercise the same code. `87dc0e4` is a separate product fix this surfaced: `LocalDatabase.is_odoo` checks that a database exists and then connects to it, and any process can drop it in between — `odev list` inspects every database in turn and would fail outright because one went away. Interrupt handling is covered by `tests/tests/common/test_interrupts.py`: odev captures `SIGINT` around every query to cancel just that statement, so a `Ctrl+C` was previously swallowed and the run carried on. Letting it through instead abandons the connection mid-statement, so the interrupt is recorded and acted upon at the next test boundary. ## 4. Connection lifetime — `47499cb`, `cd07007` Both database context managers built a **second, unconnected** connector to close instead of the one they had connected, so `disconnect()` did nothing and the connection stayed open until the garbage collector got to it: ```python def __enter__(self): self.connector = self._connector_class(self.name).__enter__() # connector A, connected return self def __exit__(self, *args): self._connector_class(self.name).__exit__(*args) # connector B, never connected ``` `ensure_connected` runs every database method inside its own block and those blocks nest — `is_odoo` opens one and then calls `table_exists`, which opens another — so this meant a fresh backend per call. Closing the right connector is **not enough on its own**: an inner block would close the connection the enclosing one is still using. The blocks are now reentrant and share a single connector, counted in `PostgresConnectorMixin` so both classes get the same behaviour. The datastore holds its connection instead of reopening it per read — every command reads it and it lives as long as the process, which is not true of the databases odev walks through for `list` or `delete`. A connection pool keyed per database was considered and set aside: `list --all` and `delete --expression` touch **every** database on the server briefly, so a per-database pool would hold one idle backend per Odoo database until the process ends — the very exhaustion this fixes — unless it also grew a global cap and idle eviction. Measured over a full suite run: | | before | after | |---|---|---| | peak backends held | 42 | **3** | | mean backends held | 8.7 | **0.8** | | suite duration | 54.8s | **33.6s** | | three concurrent suites | died on `max_connections` | **249 passed each**, peak 7 backends | The speedup was not the goal — it is what a backend fork plus an authentication round-trip per query costs. ## Coverage | Module | Before | After | |---|---|---| | `common/string.py` | 85% | **100%** | | `common/version.py` | 96% | **100%** | | `common/postgres.py` | 81% | **93%** | | `common/connectors/git.py` | 34% | **40%** | | **Total** | **64%** | **65%** | ## Verification - `pytest tests` — **249 passed**, from 242 on the first revision - Two and three concurrent suites — **249 passed each**, repeatedly, leaving zero directories and zero databases behind - `SIGINT`, `SIGTERM` and `SIGKILL` mid-run — each verified to leave nothing behind, the last one via the next run's sweep - `odev list --all`, `odev history`, `odev version` — smoke-checked, no connections surviving the process - `pre-commit run --all-files` — clean - `basedpyright` — 3 errors, all pre-existing on `beta`; **0 new** ## Notes for reviewers - `odev/_version.py` is bumped once, to `4.29.10`. `origin/beta` is at `4.29.9`; PRs #175, #176 and #177 each bump from the same base, so whichever merges second needs a one-line rebase. - `LocalDatabase.connector: PostgresConnector | None = None` was removed as dead — `ConnectorMixin.__init__` overwrites it with the connector *class* at construction, which also meant the `if self.connector is not None` guard in `_restore` never protected anything. It is now the `isinstance` check `drop()` already used. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012tWyDuUYenE92nsBtG19Du

brinkflew added 3 commits July 26, 2026 20:09

brinkflew requested a review from sea-odoo July 26, 2026 18:38

brinkflew mentioned this pull request Jul 26, 2026

[IMP] tooling: stop importing the plugin's commands at load time odoo-odev/odev-plugin-ai-upgrade#9

Merged

brinkflew mentioned this pull request Jul 26, 2026

[FIX] common: close database connections, fix helpers, isolate the test suite #178

Merged

sea-odoo previously approved these changes Jul 30, 2026

View reviewed changes

[IMP] version: bump to 4.30.4 for the merge into beta

fb570ce

brinkflew dismissed sea-odoo’s stale review via fb570ce August 6, 2026 21:32

brinkflew merged commit 556c4a9 into beta Aug 6, 2026

brinkflew deleted the avs-startup-performance branch August 6, 2026 21:33

This was referenced Aug 6, 2026

[IMP] plugin: search, list and inspect odev plugins #174

Merged

[FIX] common: only report an available update when there is one #175

Closed

[FIX] odoobin: read the repository name from its git remote #179

Closed

Navigation Menu

Uh oh!

[IMP] common: halve the time odev takes to start and exit - #176

[IMP] common: halve the time odev takes to start and exit#176
brinkflew merged 4 commits into
betafrom
avs-startup-performance

brinkflew commented Jul 26, 2026 •

edited

Loading

Uh oh!

brinkflew commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

brinkflew commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Results

What changed

Two bugs found on the way

Notes for reviewers

Verification

Follow-up (not in this PR)

Uh oh!

brinkflew commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

brinkflew commented Jul 26, 2026 •

edited

Loading