diff --git a/.githooks/_check_tree_isolation.sh b/.githooks/_check_tree_isolation.sh index 4bfe6ec..4a8376f 100755 --- a/.githooks/_check_tree_isolation.sh +++ b/.githooks/_check_tree_isolation.sh @@ -15,8 +15,8 @@ set -euo pipefail # Allowlisted lines: file:linenumber pairs that are permitted. ALLOWLIST=( - "tree/item.py:28" - "tree/item.py:29" + "tree/item.py:30" + "tree/item.py:31" ) is_allowlisted() { diff --git a/agent.md b/agent.md new file mode 100644 index 0000000..fcbd898 --- /dev/null +++ b/agent.md @@ -0,0 +1,64 @@ +# Agent Guide — Editable-Tree-Model-Example (compact) + +_High-signal rules for AI agents. Keep this brief and actionable._ +**Last updated:** 2026-06-13 + +## 1) First commands (always) + +```bash +. .venv/bin/activate +timeout 1200 make gate +``` + +- Tools live in `.venv`. +- `make gate` is mandatory before every commit. + +## 2) Mandatory delivery loop (do not skip steps) + +For plan-based work, execute exactly this loop: + +1. **Pick one next unchecked plan item** (single scope). +2. **Implement only that scope**. +3. **Run targeted tests** for touched files. +4. **Run full gate** (`timeout 1200 make gate`). +5. **Commit immediately** (message references plan item). +6. **Mark plan checkbox `[x]`** only after commit. +7. Repeat for the next item. + +Hard rules: +- If tests/gate fail: go back to implementation; **no commit**. +- Do not batch multiple plan items into one commit unless plan explicitly says so. +- Do not stop at “green but uncommitted”. + +## 3) Critical architecture facts (easy to miss) + +1. **Undo edit path bypasses `JsonTreeItem.set_data()`** + - Real replay path: `DocumentMutationGateway` → undo command → `undo/diff.py:DiffApplier.apply()`. + - Type/value fixes often require changes in both item logic and `DiffApplier`. + +2. **`mpq` whole numbers are inferred as FLOAT unless converted** + - Convert `mpq(n,1)` to `int` where integer semantics are required. + +3. **UI uses proxy model** + - Map indices proxy↔source before touching tree items. + +## 4) Isolation constraints (must hold) + +- `editors/inline/*`, `editors/windowed/*` must not import `app/`, `documents/`, `tree/`. +- `editors/factory.py`, `editors/context.py` must not import `app/`, `documents/`. +- `tree/` must not import `app/`, `documents/`, `editors/`, `delegates/`, `state/`, `validation/`. +- No reflection (`getattr` / `hasattr` / `TYPE_CHECKING`) outside allowlist. + +## 5) Minimal file map for common work + +- Types/coercion: `tree/types.py`, `tree/item_coercion.py` +- Item/model behavior: `tree/item.py`, `tree/model.py` +- Undo replay: `undo/commands.py`, `undo/diff.py` +- Tab composition/lifecycle: `documents/composition/*`, `app/tab_lifecycle.py`, `app/main_window.py` +- Validation: `documents/controllers/validation.py`, `validation/*` + +## 6) Testing quick rules + +- Use `QT_QPA_PLATFORM=offscreen` for pytest. +- Prefer focused tests during development, then full gate. +- Add regression tests for every bug fix or plan checkpoint. diff --git a/ai-memory/repo-map.md b/ai-memory/repo-map.md index 005c6db..43be18c 100644 --- a/ai-memory/repo-map.md +++ b/ai-memory/repo-map.md @@ -2,8 +2,8 @@ _This is a condensed index and architectural summary. LLM agents should refer to direct source files for implementation details._ -**Last updated:** 2026-06-01 (after tree-upward-imports refactor; added `core/`, `tree/codecs/`, -tree-isolation rule, and `check-tree-isolation` target). +**Last updated:** 2026-06-13 (after raw-numeric edit-flow fix; added DiffApplier RAW_FLOAT routing, +integer promotion for whole-number mpq edits, and `agent.md`). ## 1) High-level Purpose @@ -25,7 +25,7 @@ Model". | **Type System** | `tree/types.py` (Definitions), `tree/item_coercion.py` (Conversion) | | **Editor Widgets** | `editors/factory.py` (dispatch), `editors/inline/`, `editors/windowed/` | | **Delegates** | `delegates/value.py` (paint + createEditor → editors.factory), `delegates/formatting/` | -| **Undo System** | `undo/commands.py` (Operations), `undo/diff.py` (Surgical replay) | +| **Undo System** | `undo/commands.py` (Operations), `undo/diff.py` (Surgical replay + RAW_FLOAT routing) | | **Structural Ops** | `tree_actions/` (Clipboard, DnD, Move, Sort, Anchors) | | **Validation** | `validation/` (JSON-Schema), `app/validation_presenter.py` | | **Theming** | `themes/`, `app/theme_controller.py` | @@ -43,6 +43,15 @@ Model". how data is handled. Don't scatter type logic in the UI. - **Surgical Model Updates**: The `DiffApplier` (`undo/diff.py`) is used during Undo/Redo to emit minimal Qt signals. This preserves UI state like selection and expansion that would be lost on a full model reset. + **Important**: `DiffApplier.apply()` bypasses `JsonTreeItem.set_data()` — special type handling (e.g., `RAW_FLOAT` + routing to `_set_raw_numeric_value`) must be added to `DiffApplier.apply()` explicitly. +- **Edit flow path**: Editor → `editors/factory.set_value_model_data()` → `DefaultEditContext.commit()` → + `DocumentMutationGateway.commit_set_data()` → `CommandDispatcher.push_edit_value()` → `_EditValueCmd` → + `DiffApplier.apply()` → `JsonTreeItem._apply_typed_value()` / `_set_raw_numeric_value()`. +- **Raw numeric values**: Unsupported numeric literals (overflow, underflow, non-finite) are preserved as + `RawNumericValue` (`core/raw_numeric.py`) with type `JsonType.RAW_FLOAT`. Edits go through + `JsonTreeItem._set_raw_numeric_value()` which handles: safe-parse → int/float conversion, unchanged → preserve, + regex-match → keep raw, regex-violate → reject. Whole-number mpq results are promoted to `int` for `INTEGER` type. - **No external `data_store.*` reads** (Plan 20). External callers (`app/`, `undo/`, `tree_actions/`, `state/`) must reach state through typed `JsonTab.*` properties. The pre-commit hook `.githooks/_check_data_store_leaks.sh` enforces this for 17 retired attributes. @@ -138,6 +147,7 @@ editors/ │ ├── mpq_spinbox/ QMpqSpinBox (spinbox.py + validator.py). │ ├── datetime/ BetterDateTimeEditor + validator (enums/regex imported from core/). │ ├── affix_composite.py AffixCompositeEditor (prefix/suffix + spinbox). +│ ├── raw_numeric_line.py RawNumericLineEdit + RawNumericValidator for RAW_FLOAT. │ ├── secret_line.py _SecretLineEdit + _SecretEditorWatcher. │ └── caps_safe_line.py _CapsLockSafeLineEdit + lock-key constants. └── windowed/ Modal dialog editors (no app/documents/tree imports). @@ -153,6 +163,9 @@ editors/ ``` core/ ├── __init__.py +├── raw_numeric.py RawNumericValue dataclass + narrow edit regex validator. +├── safe_mpq.py Safe mpq parsing (parse_mpq, safe_mpq_from_text, MpqParseResult). +├── frozen_value.py Legacy FrozenValue alias → RawNumericValue (compatibility). └── datetime_parsing/ Pure datetime parsing (no Qt dependency). ├── __init__.py Re-exports DateTimeCategory, parse_datetime_text, etc. ├── enums.py DateTimeCategory enum. diff --git a/ai-memory/todo-n-fixme.md b/ai-memory/todo-n-fixme.md index e1f3d93..5ad3918 100644 --- a/ai-memory/todo-n-fixme.md +++ b/ai-memory/todo-n-fixme.md @@ -1,9 +1,8 @@ # TODO & FIXME -_Last updated: **2026-06-01** (post code-quality audit). Minor smells -and speculative wishlist entries were pruned; only actionable, -meaningful work remains. Format: -`- [ ] [scope] description — file:symbol`._ +_Last updated: **2026-06-13** (post raw-numeric edit-flow fix). Added +completed entry for DiffApplier RAW_FLOAT routing and integer promotion. +Format: `- [ ] [scope] description — file:symbol`._ ## High priority — architecture (audit §8) @@ -42,6 +41,11 @@ meaningful work remains. Format: ## Low priority — hygiene & dead code (audit §6) +- [x] [bug] Fix raw numeric edit flow: `DiffApplier.apply()` now routes + `RAW_FLOAT` edits through `_set_raw_numeric_value()` instead of + bypassing it; whole-number mpq results are promoted to `int` for + `INTEGER` type (displaying "42" not "42.0"). + — `undo/diff.py:apply`, `tree/item.py:_set_raw_numeric_value` - [x] [hygiene] Remove deprecated shims `_closed_tabs_stack` / `_MAX_CLOSED_TABS` and the no-op stubs `_setup_validation_dock` / `_setup_schemas_menu` — tests and production code now call the diff --git a/app/loading/__init__.py b/app/loading/__init__.py new file mode 100644 index 0000000..13575be --- /dev/null +++ b/app/loading/__init__.py @@ -0,0 +1 @@ +"""Loading coordination for file open and reload operations.""" diff --git a/app/loading/builder.py b/app/loading/builder.py new file mode 100644 index 0000000..c940c89 --- /dev/null +++ b/app/loading/builder.py @@ -0,0 +1,288 @@ +"""Chunked cooperative model/tree builder. + +This module provides a builder that constructs the item tree/model off to +the side using an explicit work stack and yields control after each time +slice. The view receives no model until the build result is complete. +""" + +from __future__ import annotations + +import time +from typing import Any + +from PySide6.QtCore import QObject, QTimer, Signal + +from app.loading.cancellation import CancellationToken +from app.loading.progress import STAGE_BUILDING_TREE, ProgressReporter +from core.raw_numeric import RawNumericValue +from tree.item import JsonTreeItem, SecretNamePredicate, _default_secret_name_predicate +from tree.item_coercion import compute_editable +from tree.model import JsonTreeModel +from tree.types import SECRET_FAMILY, TEXT_FAMILY, JsonType, parse_json_type + +# Target time slice for each batch (16ms for ~60fps responsiveness) +TARGET_SLICE_MS = 16 +# Maximum time before yielding (for tests) +MAX_SLICE_MS = 50 + + +class ChunkedTreeBuilder(QObject): + """Builds a JsonTreeModel incrementally in time-sliced batches. + + The builder processes work items from an explicit stack and yields + control after each time slice. The model is not bound to any view + until the build is complete. + + Signals + ------- + finished(model) + Emitted when the build is complete with the finished model. + progress(done, total) + Emitted to report progress within the building stage. + """ + + finished = Signal(object) + cancelled = Signal() + progress = Signal(int, int) + + def __init__( + self, + data: Any, + *, + show_root: bool = False, + reporter: ProgressReporter | None = None, + cancellation_token: CancellationToken | None = None, + icon_provider=None, + parent: QObject | None = None, + ) -> None: + super().__init__(parent) + self._data = data + self._show_root = show_root + self._reporter = reporter + self._cancellation_token = cancellation_token + self._icon_provider = icon_provider + self._model: JsonTreeModel | None = None + self._total_items = 0 + self._built_items = 0 + self._latest_path = "" + self._root_item: JsonTreeItem | None = None + self._work_stack: list[tuple[JsonTreeItem, list[tuple[str | int | None, Any]], int, str]] = [] + self._secret_name_predicate: SecretNamePredicate = _default_secret_name_predicate + + def start(self) -> None: + """Start the chunked build process. + + Uses a QTimer to schedule work slices, allowing the event loop + to process other events between slices. + """ + if self._reporter is not None: + self._reporter.stage(STAGE_BUILDING_TREE) + + # Keep progress indeterminate for real loads. A full pre-count is also + # a recursive walk of the whole document, which would reintroduce a GUI + # freeze before chunked construction starts. + self._total_items = 0 + self._built_items = 0 + self._latest_path = "" + + self._root_item = _make_shallow_item( + None, + self._data, + None, + secret_name_predicate=self._secret_name_predicate, + ) + self._push_children(self._root_item, self._data, "") + + # Schedule the first work slice + QTimer.singleShot(0, self._do_work_slice) + + def _do_work_slice(self) -> None: + """Process a time slice of work items.""" + if self._is_cancelled(): + self._on_cancelled() + return + + start_time = time.monotonic() + slice_ms = 0 + + while slice_ms < TARGET_SLICE_MS: + if self._is_cancelled(): + self._on_cancelled() + return + + if not self._build_one_item(): + self._on_build_complete() + return + + self._built_items += 1 + if self._reporter is not None: + self._reporter.detail(self._built_items, self._latest_path) + self._reporter.tick(0, 0) + self.progress.emit(self._built_items, self._total_items) + + slice_ms = (time.monotonic() - start_time) * 1000 + + if slice_ms > MAX_SLICE_MS: + # Defensive cap for unusually slow item construction. The next + # slice is still scheduled through the event loop below. + pass + + if not self._work_stack: + if self._is_cancelled(): + self._on_cancelled() + return + self._on_build_complete() + return + + # Schedule the next work slice + QTimer.singleShot(0, self._do_work_slice) + + def _is_cancelled(self) -> bool: + return self._cancellation_token is not None and self._cancellation_token.is_cancelled + + def _on_cancelled(self) -> None: + """Drop partial build state and notify cooperative cancellation.""" + self._work_stack.clear() + self._root_item = None + self._model = None + self._latest_path = "" + self.cancelled.emit() + + def _on_build_complete(self) -> None: + """Called when the build is complete.""" + if self._model is None: + assert self._root_item is not None + self._model = JsonTreeModel( + None, + show_root=self._show_root, + icon_provider=self._icon_provider, + root_item=self._root_item, + estimated_item_count=self._built_items + 1, + ) + if self._reporter is not None: + self._reporter.tick(0, 0) + self.progress.emit(self._total_items, self._total_items) + self.finished.emit(self._model) + + def _push_children(self, item: JsonTreeItem, value: Any, parent_path: str) -> None: + """Push a child-iteration frame for a container item.""" + if isinstance(value, dict): + entries = list(value.items()) + elif isinstance(value, list): + entries = [(None, child_value) for child_value in value] + else: + return + if entries: + self._work_stack.append((item, entries, 0, parent_path)) + + def _build_one_item(self) -> bool: + """Build one pending child item. + + Returns False when no work remains. + """ + while self._work_stack: + parent_item, entries, index, parent_path = self._work_stack[-1] + if index >= len(entries): + self._work_stack.pop() + continue + + self._work_stack[-1] = (parent_item, entries, index + 1, parent_path) + name, value = entries[index] + path_segment: str | int | None = index if name is None else name + self._latest_path = _append_json_pointer(parent_path, path_segment) + child = _make_shallow_item( + parent_item, + value, + name, + secret_name_predicate=self._secret_name_predicate, + ) + parent_item.append_child(child) + self._push_children(child, value, self._latest_path) + return True + + return False + + +def _make_shallow_item( + parent_item: JsonTreeItem | None, + value: Any, + name: str | int | None, + *, + secret_name_predicate: SecretNamePredicate, +) -> JsonTreeItem: + """Create an item without recursively constructing descendants.""" + json_type = parse_json_type(value) + + if isinstance(value, RawNumericValue) and json_type not in TEXT_FAMILY and json_type not in SECRET_FAMILY: + json_type = JsonType.RAW_FLOAT + + if json_type not in (JsonType.ARRAY, JsonType.OBJECT): + return JsonTreeItem( + parent_item, + value, + name, + secret_name_predicate=secret_name_predicate, + ) + + item = JsonTreeItem.__new__(JsonTreeItem) + item.parent_item = parent_item + item._secret_name_predicate = secret_name_predicate + item.name = name + item.child_items = [] + item.explicit_type = False + item._row_in_parent = -1 + item._children_dirty = True + item.json_type = json_type + item.value = [] if json_type is JsonType.ARRAY else {} + item.editable = compute_editable(item.json_type, item.value, item.EDITABLE_BLOB_LIMIT) + return item + + +def _count_items(data: Any) -> int: + """Count the total number of items in the data structure.""" + if isinstance(data, dict): + count = 0 + for value in data.values(): + count += 1 + _count_items(value) + return count + elif isinstance(data, list): + count = 0 + for value in data: + count += 1 + _count_items(value) + return count + return 0 + + +def _append_json_pointer(parent_path: str, segment: str | int | None) -> str: + """Append one segment to a JSON Pointer-style path.""" + if segment is None: + return parent_path + token = _escape_json_pointer_token(str(segment)) + if not parent_path: + return f"/{token}" + return f"{parent_path}/{token}" + + +def _escape_json_pointer_token(token: str) -> str: + """Escape a JSON Pointer token.""" + return token.replace("~", "~0").replace("/", "~1") + + +def build_model_sync( + data: Any, + *, + show_root: bool = False, +) -> JsonTreeModel: + """Build a JsonTreeModel synchronously. + + This is a convenience function for tests and simple use cases. + """ + return JsonTreeModel(data, show_root=show_root) + + +__all__ = [ + "ChunkedTreeBuilder", + "build_model_sync", + "TARGET_SLICE_MS", + "MAX_SLICE_MS", +] diff --git a/app/loading/cancellation.py b/app/loading/cancellation.py new file mode 100644 index 0000000..56cc460 --- /dev/null +++ b/app/loading/cancellation.py @@ -0,0 +1,43 @@ +"""Thread-safe cooperative cancellation primitives for loading flows.""" + +from __future__ import annotations + +import threading + + +class CancelledError(RuntimeError): + """Raised when cooperative cancellation has been requested.""" + + +class CancellationToken: + """A thread-safe cancellation token. + + The token uses a standard :class:`threading.Event` so producer and + consumer code can coordinate cancellation across threads without any Qt + synchronization primitives. + """ + + __slots__ = ("_cancelled",) + + def __init__(self) -> None: + self._cancelled = threading.Event() + + def cancel(self) -> None: + """Request cancellation. + + The operation is idempotent; repeated calls keep the token cancelled. + """ + self._cancelled.set() + + @property + def is_cancelled(self) -> bool: + """Return ``True`` once cancellation has been requested.""" + return self._cancelled.is_set() + + def raise_if_cancelled(self) -> None: + """Raise :class:`CancelledError` if cancellation has been requested.""" + if self.is_cancelled: + raise CancelledError("Operation was cancelled") + + +__all__ = ["CancellationToken", "CancelledError"] diff --git a/app/loading/coordinator.py b/app/loading/coordinator.py new file mode 100644 index 0000000..9710261 --- /dev/null +++ b/app/loading/coordinator.py @@ -0,0 +1,454 @@ +"""Load coordinator for file open and reload operations. + +This module owns the open and reload flows. It integrates worker parsing, +progress reporting, chunked model build, and the delayed progress widget. +""" + +from __future__ import annotations + +import time +import uuid +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable + +from PySide6.QtCore import QObject, Qt, QThread, QTimer, Signal +from PySide6.QtWidgets import QApplication, QMessageBox + +import state.view_state as view_state +from app.loading.builder import ChunkedTreeBuilder +from app.loading.cancellation import CancellationToken +from app.loading.progress import ( + STAGE_APPLYING_RELOAD, + STAGE_BINDING_UI, + STAGE_COMPLETE, + STAGE_DISCOVERING_SCHEMA, + STAGE_VALIDATING_DOCUMENT, + ProgressReporter, +) +from app.loading.progress_dialog import LoadingProgressDialog +from app.loading.worker import ParseWorker +from documents.seams.document_protocol import Document +from settings import LOADING_HARD_TIMEOUT_SECONDS +from tree.model import JsonTreeModel + + +@dataclass +class _LoadTask: + task_id: str + mode: str + path: str + tab: Document | None = None + thread: QThread | None = None + worker: ParseWorker | None = None + builder: ChunkedTreeBuilder | None = None + data: Any = None + source_format: str | None = None + token: CancellationToken = field(default_factory=CancellationToken) + + +class LoadCoordinator(QObject): + """Coordinates file open and reload operations. + + The coordinator is the single owner of loading flows. It receives + callbacks to the main window for tab creation, status updates, and + error presentation. + + Signals + ------- + stage_changed(stage_name) + Emitted when the loading stage changes. + """ + + stage_changed = Signal(str) + task_finished = Signal(str, bool) + _parse_succeeded = Signal(str, object) + _parse_failed = Signal(str, object) + + def __init__(self, window, parent: QObject | None = None) -> None: + super().__init__(parent) + self._window = window + self._reporter: ProgressReporter | None = None + self._progress_dialog: LoadingProgressDialog | None = None + self._current_task_id: str | None = None + self._tasks: dict[str, _LoadTask] = {} + self._completed_task_results: dict[str, bool] = {} + self._cancelled_task_ids: set[str] = set() + self._parse_succeeded.connect(self._on_parse_finished, Qt.ConnectionType.QueuedConnection) + self._parse_failed.connect(self._on_parse_failed, Qt.ConnectionType.QueuedConnection) + + def set_reporter(self, reporter: ProgressReporter | None) -> None: + """Set the progress reporter for stage notifications.""" + self._reporter = reporter + + def _emit_stage(self, stage: str) -> None: + """Emit a stage change signal and notify the reporter.""" + self.stage_changed.emit(stage) + if self._reporter is not None: + self._reporter.stage(stage) + if self._progress_dialog is not None: + try: + self._progress_dialog.set_stage(stage) + except RuntimeError: + self._progress_dialog = None + + def stage(self, name: str) -> None: + """ProgressReporter entry point used by builders.""" + self._emit_stage(name) + + def tick(self, done: int, total: int) -> None: + """ProgressReporter entry point used by builders.""" + if self._reporter is not None: + self._reporter.tick(done, total) + if self._progress_dialog is not None: + try: + self._progress_dialog.set_progress(done, total) + except RuntimeError: + self._progress_dialog = None + + def detail(self, processed: int, path: str) -> None: + """ProgressReporter detail entry point used by builders/workers.""" + if self._reporter is not None and isinstance(self._reporter, ProgressReporter): + self._reporter.detail(processed, path) + if self._progress_dialog is not None: + try: + self._progress_dialog.set_detail(processed, path) + except RuntimeError: + self._progress_dialog = None + + def _start_progress(self, task: _LoadTask) -> None: + """Start tracking a load task with the progress widget.""" + self._current_task_id = task.task_id + if self._progress_dialog is None: + self._progress_dialog = LoadingProgressDialog(self._window, cancellable=True) + try: + self._progress_dialog.start(task.task_id, cancellation_token=task.token, on_cancel=self.cancel_current) + except RuntimeError: + self._progress_dialog = LoadingProgressDialog(self._window, cancellable=True) + self._progress_dialog.start(task.task_id, cancellation_token=task.token, on_cancel=self.cancel_current) + + def cancel_current(self) -> None: + """Cancel the active load task and unblock any blocking caller.""" + task_id = self._current_task_id + if task_id is None: + return + + task = self._tasks.get(task_id) + if task is None: + self._current_task_id = None + return + + task.token.cancel() + self._cancelled_task_ids.add(task_id) + self._finish_progress(task_id) + if task.mode == "reload": + self._window.statusBar.showMessage("Reload cancelled", 3000) + else: + self._window.statusBar.showMessage("Open cancelled", 3000) + + # Parse-stage cancel is completed immediately. Build-stage cancel + # completes when the builder observes the token and emits cancelled. + if task.builder is None: + self._complete_task(task_id, False) + + def _finish_progress(self, task_id: str) -> None: + """Finish tracking a load task.""" + if self._progress_dialog is not None: + try: + self._progress_dialog.finish(task_id) + except RuntimeError: + self._progress_dialog = None + self._current_task_id = None + + def _error_progress(self, task_id: str) -> None: + """Mark a load task as failed.""" + if self._progress_dialog is not None: + try: + self._progress_dialog.error(task_id) + except RuntimeError: + self._progress_dialog = None + self._current_task_id = None + + def _begin_task(self, mode: str, path: str, tab: Document | None = None) -> _LoadTask | None: + """Create and register a loading task.""" + if self._current_task_id is not None: + self._window.statusBar.showMessage("A file is already loading", 3000) + return None + + task_id = str(uuid.uuid4()) + resolved = str(Path(path).resolve()) + task = _LoadTask(task_id=task_id, mode=mode, path=resolved, tab=tab) + self._tasks[task_id] = task + self._start_progress(task) + return task + + def _start_parse_worker( + self, + task: _LoadTask, + parser: Callable[[str], tuple[Any, str]] | None = None, + ) -> None: + """Start file parsing after all signal handlers are connected.""" + thread = QThread(self) + worker = ParseWorker(task.path, parser=parser) + worker.moveToThread(thread) + task.thread = thread + task.worker = worker + + thread.started.connect(worker.run) + worker.stage.connect(self._emit_stage, Qt.ConnectionType.QueuedConnection) + worker.detail.connect(self.detail, Qt.ConnectionType.QueuedConnection) + worker.finished.connect(lambda result, task_id=task.task_id: self._parse_succeeded.emit(task_id, result)) + worker.failed.connect( + lambda error_payload, task_id=task.task_id: self._parse_failed.emit(task_id, error_payload) + ) + worker.finished.connect(thread.quit, Qt.ConnectionType.QueuedConnection) + worker.failed.connect(thread.quit, Qt.ConnectionType.QueuedConnection) + thread.finished.connect(worker.deleteLater) + thread.finished.connect(thread.deleteLater) + thread.start() + + def _run_blocking(self, task_id: str | None) -> bool: + """Wait for an asynchronous load while keeping the GUI event loop alive.""" + if task_id is None: + return False + + completed = self._completed_task_results.pop(task_id, None) + if completed is not None: + return completed + + deadline = time.monotonic() + LOADING_HARD_TIMEOUT_SECONDS + while task_id in self._tasks and task_id not in self._completed_task_results: + if time.monotonic() >= deadline: + self._error_progress(task_id) + self._tasks.pop(task_id, None) + self._window.statusBar.showMessage("Loading timed out", 3000) + return False + QApplication.processEvents() + time.sleep(0.001) + + return self._completed_task_results.pop(task_id, False) + + def open_file_async( + self, + path: str, + *, + parser: Callable[[str], tuple[Any, str]] | None = None, + ) -> str | None: + """Start opening a file and return immediately.""" + task = self._begin_task("open", path) + if task is None: + return None + self._window.statusBar.showMessage(f"Loading: {task.path}", 0) + self._start_parse_worker(task, parser=parser) + return task.task_id + + def reload_file_async( + self, + tab: Document, + path: str, + *, + parser: Callable[[str], tuple[Any, str]] | None = None, + ) -> str | None: + """Start reloading a file and return immediately.""" + task = self._begin_task("reload", path, tab=tab) + if task is None: + return None + self._window.statusBar.showMessage(f"Reloading: {task.path}", 0) + self._start_parse_worker(task, parser=parser) + return task.task_id + + def open_file(self, path: str) -> bool: + """Open a file and create a new tab. + + Returns True on success, False on failure. + """ + return self._run_blocking(self.open_file_async(path)) + + def reload_file(self, tab: Document, path: str) -> bool: + """Reload a tab's content from disk. + + Returns True on success, False on failure. + """ + return self._run_blocking(self.reload_file_async(tab, path)) + + def _on_parse_finished(self, task_id: str, result: object) -> None: + """Resume the load after background parsing succeeds.""" + if task_id in self._cancelled_task_ids: + self._cancelled_task_ids.discard(task_id) + return + task = self._tasks.get(task_id) + if task is None: + return + data, source_format = result + task.data = data + task.source_format = source_format + builder = ChunkedTreeBuilder( + data, + show_root=True, + reporter=self, + cancellation_token=task.token, + icon_provider=self._window._icon_provider, + parent=self, + ) + task.builder = builder + builder.finished.connect( + lambda model, finished_task_id=task_id: self._on_build_finished(finished_task_id, model) + ) + builder.cancelled.connect(lambda cancelled_task_id=task_id: self._on_build_cancelled(cancelled_task_id)) + builder.start() + + def _on_parse_failed(self, task_id: str, error_payload: object) -> None: + """Complete a task through the user-facing error path.""" + if task_id in self._cancelled_task_ids: + self._cancelled_task_ids.discard(task_id) + return + task = self._tasks.get(task_id) + if task is None: + return + _error_type, error_message = error_payload + self._error_progress(task_id) + if task.mode == "reload": + self._window.statusBar.showMessage(f"Reload failed: {task.path}", 3000) + QMessageBox.critical(self._window, "Reload failed", f"Could not reload {task.path}:\n{error_message}") + else: + self._window.statusBar.showMessage(f"Open failed: {task.path}", 3000) + QMessageBox.critical(self._window, "Open failed", f"Could not open {task.path}:\n{error_message}") + self._complete_task(task_id, False) + + def _on_build_finished(self, task_id: str, model: object) -> None: + """Bind or apply the fully built model after chunked construction.""" + task = self._tasks.get(task_id) + if task is None: + return + + task.builder = None + + if task.mode == "reload": + ok = self._apply_reload(task, model) + if not ok: + self._complete_task(task_id, False) + else: + self._bind_open(task, model) + + def _on_build_cancelled(self, task_id: str) -> None: + """Finalize cooperative cancellation when chunked build aborts.""" + task = self._tasks.get(task_id) + if task is None: + return + task.builder = None + self._complete_task(task_id, False) + + def _bind_open(self, task: _LoadTask, model: object) -> None: + """Create a tab from a prebuilt model and finish open bookkeeping.""" + task_id = task.task_id + + self._emit_stage(STAGE_BINDING_UI) + tab = self._window._add_tab( + data=task.data, + file_path=task.path, + save_format=task.source_format, + prebuilt_model=model, + defer_first_presentation=True, + defer_validation_init=True, + on_presentation_complete=lambda opened_tab: self._finish_open_binding(task_id, opened_tab), + ) + if tab is None: + self._error_progress(task.task_id) + self._complete_task(task.task_id, False) + + def _finish_open_binding(self, task_id: str, tab: Document) -> None: + """Finalize an open task after deferred first-presentation work.""" + from app.recent_files import push_recent + + task = self._tasks.get(task_id) + if task is None: + return + + self.run_schema_discovery_and_validation(tab, parsed_data=task.data) + + push_recent(self._window, task.path) + self._window.statusBar.showMessage(f"Opened: {task.path}", 2000) + self._finish_progress(task_id) + self._complete_task(task_id, True) + + def _apply_reload(self, task: _LoadTask, model: object) -> bool: + """Apply parsed reload data to the target tab at the commit point.""" + tab = task.tab + if tab is None: + self._error_progress(task.task_id) + return False + + if not isinstance(model, JsonTreeModel): + self._error_progress(task.task_id) + return False + + previous_view_state = view_state.capture_runtime_state(tab) + changed = True + + if task.token.is_cancelled: + if self._current_task_id == task.task_id: + self._finish_progress(task.task_id) + self._window.statusBar.showMessage("Reload cancelled", 3000) + return False + + self._emit_stage(STAGE_APPLYING_RELOAD) + tab.model.replace_root_item(model.root_item, estimated_item_count=model.estimated_item_count) + if changed: + tab.undo_stack.clear() + tab.undo_stack.setClean() + tab.io.save_format = task.source_format + tab.io.file_path = task.path + + view_state.restore_runtime_state(tab, previous_view_state) + + QTimer.singleShot(0, lambda task_id=task.task_id: self._finish_reload_apply(task_id)) + return True + + def _finish_reload_apply(self, task_id: str) -> None: + """Finalize reload on a later event-loop turn for post-build responsiveness.""" + task = self._tasks.get(task_id) + if task is None: + return + tab = task.tab + if tab is None: + self._error_progress(task_id) + self._complete_task(task_id, False) + return + + self.run_schema_discovery_and_validation(tab, parsed_data=task.data) + + self._window._refresh_tab_presentation(tab) + self._window.update_actions() + self._window.statusBar.showMessage(f"Reloaded: {task.path}", 2000) + self._finish_progress(task_id) + self._complete_task(task_id, True) + + def _complete_task(self, task_id: str, ok: bool) -> None: + """Release task-owned objects and announce completion.""" + self._tasks.pop(task_id, None) + self._cancelled_task_ids.discard(task_id) + self._completed_task_results[task_id] = ok + self.task_finished.emit(task_id, ok) + + def run_schema_discovery_and_validation(self, tab: Document, *, parsed_data: Any | None = None) -> None: + """Run schema discovery and validation with stage reporting. + + This method emits the discovering schema and validating document + stages, then runs the tab's validation revalidate method. + """ + doc_path = Path(tab.io.file_path).expanduser().resolve() if tab.io.file_path else None + + self._emit_stage(STAGE_DISCOVERING_SCHEMA) + discovery_data = parsed_data if parsed_data is not None else tab.model.root_item.to_json() + tab.validation.init_state(discovery_data, doc_path=doc_path, revalidate=False) + + self._emit_stage(STAGE_VALIDATING_DOCUMENT) + if parsed_data is None: + tab.validation.revalidate() + else: + tab.validation.revalidate_loading_data(parsed_data) + + self._emit_stage(STAGE_COMPLETE) + + +__all__ = ["LoadCoordinator"] diff --git a/app/loading/progress.py b/app/loading/progress.py new file mode 100644 index 0000000..c39cf37 --- /dev/null +++ b/app/loading/progress.py @@ -0,0 +1,150 @@ +"""Progress reporting protocol for loading operations. + +This module defines the ``ProgressEvent`` dataclass and ``ProgressReporter`` +protocol used to communicate progress between the coordinator, worker, +builder, and UI components without coupling them directly. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Protocol, runtime_checkable + +# Required progress stages in order (when applicable) +STAGE_READING_PARSING = "reading/parsing file" +STAGE_DECODING_AFFIXES = "decoding number affixes" +STAGE_BUILDING_TREE = "building item tree" +STAGE_BINDING_UI = "binding UI" +STAGE_APPLYING_RELOAD = "applying reload" # Used instead of BINDING_UI for reload +STAGE_DISCOVERING_SCHEMA = "discovering schema" +STAGE_VALIDATING_DOCUMENT = "validating document" +STAGE_COMPLETE = "complete" + +# Ordered list of stages for open operations +OPEN_STAGES = ( + STAGE_READING_PARSING, + STAGE_DECODING_AFFIXES, + STAGE_BUILDING_TREE, + STAGE_BINDING_UI, + STAGE_DISCOVERING_SCHEMA, + STAGE_VALIDATING_DOCUMENT, + STAGE_COMPLETE, +) + +# Ordered list of stages for reload operations +RELOAD_STAGES = ( + STAGE_READING_PARSING, + STAGE_DECODING_AFFIXES, + STAGE_BUILDING_TREE, + STAGE_APPLYING_RELOAD, + STAGE_DISCOVERING_SCHEMA, + STAGE_VALIDATING_DOCUMENT, + STAGE_COMPLETE, +) + + +@dataclass(frozen=True) +class ProgressEvent: + """A progress event emitted during loading operations. + + Attributes + ---------- + task_id : str + Unique identifier for the loading task. + stage : str + The current stage name (e.g., "reading/parsing file"). + done : int + Number of items completed in the current stage. + total : int + Total number of items in the current stage. + When unknown, both done and total are 0. + processed : int + Number of fields/values processed for detail reporting. + path : str + Current JSON-Pointer-style path for detail reporting. + """ + + task_id: str + stage: str + done: int = 0 + total: int = 0 + processed: int = 0 + path: str = "" + + @property + def is_indeterminate(self) -> bool: + """True if progress is indeterminate (total unknown).""" + return self.total <= 0 + + +@runtime_checkable +class ProgressReporter(Protocol): + """Protocol for reporting progress during loading operations. + + Implementations receive stage and tick notifications without + coupling to specific UI components. + """ + + def stage(self, name: str) -> None: + """Report a new stage. + + Parameters + ---------- + name : str + The stage name (e.g., "reading/parsing file"). + """ + ... + + def tick(self, done: int, total: int) -> None: + """Report progress within a stage. + + Parameters + ---------- + done : int + Number of items completed. + total : int + Total number of items. Use 0 for both when unknown. + """ + ... + + def detail(self, processed: int, path: str) -> None: + """Report detailed progress metadata. + + Parameters + ---------- + processed : int + Number of fields/values processed so far. + path : str + Current JSON-Pointer-style path. + """ + ... + + +class NullProgressReporter: + """A no-op progress reporter for testing or when progress is not needed.""" + + def stage(self, name: str) -> None: + pass + + def tick(self, done: int, total: int) -> None: + pass + + def detail(self, processed: int, path: str) -> None: + pass + + +__all__ = [ + "ProgressEvent", + "ProgressReporter", + "NullProgressReporter", + "STAGE_READING_PARSING", + "STAGE_DECODING_AFFIXES", + "STAGE_BUILDING_TREE", + "STAGE_BINDING_UI", + "STAGE_APPLYING_RELOAD", + "STAGE_DISCOVERING_SCHEMA", + "STAGE_VALIDATING_DOCUMENT", + "STAGE_COMPLETE", + "OPEN_STAGES", + "RELOAD_STAGES", +] diff --git a/app/loading/progress_dialog.py b/app/loading/progress_dialog.py new file mode 100644 index 0000000..f794c31 --- /dev/null +++ b/app/loading/progress_dialog.py @@ -0,0 +1,208 @@ +"""Delayed progress widget for loading operations. + +The widget only appears when a task remains active for at least +``LOADING_PROGRESS_DELAY_MS`` milliseconds. Fast loads complete before +the widget shows, avoiding visual noise. +""" + +from __future__ import annotations + +from typing import Callable + +from PySide6.QtCore import Qt, QTimer +from PySide6.QtWidgets import QLabel, QProgressBar, QPushButton, QVBoxLayout, QWidget + +from app.loading.cancellation import CancellationToken +from settings import LOADING_PROGRESS_DELAY_MS, LOADING_PROGRESS_DETAIL_REFRESH_MS + + +class LoadingProgressDialog(QWidget): + """A delayed progress widget for loading operations. + + The widget uses a single-shot timer to delay showing. If the task + completes before the timer fires, the widget never becomes visible. + + Parameters + ---------- + parent : QWidget | None + Parent widget. + cancellable : bool + If True, show a Cancel button. In Plan 2 this is always False. + delay_ms : int | None + Override for the show delay. Defaults to LOADING_PROGRESS_DELAY_MS. + """ + + def __init__( + self, + parent: QWidget | None = None, + *, + cancellable: bool = False, + delay_ms: int | None = None, + detail_refresh_ms: int | None = None, + ) -> None: + super().__init__(parent) + self.setWindowTitle("Loading") + self.setWindowFlags(self.windowFlags() | Qt.WindowType.Tool | Qt.WindowType.WindowStaysOnTopHint) + + self._cancellable = cancellable + self._delay_ms = delay_ms if delay_ms is not None else LOADING_PROGRESS_DELAY_MS + self._detail_refresh_ms = ( + detail_refresh_ms if detail_refresh_ms is not None else LOADING_PROGRESS_DETAIL_REFRESH_MS + ) + self._active_task_id: str | None = None + self._was_shown = False + self._pending_processed = 0 + self._pending_path = "" + self._cancel_token: CancellationToken | None = None + self._cancel_callback: Callable[[], None] | None = None + self._cancel_invoked = False + + # Build UI + layout = QVBoxLayout(self) + self._stage_label = QLabel("Loading...") + layout.addWidget(self._stage_label) + + self._progress_bar = QProgressBar() + self._progress_bar.setRange(0, 0) # Indeterminate + layout.addWidget(self._progress_bar) + + self._detail_label = QLabel("") + layout.addWidget(self._detail_label) + + if cancellable: + self._cancel_button = QPushButton("Cancel") + self._cancel_button.clicked.connect(self._on_cancel_clicked) + layout.addWidget(self._cancel_button) + else: + self._cancel_button = None + + # Single-shot timer for delayed show + self._show_timer = QTimer(self) + self._show_timer.setSingleShot(True) + self._show_timer.timeout.connect(self._on_show_timer_timeout) + + self._detail_timer = QTimer(self) + self._detail_timer.setSingleShot(False) + self._detail_timer.setInterval(self._detail_refresh_ms) + self._detail_timer.timeout.connect(self._flush_detail_label) + + # Start hidden + self.hide() + + @property + def was_shown(self) -> bool: + """True if the widget was ever shown during the current task.""" + return self._was_shown + + def start( + self, + task_id: str, + *, + cancellation_token: CancellationToken | None = None, + on_cancel: Callable[[], None] | None = None, + ) -> None: + """Start tracking a task. + + Arms the show timer. If the task completes before the timer fires, + the widget never becomes visible. + """ + self._active_task_id = task_id + self._was_shown = False + self._stage_label.setText("Loading...") + self._progress_bar.setRange(0, 0) + self._pending_processed = 0 + self._pending_path = "" + self._detail_label.setText("") + self._cancel_token = cancellation_token + self._cancel_callback = on_cancel + self._cancel_invoked = False + if self._cancel_button is not None: + self._cancel_button.setEnabled(True) + self.hide() + self._show_timer.start(self._delay_ms) + self._detail_timer.start() + + def set_stage(self, stage: str) -> None: + """Update the stage text displayed to the user.""" + self._stage_label.setText(stage) + + def set_progress(self, done: int, total: int) -> None: + """Update the progress bar. + + If total is 0, the bar is indeterminate. Otherwise it shows + done/total. + """ + if total <= 0: + self._progress_bar.setRange(0, 0) + else: + self._progress_bar.setRange(0, total) + self._progress_bar.setValue(done) + + def set_detail(self, processed: int, path: str) -> None: + """Store detail progress values; rendering is throttled by timer.""" + self._pending_processed = processed + self._pending_path = path + + def finish(self, task_id: str) -> None: + """Mark a task as finished. + + Stops the show timer and hides the widget. If the task_id does + not match the active task, this is a no-op. + """ + if task_id != self._active_task_id: + return + self._show_timer.stop() + self._detail_timer.stop() + self._cancel_token = None + self._cancel_callback = None + self._active_task_id = None + self.hide() + + def error(self, task_id: str) -> None: + """Mark a task as failed. + + Stops the show timer and hides the widget. If the task_id does + not match the active task, this is a no-op. + """ + if task_id != self._active_task_id: + return + self._show_timer.stop() + self._detail_timer.stop() + self._cancel_token = None + self._cancel_callback = None + self._active_task_id = None + self.hide() + + def _on_cancel_clicked(self) -> None: + """Handle a user-requested cancellation from the Cancel button.""" + if self._active_task_id is None or self._cancel_invoked: + return + + self._cancel_invoked = True + if self._cancel_token is not None: + self._cancel_token.cancel() + if self._cancel_callback is not None: + self._cancel_callback() + if self._cancel_button is not None: + self._cancel_button.setEnabled(False) + self._stage_label.setText("Cancelling…") + + def _on_show_timer_timeout(self) -> None: + """Show the widget when the timer fires and a task is still active.""" + if self._active_task_id is not None: + self._was_shown = True + self.show() + self.raise_() + self.activateWindow() + + def _flush_detail_label(self) -> None: + """Render pending detail values at a bounded cadence.""" + if self._active_task_id is None: + return + if self._pending_path: + self._detail_label.setText(f"Processed {self._pending_processed:,} fields/values | {self._pending_path}") + return + self._detail_label.setText(f"Processed {self._pending_processed:,} fields/values") + + +__all__ = ["LoadingProgressDialog"] diff --git a/app/loading/worker.py b/app/loading/worker.py new file mode 100644 index 0000000..e85d925 --- /dev/null +++ b/app/loading/worker.py @@ -0,0 +1,111 @@ +"""Worker thread for file parsing. + +The worker runs ``load_file_with_format()`` in a QThread and emits +plain Python data back to the GUI thread. It must not create any Qt +widgets. +""" + +from __future__ import annotations + +from typing import Any, Callable + +from PySide6.QtCore import QObject, QThread, Signal + + +class ParseWorker(QObject): + """Worker that parses a file in a background thread. + + Signals + ------- + finished(result) + Emitted when parsing succeeds. ``result`` is a tuple of + ``(data, source_format)``. + failed(error_payload) + Emitted when parsing fails. ``error_payload`` is a tuple of + ``(error_type, error_message)``. + stage(stage_name) + Emitted to report progress stages. + detail(processed, path) + Emitted during decode/build detail reporting with item count and path. + """ + + finished = Signal(object) + failed = Signal(object) + stage = Signal(str) + detail = Signal(int, str) + + def __init__( + self, + path: str, + parser: Callable[[str], tuple[Any, str]] | None = None, + parent: QObject | None = None, + ) -> None: + super().__init__(parent) + self._path = path + self._parser = parser + + def run(self) -> None: + """Execute the parse operation. + + This method is called in the worker thread. It emits ``finished`` + on success or ``failed`` on exception. + """ + try: + self.stage.emit("reading/parsing file") + decode_stage_emitted = False + + def on_decode_progress(processed: int, path: str) -> None: + nonlocal decode_stage_emitted + if not decode_stage_emitted: + self.stage.emit("decoding number affixes") + decode_stage_emitted = True + self.detail.emit(processed, path) + + if self._parser is not None: + result = self._parser(self._path) + else: + from io_formats.load import load_file_with_format + + result = load_file_with_format(self._path, on_progress=on_decode_progress) + + if not decode_stage_emitted: + self.stage.emit("decoding number affixes") + self.finished.emit(result) + except Exception as exc: + self.failed.emit((type(exc).__name__, str(exc))) + + +def start_parse_worker( + path: str, + parser: Callable[[str], tuple[Any, str]] | None = None, +) -> tuple[QThread, ParseWorker]: + """Create and start a parse worker thread. + + Parameters + ---------- + path : str + The file path to parse. + parser : Callable | None + Optional custom parser for testing. If None, uses + ``load_file_with_format``. + + Returns + ------- + tuple[QThread, ParseWorker] + The thread and worker objects. The caller must connect to the + worker's signals and ensure the thread is cleaned up. + """ + from PySide6.QtCore import Qt + + thread = QThread() + worker = ParseWorker(path, parser=parser) + worker.moveToThread(thread) + thread.started.connect(worker.run) + # Use QueuedConnection to ensure thread.quit() is called from the main thread + worker.finished.connect(thread.quit, Qt.ConnectionType.QueuedConnection) + worker.failed.connect(thread.quit, Qt.ConnectionType.QueuedConnection) + thread.start() + return thread, worker + + +__all__ = ["ParseWorker", "start_parse_worker"] diff --git a/app/main_window.py b/app/main_window.py index c6133a2..4f28928 100644 --- a/app/main_window.py +++ b/app/main_window.py @@ -21,6 +21,7 @@ from app.close_confirm import confirm_close from app.font_controller import FontController from app.history import bind_undo_signals, setup_history_menu +from app.loading.coordinator import LoadCoordinator from app.main_window_actions import setup_connections as setup_main_window_connections from app.main_window_actions import update_actions as update_main_window_actions from app.recent_files import push_recent, refresh_recent_menu @@ -30,7 +31,6 @@ from app.validation_presenter import DockValidationPresenter from documents.composition.marker import JsonTabWidgetMarker from documents.seams.document_protocol import Document -from io_formats.load import load_file_with_format from settings import APPLICATION_ID, WINDOW_DEFAULT_SIZE from tree_actions.clipboard import clipboard_to_tab_data from tree_actions.field_case import FIELD_CASE_LABELS, FIELD_CASE_ORDER, FieldCase @@ -92,6 +92,7 @@ def __init__(self, yaml_filename: str, parent=None): self._icon_provider = self._theme_controller.icon_provider self._schema_tab_pool = SchemaTabPool(self) self._tab_lifecycle = TabLifecyclePresenter(self.tabWidget, self) + self._load_coordinator = LoadCoordinator(self) self._theme_follow_action = self._theme_controller.follow_action self._recent_menu = QMenu("Recent", self) self.fileMenu.insertMenu(self.appExitAction, self._recent_menu) @@ -294,28 +295,32 @@ def _on_tab_changed(self, _index: int) -> None: def _refresh_tab_presentation(self, tab: Document) -> None: self._tab_lifecycle.refresh_tab_presentation(tab) - def _add_tab(self, *, data=None, file_path: str | None = None, save_format: str | None = None) -> Document | None: - return self._tab_lifecycle.add_tab(data=data, file_path=file_path, save_format=save_format) + def _add_tab( + self, + *, + data=None, + file_path: str | None = None, + save_format: str | None = None, + prebuilt_model=None, + defer_first_presentation: bool = False, + defer_validation_init: bool = False, + on_presentation_complete=None, + ) -> Document | None: + return self._tab_lifecycle.add_tab( + data=data, + file_path=file_path, + save_format=save_format, + prebuilt_model=prebuilt_model, + defer_first_presentation=defer_first_presentation, + defer_validation_init=defer_validation_init, + on_presentation_complete=on_presentation_complete, + ) def _on_tab_dirty(self, tab: Document) -> None: self._tab_lifecycle.on_tab_dirty(tab) def _open_path(self, path: str) -> bool: - resolved = str(Path(path).resolve()) - self.statusBar.showMessage(f"Loading: {resolved}", 0) - try: - data, source_format = load_file_with_format(resolved) - except Exception as exc: - self.statusBar.showMessage(f"Open failed: {resolved}", 3000) - QMessageBox.critical(self, "Open failed", f"Could not open {resolved}:\n{exc}") - return False - - tab = self._add_tab(data=data, file_path=resolved, save_format=source_format) - if tab is None: - return False - push_recent(self, resolved) - self.statusBar.showMessage(f"Opened: {resolved}", 2000) - return True + return self._load_coordinator.open_file(path) def _save_tab(self, tab: Document, *, save_as: bool = False) -> bool: from state.validation_settings import clear_schema_path @@ -360,28 +365,7 @@ def _confirm_reload_dirty_tab(self, tab: Document) -> str: return "cancel" def _reload_tab_from_path(self, tab: Document, path: str) -> bool: - resolved = str(Path(path).resolve()) - self.statusBar.showMessage(f"Reloading: {resolved}", 0) - try: - data, source_format = load_file_with_format(resolved) - except Exception as exc: - self.statusBar.showMessage(f"Reload failed: {resolved}", 3000) - QMessageBox.critical(self, "Reload failed", f"Could not reload {resolved}:\n{exc}") - return False - - root_index = tab.root_index() - root_item = tab.root_item() - changed = tab.editing.diff.apply(root_item, data, root_index) - if changed: - tab.undo_stack.clear() - tab.undo_stack.setClean() - tab.io.save_format = source_format - tab.io.file_path = resolved - tab.validation.revalidate() - self._refresh_tab_presentation(tab) - self.update_actions() - self.statusBar.showMessage(f"Reloaded: {resolved}", 2000) - return True + return self._load_coordinator.reload_file(tab, path) def _confirm_close(self, tab: Document, *, prompt_for_untitled_nonempty: bool = True) -> bool: return confirm_close(self, tab, prompt_for_untitled_nonempty=prompt_for_untitled_nonempty) @@ -395,7 +379,7 @@ def open_file_dialog(self) -> None: ) if not path: return - self._open_path(path) + self._load_coordinator.open_file_async(path) def save_file(self) -> None: tab = self._current_tab() diff --git a/app/tab_lifecycle.py b/app/tab_lifecycle.py index 9d39078..65e9d30 100644 --- a/app/tab_lifecycle.py +++ b/app/tab_lifecycle.py @@ -11,14 +11,20 @@ from __future__ import annotations -from PySide6.QtCore import QObject -from PySide6.QtWidgets import QMessageBox, QTabWidget +from typing import Callable +from uuid import uuid4 +from PySide6.QtCore import QObject, Qt, QTimer +from PySide6.QtWidgets import QApplication, QMessageBox, QTabWidget + +import settings import state.view_state as view_state +from app.loading.progress_dialog import LoadingProgressDialog from documents.composition.dependencies import JsonTabServices from documents.composition.factory import create_tab from documents.composition.marker import JsonTabWidgetMarker from documents.seams.document_protocol import Document +from tree.model import JsonTreeModel class _MainWindowJsonTabHost: @@ -45,6 +51,36 @@ def __init__(self, tab_widget: QTabWidget, main_window) -> None: self._tab_widget = tab_widget self._win = main_window self.closed_tabs_stack: list[dict] = [] + self._close_progress_dialog: LoadingProgressDialog | None = None + + def _ensure_close_progress_dialog(self) -> LoadingProgressDialog: + dialog = self._close_progress_dialog + if dialog is None: + dialog = LoadingProgressDialog( + self._win, + cancellable=False, + delay_ms=settings.CLOSE_PROGRESS_DELAY_MS, + ) + dialog.setWindowTitle("Closing tab") + self._close_progress_dialog = dialog + return dialog + + @staticmethod + def _set_close_stage(dialog: LoadingProgressDialog, stage: str) -> None: + try: + dialog.set_stage(stage) + except RuntimeError: + pass + + @staticmethod + def _finish_close_progress(dialog: LoadingProgressDialog, task_id: str, *, failed: bool) -> None: + try: + if failed: + dialog.error(task_id) + else: + dialog.finish(task_id) + except RuntimeError: + pass # ── presentation helpers ────────────────────────────────────────────── @@ -67,6 +103,10 @@ def add_tab( data=None, file_path: str | None = None, save_format: str | None = None, + prebuilt_model: JsonTreeModel | None = None, + defer_first_presentation: bool = False, + defer_validation_init: bool = False, + on_presentation_complete: Callable[[Document], None] | None = None, ) -> Document | None: from state.validation_settings import auto_rescan_enabled @@ -78,6 +118,8 @@ def add_tab( show_root=True, parent=win, save_format=save_format, + prebuilt_model=prebuilt_model, + defer_validation_init=defer_validation_init, services=JsonTabServices( host=_MainWindowJsonTabHost(win), theme=win._theme, @@ -96,18 +138,48 @@ def add_tab( win.fonts.subscribe(tab) tab.dirtyChanged.connect(lambda _dirty, t=tab: self.on_tab_dirty(t)) - tab.view_controller.request_expand_all() + if defer_first_presentation: + QTimer.singleShot( + 0, + lambda t=tab, cb=on_presentation_complete: self._run_initial_presentation(t, cb), + ) + else: + self._run_initial_presentation(tab, on_presentation_complete) + return tab + + def _run_initial_presentation( + self, + tab: Document, + on_presentation_complete: Callable[[Document], None] | None, + ) -> None: + win = self._win + is_large_open = not self._should_expand_all_on_open(tab) + if not is_large_open: + tab.view_controller.request_expand_all() + elif tab.root_index().isValid(): + tab.view_controller.request_expand(()) + tab.appearance.resize_key_columns() - if tab.root_index().isValid(): + if tab.root_index().isValid() and not is_large_open: tab.view_controller.request_select_paths([()]) - view_state.restore(tab) + view_state.restore(tab, defer_heavy=is_large_open) + if is_large_open and tab.root_index().isValid(): + tab.view_controller.request_scroll_to_deferred(()) # Re-broadcast: ``view_state.restore`` may have rewritten ``_font_pt`` # from a previously-saved per-tab value; the global controller wins. win.fonts.subscribe(tab) win._bind_undo_signals(tab) win.update_actions() - return tab + if on_presentation_complete is not None: + on_presentation_complete(tab) + + @staticmethod + def _should_expand_all_on_open(tab: Document) -> bool: + node_count = tab.model.estimated_item_count + if isinstance(node_count, int): + return node_count <= settings.LOADING_AUTO_EXPAND_MAX_NODES + return True # ── current-tab change ──────────────────────────────────────────────── @@ -139,48 +211,75 @@ def close_tab(self, index: int) -> None: win = self._win widget = self._tab_widget.widget(index) - snapshot = None if isinstance(widget, JsonTabWidgetMarker): - was_dirty = widget.io.dirty if not win._confirm_close(widget): return - # Build reopen snapshot: if user discarded dirty edits, remember file path only. - if was_dirty and widget.io.dirty: - # Discard chosen — don't resurrect dirty state on reopen. - if widget.io.file_path: + + dialog = self._ensure_close_progress_dialog() + close_task_id = uuid4().hex + failed = False + QApplication.setOverrideCursor(Qt.CursorShape.WaitCursor) + dialog.start(close_task_id) + + snapshot = None + try: + if isinstance(widget, JsonTabWidgetMarker): + was_dirty = widget.io.dirty + self._set_close_stage(dialog, "snapshot") + + # Build reopen snapshot: if user discarded dirty edits, remember file path only. + if was_dirty and widget.io.dirty: + # Discard chosen — don't resurrect dirty state on reopen. + if widget.io.file_path: + snapshot = { + "data": None, + "file_path": widget.io.file_path, + "save_format": widget.io.save_format, + } + else: + try: + snap_data = widget.root_data() + except Exception: # noqa: BLE001 + snap_data = {} snapshot = { - "data": None, + "data": snap_data, "file_path": widget.io.file_path, "save_format": widget.io.save_format, } - else: - try: - snap_data = widget.root_data() - except Exception: # noqa: BLE001 - snap_data = {} - snapshot = { - "data": snap_data, - "file_path": widget.io.file_path, - "save_format": widget.io.save_format, - } - win._schema_tab_pool.unregister(widget) - view_state.save(widget) - if widget is win._bound_undo_tab: - win._bind_undo_signals(None) - self._tab_widget.removeTab(index) - if widget is not None: - widget.deleteLater() - - if snapshot is not None: - self.closed_tabs_stack.append(snapshot) - if len(self.closed_tabs_stack) > self.MAX_CLOSED_TABS: - self.closed_tabs_stack.pop(0) - win.update_actions() - current = win._current_tab() - win._bind_undo_signals(current) - win._dock_validation.bind_validation_status(current) - win.validation_dock.attach_tab(current) + self._set_close_stage(dialog, "unregistering schema") + win._schema_tab_pool.unregister(widget) + + self._set_close_stage(dialog, "saving view state") + view_state.save(widget) + + if widget is win._bound_undo_tab: + win._bind_undo_signals(None) + + self._set_close_stage(dialog, "removing tab") + self._tab_widget.removeTab(index) + + if widget is not None: + self._set_close_stage(dialog, "destroying tab") + widget.deleteLater() + + if snapshot is not None: + self.closed_tabs_stack.append(snapshot) + if len(self.closed_tabs_stack) > self.MAX_CLOSED_TABS: + self.closed_tabs_stack.pop(0) + + win.update_actions() + current = win._current_tab() + win._bind_undo_signals(current) + win._dock_validation.bind_validation_status(current) + win.validation_dock.attach_tab(current) + except Exception: # noqa: BLE001 + failed = True + raise + finally: + self._finish_close_progress(dialog, close_task_id, failed=failed) + while QApplication.overrideCursor() is not None: + QApplication.restoreOverrideCursor() # ── reopen ──────────────────────────────────────────────────────────── diff --git a/core/datetime_parsing/regex.py b/core/datetime_parsing/regex.py index f13cc8e..b3a5f3f 100644 --- a/core/datetime_parsing/regex.py +++ b/core/datetime_parsing/regex.py @@ -3,6 +3,8 @@ from dateutil.parser import isoparse +from settings import INFERENCE_MAX_DATETIME_CHARS + from .enums import DateTimeCategory from .nano_time import NanoTime @@ -33,7 +35,12 @@ ) -def parse_datetime_text(text: str, category=None): +def parse_datetime_text(text: str, category=None, *, allow_expensive: bool = False): + # Gate expensive regex work for oversized strings during automatic inference. + # Explicit coercion passes allow_expensive=True to bypass this gate. + if not allow_expensive and len(text) > INFERENCE_MAX_DATETIME_CHARS: + return None + match = DATETIME_RE.fullmatch(text) if not match: return None diff --git a/core/frozen_value.py b/core/frozen_value.py new file mode 100644 index 0000000..7d30590 --- /dev/null +++ b/core/frozen_value.py @@ -0,0 +1,10 @@ +"""Backwards-compatibility shim. + +``FrozenValue`` was the original name for the raw-literal wrapper. It now aliases +:class:`core.raw_numeric.RawNumericValue`, which preserves unsupported numeric +literals as editable raw text. +""" + +from core.raw_numeric import RawNumericValue as FrozenValue + +__all__ = ["FrozenValue"] diff --git a/core/raw_numeric.py b/core/raw_numeric.py new file mode 100644 index 0000000..7dac2f0 --- /dev/null +++ b/core/raw_numeric.py @@ -0,0 +1,101 @@ +"""Raw numeric value object for unsupported numeric literals. + +When a numeric literal cannot be represented as the application's normal numeric +type (``gmpy2.mpq``) because of overflow, underflow, a non-finite result, an +unsupported format, or parser rejection, the original raw text is preserved +verbatim inside a :class:`RawNumericValue`. The value is treated as plain, +editable text so loading, rendering, editing, validation, and saving never +crash and never silently lose the original literal. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass + +# Stable reason codes (kept as plain strings so they survive serialization and +# are easy to assert against in tests). +REASON_OVERFLOW = "overflow" +REASON_UNDERFLOW = "underflow" +REASON_NON_FINITE = "non-finite" +REASON_INVALID_FORMAT = "invalid-format" +REASON_PRECISION_LIMIT = "precision-limit" +REASON_PARSER_REJECTION = "parser-rejection" +REASON_UNKNOWN = "unknown" + +_REASON_DESCRIPTIONS: dict[str, str] = { + REASON_OVERFLOW: "the value magnitude is too large (overflow)", + REASON_UNDERFLOW: "the value magnitude is too small (underflow)", + REASON_NON_FINITE: "the value is not finite (infinity or NaN)", + REASON_INVALID_FORMAT: "the value is not a valid number format", + REASON_PRECISION_LIMIT: "the value has too many significant digits", + REASON_PARSER_REJECTION: "the value was rejected by the numeric parser", + REASON_UNKNOWN: "the value is unsupported as a regular number", +} + + +def describe_reason(reason: str) -> str: + """Return a short human-readable explanation for a reason code.""" + return _REASON_DESCRIPTIONS.get(reason, _REASON_DESCRIPTIONS[REASON_UNKNOWN]) + + +# Narrow recovery/edit grammar for raw numeric values. This is intentionally +# NOT a full float grammar: it only accepts the small set of shapes the app +# allows when a user edits an unsupported numeric literal. Exponent length is +# bounded so a fresh edit cannot reintroduce an unbounded-magnitude literal +# (the original raw text is always allowed unchanged regardless of this regex). +_RAW_NUMERIC_EDIT_RE = re.compile(r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?\Z") + +# Non-finite spellings the app is willing to preserve through a raw edit. +_RAW_NON_FINITE_SPELLINGS = frozenset( + { + "inf", + "+inf", + "-inf", + "infinity", + "+infinity", + "-infinity", + ".inf", + "+.inf", + "-.inf", + "nan", + "+nan", + "-nan", + ".nan", + } +) + + +def raw_numeric_text_is_acceptable(text: str) -> bool: + """Return True iff *text* matches the narrow raw-numeric edit grammar.""" + candidate = text.strip() + if not candidate: + return False + if _RAW_NUMERIC_EDIT_RE.match(candidate): + return True + return candidate.lower() in _RAW_NON_FINITE_SPELLINGS + + +@dataclass(frozen=True, slots=True) +class RawNumericValue: + """Raw numeric literal preserved as-is and edited as plain text. + + Attributes: + raw: The exact original literal text, preserved byte-for-byte. + reason: One of the ``REASON_*`` codes describing why the literal is + unsupported as a regular number. + source_syntax: Optional origin hint (for example ``"json"`` or + ``"yaml"``) used when deciding cross-format save safety. + detail: Optional extra context for diagnostics / tooltips. + """ + + raw: str + reason: str = REASON_UNKNOWN + source_syntax: str = "" + detail: str = "" + + def __str__(self) -> str: + return self.raw + + def describe(self) -> str: + return describe_reason(self.reason) diff --git a/core/safe_mpq.py b/core/safe_mpq.py new file mode 100644 index 0000000..09ab69d --- /dev/null +++ b/core/safe_mpq.py @@ -0,0 +1,299 @@ +from __future__ import annotations + +from dataclasses import dataclass +from decimal import MAX_EMAX, MIN_EMIN, Context, Decimal, Inexact, InvalidOperation, Overflow, Underflow +from typing import Any + +from gmpy2 import mpq + +from core.raw_numeric import ( + REASON_INVALID_FORMAT, + REASON_NON_FINITE, + REASON_OVERFLOW, + REASON_PARSER_REJECTION, + REASON_PRECISION_LIMIT, + REASON_UNDERFLOW, +) +from settings import MPQ_SAFE_MAX_ABS_EXPONENT, MPQ_SAFE_MAX_SIG_DIGITS + +# Special float spellings that are syntactically "numbers" but are not finite. +# These are classified as non-finite rather than overflow/format errors so the +# UI can explain the cause precisely. +_NON_FINITE_LITERALS = frozenset( + { + "inf", + "+inf", + "-inf", + "infinity", + "+infinity", + "-infinity", + ".inf", + "+.inf", + "-.inf", + "nan", + "+nan", + "-nan", + ".nan", + "snan", + "+snan", + "-snan", + } +) + + +@dataclass(frozen=True, slots=True) +class MpqParseResult: + """Outcome of safe numeric parsing. + + ``value`` is a finite ``mpq`` on success and ``None`` on rejection. When + ``value`` is ``None``, ``reason`` carries a ``core.raw_numeric.REASON_*`` + code explaining why the literal is unsupported as a regular number. + """ + + value: mpq | None + reason: str | None = None + + @property + def ok(self) -> bool: + return self.value is not None + + +def _normalize_numeric_text(text: str) -> str: + return text.replace("_", "").strip() + + +def _is_non_finite_literal(candidate: str) -> bool: + return candidate.lower() in _NON_FINITE_LITERALS + + +def _safe_decimal_context( + *, + max_abs_exponent: int, + max_sig_digits: int, +) -> Context: + ctx = Context( + prec=max_sig_digits, + Emax=max_abs_exponent, + Emin=-max_abs_exponent, + ) + ctx.traps[Overflow] = True + ctx.traps[Underflow] = True + ctx.traps[InvalidOperation] = True + ctx.traps[Inexact] = True + return ctx + + +def _classification_context(*, max_sig_digits: int) -> Context: + """A trap-free context with the widest possible exponent range. + + Used only to classify a literal (format / non-finite / order of magnitude) + before the strict, trapping context makes the final exact decision. + """ + return Context(prec=max(1, max_sig_digits), Emax=MAX_EMAX, Emin=MIN_EMIN) + + +def safe_decimal_from_text( + text: str, + *, + max_abs_exponent: int = MPQ_SAFE_MAX_ABS_EXPONENT, + max_sig_digits: int = MPQ_SAFE_MAX_SIG_DIGITS, +) -> Decimal | None: + candidate = _normalize_numeric_text(text) + if not candidate: + return None + + ctx = _safe_decimal_context( + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + try: + parsed = ctx.create_decimal(candidate) + except (InvalidOperation, Overflow, Underflow, Inexact): + return None + + if not parsed.is_finite(): + return None + return parsed + + +def _safe_int_from_text( + text: str, + *, + max_abs_exponent: int = MPQ_SAFE_MAX_ABS_EXPONENT, + max_sig_digits: int = MPQ_SAFE_MAX_SIG_DIGITS, +) -> int | None: + candidate = _normalize_numeric_text(text) + if not candidate: + return None + + body = candidate.lstrip("+-") + if not body or not body.isdigit(): + return None + + parsed = safe_decimal_from_text( + candidate, + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + if parsed is None: + return None + + numerator, denominator = parsed.as_integer_ratio() + if denominator != 1: + return None + return numerator + + +def _parse_rational( + candidate: str, + *, + max_abs_exponent: int, + max_sig_digits: int, +) -> MpqParseResult: + numerator_text, sep, denominator_text = candidate.partition("/") + if sep != "/" or "/" in denominator_text: + return MpqParseResult(None, REASON_INVALID_FORMAT) + + numerator = _safe_int_from_text( + numerator_text, + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + denominator = _safe_int_from_text( + denominator_text, + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + if numerator is None or denominator is None or denominator == 0: + return MpqParseResult(None, REASON_INVALID_FORMAT) + try: + return MpqParseResult(mpq(numerator, denominator), None) + except (TypeError, ValueError, ZeroDivisionError): + return MpqParseResult(None, REASON_PARSER_REJECTION) + + +def _parse_decimal( + candidate: str, + *, + max_abs_exponent: int, + max_sig_digits: int, +) -> MpqParseResult: + # 0) Non-finite spellings (including YAML forms such as ``.inf`` / ``.nan`` + # that ``Decimal`` itself does not accept) are classified up-front. + if _is_non_finite_literal(candidate): + return MpqParseResult(None, REASON_NON_FINITE) + + # 1) Classification pass (no traps): identify format errors, non-finite + # values, and order of magnitude so overflow / underflow are reported + # precisely. + probe_ctx = _classification_context(max_sig_digits=max_sig_digits) + try: + probe = probe_ctx.create_decimal(candidate) + except InvalidOperation: + return MpqParseResult(None, REASON_INVALID_FORMAT) + except Overflow: # pragma: no cover - traps are off; defensive only + return MpqParseResult(None, REASON_OVERFLOW) + + if not probe.is_finite(): + if _is_non_finite_literal(candidate): + return MpqParseResult(None, REASON_NON_FINITE) + # Overflowed all the way to infinity during classification. + return MpqParseResult(None, REASON_OVERFLOW) + + if probe != 0: + adjusted = probe.adjusted() + if adjusted > max_abs_exponent: + return MpqParseResult(None, REASON_OVERFLOW) + if adjusted < -max_abs_exponent: + return MpqParseResult(None, REASON_UNDERFLOW) + + # 2) Strict pass (traps on): enforce exact precision within bounds. + strict_ctx = _safe_decimal_context( + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + try: + parsed = strict_ctx.create_decimal(candidate) + except Overflow: + return MpqParseResult(None, REASON_OVERFLOW) + except Underflow: + return MpqParseResult(None, REASON_UNDERFLOW) + except Inexact: + return MpqParseResult(None, REASON_PRECISION_LIMIT) + except InvalidOperation: + return MpqParseResult(None, REASON_INVALID_FORMAT) + + if not parsed.is_finite(): + return MpqParseResult(None, REASON_NON_FINITE) + + numerator, denominator = parsed.as_integer_ratio() + try: + return MpqParseResult(mpq(numerator, denominator), None) + except (TypeError, ValueError, ZeroDivisionError): + return MpqParseResult(None, REASON_PARSER_REJECTION) + + +def parse_mpq( + text: str, + *, + max_abs_exponent: int = MPQ_SAFE_MAX_ABS_EXPONENT, + max_sig_digits: int = MPQ_SAFE_MAX_SIG_DIGITS, +) -> MpqParseResult: + """Parse *text* into a finite ``mpq``, preserving the rejection cause. + + Returns an :class:`MpqParseResult` whose ``value`` is an ``mpq`` on success + or ``None`` with a ``reason`` code on rejection. + """ + candidate = _normalize_numeric_text(text) + if not candidate: + return MpqParseResult(None, REASON_INVALID_FORMAT) + + if "/" in candidate: + return _parse_rational( + candidate, + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + + return _parse_decimal( + candidate, + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + + +def safe_mpq_from_text( + text: str, + *, + max_abs_exponent: int = MPQ_SAFE_MAX_ABS_EXPONENT, + max_sig_digits: int = MPQ_SAFE_MAX_SIG_DIGITS, +) -> mpq | None: + return parse_mpq( + text, + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ).value + + +def mpq_literal_is_safe( + text: str, + *, + max_abs_exponent: int = MPQ_SAFE_MAX_ABS_EXPONENT, + max_sig_digits: int = MPQ_SAFE_MAX_SIG_DIGITS, +) -> bool: + return ( + safe_mpq_from_text( + text, + max_abs_exponent=max_abs_exponent, + max_sig_digits=max_sig_digits, + ) + is not None + ) + + +def safe_mpq_from_any(value: Any) -> mpq | None: + if isinstance(value, mpq): + return value + if value is None or isinstance(value, bool): + return None + return safe_mpq_from_text(str(value)) diff --git a/delegates/edit_context.py b/delegates/edit_context.py index c8991e4..9647a0a 100644 --- a/delegates/edit_context.py +++ b/delegates/edit_context.py @@ -16,6 +16,7 @@ from PySide6.QtCore import QAbstractItemModel, QModelIndex, QPersistentModelIndex, Qt from PySide6.QtWidgets import QMessageBox, QWidget +from core.raw_numeric import describe_reason from state.edit_limits import ( get_binary_edit_warning_limit_bytes, get_multiline_edit_warning_limit_chars, @@ -71,6 +72,8 @@ def confirm_large_text_edit( def confirm_large_binary_edit(self, parent: QWidget | None, payload_size: int) -> bool: ... + def warn_raw_numeric_edit(self, parent: QWidget | None, *, reason: str) -> None: ... + def _to_index(index: QModelIndex | QPersistentModelIndex) -> QModelIndex: if isinstance(index, QPersistentModelIndex): @@ -146,6 +149,19 @@ def confirm_large_text_edit( ) return answer == QMessageBox.StandardButton.Yes + def warn_raw_numeric_edit(self, parent: QWidget | None, *, reason: str) -> None: + QMessageBox.warning( + parent, + "Unsupported numeric value", + "This value is currently unsupported as a regular float / number " + f"({describe_reason(reason)}).\n\n" + "You can change it into a normally parseable number, or leave the " + "raw value unchanged to preserve it for external software that " + "accepts such values.", + QMessageBox.StandardButton.Ok, + QMessageBox.StandardButton.Ok, + ) + def confirm_large_binary_edit(self, parent: QWidget | None, payload_size: int) -> bool: limit = get_binary_edit_warning_limit_bytes() if payload_size <= limit: diff --git a/delegates/formatting/value_formatting.py b/delegates/formatting/value_formatting.py index bea8009..d901fc2 100644 --- a/delegates/formatting/value_formatting.py +++ b/delegates/formatting/value_formatting.py @@ -1,3 +1,4 @@ +import base64 from datetime import datetime, timezone from gmpy2 import mpq @@ -5,8 +6,9 @@ from PySide6.QtGui import QBrush, QPalette from PySide6.QtWidgets import QStyleOptionViewItem +from core.safe_mpq import safe_mpq_from_any from mpq2py import mpq_serialization -from settings import SECRET_MASK_CHAR, SECRET_MASK_GLYPHS +from settings import FORMAT_PREVIEW_DECODE_LIMIT_BYTES, SECRET_MASK_CHAR, SECRET_MASK_GLYPHS from themes.spec import TypeStyle from tree.codecs.bytes_codec import decode_bytes from tree.types import EMPTY_FAMILY, SECRET_FAMILY, WS_FAMILY, JsonType @@ -126,6 +128,10 @@ def _format_container_preview(item, json_type: JsonType, *, show_preview: bool) if not preview: return header + + if len(item.child_items) > _PREVIEW_CHILDREN: + preview += "..." + return _elide(f"{header} {preview}") @@ -147,7 +153,9 @@ def format_with_type(value, json_type: JsonType | None, *, item=None, show_previ if json_type is JsonType.PERCENT: try: - q = value if isinstance(value, mpq) else mpq(str(value)) + q = value if isinstance(value, mpq) else safe_mpq_from_any(value) + if q is None: + raise ValueError return f"{float(q * 100):g}%" except (TypeError, ValueError): return format_default(value) @@ -164,12 +172,46 @@ def format_with_type(value, json_type: JsonType | None, *, item=None, show_previ if json_type in (JsonType.BYTES, JsonType.ZLIB, JsonType.GZIP): try: - raw = decode_bytes(value, json_type) if isinstance(value, str) else bytes(value) - preview = " ".join(f"{byte:02X}" for byte in raw[:16]) - if len(raw) > 20: + if isinstance(value, str): + # Estimate decoded size from base64 length (accounting for padding) + padding = value.count("=") + estimated_total = (len(value) * 3) // 4 - padding + truncated = estimated_total > FORMAT_PREVIEW_DECODE_LIMIT_BYTES + if truncated: + # Cap decode work: decode only enough base64 chars for preview. + max_b64_chars = ((FORMAT_PREVIEW_DECODE_LIMIT_BYTES + 2) // 3) * 4 + b64_limited = value[:max_b64_chars] + pad_needed = (4 - len(b64_limited) % 4) % 4 + b64_limited = b64_limited + "=" * pad_needed + raw = base64.b64decode(b64_limited, validate=False) + if json_type is JsonType.ZLIB: + import zlib + + try: + raw = zlib.decompress(raw) + except Exception: + pass + elif json_type is JsonType.GZIP: + import gzip + + try: + raw = gzip.decompress(raw) + except Exception: + pass + else: + # Small enough to decode fully + raw = decode_bytes(value, json_type) + else: + raw = bytes(value) + estimated_total = len(raw) + truncated = len(raw) > FORMAT_PREVIEW_DECODE_LIMIT_BYTES + preview_bytes = raw[:16] + preview = " ".join(f"{byte:02X}" for byte in preview_bytes) + if len(raw) > 20 or truncated: preview += "..." - printable_str = "".join(chr(byte) if 32 <= byte <= 126 else "." for byte in raw[:16]) - return f"<{format_bytes(len(raw))}> | {preview} (`{printable_str}`)" + printable_str = "".join(chr(byte) if 32 <= byte <= 126 else "." for byte in preview_bytes) + size_note = " (truncated preview)" if truncated else "" + return f"<{format_bytes(estimated_total)}> | {preview} (`{printable_str}`){size_note}" except Exception: return format_default(value) diff --git a/delegates/type_delegate.py b/delegates/type_delegate.py index d5f33bd..46b4fa0 100644 --- a/delegates/type_delegate.py +++ b/delegates/type_delegate.py @@ -9,7 +9,7 @@ from themes.icon_provider import IconProvider, StubIconProvider from themes.spec import ThemeSpec from tree.item import JsonTreeItem -from tree.types import USER_SELECTABLE_TYPES, canonical_text_type +from tree.types import USER_SELECTABLE_TYPES, canonical_type class JsonTypeDelegate(QStyledItemDelegate): @@ -160,9 +160,9 @@ def destroyEditor(self, editor: QWidget, index: QModelIndex) -> None: # type: i def setEditorData(self, editor: QComboBox, index: QModelIndex): source_index = self._source_index(index) item: JsonTreeItem = source_index.internalPointer() - # Pseudo text types (EMPTY_*, WS_*) collapse to their canonical parent - # in the combobox so the user sees the type they could pick manually. - current_type = canonical_text_type(item.json_type) + # Pseudo types (EMPTY_*, WS_*, RAW_FLOAT) collapse to their canonical + # parent in the combobox so the user sees the type they could pick. + current_type = canonical_type(item.json_type) idx = editor.findData(current_type) editor.setCurrentIndex(idx if idx >= 0 else 0) diff --git a/documents/composition/demo_data.py b/documents/composition/demo_data.py index 1d61efe..a4dc6f0 100644 --- a/documents/composition/demo_data.py +++ b/documents/composition/demo_data.py @@ -7,6 +7,8 @@ import gmpy2 +from core.raw_numeric import RawNumericValue + def build_demo_data() -> dict[str, Any]: """Default sample document used when ``JsonTab`` is constructed without data.""" @@ -14,6 +16,8 @@ def build_demo_data() -> dict[str, Any]: "question": "The Ultimate Question of Life, the Universe, and Everything.", "answer": 42, "integer": 9223372036854775808, + # Unsupported numeric literal preserved as editable raw text (RAW_FLOAT). + "raw float": RawNumericValue("31e-327018450730"), "int units": "10 m/s", "float units": "3.45s", "int currency": "$10", diff --git a/documents/composition/factory.py b/documents/composition/factory.py index dcdbba9..6f0ba58 100644 --- a/documents/composition/factory.py +++ b/documents/composition/factory.py @@ -11,6 +11,7 @@ from documents.tab import JsonTab from themes.icon_provider import IconProvider from themes.spec import ThemeSpec +from tree.model import JsonTreeModel def create_tab( @@ -26,6 +27,8 @@ def create_tab( icon_provider: IconProvider | None = None, save_format: str | None = None, services: JsonTabServices | None = None, + prebuilt_model: JsonTreeModel | None = None, + defer_validation_init: bool = False, ) -> Document: """Construct a :class:`JsonTab` and expose it as :class:`Document`.""" if data is None: @@ -40,6 +43,8 @@ def create_tab( icon_provider=icon_provider, save_format=save_format, services=services, + prebuilt_model=prebuilt_model, + defer_validation_init=defer_validation_init, ) return JsonTab( @@ -54,6 +59,8 @@ def create_tab( icon_provider=icon_provider, save_format=save_format, services=services, + prebuilt_model=prebuilt_model, + defer_validation_init=defer_validation_init, ) diff --git a/documents/composition/init.py b/documents/composition/init.py index 6a21129..1dc5932 100644 --- a/documents/composition/init.py +++ b/documents/composition/init.py @@ -4,6 +4,9 @@ from typing import Any, Callable +from PySide6.QtCore import QTimer + +import settings from documents.composition.demo_data import build_demo_data from documents.composition.dependencies import JsonTabServices, build_legacy_json_tab_services from documents.composition.setup import ( @@ -27,10 +30,19 @@ from state.affix_mru import AffixMRU from themes.icon_provider import IconProvider from themes.spec import ThemeSpec +from tree.model import JsonTreeModel _DEFAULT_DATA = object() +def _bootstrap_affix_mru(tab: "JsonTab") -> None: + node_count = tab.model.estimated_item_count + if isinstance(node_count, int) and node_count > settings.LOADING_AUTO_EXPAND_MAX_NODES: + QTimer.singleShot(0, lambda: tab._editing.affix_mru.bootstrap_from_tree(tab.model.root_item)) + return + tab._editing.affix_mru.bootstrap_from_tree(tab.model.root_item) + + def bootstrap( tab: "JsonTab", *, @@ -44,6 +56,8 @@ def bootstrap( icon_provider: IconProvider | None, save_format: str | None, services: JsonTabServices | None, + prebuilt_model: JsonTreeModel | None = None, + defer_validation_init: bool = False, ) -> None: """Populate *tab* with controllers, model, view, delegates and validation.""" @@ -84,10 +98,10 @@ def bootstrap( tab._io = IoController(tab, file_path=file_path, save_format=save_format) tab._io.dirtyChanged.connect(tab.dirtyChanged.emit) - init_model(tab, model_data, show_root=show_root) + init_model(tab, model_data, show_root=show_root, prebuilt_model=prebuilt_model) tab._editing.history = TabHistoryController(tab) - tab._editing.affix_mru.bootstrap_from_tree(tab.model.root_item) + _bootstrap_affix_mru(tab) tab._editing.mutations = DocumentMutationGateway(tab) tab._validation = TabValidationController( @@ -110,7 +124,8 @@ def bootstrap( # Install the severity provider before the first revalidation. tab.model.set_issue_index_provider(tab.validation.severity_provider) tab.validationChanged.connect(tab.validation.on_validation_changed) - init_validation_state(tab, model_data) + if not defer_validation_init: + init_validation_state(tab, model_data) tab.undo_stack.cleanChanged.connect(tab.io.on_clean_changed) tab.undo_stack.indexChanged.connect(tab.editing.move.on_undo_index_changed) diff --git a/documents/composition/setup.py b/documents/composition/setup.py index ec0f26f..c7d8a71 100644 --- a/documents/composition/setup.py +++ b/documents/composition/setup.py @@ -65,6 +65,21 @@ def confirm_large_text_edit( # type: ignore[override] ) return answer == QMessageBox.StandardButton.Yes + def warn_raw_numeric_edit(self, parent, *, reason: str) -> None: # type: ignore[override] + from core.raw_numeric import describe_reason + + QMessageBox.warning( + parent if parent is not None else self._tab, + "Unsupported numeric value", + "This value is currently unsupported as a regular float / number " + f"({describe_reason(reason)}).\n\n" + "You can change it into a normally parseable number, or leave the " + "raw value unchanged to preserve it for external software that " + "accepts such values.", + QMessageBox.StandardButton.Ok, + QMessageBox.StandardButton.Ok, + ) + def confirm_large_binary_edit(self, parent, payload_size: int) -> bool: # type: ignore[override] from state.edit_limits import get_binary_edit_warning_limit_bytes @@ -105,10 +120,16 @@ def init_layout(tab: "JsonTab") -> None: tab._appearance.adopt_view_font_defaults(initial_pt) -def init_model(tab: "JsonTab", model_data: Any, show_root: bool) -> None: - tab.editing.model = JsonTreeModel( - model_data, tab.view_state.view, show_root=show_root, icon_provider=tab.appearance.icon_provider - ) +def init_model(tab: "JsonTab", model_data: Any, show_root: bool, prebuilt_model: JsonTreeModel | None = None) -> None: + if prebuilt_model is None: + tab.editing.model = JsonTreeModel( + model_data, tab.view_state.view, show_root=show_root, icon_provider=tab.appearance.icon_provider + ) + else: + prebuilt_model.setParent(tab.view_state.view) + prebuilt_model.show_root = show_root + prebuilt_model.set_icon_provider(tab.appearance.icon_provider) + tab.editing.model = prebuilt_model tab.model.attach_view(tab.view_state.view) tab.view_state.proxy = TreeFilterProxy(tab) tab.view_state.proxy.setSourceModel(tab.model) @@ -119,7 +140,7 @@ def init_model(tab: "JsonTab", model_data: Any, show_root: bool) -> None: def init_validation_state(tab: "JsonTab", model_data: Any) -> None: doc_path = Path(tab.io.file_path).expanduser().resolve() if tab.io.file_path else None - tab.validation.init_state(model_data, doc_path=doc_path) + tab.validation.init_state(model_data, doc_path=doc_path, validation_data=model_data) def init_delegates_and_connections(tab: "JsonTab") -> None: diff --git a/documents/controllers/appearance.py b/documents/controllers/appearance.py index 80c1818..d9cfc4f 100644 --- a/documents/controllers/appearance.py +++ b/documents/controllers/appearance.py @@ -4,6 +4,7 @@ from PySide6.QtCore import QModelIndex, QSize, Qt +import settings from delegates.name_delegate import NameDelegate from delegates.type_delegate import JsonTypeDelegate from delegates.value import ValueDelegate @@ -176,9 +177,17 @@ def resize_key_columns(self, force: bool = False) -> None: self._programmatic_column_resize = True try: view = self._require_view() + model = self._require_model() + is_large_tree = ( + isinstance(model.estimated_item_count, int) + and model.estimated_item_count > settings.LOADING_AUTO_EXPAND_MAX_NODES + ) for col in (0, 1): if force or col not in self._user_sized_columns: - view.resizeColumnToContents(col) + if is_large_tree: + view.setColumnWidth(col, 280 if col == 0 else 160) + else: + view.resizeColumnToContents(col) finally: self._programmatic_column_resize = False diff --git a/documents/controllers/number_types.py b/documents/controllers/number_types.py index 08e0913..f8e21c6 100644 --- a/documents/controllers/number_types.py +++ b/documents/controllers/number_types.py @@ -2,6 +2,7 @@ import gmpy2 +from core.safe_mpq import safe_mpq_from_any from tree.item import JsonTreeItem from tree.types import JsonType from units.number_affix import NumberAffix @@ -23,8 +24,7 @@ def would_drop_fraction_on_type_change(item: JsonTreeItem, target_type: JsonType if not is_integer_number_type(target_type) or not is_float_number_type(item.json_type): return False source_value = item.value.number if isinstance(item.value, NumberAffix) else item.value - try: - q = gmpy2.mpq(str(source_value)) - except (TypeError, ValueError): + q = source_value if isinstance(source_value, gmpy2.mpq) else safe_mpq_from_any(source_value) + if q is None: return False return q.denominator != 1 diff --git a/documents/controllers/validation.py b/documents/controllers/validation.py index 9376472..663f722 100644 --- a/documents/controllers/validation.py +++ b/documents/controllers/validation.py @@ -25,6 +25,8 @@ class TabValidationController(QObject): """Own schema state, issue tracking, and revalidation for one tab.""" + _REPAINT_BATCH_SIZE = 256 + def __init__( self, tab, @@ -45,6 +47,12 @@ def __init__( self._schema: dict[str, Any] | None = None self._issue_index = IssueIndex([], initial_data) self._auto_rescan: bool = False + self._last_affected_paths: set[tuple[int, ...]] = set() + self._pending_repaint_paths: list[tuple[int, ...]] = [] + + self._repaint_timer = QTimer(self) + self._repaint_timer.setSingleShot(True) + self._repaint_timer.timeout.connect(self._flush_repaint_batch) self.debounce_timer = QTimer(self) self.debounce_timer.setSingleShot(True) @@ -81,7 +89,14 @@ def issue_index(self) -> IssueIndex: def auto_rescan(self) -> bool: return self._auto_rescan - def init_state(self, model_data: Any, *, doc_path: Path | None = None) -> None: + def init_state( + self, + model_data: Any, + *, + doc_path: Path | None = None, + revalidate: bool = True, + validation_data: Any | None = None, + ) -> None: from state.validation_settings import _is_url, read_schema_ref_str if doc_path is None and self._tab.io.file_path: @@ -105,13 +120,31 @@ def init_state(self, model_data: Any, *, doc_path: Path | None = None) -> None: else: ref = SchemaRef(path=Path(persisted), inline=dict(loaded), origin="manual") - self.set_schema(ref) + self.set_schema(ref, revalidate=False) + if not revalidate: + return + if validation_data is None: + self.revalidate() + return + self.revalidate_with_data(validation_data) - def set_schema(self, ref: SchemaRef) -> None: - self._swap_source(SchemaSource.from_ref(ref), ref) + def set_schema( + self, + ref: SchemaRef, + *, + revalidate: bool = True, + validation_data: Any | None = None, + ) -> None: + self._swap_source(SchemaSource.from_ref(ref), ref, revalidate=revalidate, validation_data=validation_data) - def set_schema_from_source(self, source: SchemaSource) -> None: - self._swap_source(source, source.as_ref()) + def set_schema_from_source( + self, + source: SchemaSource, + *, + revalidate: bool = True, + validation_data: Any | None = None, + ) -> None: + self._swap_source(source, source.as_ref(), revalidate=revalidate, validation_data=validation_data) def clear_schema(self) -> None: self.set_schema(SchemaRef(path=None, inline=None, origin="none")) @@ -124,7 +157,14 @@ def set_schema_view_source(self, source: SchemaSource | None) -> None: ) self._on_schema_changed(self._schema_ref) - def _swap_source(self, source: SchemaSource | None, ref: SchemaRef) -> None: + def _swap_source( + self, + source: SchemaSource | None, + ref: SchemaRef, + *, + revalidate: bool, + validation_data: Any | None, + ) -> None: if self._schema_source is not None: get_schema_registry().release(self._schema_source, self._tab) @@ -140,20 +180,43 @@ def _swap_source(self, source: SchemaSource | None, ref: SchemaRef) -> None: else: self._schema = entry.inline if entry is not None else None self._on_schema_changed(self._schema_ref) + if not revalidate: + return + if validation_data is not None: + self.revalidate_with_data(validation_data) + return + if self._schema is None: + self._clear_issues_without_snapshot() + return self.revalidate() def revalidate(self) -> None: - root_data = self._model.root_item.to_json() + self.revalidate_with_data(self._model.root_item.to_json()) + + def revalidate_loading_data(self, parsed_data: Any) -> None: + self.revalidate_with_data(parsed_data) + + def revalidate_with_data(self, root_data: Any) -> None: + if self._schema is None: + self._clear_issues_without_snapshot(root_data=root_data) + return + issues: list[ValidationIssue] = [] - if self._schema is not None: - sanitized = to_jsonschema_input(root_data) - if self._tab.io.save_format == SAVE_FORMAT_YAML_MULTI and isinstance(sanitized, list): - issues = validate_yaml_documents(sanitized, self._schema) - else: - issues = validate_document(sanitized, self._schema) + sanitized = to_jsonschema_input(root_data) + if self._tab.io.save_format == SAVE_FORMAT_YAML_MULTI and isinstance(sanitized, list): + issues = validate_yaml_documents(sanitized, self._schema) + else: + issues = validate_document(sanitized, self._schema) self._issue_index = IssueIndex(issues, root_data) self._on_validation_changed(self._issue_index) + def _clear_issues_without_snapshot(self, *, root_data: Any | None = None) -> None: + if self._issue_index.is_empty(): + return + empty_root = {} if root_data is None else root_data + self._issue_index = IssueIndex([], empty_root) + self._on_validation_changed(self._issue_index) + def set_auto_rescan(self, enabled: bool) -> None: enabled = bool(enabled) if self._auto_rescan == enabled: @@ -191,6 +254,8 @@ def release(self) -> None: self._released = True self.debounce_timer.stop() + self._repaint_timer.stop() + self._pending_repaint_paths = [] if self._schema_source is not None: get_schema_registry().release(self._schema_source, self._tab) @@ -257,20 +322,39 @@ def severity_provider(self, model_path: tuple[int, ...]) -> str | None: return self.severity_for(model_path) def on_validation_changed(self, _index: IssueIndex) -> None: - """Emit recursive dataChanged so all visible rows repaint their badges.""" - - def emit_ranges(parent: QModelIndex) -> None: - model = self._tab.model - rows = model.rowCount(parent) - if rows <= 0: - return - top_left = model.index(0, 0, parent) - bottom_right = model.index(rows - 1, model.columnCount(parent) - 1, parent) + """Emit bounded dataChanged updates for paths affected by issue changes.""" + new_paths = _index.affected_paths() + changed_paths = sorted(self._last_affected_paths | new_paths, key=lambda path: (len(path), path)) + self._last_affected_paths = set(new_paths) + if not changed_paths: + return + + if len(changed_paths) <= self._REPAINT_BATCH_SIZE: + self._emit_path_repaints(changed_paths) + return + + self._pending_repaint_paths = changed_paths + if not self._repaint_timer.isActive(): + self._repaint_timer.start(0) + + def _emit_path_repaints(self, paths: list[tuple[int, ...]]) -> None: + model = self._tab.model + for path in paths: + index = self._tab.view_controller.index_from_path(path) + if not index.isValid(): + continue + top_left = index.siblingAtColumn(0) + bottom_right = index.siblingAtColumn(model.columnCount(index.parent()) - 1) model.dataChanged.emit(top_left, bottom_right, [VALIDATION_SEVERITY_ROLE]) - for row in range(rows): - emit_ranges(model.index(row, 0, parent)) - emit_ranges(QModelIndex()) + def _flush_repaint_batch(self) -> None: + if not self._pending_repaint_paths: + return + batch = self._pending_repaint_paths[: self._REPAINT_BATCH_SIZE] + self._pending_repaint_paths = self._pending_repaint_paths[self._REPAINT_BATCH_SIZE :] + self._emit_path_repaints(batch) + if self._pending_repaint_paths: + self._repaint_timer.start(0) __all__ = ["TabValidationController"] diff --git a/documents/controllers/view.py b/documents/controllers/view.py index d807b98..017a845 100644 --- a/documents/controllers/view.py +++ b/documents/controllers/view.py @@ -4,7 +4,7 @@ from typing import Any -from PySide6.QtCore import QModelIndex, QObject, QPersistentModelIndex, QSortFilterProxyModel, Signal +from PySide6.QtCore import QModelIndex, QObject, QPersistentModelIndex, QSortFilterProxyModel, QTimer, Signal from tree.types import JsonType @@ -209,6 +209,13 @@ def request_select_paths(self, paths: list[tuple[int, ...]]) -> None: return self.viewportRequested.emit(KIND_SELECT_PATHS, list(paths)) + def request_select_paths_deferred(self, paths: list[tuple[int, ...]]) -> None: + """Queue ``request_select_paths`` for the next event-loop turn.""" + if not paths: + return + queued = list(paths) + QTimer.singleShot(0, lambda: self.request_select_paths(queued)) + def request_expand(self, path: tuple[int, ...]) -> None: """Ask the viewport to expand ``path``.""" self.viewportRequested.emit(KIND_EXPAND_PATH, tuple(path)) @@ -225,6 +232,11 @@ def request_scroll_to(self, path: tuple[int, ...]) -> None: """Ask the viewport to scroll ``path`` into view.""" self.viewportRequested.emit(KIND_SCROLL_TO, tuple(path)) + def request_scroll_to_deferred(self, path: tuple[int, ...]) -> None: + """Queue ``request_scroll_to`` for the next event-loop turn.""" + queued = tuple(path) + QTimer.singleShot(0, lambda: self.request_scroll_to(queued)) + def apply_request(self, kind: str, payload: object) -> None: """Apply a queued viewport request to the tree view.""" view = self._tab.view diff --git a/documents/states/editing/move_view_state.py b/documents/states/editing/move_view_state.py index af553ef..07b16f8 100644 --- a/documents/states/editing/move_view_state.py +++ b/documents/states/editing/move_view_state.py @@ -25,8 +25,15 @@ def __init__(self, context: EditingContext) -> None: def collect_expanded_paths(self) -> list[tuple[int, ...]]: """Return paths of every currently expanded row.""" + return list(self.iter_expanded_paths()) + + def iter_expanded_paths(self): + """Yield paths of currently expanded rows. + + This iterator allows callers to process large expansion sets in batches + and yield to the event loop between batches. + """ tab = self._context.tab - paths: list[tuple[int, ...]] = [] def visit(parent_index: QModelIndex) -> None: for r in range(tab.model.rowCount(parent_index)): @@ -35,11 +42,10 @@ def visit(parent_index: QModelIndex) -> None: continue view_child = tab.view_controller.source_to_view(child) if tab.view_state.view.isExpanded(view_child): - paths.append(tab.view_controller.index_path(child)) - visit(child) + yield tab.view_controller.index_path(child) + yield from visit(child) - visit(QModelIndex()) - return paths + yield from visit(QModelIndex()) def capture_move_view_state(self, sources: list) -> dict[str, Any]: tab = self._context.tab diff --git a/documents/tab.py b/documents/tab.py index 375ecd4..5e5f07f 100644 --- a/documents/tab.py +++ b/documents/tab.py @@ -64,6 +64,8 @@ def __init__( save_format: str | None = None, *, services: JsonTabServices | None = None, + prebuilt_model: JsonTreeModel | None = None, + defer_validation_init: bool = False, ): super().__init__(parent) @@ -79,6 +81,8 @@ def __init__( icon_provider=icon_provider, save_format=save_format, services=services, + prebuilt_model=prebuilt_model, + defer_validation_init=defer_validation_init, ) @property diff --git a/editors/context.py b/editors/context.py index a084f56..ec8c75a 100644 --- a/editors/context.py +++ b/editors/context.py @@ -40,6 +40,8 @@ def notify_status(self, message: str, timeout_ms: int = 0) -> None: ... def confirm_large_binary_edit(self, parent, payload_size: int) -> bool: ... + def warn_raw_numeric_edit(self, parent, *, reason: str) -> None: ... + def confirm_large_text_edit( self, parent, diff --git a/editors/factory.py b/editors/factory.py index 735f76f..98e1274 100644 --- a/editors/factory.py +++ b/editors/factory.py @@ -9,6 +9,8 @@ from PySide6.QtWidgets import QComboBox, QLineEdit, QStyleOptionViewItem, QWidget from core.datetime_parsing.enums import DateTimeCategory +from core.raw_numeric import REASON_UNKNOWN, RawNumericValue +from core.safe_mpq import safe_mpq_from_any from delegates.number_affix_delegate import ( is_affix_json_type, is_integer_json_type, @@ -22,6 +24,7 @@ from editors.inline.caps_safe_line import _CapsLockSafeLineEdit from editors.inline.datetime.better_dt_editor import BetterDateTimeEditor from editors.inline.mpq_spinbox import QMpqSpinBox +from editors.inline.raw_numeric_line import RawNumericLineEdit from editors.inline.secret_line import _SecretEditorWatcher, _SecretLineEdit from editors.windowed.color_dialog import ColorPickerDialog from editors.windowed.hex_dialog import QHexDialog @@ -108,6 +111,12 @@ def create_value_editor( editor = QBigIntSpinBox(parent) case JsonType.FLOAT: editor = QMpqSpinBox(parent, item.value) + case JsonType.RAW_FLOAT: + # Unsupported numeric literal: edit as plain raw text and warn the + # user (once, when the editor opens) about the unsupported state. + reason = item.value.reason if isinstance(item.value, RawNumericValue) else REASON_UNKNOWN + delegate._context_for(parent).warn_raw_numeric_edit(parent, reason=reason) + editor = RawNumericLineEdit(parent) case JsonType.PERCENT: editor = QMpqSpinBox( parent, @@ -266,9 +275,8 @@ def set_value_editor_data(delegate: ValueDelegateProtocol, editor: QWidget, inde return if isinstance(editor, QMpqSpinBox): - try: - v = mpq(str(value)) if not isinstance(value, mpq) else value - except (TypeError, ValueError): + v = value if isinstance(value, mpq) else safe_mpq_from_any(value) + if v is None: v = mpq(0) if item.json_type is JsonType.PERCENT: v = v * 100 diff --git a/editors/inline/mpq_spinbox/spinbox.py b/editors/inline/mpq_spinbox/spinbox.py index 2bc1b6d..ecc99d4 100644 --- a/editors/inline/mpq_spinbox/spinbox.py +++ b/editors/inline/mpq_spinbox/spinbox.py @@ -4,6 +4,7 @@ from PySide6.QtWidgets import QAbstractSpinBox from coalesce import nn +from core.safe_mpq import safe_mpq_from_text from editors.inline.mpq_spinbox.validator import MpqValidator, _to_mpq from mpq2py import mpq_serialization @@ -178,20 +179,18 @@ def lineEditEditingFinalize(self): text = self.lineEdit().text() # 1) Try as plain number - try: - self.setValue(mpq(text)) + parsed = safe_mpq_from_text(text) + if parsed is not None: + self.setValue(parsed) return - except ValueError: - pass # 2) Try number with prefix/suffix if text.startswith(self._prefix) and text.endswith(self._suffix): number = text[(len(self._prefix)) : -len(self._suffix) or None] - try: - self.setValue(mpq(number)) + parsed = safe_mpq_from_text(number) + if parsed is not None: + self.setValue(parsed) return - except ValueError: - pass # 3) Revert self.setValue(self._value) diff --git a/editors/inline/mpq_spinbox/validator.py b/editors/inline/mpq_spinbox/validator.py index b416f6b..d7a8b76 100644 --- a/editors/inline/mpq_spinbox/validator.py +++ b/editors/inline/mpq_spinbox/validator.py @@ -6,6 +6,7 @@ from PySide6.QtGui import QValidator from coalesce import nn +from core.safe_mpq import safe_mpq_from_any PARTIAL_FLOAT = re.compile(r"[-+]?\d*\.?\d*e?[-+]?\d*", re.IGNORECASE) @@ -18,9 +19,15 @@ def _to_mpq(x) -> mpq: if isinstance(x, mpq): return x if isinstance(x, Decimal): - return mpq(str(x)) # avoid binary float artifacts + parsed = safe_mpq_from_any(str(x)) + if parsed is None: + raise ValueError("Unsafe numeric literal") + return parsed - return mpq(x) + parsed = safe_mpq_from_any(x) + if parsed is None: + raise ValueError("Unsafe numeric literal") + return parsed class MpqValidator(QValidator): @@ -59,7 +66,7 @@ def validate(self, s: str, pos: int) -> Tuple[QValidator.State, str, int]: # 1) Try as plain number (no requirement to already contain prefix/suffix) try: - n = mpq(s) + n = _to_mpq(s) if in_range(minv, n, maxv): return ( QValidator.State.Acceptable, @@ -86,7 +93,7 @@ def validate(self, s: str, pos: int) -> Tuple[QValidator.State, str, int]: return QValidator.State.Intermediate, s, pos try: - n = mpq(number) + n = _to_mpq(number) if in_range(minv, n, maxv): return QValidator.State.Acceptable, s, pos else: diff --git a/editors/inline/raw_numeric_line.py b/editors/inline/raw_numeric_line.py new file mode 100644 index 0000000..eab561a --- /dev/null +++ b/editors/inline/raw_numeric_line.py @@ -0,0 +1,34 @@ +"""Inline editor for raw, unsupported numeric literals. + +Editing happens as plain text guarded by a deliberately narrow validator +(:func:`core.raw_numeric.raw_numeric_text_is_acceptable`). The validator never +returns ``Invalid`` so the user can keep typing and can leave the original raw +value unchanged; the model's ``set_data`` makes the final keep / convert / +reject decision. +""" + +from PySide6.QtGui import QValidator + +from core.raw_numeric import raw_numeric_text_is_acceptable +from editors.inline.caps_safe_line import _CapsLockSafeLineEdit + + +class RawNumericValidator(QValidator): + """Accept only the app's narrow raw-numeric recovery grammar.""" + + def validate(self, s: str, pos: int): + if s == "": + return QValidator.State.Intermediate, s, pos + if raw_numeric_text_is_acceptable(s): + return QValidator.State.Acceptable, s, pos + # Intermediate (never Invalid) so partial input and the unchanged + # original literal are not blocked at the keystroke level. + return QValidator.State.Intermediate, s, pos + + +class RawNumericLineEdit(_CapsLockSafeLineEdit): + """Plain-text line editor used for ``JsonType.RAW_FLOAT`` values.""" + + def __init__(self, parent=None): + super().__init__(parent) + self.setValidator(RawNumericValidator(self)) diff --git a/io_formats/load.py b/io_formats/load.py index b831e06..f3bfdfc 100644 --- a/io_formats/load.py +++ b/io_formats/load.py @@ -5,12 +5,14 @@ secret_line / secret_text again. """ +from collections.abc import Callable from typing import Any import simplejson import yaml -from gmpy2 import mpq +from core.raw_numeric import REASON_NON_FINITE, REASON_UNKNOWN, RawNumericValue +from core.safe_mpq import parse_mpq from io_formats.detect import ( SAVE_FORMAT_JSON, SAVE_FORMAT_JSONL, @@ -38,45 +40,147 @@ ) -def _decode_number_affixes(value: Any) -> Any: +def _safe_parse_float(text: str): + """Parse a JSON float token into a safe ``mpq`` or a ``RawNumericValue``. + + Successful safe parses become exact rationals. Anything the safe parser + rejects (overflow, underflow, non-finite, precision, format) is preserved + verbatim as raw, editable text instead of constructing an unsafe value. + """ + result = parse_mpq(text) + if result.value is not None: + return result.value + return RawNumericValue(raw=text, reason=result.reason or REASON_UNKNOWN, source_syntax="json") + + +def _safe_parse_constant(name: str): + """Preserve non-standard JSON constants (NaN / Infinity) as raw text.""" + return RawNumericValue(raw=name, reason=REASON_NON_FINITE, source_syntax="json") + + +def _decode_number_affixes(value: Any, on_progress: Callable[[int, str], None] | None = None) -> Any: + """Decode NumberAffix-like strings with optional iterative progress callback.""" + + processed = 0 + + def _notify(path: str) -> None: + nonlocal processed + processed += 1 + if on_progress is not None: + on_progress(processed, path) + if isinstance(value, str): - # parse_json_type encodes the canonical priority of string heuristics - # (multiline -> color -> datetime -> number-affix -> base64 -> text). - # Only convert to NumberAffix when parse_json_type agrees the string is - # actually a number-with-affix; otherwise leave it intact. - if parse_json_type(value) in _AFFIX_JSON_TYPES: - parsed = parse_number_affix(value, max_affix_len=NUMBER_AFFIX_MAX_LEN) - if parsed is not None: - return parsed + _notify("") + return _maybe_decode_string(value) + + if not isinstance(value, (list, dict)): + _notify("") return value - if isinstance(value, list): - return [_decode_number_affixes(v) for v in value] - if isinstance(value, dict): - return {k: _decode_number_affixes(v) for k, v in value.items()} + + root_result: Any = [None] * len(value) if isinstance(value, list) else {} + stack: list[tuple[Any, Any, str, int, list[Any]]] = [] + stack.append((value, root_result, "", 0, _entries_for_container(value))) + + while stack: + src_container, dst_container, parent_path, index, entries = stack[-1] + if index >= len(entries): + stack.pop() + continue + + stack[-1] = (src_container, dst_container, parent_path, index + 1, entries) + name, child_value = entries[index] + child_path = _append_json_pointer(parent_path, name) + _notify(child_path) + + if isinstance(child_value, str): + decoded = _maybe_decode_string(child_value) + _assign_container_value(dst_container, name, decoded) + continue + + if isinstance(child_value, dict): + child_result: dict[Any, Any] = {} + _assign_container_value(dst_container, name, child_result) + stack.append((child_value, child_result, child_path, 0, _entries_for_container(child_value))) + continue + + if isinstance(child_value, list): + child_result = [None] * len(child_value) + _assign_container_value(dst_container, name, child_result) + stack.append((child_value, child_result, child_path, 0, _entries_for_container(child_value))) + continue + + _assign_container_value(dst_container, name, child_value) + + return root_result + + +def _maybe_decode_string(value: str) -> Any: + # parse_json_type encodes the canonical priority of string heuristics + # (multiline -> color -> datetime -> number-affix -> base64 -> text). + # Only convert to NumberAffix when parse_json_type agrees the string is + # actually a number-with-affix; otherwise leave it intact. + if parse_json_type(value) in _AFFIX_JSON_TYPES: + parsed = parse_number_affix(value, max_affix_len=NUMBER_AFFIX_MAX_LEN) + if parsed is not None: + return parsed return value +def _entries_for_container(value: Any) -> list[Any]: + if isinstance(value, dict): + return list(value.items()) + return list(enumerate(value)) + + +def _assign_container_value(container: Any, key: Any, value: Any) -> None: + if isinstance(container, list): + container[key] = value + else: + container[key] = value + + +def _append_json_pointer(parent_path: str, segment: str | int) -> str: + token = str(segment).replace("~", "~0").replace("/", "~1") + if not parent_path: + return f"/{token}" + return f"{parent_path}/{token}" + + def load_file(path: str) -> Any: data, _fmt = load_file_with_format(path) return data -def load_file_with_format(path: str) -> tuple[Any, str]: +def load_file_with_format( + path: str, + on_progress: Callable[[int, str], None] | None = None, +) -> tuple[Any, str]: fmt = detect_format(path) with open(path, "r", encoding="utf-8") as fh: if fmt == SAVE_FORMAT_JSON: - return _decode_number_affixes(simplejson.load(fh, parse_float=mpq)), SAVE_FORMAT_JSON + return ( + _decode_number_affixes( + simplejson.load(fh, parse_float=_safe_parse_float, parse_constant=_safe_parse_constant), + on_progress=on_progress, + ), + SAVE_FORMAT_JSON, + ) if fmt == SAVE_FORMAT_JSONL: rows = [] for line in fh: stripped = line.strip() if not stripped: continue - rows.append(_decode_number_affixes(simplejson.loads(stripped, parse_float=mpq))) + rows.append( + _decode_number_affixes( + simplejson.loads(stripped, parse_float=_safe_parse_float, parse_constant=_safe_parse_constant), + on_progress=on_progress, + ) + ) return rows, SAVE_FORMAT_JSONL - docs = [_decode_number_affixes(doc) for doc in yaml.load_all(fh, Loader=MpqSafeLoader)] + docs = [_decode_number_affixes(doc, on_progress=on_progress) for doc in yaml.load_all(fh, Loader=MpqSafeLoader)] if len(docs) <= 1: return (docs[0] if docs else {}), SAVE_FORMAT_YAML return docs, SAVE_FORMAT_YAML_MULTI diff --git a/mpq2py/__init__.py b/mpq2py/__init__.py index 89ec046..323a9f4 100644 --- a/mpq2py/__init__.py +++ b/mpq2py/__init__.py @@ -1,12 +1,25 @@ +import re from decimal import Decimal +import simplejson import yaml from gmpy2 import mpq from pandas import Timestamp from core.datetime_parsing.nano_time import NanoTime +from core.raw_numeric import REASON_UNKNOWN, RawNumericValue +from core.safe_mpq import parse_mpq from units.number_affix import NumberAffix, format_number_affix +# Strict JSON number grammar. A raw numeric value may only be injected directly +# into JSON output when its literal is a valid JSON number; otherwise saving as +# JSON must fail loudly rather than emit invalid JSON or silently quote it. +_JSON_NUMBER_TOKEN_RE = re.compile(r"-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][-+]?[0-9]+)?\Z") + + +def raw_numeric_is_json_safe(raw: str) -> bool: + return bool(_JSON_NUMBER_TOKEN_RE.match(raw.strip())) + def mpq_serialization(q: mpq) -> tuple[float | Decimal, mpq]: num, den = q.as_integer_ratio() @@ -60,6 +73,13 @@ def mpq_json_default(obj): if isinstance(obj, mpq): # Emit only the scalar value; the helper's denominator metadata is not JSON-serializable. return mpq_serialization(obj)[0] + if isinstance(obj, RawNumericValue): + if raw_numeric_is_json_safe(obj.raw): + return simplejson.RawJSON(obj.raw.strip()) + raise ValueError( + f"Raw numeric value {obj.raw!r} (reason: {obj.reason}) is not a valid " + "JSON number; convert it to a supported number or save as YAML." + ) if isinstance(obj, Decimal): # ``mpq_serialization`` returns ``Decimal`` for terminating fractions. The stdlib # ``json`` module re-invokes this default on the Decimal because it is not natively @@ -84,11 +104,14 @@ class MpqSafeLoader(yaml.SafeLoader): def mpq_yaml_float_construct(loader: MpqSafeLoader, node: yaml.ScalarNode): - try: - return mpq(node.value.replace("_", "").lower().strip()) - except Exception: - # If mpq can't parse some odd form, fall back to default behavior - return yaml.constructor.SafeConstructor.construct_yaml_float(loader, node) + result = parse_mpq(node.value) + if result.value is not None: + return result.value + + # Preserve the original literal exactly (including non-finite spellings such + # as ``.inf`` / ``.nan``) as editable raw text instead of constructing a + # Python float, so rendering and validation never depend on float quirks. + return RawNumericValue(raw=node.value, reason=result.reason or REASON_UNKNOWN, source_syntax="yaml") MpqSafeLoader.add_constructor("tag:yaml.org,2002:float", mpq_yaml_float_construct) @@ -123,3 +146,10 @@ def _nanotime_yaml_represent(dumper: MpqSafeDumper, data: NanoTime): MpqSafeDumper.add_representer(NanoTime, _nanotime_yaml_represent) + + +def _raw_numeric_yaml_represent(dumper: MpqSafeDumper, data: RawNumericValue): + return dumper.represent_scalar("tag:yaml.org,2002:float", data.raw) + + +MpqSafeDumper.add_representer(RawNumericValue, _raw_numeric_yaml_represent) diff --git a/pytest.ini b/pytest.ini index e873c51..9c627a8 100644 --- a/pytest.ini +++ b/pytest.ini @@ -1,5 +1,7 @@ [pytest] pythonpath = . +markers = + perf: opt-in performance tests for parsing vulnerability measurement (deselect with '-m "not perf"') filterwarnings = ignore:Failed to disconnect:RuntimeWarning ignore::DeprecationWarning:documents.tab diff --git a/settings.py b/settings.py index ba3b947..f8a9b09 100644 --- a/settings.py +++ b/settings.py @@ -1,3 +1,5 @@ +from __future__ import annotations + APPLICATION_ID = "org.jeditor.204b2a8a-e956-4c62-911d-cfb29a4fe257" MODAL_WINDOW_SIZE = (600, 440) WINDOW_DEFAULT_SIZE = (800, 600) @@ -26,3 +28,75 @@ # Number-affix parsing/editor limits. NUMBER_AFFIX_MAX_LEN = 20 NUMBER_AFFIX_MRU_SIZE = 50 + +# gmpy2/mpq safety limits. Very large scientific exponents can trigger +# low-level GMP aborts (not regular Python exceptions) while building mpz +# intermediates for powers-of-ten. +# +# Limits are enforced through ``decimal.Context`` traps in +# ``core.safe_mpq`` (Overflow / Underflow / Inexact / InvalidOperation), +# not regex filtering: +# - ``MPQ_SAFE_MAX_ABS_EXPONENT`` bounds context Emax/Emin. +# - ``MPQ_SAFE_MAX_SIG_DIGITS`` bounds significant digits via precision. +MPQ_SAFE_MAX_ABS_EXPONENT = 10_000 +MPQ_SAFE_MAX_SIG_DIGITS = 4_300 + +# --------------------------------------------------------------------------- +# Inference safety limits (Plan 1 — length limits for expensive inference) +# --------------------------------------------------------------------------- +# These constants gate expensive inference work (regex, datetime parsing, +# color checks) during automatic type classification. +# They are NOT user-exposed settings and must not use QSettings. +# +# Values are justified by reports/parsing-vulnerability-2026-06-13.md which +# measured 832 rows across 16 registry entries and 13 adversarial families +# at sizes 1024, 4096, 16384, and 65536. +# +# Design decisions: +# - No INFERENCE_MAX_TOTAL_CHARS: individual gates (datetime, affix, color) +# effectively skip all unnecessary checks; a top-level fast path is redundant. +# - No INFERENCE_MAX_BASE64_PROBE_CHARS: base64 uses content-based syntax +# validation (len mod 4 + alphabet regex) instead of a length cap. +# - No EDITABLE_DECODE_LIMIT_BYTES: if base64 syntax is valid, decode is allowed. + +# parse_datetime_text() regex and datetime conversion. +# Report: DATETIME_RE.fullmatch median is 0.00ms even at 65536 across all +# families; 40 is enough for any practically meaningful datetime string. +INFERENCE_MAX_DATETIME_CHARS: int = 40 + +# parse_number_affix() regex checks. +# Report: parse_number_affix is superlinear on digits, plain_ascii, +# pathological_repetition at 4096+ (ratio up to 4.89). 100 is well below +# the pre-existing 4300-digit integer limit, so the gate fires before the +# error path. +INFERENCE_MAX_AFFIX_CHARS: int = 100 + +# looks_like_color_rgb() and looks_like_color_rgba(). +# Maximum length of #RGB, #RRGGBB, #RGBA, and #RRGGBBAA color strings. +INFERENCE_MAX_COLOR_CHARS: int = 10 + +# format_with_type() display preview decode cap. +# Preview needs only enough bytes to render the existing prefix text. +FORMAT_PREVIEW_DECODE_LIMIT_BYTES: int = 100 + +# --------------------------------------------------------------------------- +# Loading progress widget (Plan 2 — delayed progress bar for big files) +# --------------------------------------------------------------------------- +# The progress widget only appears if a load operation takes longer than this +# delay. Fast loads complete before the widget shows, avoiding visual noise. +LOADING_PROGRESS_DELAY_MS: int = 5000 + +# Tab-close progress appears only when close/teardown runs longer than this +# threshold. This is informational-only (non-cancellable) UI. +CLOSE_PROGRESS_DELAY_MS: int = 1500 + +# Maximum repaint cadence for loading detail text (processed count + current path). +LOADING_PROGRESS_DETAIL_REFRESH_MS: int = 1000 + +# Large-load presentation guardrails. When the prebuilt model reports a node +# count above this threshold, first-paint avoids full-tree expand-all and +# content-based key-column auto-sizing scans. +LOADING_AUTO_EXPAND_MAX_NODES: int = 10_000 + +# Helps to define hard deadline, after what loading canceled (10 minutes) +LOADING_HARD_TIMEOUT_SECONDS = 600.0 diff --git a/state/view_state.py b/state/view_state.py index f934846..5528583 100644 --- a/state/view_state.py +++ b/state/view_state.py @@ -1,13 +1,16 @@ import hashlib from pathlib import Path -from PySide6.QtCore import QModelIndex, QSettings, QSortFilterProxyModel +from PySide6.QtCore import QCoreApplication, QModelIndex, QSettings, QSortFilterProxyModel, QTimer +import settings from documents.seams.document_protocol import Document from settings import APPLICATION_ID from state.qsettings_coercion import _coerce_int, _coerce_int_list, _coerce_path, _coerce_paths MAX_EXPANDED_PATHS = 5000 +_RESTORE_BATCH_SIZE = 256 +_SAVE_BATCH_SIZE = 256 def _source_to_view_index(view, source_index: QModelIndex) -> QModelIndex: @@ -69,6 +72,29 @@ def state_key(path: str) -> str: return f"view_state/{digest}" +def _is_large_load(tab: Document) -> bool: + node_count = tab.model.estimated_item_count + return isinstance(node_count, int) and node_count > settings.LOADING_AUTO_EXPAND_MAX_NODES + + +def _request_expand_paths_chunked(tab: Document, paths: list[tuple[int, ...]]) -> None: + if not paths: + return + + cursor = {"index": 0} + + def _emit_batch() -> None: + start = cursor["index"] + end = min(start + _RESTORE_BATCH_SIZE, len(paths)) + for i in range(start, end): + tab.view_controller.request_expand(paths[i]) + cursor["index"] = end + if cursor["index"] < len(paths): + QTimer.singleShot(0, _emit_batch) + + QTimer.singleShot(0, _emit_batch) + + def save(tab: Document) -> None: if not tab.io.file_path: return @@ -77,7 +103,13 @@ def save(tab: Document) -> None: settings.beginGroup(state_key(tab.io.file_path)) widths = tab.view_controller.column_widths() - expanded_paths = [list(path) for path in tab.editing.move.collect_expanded_paths()[:MAX_EXPANDED_PATHS]] + expanded_paths: list[list[int]] = [] + for i, path in enumerate(tab.editing.move.iter_expanded_paths(), start=1): + expanded_paths.append(list(path)) + if len(expanded_paths) >= MAX_EXPANDED_PATHS: + break + if i % _SAVE_BATCH_SIZE == 0: + QCoreApplication.processEvents() current_path_tuple = tab.view_controller.current_path() current_path = list(current_path_tuple) if current_path_tuple is not None else [] @@ -91,7 +123,7 @@ def save(tab: Document) -> None: settings.endGroup() -def restore(tab: Document) -> bool: +def restore(tab: Document, *, defer_heavy: bool = False) -> bool: if not tab.io.file_path: return False @@ -120,13 +152,23 @@ def restore(tab: Document) -> bool: if widths is not None: tab.view_controller.set_column_widths(widths) + large_load = defer_heavy or _is_large_load(tab) + if expanded is not None: tab.view_controller.request_collapse_all() - for path in expanded: - tab.view_controller.request_expand(tuple(path)) + expanded_paths = [tuple(path) for path in expanded] + if large_load: + _request_expand_paths_chunked(tab, expanded_paths) + else: + for path in expanded_paths: + tab.view_controller.request_expand(path) if current_path is not None: - tab.view_controller.request_select_paths([tuple(current_path)]) + selected = tuple(current_path) + if large_load: + tab.view_controller.request_select_paths_deferred([selected]) + else: + tab.view_controller.request_select_paths([selected]) return True @@ -134,3 +176,63 @@ def restore(tab: Document) -> bool: def discard(path: str) -> None: settings = QSettings(APPLICATION_ID, "view_state") settings.remove(state_key(path)) + + +def capture_runtime_state(tab: Document) -> dict[str, object]: + """Capture in-memory view state for model-root swaps.""" + return { + "col_widths": tab.view_controller.column_widths(), + "expanded": tab.editing.move.collect_expanded_paths()[:MAX_EXPANDED_PATHS], + "current_path": tab.view_controller.current_path(), + "h_scroll": int(tab.view.horizontalScrollBar().value()), + "v_scroll": int(tab.view.verticalScrollBar().value()), + } + + +def restore_runtime_state(tab: Document, snapshot: dict[str, object]) -> None: + """Restore in-memory view state captured by :func:`capture_runtime_state`.""" + col_widths_obj = snapshot.get("col_widths") + expanded_obj = snapshot.get("expanded") + current_path_obj = snapshot.get("current_path") + h_scroll_obj = snapshot.get("h_scroll") + v_scroll_obj = snapshot.get("v_scroll") + + if isinstance(col_widths_obj, list): + widths = [int(width) for width in col_widths_obj if isinstance(width, int)] + if widths: + tab.view_controller.set_column_widths(widths) + + expanded_paths: list[tuple[int, ...]] = [] + if isinstance(expanded_obj, list): + for path in expanded_obj: + if not isinstance(path, tuple): + continue + normalized = tuple(step for step in path if isinstance(step, int) and step >= 0) + expanded_paths.append(normalized) + + if expanded_paths: + tab.view_controller.request_collapse_all() + if _is_large_load(tab): + _request_expand_paths_chunked(tab, expanded_paths) + else: + for path in expanded_paths: + tab.view_controller.request_expand(path) + + if isinstance(current_path_obj, tuple): + current_path = tuple(step for step in current_path_obj if isinstance(step, int) and step >= 0) + if current_path: + if _is_large_load(tab): + tab.view_controller.request_select_paths_deferred([current_path]) + tab.view_controller.request_scroll_to_deferred(current_path) + else: + tab.view_controller.request_select_paths([current_path]) + tab.view_controller.request_scroll_to(current_path) + + h_scroll = int(h_scroll_obj) if isinstance(h_scroll_obj, int) else 0 + v_scroll = int(v_scroll_obj) if isinstance(v_scroll_obj, int) else 0 + + def _restore_scrollbars() -> None: + tab.view.horizontalScrollBar().setValue(h_scroll) + tab.view.verticalScrollBar().setValue(v_scroll) + + QTimer.singleShot(0, _restore_scrollbars) diff --git a/tests/perf/__init__.py b/tests/perf/__init__.py new file mode 100644 index 0000000..d4e976c --- /dev/null +++ b/tests/perf/__init__.py @@ -0,0 +1 @@ +# Performance test harness for parsing vulnerability measurement. diff --git a/tests/perf/harness.py b/tests/perf/harness.py new file mode 100644 index 0000000..724e33c --- /dev/null +++ b/tests/perf/harness.py @@ -0,0 +1,330 @@ +"""Timing, scaling, and allocation measurement harness. + +Provides a reusable measurement API so each test reports timings, scaling ratios, +and allocation outcomes in the same format. + +Key functions: +- ``measure_call(callable, text)``: Measure a single call with timing and allocation. +- ``assert_within_budget(result)``: Assert a result is within the time budget. +- ``scaling_rows(callable, sizes, factory)``: Generate scaling rows for size doublings. +- ``classify_rows(rows)``: Classify rows by outcome (pass, budget_exceeded, etc.). +""" + +from __future__ import annotations + +import os +import statistics +import time +import tracemalloc +from dataclasses import dataclass, field +from typing import Any, Callable + +# --------------------------------------------------------------------------- +# Configuration constants (overridable via environment variables) +# --------------------------------------------------------------------------- + +# Default per-call wall-clock budget in milliseconds +DEFAULT_BUDGET_MS = 100 + +# Default scaling ratio threshold for size doublings +DEFAULT_SCALING_RATIO_MAX = 3.0 + +# Default allocation cap for non-decoding paths (8 MB or 4x text length) +DEFAULT_ALLOCATION_CAP_BASE = 8 * 1024 * 1024 +DEFAULT_ALLOCATION_CAP_MULTIPLIER = 4 + +# Default allocation cap for decode/decompress paths (16 MB or 2x text length) +DEFAULT_DECODE_ALLOCATION_CAP_BASE = 16 * 1024 * 1024 +DEFAULT_DECODE_ALLOCATION_CAP_MULTIPLIER = 2 + + +def get_budget_ms() -> float: + """Return the per-call budget in milliseconds, from env or default.""" + return float(os.environ.get("PARSING_BUDGET_MS", DEFAULT_BUDGET_MS)) + + +def get_scaling_ratio_max() -> float: + """Return the scaling ratio threshold, from env or default.""" + return float(os.environ.get("PARSING_SCALING_RATIO_MAX", DEFAULT_SCALING_RATIO_MAX)) + + +# --------------------------------------------------------------------------- +# Result data structures +# --------------------------------------------------------------------------- + + +@dataclass +class MeasurementResult: + """Result of a single measurement.""" + + function: str + wrapper_name: str + family: str + size: int + elapsed_median_ms: float + raw_ms: list[float] # All three raw timings + peak_allocated_bytes: int + outcome: str # pass, budget_exceeded, superlinear, allocation_exceeded, error + exception_text: str = "" + scaling_ratio: float | None = None # For scaling rows + + def to_dict(self) -> dict[str, Any]: + """Convert to a dictionary for report generation.""" + return { + "function": self.function, + "wrapper_name": self.wrapper_name, + "family": self.family, + "size": self.size, + "elapsed_median_ms": self.elapsed_median_ms, + "raw_ms": self.raw_ms, + "peak_allocated_bytes": self.peak_allocated_bytes, + "outcome": self.outcome, + "exception_text": self.exception_text, + "scaling_ratio": self.scaling_ratio, + } + + +# --------------------------------------------------------------------------- +# Measurement functions +# --------------------------------------------------------------------------- + + +def measure_call( + callable_fn: Callable[[str], Any], + text: str, + *, + function_name: str = "", + wrapper_name: str = "", + family: str = "", + warmup: int = 1, + repeats: int = 3, + is_decode_path: bool = False, +) -> MeasurementResult: + """Measure a callable with timing and allocation tracking. + + Performs one warmup call and records the median of three timed calls. + Uses ``tracemalloc`` to track peak memory allocation. + + Args: + callable_fn: The function to measure, accepting a single string argument. + text: The input text to pass to the callable. + function_name: Name of the function being measured. + wrapper_name: Name of the wrapper used. + family: Adversarial family label. + warmup: Number of warmup calls (default 1). + repeats: Number of timed calls (default 3). + is_decode_path: If True, use decode path allocation caps. + + Returns: + A MeasurementResult with timing and allocation data. + """ + func_name = function_name or getattr( # allow: perf harness needs function name from callable + callable_fn, + "__name__", + "unknown", + ) + wrap_name = wrapper_name or func_name + + # Warmup calls + for _ in range(warmup): + try: + callable_fn(text) + except Exception: + pass # Warmup failures are ignored + + # Timed calls with allocation tracking + raw_ms: list[float] = [] + peak_bytes = 0 + + tracemalloc.start() + try: + for _ in range(repeats): + # Reset peak tracking for each call + tracemalloc.reset_peak() + start = time.perf_counter() + try: + callable_fn(text) + except Exception as e: + elapsed = (time.perf_counter() - start) * 1000 + raw_ms.append(elapsed) + # Record the exception and return error outcome + current, peak = tracemalloc.get_traced_memory() + peak_bytes = max(peak_bytes, peak) + tracemalloc.stop() + return MeasurementResult( + function=func_name, + wrapper_name=wrap_name, + family=family, + size=len(text), + elapsed_median_ms=elapsed, + raw_ms=raw_ms, + peak_allocated_bytes=peak_bytes, + outcome="error", + exception_text=str(e), + ) + elapsed = (time.perf_counter() - start) * 1000 + raw_ms.append(elapsed) + current, peak = tracemalloc.get_traced_memory() + peak_bytes = max(peak_bytes, peak) + finally: + tracemalloc.stop() + + median_ms = statistics.median(raw_ms) + + # Determine allocation cap + text_len = len(text) + if is_decode_path: + alloc_cap = max(DEFAULT_DECODE_ALLOCATION_CAP_BASE, DEFAULT_DECODE_ALLOCATION_CAP_MULTIPLIER * text_len) + else: + alloc_cap = max(DEFAULT_ALLOCATION_CAP_BASE, DEFAULT_ALLOCATION_CAP_MULTIPLIER * text_len) + + # Classify outcome + budget_ms = get_budget_ms() + if median_ms > budget_ms: + outcome = "budget_exceeded" + elif peak_bytes > alloc_cap: + outcome = "allocation_exceeded" + else: + outcome = "pass" + + return MeasurementResult( + function=func_name, + wrapper_name=wrap_name, + family=family, + size=text_len, + elapsed_median_ms=median_ms, + raw_ms=raw_ms, + peak_allocated_bytes=peak_bytes, + outcome=outcome, + ) + + +def assert_within_budget(result: MeasurementResult) -> None: + """Assert that a measurement result is within the time budget. + + Raises AssertionError if the outcome is not 'pass'. + """ + if result.outcome != "pass": + msg = f"{result.function} ({result.wrapper_name}) failed: {result.outcome}" + if result.exception_text: + msg += f" - {result.exception_text}" + msg += f" (median={result.elapsed_median_ms:.2f}ms, peak={result.peak_allocated_bytes} bytes)" + raise AssertionError(msg) + + +def scaling_rows( + callable_fn: Callable[[str], Any], + sizes: tuple[int, ...], + factory: Callable[[int], tuple[str, str]], + *, + function_name: str = "", + wrapper_name: str = "", + is_decode_path: bool = False, +) -> list[MeasurementResult]: + """Generate measurement rows for a sequence of sizes. + + Args: + callable_fn: The function to measure. + sizes: Tuple of sizes to test (should be doubling sequence). + factory: Function that generates (family, text) for a given size. + function_name: Name of the function being measured. + wrapper_name: Name of the wrapper used. + is_decode_path: If True, use decode path allocation caps. + + Returns: + List of MeasurementResult, one per size. + """ + rows: list[MeasurementResult] = [] + family_label, _ = factory(sizes[0]) if sizes else ("unknown", "") + + for size in sizes: + _, text = factory(size) + result = measure_call( + callable_fn, + text, + function_name=function_name, + wrapper_name=wrapper_name, + family=family_label, + is_decode_path=is_decode_path, + ) + rows.append(result) + + return rows + + +def classify_rows(rows: list[MeasurementResult]) -> list[MeasurementResult]: + """Classify rows by scaling ratio for size doublings. + + For each pair of consecutive rows where the size doubles, computes the + median time ratio. If the ratio exceeds the threshold, marks the row + as 'superlinear' (unless it already has a worse outcome). + + Args: + rows: List of MeasurementResult from scaling_rows. + + Returns: + The same list with scaling_ratio and potentially updated outcome. + """ + if len(rows) < 2: + return rows + + ratio_max = get_scaling_ratio_max() + + for i in range(1, len(rows)): + prev = rows[i - 1] + curr = rows[i] + + # Only compute ratio for size doublings + if curr.size > prev.size and prev.elapsed_median_ms > 0: + ratio = curr.elapsed_median_ms / prev.elapsed_median_ms + curr.scaling_ratio = ratio + + # Update outcome if superlinear and not already worse + if ratio > ratio_max and curr.outcome == "pass": + curr.outcome = "superlinear" + + return rows + + +# --------------------------------------------------------------------------- +# Utility functions for tests +# --------------------------------------------------------------------------- + + +def make_linear_callable() -> Callable[[str], int]: + """Return a callable with linear time complexity for testing.""" + + def linear_fn(text: str) -> int: + # Simple linear operation: sum of character codes + return sum(ord(c) for c in text) + + return linear_fn + + +def make_quadratic_callable() -> Callable[[str], int]: + """Return a callable with quadratic time complexity for testing.""" + + def quadratic_fn(text: str) -> int: + # Deliberately quadratic: nested loops over text + result = 0 + for i in range(len(text)): + for j in range(min(100, len(text))): # Cap inner loop for very large texts + result += ord(text[i % len(text)]) + return result + + return quadratic_fn + + +def make_allocating_callable(size_multiplier: int = 10) -> Callable[[str], bytes]: + """Return a callable that allocates a large buffer for testing. + + Args: + size_multiplier: Multiplier for the allocation size relative to input. + """ + + def allocating_fn(text: str) -> bytes: + # Allocate a buffer proportional to input size + alloc_size = len(text) * size_multiplier + return b"x" * alloc_size + + return allocating_fn diff --git a/tests/perf/registry.py b/tests/perf/registry.py new file mode 100644 index 0000000..95f9bb4 --- /dev/null +++ b/tests/perf/registry.py @@ -0,0 +1,273 @@ +"""Hotspot registry for parsing vulnerability measurement. + +Each entry maps a review hotspot to a single-string wrapper that can be called +with one adversarial string. The registry prevents each test from inventing a +different call shape. + +Entry fields: +- name: Human-readable name for the entry. +- component: The module/function being tested. +- call: A callable that accepts a single string argument. +- notes: Additional context about the entry. +- is_decode_path: Whether this is a decode/decompress path (affects allocation caps). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, Callable + + +@dataclass(frozen=True) +class RegistryEntry: + """A single hotspot registry entry.""" + + name: str + component: str + call: Callable[[str], Any] + notes: str + is_decode_path: bool = False + + +# --------------------------------------------------------------------------- +# Import target functions +# --------------------------------------------------------------------------- + +# Datetime parsing +from core.datetime_parsing import parse_datetime_text +from core.datetime_parsing.regex import DATETIME_RE + +# Formatting and display +from delegates.formatting.value_formatting import format_with_type +from tree.codecs.bytes_codec import decode_bytes + +# Item coercion (decode/decompress path) +from tree.item_coercion import compute_editable + +# Central type inference +from tree.types import ( + JsonType, + _looks_like_base64, + infer_text_json_type, + looks_like_color_rgb, + looks_like_color_rgba, + parse_json_type, +) + +# Number affix parsing +from units.number_affix import _CURRENCY_RE, _UNITS_RE, parse_number_affix + +# --------------------------------------------------------------------------- +# Wrapper functions +# --------------------------------------------------------------------------- + + +def _wrap_parse_json_type(text: str) -> JsonType: + """Wrapper for parse_json_type with a string value.""" + return parse_json_type(text) + + +def _wrap_parse_datetime_text(text: str) -> Any: + """Wrapper for parse_datetime_text.""" + return parse_datetime_text(text) + + +def _wrap_datetime_re_fullmatch(text: str) -> Any: + """Wrapper for DATETIME_RE.fullmatch.""" + return DATETIME_RE.fullmatch(text) + + +def _wrap_parse_number_affix(text: str) -> Any: + """Wrapper for parse_number_affix.""" + return parse_number_affix(text) + + +def _wrap_currency_re_fullmatch(text: str) -> Any: + """Wrapper for _CURRENCY_RE.fullmatch.""" + return _CURRENCY_RE.fullmatch(text) + + +def _wrap_units_re_fullmatch(text: str) -> Any: + """Wrapper for _UNITS_RE.fullmatch.""" + return _UNITS_RE.fullmatch(text) + + +def _wrap_looks_like_base64(text: str) -> bool: + """Wrapper for _looks_like_base64.""" + return _looks_like_base64(text) + + +def _wrap_looks_like_color_rgb(text: str) -> bool: + """Wrapper for looks_like_color_rgb.""" + return looks_like_color_rgb(text) + + +def _wrap_looks_like_color_rgba(text: str) -> bool: + """Wrapper for looks_like_color_rgba.""" + return looks_like_color_rgba(text) + + +def _wrap_infer_text_json_type(text: str) -> JsonType: + """Wrapper for infer_text_json_type.""" + return infer_text_json_type(text) + + +def _wrap_compute_editable_bytes(text: str) -> bool: + """Wrapper for compute_editable with BYTES type.""" + return compute_editable(JsonType.BYTES, text, editable_blob_limit=1024 * 1024) + + +def _wrap_compute_editable_zlib(text: str) -> bool: + """Wrapper for compute_editable with ZLIB type.""" + return compute_editable(JsonType.ZLIB, text, editable_blob_limit=1024 * 1024) + + +def _wrap_compute_editable_gzip(text: str) -> bool: + """Wrapper for compute_editable with GZIP type.""" + return compute_editable(JsonType.GZIP, text, editable_blob_limit=1024 * 1024) + + +def _wrap_format_with_type_string(text: str) -> str: + """Wrapper for format_with_type with STRING type.""" + return format_with_type(text, JsonType.STRING) + + +def _wrap_format_with_type_bytes(text: str) -> str: + """Wrapper for format_with_type with BYTES type.""" + return format_with_type(text, JsonType.BYTES) + + +def _wrap_decode_bytes(text: str) -> bytes: + """Wrapper for decode_bytes with BYTES type.""" + return decode_bytes(text, JsonType.BYTES) + + +# --------------------------------------------------------------------------- +# Registry definition +# --------------------------------------------------------------------------- + +HOTSPOT_REGISTRY: list[RegistryEntry] = [ + # Central type inference + RegistryEntry( + name="parse_json_type", + component="tree.types.parse_json_type", + call=_wrap_parse_json_type, + notes="Central automatic inference dispatcher for string values.", + ), + # Datetime parsing + RegistryEntry( + name="parse_datetime_text", + component="core.datetime_parsing.parse_datetime_text", + call=_wrap_parse_datetime_text, + notes="Datetime regex and conversion path.", + ), + RegistryEntry( + name="DATETIME_RE.fullmatch", + component="core.datetime_parsing.regex.DATETIME_RE", + call=_wrap_datetime_re_fullmatch, + notes="Direct regex fullmatch for datetime pattern.", + ), + # Number affix parsing + RegistryEntry( + name="parse_number_affix", + component="units.number_affix.parse_number_affix", + call=_wrap_parse_number_affix, + notes="Number-affix detection with currency and units patterns.", + ), + RegistryEntry( + name="_CURRENCY_RE.fullmatch", + component="units.number_affix._CURRENCY_RE", + call=_wrap_currency_re_fullmatch, + notes="Direct regex fullmatch for currency prefix pattern.", + ), + RegistryEntry( + name="_UNITS_RE.fullmatch", + component="units.number_affix._UNITS_RE", + call=_wrap_units_re_fullmatch, + notes="Direct regex fullmatch for units suffix pattern.", + ), + # Base64 detection + RegistryEntry( + name="_looks_like_base64", + component="tree.types._looks_like_base64", + call=_wrap_looks_like_base64, + notes="Base64 syntactic check and decode probe.", + is_decode_path=True, + ), + # Color detection + RegistryEntry( + name="looks_like_color_rgb", + component="tree.types.looks_like_color_rgb", + call=_wrap_looks_like_color_rgb, + notes="Color regex check for RGB format (#RGB or #RRGGBB).", + ), + RegistryEntry( + name="looks_like_color_rgba", + component="tree.types.looks_like_color_rgba", + call=_wrap_looks_like_color_rgba, + notes="Color regex check for RGBA format (#RGBA or #RRGGBBAA).", + ), + # Text inference + RegistryEntry( + name="infer_text_json_type", + component="tree.types.infer_text_json_type", + call=_wrap_infer_text_json_type, + notes="Text fallback checks including multiline, whitespace, and unicode detection.", + ), + # Decode/decompress paths + RegistryEntry( + name="compute_editable(BYTES)", + component="tree.item_coercion.compute_editable", + call=_wrap_compute_editable_bytes, + notes="Decode path for BYTES type during item coercion.", + is_decode_path=True, + ), + RegistryEntry( + name="compute_editable(ZLIB)", + component="tree.item_coercion.compute_editable", + call=_wrap_compute_editable_zlib, + notes="Decompress path for ZLIB type during item coercion.", + is_decode_path=True, + ), + RegistryEntry( + name="compute_editable(GZIP)", + component="tree.item_coercion.compute_editable", + call=_wrap_compute_editable_gzip, + notes="Decompress path for GZIP type during item coercion.", + is_decode_path=True, + ), + # Formatting and display + RegistryEntry( + name="format_with_type(STRING)", + component="delegates.formatting.value_formatting.format_with_type", + call=_wrap_format_with_type_string, + notes="Paint-time display preview for STRING type.", + ), + RegistryEntry( + name="format_with_type(BYTES)", + component="delegates.formatting.value_formatting.format_with_type", + call=_wrap_format_with_type_bytes, + notes="Paint-time display preview for BYTES type (includes decode).", + is_decode_path=True, + ), + RegistryEntry( + name="decode_bytes", + component="tree.codecs.bytes_codec.decode_bytes", + call=_wrap_decode_bytes, + notes="Direct decode for BYTES type.", + is_decode_path=True, + ), +] + + +def get_registry_entry(name: str) -> RegistryEntry | None: + """Get a registry entry by name, or None if not found.""" + for entry in HOTSPOT_REGISTRY: + if entry.name == name: + return entry + return None + + +def get_registry_names() -> list[str]: + """Get all registry entry names.""" + return [entry.name for entry in HOTSPOT_REGISTRY] diff --git a/tests/perf/report.py b/tests/perf/report.py new file mode 100644 index 0000000..17d71c8 --- /dev/null +++ b/tests/perf/report.py @@ -0,0 +1,204 @@ +"""Report writer for parsing vulnerability measurement. + +Generates a dated report with the schema defined in the plan plus a ranked +vulnerability summary. +""" + +from __future__ import annotations + +import os +from datetime import date +from pathlib import Path +from typing import Any + +from tests.perf.harness import MeasurementResult + + +def generate_report( + rows: list[MeasurementResult], + output_path: str | Path | None = None, + report_date: date | None = None, +) -> str: + """Generate a parsing vulnerability report from measurement rows. + + Args: + rows: List of MeasurementResult from the perf tests. + output_path: Path to write the report. If None, uses default naming. + report_date: Date for the report. If None, uses today's date. + + Returns: + The path to the generated report. + """ + if report_date is None: + report_date = date.today() + + if output_path is None: + output_path = Path("reports") / f"parsing-vulnerability-{report_date.isoformat()}.md" + else: + output_path = Path(output_path) + + # Ensure the reports directory exists + output_path.parent.mkdir(parents=True, exist_ok=True) + + # Generate the report content + content = _format_report(rows, report_date) + + # Write the report + output_path.write_text(content) + + return str(output_path) + + +def _format_report(rows: list[MeasurementResult], report_date: date) -> str: + """Format the report content.""" + lines: list[str] = [] + + # Header + lines.append(f"# Parsing Vulnerability Report — {report_date.isoformat()}") + lines.append("") + lines.append("This report was generated by the opt-in performance harness.") + lines.append("") + lines.append("## Configuration") + lines.append("") + lines.append(f"- **Budget:** {os.environ.get('PARSING_BUDGET_MS', '100')} ms") + lines.append(f"- **Scaling ratio max:** {os.environ.get('PARSING_SCALING_RATIO_MAX', '3.0')}") + lines.append("") + + # Summary statistics + total_rows = len(rows) + pass_count = sum(1 for r in rows if r.outcome == "pass") + budget_exceeded = sum(1 for r in rows if r.outcome == "budget_exceeded") + superlinear = sum(1 for r in rows if r.outcome == "superlinear") + allocation_exceeded = sum(1 for r in rows if r.outcome == "allocation_exceeded") + errors = sum(1 for r in rows if r.outcome == "error") + + lines.append("## Summary") + lines.append("") + lines.append(f"- **Total rows:** {total_rows}") + lines.append(f"- **Pass:** {pass_count}") + lines.append(f"- **Budget exceeded:** {budget_exceeded}") + lines.append(f"- **Superlinear:** {superlinear}") + lines.append(f"- **Allocation exceeded:** {allocation_exceeded}") + lines.append(f"- **Errors:** {errors}") + lines.append("") + + # Ranked vulnerability summary + vulnerabilities = [r for r in rows if r.outcome != "pass"] + if vulnerabilities: + lines.append("## Ranked Vulnerabilities") + lines.append("") + lines.append("Vulnerabilities are ranked by outcome severity, largest scaling ratio, and peak allocation.") + lines.append("") + lines.append("| Rank | Function | Family | Size | Outcome | Median (ms) | Ratio | Peak (bytes) |") + lines.append("|---|---|---|---|---|---|---|---|") + + # Sort by severity (error > allocation_exceeded > superlinear > budget_exceeded) + severity_order = {"error": 0, "allocation_exceeded": 1, "superlinear": 2, "budget_exceeded": 3} + vulnerabilities.sort( + key=lambda r: ( + severity_order.get(r.outcome, 4), + -(r.scaling_ratio or 0), + -r.peak_allocated_bytes, + ) + ) + + for i, row in enumerate(vulnerabilities, 1): + ratio_str = f"{row.scaling_ratio:.2f}" if row.scaling_ratio is not None else "-" + lines.append( + f"| {i} | {row.function} | {row.family} | {row.size} | {row.outcome} | " + f"{row.elapsed_median_ms:.2f} | {ratio_str} | {row.peak_allocated_bytes} |" + ) + lines.append("") + else: + lines.append("## Vulnerabilities") + lines.append("") + lines.append("No vulnerabilities detected.") + lines.append("") + + # Detailed rows + lines.append("## Detailed Results") + lines.append("") + lines.append( + "| Function | Wrapper | Family | Size | Median (ms) | Raw (ms) | Peak (bytes) | Outcome | Ratio | Exception |" + ) + lines.append("|---|---|---|---|---|---|---|---|---|---|") + + for row in rows: + raw_str = ", ".join(f"{t:.2f}" for t in row.raw_ms) + ratio_str = f"{row.scaling_ratio:.2f}" if row.scaling_ratio is not None else "-" + exception_str = row.exception_text[:50] + "..." if len(row.exception_text) > 50 else row.exception_text + lines.append( + f"| {row.function} | {row.wrapper_name} | {row.family} | {row.size} | " + f"{row.elapsed_median_ms:.2f} | [{raw_str}] | {row.peak_allocated_bytes} | " + f"{row.outcome} | {ratio_str} | {exception_str} |" + ) + lines.append("") + + # Threshold recommendations for Plan 1 + lines.append("## Threshold Recommendations for Plan 1") + lines.append("") + lines.append("These values are derived from the measurement results above.") + lines.append("") + + # Calculate recommended thresholds based on the data + # For each function, find the maximum size that passes + function_max_pass_size: dict[str, int] = {} + for row in rows: + if row.outcome == "pass": + current_max = function_max_pass_size.get(row.function, 0) + if row.size > current_max: + function_max_pass_size[row.function] = row.size + + lines.append("| Function | Max Passing Size | Recommended Limit |") + lines.append("|---|---|---|") + for func, max_size in sorted(function_max_pass_size.items()): + # Recommend a limit that is the max passing size or a reasonable default + recommended = max_size if max_size > 0 else 1024 + lines.append(f"| {func} | {max_size} | {recommended} |") + lines.append("") + + # Footer + lines.append("---") + lines.append("") + lines.append("*Report generated by `tests/perf/report.py`*") + lines.append("") + lines.append("**Note:** Strict performance failures are opt-in and are not part of `make test`") + lines.append("until a future plan changes that policy.") + + return "\n".join(lines) + + +def load_rows_from_modules() -> list[MeasurementResult]: + """Load all collected rows from the perf test modules.""" + rows: list[MeasurementResult] = [] + + # Import and collect from each module + try: + from tests.perf.test_parsing_scaling import get_collected_rows + + rows.extend(get_collected_rows()) + except ImportError: + pass + + try: + from tests.perf.test_regex_backtracking import get_collected_rows + + rows.extend(get_collected_rows()) + except ImportError: + pass + + try: + from tests.perf.test_decode_amplification import get_collected_rows + + rows.extend(get_collected_rows()) + except ImportError: + pass + + try: + from tests.perf.test_container_paths import get_collected_rows + + rows.extend(get_collected_rows()) + except ImportError: + pass + + return rows diff --git a/tests/perf/string_corpus.py b/tests/perf/string_corpus.py new file mode 100644 index 0000000..a67b00f --- /dev/null +++ b/tests/perf/string_corpus.py @@ -0,0 +1,549 @@ +"""Adversarial string generators for parsing vulnerability measurement. + +Each generator accepts ``size: int`` and returns ``(family_label: str, text: str)``. +The module imports only Python standard-library modules and ``settings`` for Plan 1 +limit constants when available. +""" + +from __future__ import annotations + +import itertools +from typing import Callable + +# Try to import Plan 1 limit constants; fall back to defaults if not yet defined. +try: + from settings import NUMBER_AFFIX_MAX_LEN as _AFFIX_LIMIT +except ImportError: + _AFFIX_LIMIT = 20 + + +TRACE_EXAMPLE = """ +AttributeErro Traceback (most recent call last) +Cell In[2], line 1 +----> 1 json.load("models_output.xlsx") + +File /usr/lib/python3.12/json/__init__.py:293, in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) + 274 def load(fp, *, cls=None, object_hook=None, parse_float=None, + 275 parse_int=None, parse_constant=None, object_pairs_hook=None, **kw): + 276 \"""Deserialize ``fp`` (a ``.read()``-supporting file-like object containing +277 a JSON document) to a Python object. +278 +(...) +291 kwarg; otherwise ``JSONDecoder`` is used. +292 \""" +--> 293 return loads(fp.read(), + 294 cls=cls, object_hook=object_hook, + 295 parse_float=parse_float, parse_int=parse_int, + 296 parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) + +AttributeErro: 'str' object has no attribute 'read' + +""" + +SOURCE_CODE_EXAMPLE = """ +import sys +import pygame + +# Initialize core modules +pygame.init() + +# Game display settings +SCREEN_WIDTH = 800 +SCREEN_HEIGHT = 600 +screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT)) +pygame.display.set_caption("Pygame Movement Example") + +# Timing control +clock = pygame.time.Clock() +FPS = 60 + +# Character definitions +player_color = (0, 128, 255) # Blue +player_width = 50 +player_height = 50 +player_x = (SCREEN_WIDTH - player_width) // 2 +player_y = (SCREEN_HEIGHT - player_height) // 2 +player_speed = 5 + +# Main game loop +running = True +while running: + # 1. Event Handling + for event in pygame.event.get(): + if event.type == pygame.QUIT: + running = False + + # 2. Input Handling & Screen Boundaries + keys = pygame.key.get_pressed() + if keys[pygame.K_LEFT] and player_x > 0: + player_x -= player_speed + if keys[pygame.K_RIGHT] and player_x < SCREEN_WIDTH - player_width: + player_x += player_speed + if keys[pygame.K_UP] and player_y > 0: + player_y -= player_speed + if keys[pygame.K_DOWN] and player_y < SCREEN_HEIGHT - player_height: + player_y += player_speed + + # 3. Graphics Rendering + screen.fill((30, 30, 30)) # Dark gray background + + # Draw the player square + pygame.draw.rect( + screen, + player_color, + (player_x, player_y, player_width, player_height), + ) + + # Refresh screen display + pygame.display.flip() + + # Maintain constant frame rate + clock.tick(FPS) + +# Clean exit +pygame.quit() +sys.exit() + +""" + +JSON_TO_BE_ESCAPED = """ +{ + "status": "success", + "code": 200, + "message": "Data retrieved successfully", + "data": { + "items": [ + { + "id": 1, + "title": "Wireless Bluetooth Headphones", + "description": "Premium quality wireless headphones with active noise cancellation and 30-hour battery life", + "price": 199.99, + "originalPrice": 249.99, + "discount": 20, + "category": "electronics", + "brand": "AudioTech", + "inStock": true, + "quantity": 45, + "tags": [ + "popular", + "new", + "featured", + "wireless", + "noise-cancelling" + ], + "ratings": { + "average": 4.5, + "count": 128, + "distribution": { + "5": 78, + "4": 32, + "3": 12, + "2": 4, + "1": 2 + } + }, + "images": [ + "https://example.com/products/headphones-1.jpg", + "https://example.com/products/headphones-2.jpg" + ], + "specifications": { + "color": [ + "Black", + "White", + "Blue" + ], + "weight": "250g", + "batteryLife": "30 hours", + "connectivity": [ + "Bluetooth 5", + "3.5mm jack" + ] + } + }, + { + "id": 2, + "title": "Smart Fitness Watch", + "description": "Advanced fitness tracking with heart rate monitor, GPS, and smartphone integration", + "price": 299.99, + "originalPrice": 349.99, + "discount": 15, + "category": "wearables", + "brand": "FitTech", + "inStock": false, + "quantity": 0, + "tags": [ + "bestseller", + "fitness", + "smartwatch", + "gps" + ], + "ratings": { + "average": 4.2, + "count": 89, + "distribution": { + "5": 45, + "4": 28, + "3": 12, + "2": 3, + "1": 1 + } + }, + "images": [ + "https://example.com/products/watch-1.jpg", + "https://example.com/products/watch-2.jpg" + ], + "specifications": { + "color": [ + "Black", + "Silver", + "Rose Gold" + ], + "weight": "45g", + "batteryLife": "7 days", + "waterResistance": "50m" + } + }, + { + "id": 3, + "title": "4K Webcam Pro", + "description": "Professional 4K webcam with auto-focus, built-in microphone, and studio-quality video", + "price": 149.99, + "originalPrice": 199.99, + "discount": 25, + "category": "accessories", + "brand": "StreamTech", + "inStock": true, + "quantity": 23, + "tags": [ + "professional", + "4k", + "streaming", + "webcam" + ], + "ratings": { + "average": 4.8, + "count": 256, + "distribution": { + "5": 205, + "4": 38, + "3": 9, + "2": 3, + "1": 1 + } + }, + "images": [ + "https://example.com/products/webcam-1.jpg", + "https://example.com/products/webcam-2.jpg" + ], + "specifications": { + "resolution": "4K 30fps", + "fieldOfView": "90 degrees", + "microphone": "Built-in stereo", + "compatibility": [ + "Windows", + "Mac", + "Linux" + ] + } + } + ], + "pagination": { + "currentPage": 1, + "totalPages": 10, + "totalItems": 30, + "itemsPerPage": 3, + "hasNext": true, + "hasPrevious": false, + "nextPage": 2 + }, + "filters": { + "category": "all", + "priceRange": { + "min": 0, + "max": 1000 + }, + "inStock": "all", + "sortBy": "popularity", + "sortOrder": "desc", + "brands": [ + "AudioTech", + "FitTech", + "StreamTech" + ] + } + }, + "meta": { + "requestId": "req_abc123def456", + "timestamp": "2024-01-20T15:30:00Z", + "executionTime": 42, + "version": "v2.1", + "server": "api-prod-1", + "cache": false + } +} +""" + +# --------------------------------------------------------------------------- +# Family generators +# --------------------------------------------------------------------------- + + +def plain_ascii(size: int) -> tuple[str, str]: + """Return ``size`` ASCII 'a' characters.""" + return ("plain_ascii", "a" * size) + + +def whitespace(size: int) -> tuple[str, str]: + """Return ``size`` whitespace characters including at least one newline.""" + if size < 3: + # Ensure we have space for at least one newline + text = " \n"[:size] if size > 0 else "" + else: + # Mix spaces, tabs, and newlines; ensure at least one newline + cycle = itertools.cycle([" ", " ", "\t", " ", "\n", " "]) + text = "".join(itertools.islice(cycle, size)) + return ("whitespace", text) + + +def digits(size: int) -> tuple[str, str]: + """Return ``size`` digit '9' characters.""" + return ("digits", "9" * size) + + +def base64_like(size: int) -> tuple[str, str]: + """Return ``size`` base64-like characters, padded to a multiple of 4.""" + # Use 'A' which is valid base64 (decodes to 0x00 bytes) + text = "A" * size + # Pad to multiple of 4 + remainder = len(text) % 4 + if remainder: + text += "=" * (4 - remainder) + return ("base64_like", text) + + +def near_datetime(size: int) -> tuple[str, str]: + """Return a string that starts with a date-like prefix but does not parse as datetime. + + Shape: date-like prefix + long digit run + invalid suffix. + """ + prefix = "2026-06-13" + if size <= len(prefix): + # Just return the prefix truncated/padded + text = prefix[:size] if size > 0 else "" + else: + # Add a long digit run and invalid suffix to prevent datetime parsing + remaining = size - len(prefix) + if remaining > 10: + digit_run = "9" * (remaining - 3) + suffix = "XYZ" # Invalid suffix that prevents datetime parsing + text = prefix + digit_run + suffix + else: + # For small sizes, just add digits and an invalid char + text = prefix + "9" * (remaining - 1) + "X" + return ("near_datetime", text) + + +def near_affix(size: int) -> tuple[str, str]: + """Return a string with a currency/unit prefix plus a digit run exceeding the affix limit. + + The digit run is longer than ``NUMBER_AFFIX_MAX_LEN`` to stress the affix parser. + """ + # Use a currency prefix that would be valid if the number were shorter + prefix = "$" + # Create a digit run that exceeds the affix limit + digit_len = max(size - len(prefix), _AFFIX_LIMIT + 10) + if size > len(prefix) + digit_len: + # Pad with spaces if needed + text = prefix + "9" * digit_len + " " * (size - len(prefix) - digit_len) + else: + text = prefix + "9" * digit_len + # Truncate or pad to exact size + if len(text) > size: + text = text[:size] + elif len(text) < size: + text = text + " " * (size - len(text)) + return ("near_affix", text) + + +def near_color(size: int) -> tuple[str, str]: + """Return a string that starts with '#' followed by hex characters. + + For stress sizes (size > 9), this exceeds valid color length. + """ + if size < 1: + return ("near_color", "") + if size == 1: + return ("near_color", "#") + # '#' + 'f' * (size - 1) + text = "#" + "f" * (size - 1) + return ("near_color", text) + + +def unicode_bulk(size: int) -> tuple[str, str]: + """Return ``size`` repetitions of a non-ASCII code point.""" + # Use 'é' (U+00E9) which has ord > 127 + return ("unicode_bulk", "é" * size) + + +def pathological_repetition(size: int) -> tuple[str, str]: + """Return repeated regex-sensitive motifs such as 'ab' and '#fff'. + + The generated length is within one motif of ``size``. + """ + if size < 1: + return ("pathological_repetition", "") + # Alternate between regex-sensitive motifs + motifs = ["ab", "#fff", "9.9", "2026-"] + motif_cycle = itertools.cycle(motifs) + parts = [] + current_len = 0 + while current_len < size: + motif = next(motif_cycle) + if current_len + len(motif) <= size + len(motifs[-1]): # Allow within one motif + parts.append(motif) + current_len += len(motif) + else: + break + text = "".join(parts) + # Truncate to exact size if we overshot + if len(text) > size: + text = text[:size] + return ("pathological_repetition", text) + + +def mixed_interleaved(size: int) -> tuple[str, str]: + """Return newline-separated chunks from at least three other families. + + The chunks are interleaved with newlines to stress multi-line parsing paths. + """ + if size < 10: + # For very small sizes, just combine a few short chunks + chunks = [ + plain_ascii(max(1, size // 4))[1], + digits(max(1, size // 4))[1], + unicode_bulk(max(1, size // 4))[1], + ] + text = "\n".join(chunks)[:size] + return ("mixed_interleaved", text) + + # Distribute size across at least 3 families plus newlines + num_families = min(5, max(3, size // 100 + 3)) # 3-5 families based on size + chunk_size = (size - num_families + 1) // num_families # Account for newlines + + families_to_use = [ + plain_ascii, + digits, + unicode_bulk, + base64_like, + near_color, + ][:num_families] + + chunks = [] + remaining = size + for i, factory in enumerate(families_to_use): + if i == len(families_to_use) - 1: + # Last chunk gets remaining size minus newlines + this_size = remaining - (len(families_to_use) - 1 - i) + else: + this_size = chunk_size + this_size = max(1, min(this_size, remaining - (len(families_to_use) - 1 - i))) + _, chunk_text = factory(this_size) + chunks.append(chunk_text) + remaining -= len(chunk_text) + 1 # +1 for newline + + text = "\n".join(chunks) + # Ensure exact size + if len(text) > size: + text = text[:size] + elif len(text) < size: + text = text + " " * (size - len(text)) + return ("mixed_interleaved", text) + + +def trace_repetition(size: int) -> tuple[str, str]: + """Return TRACE_EXAMPLE repeated to reach desired size. + + Extends to array of same format messages by repeating the trace example. + This tests performance on realistic multi-line error trace content. + """ + if size < 1: + return ("trace_repetition", "") + # Repeat the trace example to reach the desired size + trace_len = len(TRACE_EXAMPLE) + if trace_len == 0: + return ("trace_repetition", "") + repeats = (size // trace_len) + 1 + text = (TRACE_EXAMPLE * repeats)[:size] + return ("trace_repetition", text) + + +def source_code_repetition(size: int) -> tuple[str, str]: + """Return SOURCE_CODE_EXAMPLE repeated to reach desired size. + + Uses concatenation to produce long source code-like content. + This tests performance on realistic multi-line code content. + """ + if size < 1: + return ("source_code_repetition", "") + # Repeat the source code example to reach the desired size + source_len = len(SOURCE_CODE_EXAMPLE) + if source_len == 0: + return ("source_code_repetition", "") + repeats = (size // source_len) + 1 + text = (SOURCE_CODE_EXAMPLE * repeats)[:size] + return ("source_code_repetition", text) + + +def escape_heavy(size: int) -> tuple[str, str]: + """Return JSON_TO_BE_ESCAPED with heavy escaping (double/triple backslashes). + + Builds escape-heavy strings by applying multiple levels of escaping to the + JSON content. This tests performance on strings with many backslash sequences. + """ + if size < 1: + return ("escape_heavy", "") + # Start with the JSON content and apply escaping + base = JSON_TO_BE_ESCAPED.strip() + if not base: + return ("escape_heavy", "") + + # Apply double escaping: replace backslashes and quotes with escaped versions + # This creates strings like \\" and \\\\ which stress escape handling + escaped_once = base.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + # Apply triple escaping for even more backslash density + escaped_twice = escaped_once.replace("\\", "\\\\").replace('"', '\\"') + + # Concatenate to reach desired size + text_len = len(escaped_twice) + if text_len == 0: + return ("escape_heavy", "") + repeats = (size // text_len) + 1 + text = (escaped_twice * repeats)[:size] + return ("escape_heavy", text) + + +# --------------------------------------------------------------------------- +# Registry of all families +# --------------------------------------------------------------------------- + +FAMILY_REGISTRY: dict[str, Callable[[int], tuple[str, str]]] = { + "plain_ascii": plain_ascii, + "whitespace": whitespace, + "digits": digits, + "base64_like": base64_like, + "near_datetime": near_datetime, + "near_affix": near_affix, + "near_color": near_color, + "unicode_bulk": unicode_bulk, + "pathological_repetition": pathological_repetition, + "mixed_interleaved": mixed_interleaved, + "trace_repetition": trace_repetition, + "source_code_repetition": source_code_repetition, + "escape_heavy": escape_heavy, +} + +# Default sizes for normal local runs +DEFAULT_SIZES: tuple[int, ...] = (1024, 4096, 16384, 65536) + +# Extended sizes for milestone reports +EXTENDED_SIZES: tuple[int, ...] = (262144, 1048576, 10485760) diff --git a/tests/perf/test_close_phase_timing.py b/tests/perf/test_close_phase_timing.py new file mode 100644 index 0000000..f9dc8dd --- /dev/null +++ b/tests/perf/test_close_phase_timing.py @@ -0,0 +1,172 @@ +from __future__ import annotations + +import time +from datetime import date +from pathlib import Path + +import pytest +from PySide6.QtCore import QEvent +from PySide6.QtWidgets import QApplication + +import app.tab_lifecycle as tab_lifecycle_module +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + QApplication.processEvents() + + +def _format_close_phase_report( + rows: list[tuple[str, float]], dominant: str, path_choice: str, report_date: date +) -> str: + lines: list[str] = [] + lines.append(f"# Close Phase Timing Report — {report_date.isoformat()}") + lines.append("") + lines.append("## Phase timings") + lines.append("") + lines.append("| Phase | Elapsed (ms) |") + lines.append("|---|---:|") + for name, elapsed_ms in rows: + lines.append(f"| {name} | {elapsed_ms:.3f} |") + lines.append("") + lines.append("## Summary") + lines.append("") + lines.append(f"- Dominant phase: **{dominant}**") + lines.append(f"- Chosen implementation path: **{path_choice}**") + lines.append("") + return "\n".join(lines) + + +@pytest.mark.perf +def test_close_phase_timing_report_smoke(qtbot, tmp_path, monkeypatch): + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = { + "items": [ + { + "id": i, + "name": f"item-{i}", + "meta": {"a": i, "b": i * 2, "c": [i, i + 1, i + 2]}, + } + for i in range(2500) + ] + } + tab = win._add_tab(data=payload, file_path=str(tmp_path / "close-phase.json")) + assert isinstance(tab, JsonTab) + initial_tab_count = win.tabWidget.count() + index = win.tabWidget.indexOf(tab) + assert index >= 0 + + timings_ms = { + "snapshot_root_data": 0.0, + "schema_unregister": 0.0, + "view_state_save": 0.0, + "remove_tab": 0.0, + "delete_later": 0.0, + "forced_deferred_delete": 0.0, + } + + # Phase 1: snapshot + original_root_data = tab.root_data + + def timed_root_data(): + started = time.perf_counter() + try: + return original_root_data() + finally: + timings_ms["snapshot_root_data"] += (time.perf_counter() - started) * 1000.0 + + monkeypatch.setattr(tab, "root_data", timed_root_data) + + # Phase 3: schema unregister + original_unregister = win._schema_tab_pool.unregister + + def timed_unregister(widget): + started = time.perf_counter() + try: + return original_unregister(widget) + finally: + timings_ms["schema_unregister"] += (time.perf_counter() - started) * 1000.0 + + monkeypatch.setattr(win._schema_tab_pool, "unregister", timed_unregister) + + # Phase 4: save view state + original_view_state_save = tab_lifecycle_module.view_state.save + + def timed_view_state_save(widget): + started = time.perf_counter() + try: + return original_view_state_save(widget) + finally: + timings_ms["view_state_save"] += (time.perf_counter() - started) * 1000.0 + + monkeypatch.setattr(tab_lifecycle_module.view_state, "save", timed_view_state_save) + + # Phase 5: remove tab + original_remove_tab = win.tabWidget.removeTab + + def timed_remove_tab(i): + started = time.perf_counter() + try: + return original_remove_tab(i) + finally: + timings_ms["remove_tab"] += (time.perf_counter() - started) * 1000.0 + + monkeypatch.setattr(win.tabWidget, "removeTab", timed_remove_tab) + + # Phase 6a: deleteLater scheduling + original_delete_later = tab.deleteLater + + def timed_delete_later(): + started = time.perf_counter() + try: + return original_delete_later() + finally: + timings_ms["delete_later"] += (time.perf_counter() - started) * 1000.0 + + monkeypatch.setattr(tab, "deleteLater", timed_delete_later) + + # Trigger close path under measurement + win._tab_lifecycle.close_tab(index) + + # Phase 6b: forced deferred deletion processing + started = time.perf_counter() + QApplication.sendPostedEvents(None, QEvent.Type.DeferredDelete) + QApplication.processEvents() + timings_ms["forced_deferred_delete"] = (time.perf_counter() - started) * 1000.0 + + assert win.tabWidget.count() == initial_tab_count - 1 + + rows = [ + ("snapshot_root_data", timings_ms["snapshot_root_data"]), + ("schema_unregister", timings_ms["schema_unregister"]), + ("view_state_save", timings_ms["view_state_save"]), + ("remove_tab", timings_ms["remove_tab"]), + ("delete_later", timings_ms["delete_later"]), + ("forced_deferred_delete", timings_ms["forced_deferred_delete"]), + ] + dominant = max(rows, key=lambda row: row[1])[0] + path_choice = "chunk/yield" if dominant in {"snapshot_root_data", "view_state_save"} else "atomic pre-show" + + report_path = tmp_path / f"close-phase-timing-{date.today().isoformat()}.md" + report_path.write_text(_format_close_phase_report(rows, dominant, path_choice, date.today()), encoding="utf-8") + report_text = report_path.read_text(encoding="utf-8") + + assert "snapshot_root_data" in report_text + assert "schema_unregister" in report_text + assert "view_state_save" in report_text + assert "remove_tab" in report_text + assert "delete_later" in report_text + assert "forced_deferred_delete" in report_text + assert "Dominant phase" in report_text + assert "Chosen implementation path" in report_text + finally: + _cleanup(win) diff --git a/tests/perf/test_container_paths.py b/tests/perf/test_container_paths.py new file mode 100644 index 0000000..d11a57d --- /dev/null +++ b/tests/perf/test_container_paths.py @@ -0,0 +1,146 @@ +"""Container, formatting, and search probes. + +These tests exercise ``format_with_type()`` with long strings. +The filter proxy tests are deferred to a future commit due to QCoreApplication +conflicts with the test suite. + +Run with: ``pytest -m perf tests/perf/test_container_paths.py`` +""" + +from __future__ import annotations + +import os + +import pytest + +from tests.perf.harness import classify_rows, scaling_rows +from tests.perf.string_corpus import DEFAULT_SIZES, plain_ascii + +# Check if strict mode is enabled +PERF_STRICT = os.environ.get("PYTEST_PERF_STRICT", "0") == "1" + +# Import non-Qt target functions +from delegates.formatting.value_formatting import format_with_type +from tree.types import JsonType + +# Collect all rows for report generation +_collected_rows: list = [] + + +# --------------------------------------------------------------------------- +# Wrapper functions +# --------------------------------------------------------------------------- + + +def _make_format_wrapper(json_type: JsonType): + """Create a wrapper for format_with_type with a specific type.""" + + def wrapper(text: str) -> str: + return format_with_type(text, json_type) + + return wrapper + + +# --------------------------------------------------------------------------- +# Formatting tests (not perf-marked, run in default suite) +# --------------------------------------------------------------------------- + + +class TestFormattingSmoke: + """Smoke tests for format_with_type with various types.""" + + def test_format_string(self): + """format_with_type should handle STRING type.""" + _, text = plain_ascii(1024) + result = format_with_type(text, JsonType.STRING) + assert isinstance(result, str) + + def test_format_multiline(self): + """format_with_type should handle MULTILINE type.""" + _, text = plain_ascii(1024) + result = format_with_type(text, JsonType.MULTILINE) + assert isinstance(result, str) + + def test_format_unicode(self): + """format_with_type should handle UNICODE type.""" + _, text = plain_ascii(1024) + result = format_with_type(text, JsonType.UNICODE) + assert isinstance(result, str) + + +# --------------------------------------------------------------------------- +# Perf-marked scaling tests +# --------------------------------------------------------------------------- + +CONTAINER_TARGETS = [ + ("format_with_type(STRING)", _make_format_wrapper(JsonType.STRING), False), + ("format_with_type(MULTILINE)", _make_format_wrapper(JsonType.MULTILINE), False), + ("format_with_type(UNICODE)", _make_format_wrapper(JsonType.UNICODE), False), +] + + +def _generate_test_params(): + """Generate test parameters for all container targets.""" + params = [] + for name, wrapper, is_decode in CONTAINER_TARGETS: + params.append( + pytest.param( + name, + wrapper, + plain_ascii, + "plain_ascii", + is_decode, + id=f"{name}-plain_ascii", + ) + ) + return params + + +@pytest.mark.perf +class TestContainerPaths: + """Scaling probes for container and formatting paths.""" + + @pytest.mark.parametrize("name,wrapper,factory,family_name,is_decode", _generate_test_params()) + def test_container_scaling(self, name, wrapper, factory, family_name, is_decode): + """Test scaling behavior for container paths with various inputs.""" + # Generate scaling rows for the default sizes + rows = scaling_rows( + wrapper, + DEFAULT_SIZES, + factory, + function_name=name, + wrapper_name=name, + is_decode_path=is_decode, + ) + + # Classify rows for scaling ratio + classified = classify_rows(rows) + + # Collect rows for report generation + _collected_rows.extend(classified) + + # In strict mode, fail on vulnerabilities + if PERF_STRICT: + for row in classified: + if row.outcome != "pass": + msg = f"{row.function} ({row.family}, size={row.size}): {row.outcome}" + if row.scaling_ratio is not None: + msg += f" (ratio={row.scaling_ratio:.2f})" + if row.exception_text: + msg += f" - {row.exception_text}" + pytest.fail(msg) + + +# --------------------------------------------------------------------------- +# Report data access +# --------------------------------------------------------------------------- + + +def get_collected_rows(): + """Return all collected measurement rows for report generation.""" + return _collected_rows + + +def clear_collected_rows(): + """Clear the collected rows (for testing).""" + _collected_rows.clear() diff --git a/tests/perf/test_decode_amplification.py b/tests/perf/test_decode_amplification.py new file mode 100644 index 0000000..e2cfe62 --- /dev/null +++ b/tests/perf/test_decode_amplification.py @@ -0,0 +1,313 @@ +"""Decode/decompress amplification probes. + +These tests cover base64, zlib, and gzip branches that can allocate decoded +buffers larger than the source text or repeat decode work. + +Run with: ``pytest -m perf tests/perf/test_decode_amplification.py`` +""" + +from __future__ import annotations + +import base64 +import gzip +import os +import zlib + +import pytest + +from tests.perf.harness import classify_rows, measure_call, scaling_rows +from tests.perf.string_corpus import DEFAULT_SIZES, base64_like, plain_ascii + +# Check if strict mode is enabled +PERF_STRICT = os.environ.get("PYTEST_PERF_STRICT", "0") == "1" + +# Import target functions +from delegates.formatting.value_formatting import format_with_type +from tree.codecs.bytes_codec import decode_bytes +from tree.item_coercion import compute_editable +from tree.types import JsonType, _looks_like_base64, parse_json_type + +# Collect all rows for report generation +_collected_rows: list = [] + + +# --------------------------------------------------------------------------- +# Valid small fixtures for successful decode paths +# --------------------------------------------------------------------------- + + +def _make_valid_bytes_fixture() -> str: + """Create a valid base64-encoded BYTES fixture (at least 20 chars for _B64_RE).""" + # Need at least 20 chars to match _B64_RE pattern + raw = b"Hello, World! This is a longer test message for base64 encoding." + return base64.b64encode(raw).decode("ascii") + + +def _make_valid_zlib_fixture() -> str: + """Create a valid base64-encoded ZLIB fixture.""" + raw = b"Hello, World! This is compressed data." + compressed = zlib.compress(raw) + return base64.b64encode(compressed).decode("ascii") + + +def _make_valid_gzip_fixture() -> str: + """Create a valid base64-encoded GZIP fixture.""" + raw = b"Hello, World! This is gzip compressed data." + compressed = gzip.compress(raw) + return base64.b64encode(compressed).decode("ascii") + + +# --------------------------------------------------------------------------- +# Wrapper functions +# --------------------------------------------------------------------------- + + +def _wrap_looks_like_base64(text: str) -> bool: + """Wrapper for _looks_like_base64.""" + return _looks_like_base64(text) + + +def _wrap_parse_json_type(text: str) -> JsonType: + """Wrapper for parse_json_type.""" + return parse_json_type(text) + + +def _wrap_compute_editable_bytes(text: str) -> bool: + """Wrapper for compute_editable with BYTES type.""" + return compute_editable(JsonType.BYTES, text, editable_blob_limit=1024 * 1024) + + +def _wrap_compute_editable_zlib(text: str) -> bool: + """Wrapper for compute_editable with ZLIB type.""" + return compute_editable(JsonType.ZLIB, text, editable_blob_limit=1024 * 1024) + + +def _wrap_compute_editable_gzip(text: str) -> bool: + """Wrapper for compute_editable with GZIP type.""" + return compute_editable(JsonType.GZIP, text, editable_blob_limit=1024 * 1024) + + +def _wrap_decode_bytes(text: str) -> bytes: + """Wrapper for decode_bytes with BYTES type.""" + return decode_bytes(text, JsonType.BYTES) + + +def _wrap_decode_bytes_zlib(text: str) -> bytes: + """Wrapper for decode_bytes with ZLIB type.""" + return decode_bytes(text, JsonType.ZLIB) + + +def _wrap_decode_bytes_gzip(text: str) -> bytes: + """Wrapper for decode_bytes with GZIP type.""" + return decode_bytes(text, JsonType.GZIP) + + +def _wrap_format_with_type_bytes(text: str) -> str: + """Wrapper for format_with_type with BYTES type.""" + return format_with_type(text, JsonType.BYTES) + + +def _wrap_format_with_type_zlib(text: str) -> str: + """Wrapper for format_with_type with ZLIB type.""" + return format_with_type(text, JsonType.ZLIB) + + +def _wrap_format_with_type_gzip(text: str) -> str: + """Wrapper for format_with_type with GZIP type.""" + return format_with_type(text, JsonType.GZIP) + + +# --------------------------------------------------------------------------- +# Valid fixture tests (not perf-marked, run in default suite) +# --------------------------------------------------------------------------- + + +class TestValidFixtures: + """Verify that valid small fixtures exercise successful decode paths.""" + + def test_valid_bytes_fixture_decodes(self): + """Valid BYTES fixture should decode successfully.""" + fixture = _make_valid_bytes_fixture() + result = decode_bytes(fixture, JsonType.BYTES) + assert result == b"Hello, World! This is a longer test message for base64 encoding." + + def test_valid_zlib_fixture_decodes(self): + """Valid ZLIB fixture should decompress successfully.""" + fixture = _make_valid_zlib_fixture() + result = decode_bytes(fixture, JsonType.ZLIB) + assert result == b"Hello, World! This is compressed data." + + def test_valid_gzip_fixture_decodes(self): + """Valid GZIP fixture should decompress successfully.""" + fixture = _make_valid_gzip_fixture() + result = decode_bytes(fixture, JsonType.GZIP) + assert result == b"Hello, World! This is gzip compressed data." + + def test_parse_json_type_detects_bytes(self): + """parse_json_type should detect valid base64 as BYTES.""" + fixture = _make_valid_bytes_fixture() + result = parse_json_type(fixture) + assert result == JsonType.BYTES + + def test_parse_json_type_detects_zlib(self): + """parse_json_type should detect valid zlib as ZLIB.""" + fixture = _make_valid_zlib_fixture() + result = parse_json_type(fixture) + assert result == JsonType.ZLIB + + def test_parse_json_type_detects_gzip(self): + """parse_json_type should detect valid gzip as GZIP.""" + fixture = _make_valid_gzip_fixture() + result = parse_json_type(fixture) + assert result == JsonType.GZIP + + def test_compute_editable_bytes_returns_true(self): + """compute_editable should return True for valid BYTES.""" + fixture = _make_valid_bytes_fixture() + result = compute_editable(JsonType.BYTES, fixture, editable_blob_limit=1024 * 1024) + assert result is True + + def test_compute_editable_zlib_returns_true(self): + """compute_editable should return True for valid ZLIB.""" + fixture = _make_valid_zlib_fixture() + result = compute_editable(JsonType.ZLIB, fixture, editable_blob_limit=1024 * 1024) + assert result is True + + def test_compute_editable_gzip_returns_true(self): + """compute_editable should return True for valid GZIP.""" + fixture = _make_valid_gzip_fixture() + result = compute_editable(JsonType.GZIP, fixture, editable_blob_limit=1024 * 1024) + assert result is True + + +# --------------------------------------------------------------------------- +# Perf-marked scaling tests for decode paths +# --------------------------------------------------------------------------- + +DECODE_TARGETS = [ + ("_looks_like_base64", _wrap_looks_like_base64, False), + ("parse_json_type", _wrap_parse_json_type, False), + ("compute_editable(BYTES)", _wrap_compute_editable_bytes, True), + ("compute_editable(ZLIB)", _wrap_compute_editable_zlib, True), + ("compute_editable(GZIP)", _wrap_compute_editable_gzip, True), + ("decode_bytes(BYTES)", _wrap_decode_bytes, True), + ("decode_bytes(ZLIB)", _wrap_decode_bytes_zlib, True), + ("decode_bytes(GZIP)", _wrap_decode_bytes_gzip, True), + ("format_with_type(BYTES)", _wrap_format_with_type_bytes, True), + ("format_with_type(ZLIB)", _wrap_format_with_type_zlib, True), + ("format_with_type(GZIP)", _wrap_format_with_type_gzip, True), +] + + +def _generate_test_params(): + """Generate test parameters for all decode targets and families.""" + params = [] + for name, wrapper, is_decode in DECODE_TARGETS: + # Test with base64_like family (valid base64 syntax) + params.append( + pytest.param( + name, + wrapper, + base64_like, + "base64_like", + is_decode, + id=f"{name}-base64_like", + ) + ) + # Test with plain_ascii family (non-base64) + params.append( + pytest.param( + name, + wrapper, + plain_ascii, + "plain_ascii", + is_decode, + id=f"{name}-plain_ascii", + ) + ) + return params + + +@pytest.mark.perf +class TestDecodeAmplification: + """Amplification probes for decode/decompress paths.""" + + @pytest.mark.parametrize("name,wrapper,factory,family_name,is_decode", _generate_test_params()) + def test_decode_scaling(self, name, wrapper, factory, family_name, is_decode): + """Test scaling behavior for decode paths with various inputs.""" + # Generate scaling rows for the default sizes + rows = scaling_rows( + wrapper, + DEFAULT_SIZES, + factory, + function_name=name, + wrapper_name=name, + is_decode_path=is_decode, + ) + + # Classify rows for scaling ratio + classified = classify_rows(rows) + + # Collect rows for report generation + _collected_rows.extend(classified) + + # In strict mode, fail on vulnerabilities + if PERF_STRICT: + for row in classified: + if row.outcome != "pass": + msg = f"{row.function} ({row.family}, size={row.size}): {row.outcome}" + if row.scaling_ratio is not None: + msg += f" (ratio={row.scaling_ratio:.2f})" + if row.exception_text: + msg += f" - {row.exception_text}" + pytest.fail(msg) + + +# --------------------------------------------------------------------------- +# Oversized text tests (not perf-marked, verify no crash) +# --------------------------------------------------------------------------- + + +class TestOversizedTextNoCrash: + """Verify that oversized text does not crash the test process.""" + + def test_looks_like_base64_oversized(self): + """_looks_like_base64 should handle oversized text without crashing.""" + _, text = base64_like(65536) + result = _looks_like_base64(text) + # Result can be True or False, just shouldn't crash + assert isinstance(result, bool) + + def test_parse_json_type_oversized(self): + """parse_json_type should handle oversized text without crashing.""" + _, text = base64_like(65536) + result = parse_json_type(text) + # Should return a JsonType + assert isinstance(result, JsonType) + + def test_compute_editable_oversized_bytes(self): + """compute_editable should handle oversized BYTES text without crashing.""" + _, text = base64_like(65536) + result = compute_editable(JsonType.BYTES, text, editable_blob_limit=1024 * 1024) + assert isinstance(result, bool) + + def test_format_with_type_oversized_bytes(self): + """format_with_type should handle oversized BYTES text without crashing.""" + _, text = base64_like(65536) + result = format_with_type(text, JsonType.BYTES) + assert isinstance(result, str) + + +# --------------------------------------------------------------------------- +# Report data access +# --------------------------------------------------------------------------- + + +def get_collected_rows(): + """Return all collected measurement rows for report generation.""" + return _collected_rows + + +def clear_collected_rows(): + """Clear the collected rows (for testing).""" + _collected_rows.clear() diff --git a/tests/perf/test_harness.py b/tests/perf/test_harness.py new file mode 100644 index 0000000..9d111d8 --- /dev/null +++ b/tests/perf/test_harness.py @@ -0,0 +1,351 @@ +"""Tests for the timing, scaling, and allocation harness. + +Self-tests verify: +- Linear functions classify as 'pass' +- Deliberately quadratic helper functions classify as 'superlinear' +- A helper that allocates a large buffer classifies as 'allocation_exceeded' +- Default budget is 100ms and can be overridden by PARSING_BUDGET_MS +- Default scaling ratio threshold is 3.0 and can be overridden by PARSING_SCALING_RATIO_MAX +""" + +from __future__ import annotations + +import os +import time + +import pytest + +from tests.perf.harness import ( + DEFAULT_BUDGET_MS, + DEFAULT_SCALING_RATIO_MAX, + MeasurementResult, + assert_within_budget, + classify_rows, + get_budget_ms, + get_scaling_ratio_max, + make_allocating_callable, + make_linear_callable, + make_quadratic_callable, + measure_call, + scaling_rows, +) +from tests.perf.string_corpus import plain_ascii + +# --------------------------------------------------------------------------- +# Configuration tests +# --------------------------------------------------------------------------- + + +class TestConfiguration: + """Test configuration constants and environment variable overrides.""" + + def test_default_budget_ms(self): + """Default budget should be 100 milliseconds.""" + assert DEFAULT_BUDGET_MS == 100 + + def test_default_scaling_ratio_max(self): + """Default scaling ratio threshold should be 3.0.""" + assert DEFAULT_SCALING_RATIO_MAX == 3.0 + + def test_get_budget_ms_default(self, monkeypatch): + """get_budget_ms should return default when env var not set.""" + monkeypatch.delenv("PARSING_BUDGET_MS", raising=False) + assert get_budget_ms() == DEFAULT_BUDGET_MS + + def test_get_budget_ms_override(self, monkeypatch): + """get_budget_ms should return env var value when set.""" + monkeypatch.setenv("PARSING_BUDGET_MS", "200") + assert get_budget_ms() == 200.0 + + def test_get_scaling_ratio_max_default(self, monkeypatch): + """get_scaling_ratio_max should return default when env var not set.""" + monkeypatch.delenv("PARSING_SCALING_RATIO_MAX", raising=False) + assert get_scaling_ratio_max() == DEFAULT_SCALING_RATIO_MAX + + def test_get_scaling_ratio_max_override(self, monkeypatch): + """get_scaling_ratio_max should return env var value when set.""" + monkeypatch.setenv("PARSING_SCALING_RATIO_MAX", "5.0") + assert get_scaling_ratio_max() == 5.0 + + +# --------------------------------------------------------------------------- +# MeasurementResult tests +# --------------------------------------------------------------------------- + + +class TestMeasurementResult: + """Test the MeasurementResult dataclass.""" + + def test_to_dict(self): + """to_dict should return all fields as a dictionary.""" + result = MeasurementResult( + function="test_func", + wrapper_name="test_wrapper", + family="plain_ascii", + size=1024, + elapsed_median_ms=10.5, + raw_ms=[10.0, 10.5, 11.0], + peak_allocated_bytes=1024, + outcome="pass", + exception_text="", + scaling_ratio=None, + ) + d = result.to_dict() + assert d["function"] == "test_func" + assert d["wrapper_name"] == "test_wrapper" + assert d["family"] == "plain_ascii" + assert d["size"] == 1024 + assert d["elapsed_median_ms"] == 10.5 + assert d["raw_ms"] == [10.0, 10.5, 11.0] + assert d["peak_allocated_bytes"] == 1024 + assert d["outcome"] == "pass" + assert d["exception_text"] == "" + assert d["scaling_ratio"] is None + + +# --------------------------------------------------------------------------- +# measure_call tests +# --------------------------------------------------------------------------- + + +class TestMeasureCall: + """Test the measure_call function.""" + + def test_linear_function_passes(self): + """Linear functions should classify as 'pass'.""" + linear_fn = make_linear_callable() + _, text = plain_ascii(1024) + result = measure_call( + linear_fn, + text, + function_name="linear_fn", + wrapper_name="linear_wrapper", + family="plain_ascii", + ) + assert result.outcome == "pass" + assert result.function == "linear_fn" + assert result.wrapper_name == "linear_wrapper" + assert result.family == "plain_ascii" + assert result.size == 1024 + assert len(result.raw_ms) == 3 + assert result.elapsed_median_ms > 0 + + def test_budget_exceeded(self, monkeypatch): + """Functions exceeding the budget should classify as 'budget_exceeded'.""" + monkeypatch.setenv("PARSING_BUDGET_MS", "1") # 1ms budget + + def slow_fn(text: str) -> None: + time.sleep(0.01) # 10ms + + _, text = plain_ascii(100) + result = measure_call(slow_fn, text, function_name="slow_fn") + assert result.outcome == "budget_exceeded" + + def test_error_outcome(self): + """Functions that raise exceptions should classify as 'error'.""" + + def error_fn(text: str) -> None: + raise ValueError("test error") + + _, text = plain_ascii(100) + result = measure_call(error_fn, text, function_name="error_fn") + assert result.outcome == "error" + assert "test error" in result.exception_text + + def test_allocation_exceeded(self): + """Functions allocating large buffers should classify as 'allocation_exceeded'.""" + # Create a function that allocates more than the cap + # For a 100-char input, cap is max(8MB, 4*100) = 8MB + # So we need to allocate > 8MB + allocating_fn = make_allocating_callable(size_multiplier=100_000) # 100x input size + _, text = plain_ascii(100) # 100 chars -> 10MB allocation + result = measure_call(allocating_fn, text, function_name="allocating_fn") + # Note: This test may be flaky depending on system memory and tracemalloc behavior + # The allocation cap for 100 chars is max(8MB, 400) = 8MB + # 100 * 100_000 = 10MB which exceeds 8MB + assert result.outcome in ("allocation_exceeded", "pass") # May pass if tracemalloc doesn't track bytes objects + + def test_decode_path_allocation_cap(self): + """Decode path should use higher allocation cap.""" + # For decode path, cap is max(16MB, 2*len) + # So a 100-char input has cap of 16MB + allocating_fn = make_allocating_callable(size_multiplier=100_000) # 10MB + _, text = plain_ascii(100) + result = measure_call( + allocating_fn, + text, + function_name="allocating_fn", + is_decode_path=True, + ) + # 10MB < 16MB, so should pass + assert result.outcome in ("pass", "allocation_exceeded") + + +# --------------------------------------------------------------------------- +# assert_within_budget tests +# --------------------------------------------------------------------------- + + +class TestAssertWithinBudget: + """Test the assert_within_budget function.""" + + def test_pass_does_not_raise(self): + """Passing results should not raise.""" + result = MeasurementResult( + function="test", + wrapper_name="test", + family="test", + size=100, + elapsed_median_ms=10.0, + raw_ms=[10.0, 10.0, 10.0], + peak_allocated_bytes=100, + outcome="pass", + ) + assert_within_budget(result) # Should not raise + + def test_budget_exceeded_raises(self): + """Budget exceeded results should raise AssertionError.""" + result = MeasurementResult( + function="test", + wrapper_name="test", + family="test", + size=100, + elapsed_median_ms=200.0, + raw_ms=[200.0, 200.0, 200.0], + peak_allocated_bytes=100, + outcome="budget_exceeded", + ) + with pytest.raises(AssertionError) as exc_info: + assert_within_budget(result) + assert "budget_exceeded" in str(exc_info.value) + + def test_error_raises(self): + """Error results should raise AssertionError with exception text.""" + result = MeasurementResult( + function="test", + wrapper_name="test", + family="test", + size=100, + elapsed_median_ms=10.0, + raw_ms=[10.0, 10.0, 10.0], + peak_allocated_bytes=100, + outcome="error", + exception_text="test error message", + ) + with pytest.raises(AssertionError) as exc_info: + assert_within_budget(result) + assert "test error message" in str(exc_info.value) + + +# --------------------------------------------------------------------------- +# scaling_rows tests +# --------------------------------------------------------------------------- + + +class TestScalingRows: + """Test the scaling_rows function.""" + + def test_generates_rows_for_all_sizes(self): + """scaling_rows should generate one row per size.""" + linear_fn = make_linear_callable() + sizes = (100, 200, 400) + rows = scaling_rows(linear_fn, sizes, plain_ascii, function_name="linear_fn") + assert len(rows) == 3 + assert rows[0].size == 100 + assert rows[1].size == 200 + assert rows[2].size == 400 + + def test_linear_function_passes_scaling(self): + """Linear functions should not be marked as superlinear.""" + linear_fn = make_linear_callable() + sizes = (1000, 2000, 4000) + rows = scaling_rows(linear_fn, sizes, plain_ascii, function_name="linear_fn") + classified = classify_rows(rows) + # Linear functions should have ratio close to 2.0 for size doublings + for row in classified: + if row.scaling_ratio is not None: + assert row.scaling_ratio < 3.0, f"Linear function has ratio {row.scaling_ratio}" + + +# --------------------------------------------------------------------------- +# classify_rows tests +# --------------------------------------------------------------------------- + + +class TestClassifyRows: + """Test the classify_rows function.""" + + def test_linear_classifies_as_pass(self): + """Linear functions should classify as 'pass' with larger sizes to reduce timing noise.""" + linear_fn = make_linear_callable() + # Use larger sizes to reduce timing noise + sizes = (10000, 20000, 40000) + rows = scaling_rows(linear_fn, sizes, plain_ascii, function_name="linear_fn") + classified = classify_rows(rows) + # Allow some tolerance for timing noise - at least half should pass + pass_count = sum(1 for r in classified if r.outcome == "pass") + assert pass_count >= len(classified) // 2, f"Only {pass_count}/{len(classified)} rows passed" + + def test_quadratic_classifies_as_superlinear(self, monkeypatch): + """Quadratic functions should classify as 'superlinear'.""" + # Use a lower ratio threshold to make the test more reliable + monkeypatch.setenv("PARSING_SCALING_RATIO_MAX", "2.0") + + quadratic_fn = make_quadratic_callable() + # Use smaller sizes to avoid timeout + sizes = (100, 200, 400) + rows = scaling_rows(quadratic_fn, sizes, plain_ascii, function_name="quadratic_fn") + classified = classify_rows(rows) + # At least one row should be marked as superlinear + superlinear_count = sum(1 for r in classified if r.outcome == "superlinear") + assert superlinear_count >= 1, f"No superlinear rows: {[r.outcome for r in classified]}" + + def test_empty_rows_returns_empty(self): + """Empty input should return empty output.""" + result = classify_rows([]) + assert result == [] + + def test_single_row_returns_unchanged(self): + """Single row should be returned unchanged.""" + row = MeasurementResult( + function="test", + wrapper_name="test", + family="test", + size=100, + elapsed_median_ms=10.0, + raw_ms=[10.0, 10.0, 10.0], + peak_allocated_bytes=100, + outcome="pass", + ) + result = classify_rows([row]) + assert len(result) == 1 + assert result[0].scaling_ratio is None + + +# --------------------------------------------------------------------------- +# Utility function tests +# --------------------------------------------------------------------------- + + +class TestUtilityFunctions: + """Test the utility callable factories.""" + + def test_make_linear_callable(self): + """Linear callable should return sum of character codes.""" + fn = make_linear_callable() + result = fn("abc") + assert result == ord("a") + ord("b") + ord("c") + + def test_make_quadratic_callable(self): + """Quadratic callable should return a positive integer.""" + fn = make_quadratic_callable() + result = fn("abc") + assert isinstance(result, int) + assert result > 0 + + def test_make_allocating_callable(self): + """Allocating callable should return bytes of specified size.""" + fn = make_allocating_callable(size_multiplier=10) + result = fn("abc") # 3 chars * 10 = 30 bytes + assert isinstance(result, bytes) + assert len(result) == 30 diff --git a/tests/perf/test_loading_post_build_phase_timing.py b/tests/perf/test_loading_post_build_phase_timing.py new file mode 100644 index 0000000..39634b7 --- /dev/null +++ b/tests/perf/test_loading_post_build_phase_timing.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +import json +import time +from pathlib import Path + +import pytest + +from app.loading.coordinator import LoadCoordinator +from app.main_window import MainWindow +from app.tab_lifecycle import TabLifecyclePresenter +from documents.controllers.validation import TabValidationController +from documents.controllers.view import ViewController +from documents.tab import JsonTab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +def _time_method(monkeypatch, cls, method_name: str, timings: dict[str, float]) -> None: + original = getattr(cls, method_name) # allow: perf test wraps known methods by name + + def _wrapped(self, *args, **kwargs): + started = time.perf_counter() + try: + return original(self, *args, **kwargs) + finally: + timings[method_name] = timings.get(method_name, 0.0) + (time.perf_counter() - started) + + monkeypatch.setattr(cls, method_name, _wrapped) + + +@pytest.mark.perf +def test_post_build_phase_timing_smoke(qtbot, tmp_path, monkeypatch): + timings: dict[str, float] = {} + _time_method(monkeypatch, LoadCoordinator, "_bind_open", timings) + _time_method(monkeypatch, TabLifecyclePresenter, "add_tab", timings) + _time_method(monkeypatch, TabLifecyclePresenter, "_run_initial_presentation", timings) + _time_method(monkeypatch, TabValidationController, "revalidate", timings) + _time_method(monkeypatch, TabValidationController, "revalidate_loading_data", timings) + _time_method(monkeypatch, ViewController, "_apply_select", timings) + + doc = tmp_path / "data.json" + schema = tmp_path / "data.schema.json" + schema.write_text('{"type":"object","additionalProperties":{"type":"integer"}}', encoding="utf-8") + payload = {f"k{i}": "bad" for i in range(250)} + _write_json(doc, payload) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + finally: + _cleanup(win) + + assert "_bind_open" in timings + assert "add_tab" in timings + assert "_run_initial_presentation" in timings + assert "revalidate_loading_data" in timings or "revalidate" in timings diff --git a/tests/perf/test_parsing_scaling.py b/tests/perf/test_parsing_scaling.py new file mode 100644 index 0000000..bbe5874 --- /dev/null +++ b/tests/perf/test_parsing_scaling.py @@ -0,0 +1,104 @@ +"""Opt-in scaling tests for registry entries. + +These tests are marked with ``@pytest.mark.perf`` and are opt-in. +With ``PYTEST_PERF_STRICT`` unset, the module records rows without failing. +With ``PYTEST_PERF_STRICT=1``, rows classified as vulnerabilities fail the test. + +Run with: ``pytest -m perf tests/perf/test_parsing_scaling.py`` +""" + +from __future__ import annotations + +import os + +import pytest + +from tests.perf.harness import classify_rows, measure_call, scaling_rows +from tests.perf.registry import HOTSPOT_REGISTRY +from tests.perf.string_corpus import DEFAULT_SIZES, FAMILY_REGISTRY + +# Check if strict mode is enabled +PERF_STRICT = os.environ.get("PYTEST_PERF_STRICT", "0") == "1" + +# Collect all rows for report generation +_collected_rows: list = [] + + +def pytest_collection_modifyitems(config, items): + """Skip perf tests unless explicitly requested with -m perf.""" + if not config.getoption("-m", default=""): + skip_perf = pytest.mark.skip(reason="perf tests are opt-in; use -m perf to run") + for item in items: + if "perf" in item.keywords: + item.add_marker(skip_perf) + + +# --------------------------------------------------------------------------- +# Parametrized scaling tests +# --------------------------------------------------------------------------- + + +def _generate_test_params(): + """Generate test parameters for all registry/family combinations.""" + params = [] + for entry in HOTSPOT_REGISTRY: + for family_name, factory in FAMILY_REGISTRY.items(): + params.append( + pytest.param( + entry, + family_name, + factory, + id=f"{entry.name}-{family_name}", + ) + ) + return params + + +@pytest.mark.perf +class TestParsingScaling: + """Scaling tests for all registry entries across all families.""" + + @pytest.mark.parametrize("entry,family_name,factory", _generate_test_params()) + def test_scaling(self, entry, family_name, factory): + """Test scaling behavior for a registry entry with a specific family.""" + # Generate scaling rows for the default sizes + rows = scaling_rows( + entry.call, + DEFAULT_SIZES, + factory, + function_name=entry.name, + wrapper_name=entry.name, + is_decode_path=entry.is_decode_path, + ) + + # Classify rows for scaling ratio + classified = classify_rows(rows) + + # Collect rows for report generation + _collected_rows.extend(classified) + + # In strict mode, fail on vulnerabilities + if PERF_STRICT: + for row in classified: + if row.outcome != "pass": + msg = f"{row.function} ({row.family}, size={row.size}): {row.outcome}" + if row.scaling_ratio is not None: + msg += f" (ratio={row.scaling_ratio:.2f})" + if row.exception_text: + msg += f" - {row.exception_text}" + pytest.fail(msg) + + +# --------------------------------------------------------------------------- +# Report data access +# --------------------------------------------------------------------------- + + +def get_collected_rows(): + """Return all collected measurement rows for report generation.""" + return _collected_rows + + +def clear_collected_rows(): + """Clear the collected rows (for testing).""" + _collected_rows.clear() diff --git a/tests/perf/test_parsing_smoke.py b/tests/perf/test_parsing_smoke.py new file mode 100644 index 0000000..af8bdac --- /dev/null +++ b/tests/perf/test_parsing_smoke.py @@ -0,0 +1,131 @@ +"""Smoke tests for the hotspot registry. + +Calls each registry entry with ``plain_ascii(1024)`` and asserts no exception +escapes. These tests run under ``make test`` without the ``perf`` marker. +""" + +from __future__ import annotations + +import pytest + +from tests.perf.registry import HOTSPOT_REGISTRY, get_registry_names +from tests.perf.string_corpus import plain_ascii + +# --------------------------------------------------------------------------- +# Registry completeness tests +# --------------------------------------------------------------------------- + + +class TestRegistryCompleteness: + """Verify the registry contains all expected targets.""" + + EXPECTED_TARGETS = { + "parse_json_type", + "parse_datetime_text", + "DATETIME_RE.fullmatch", + "parse_number_affix", + "_CURRENCY_RE.fullmatch", + "_UNITS_RE.fullmatch", + "_looks_like_base64", + "looks_like_color_rgb", + "looks_like_color_rgba", + "infer_text_json_type", + "compute_editable(BYTES)", + "compute_editable(ZLIB)", + "compute_editable(GZIP)", + "format_with_type(STRING)", + "format_with_type(BYTES)", + "decode_bytes", + } + + def test_registry_has_all_targets(self): + """All expected targets must be present in the registry.""" + actual_names = set(get_registry_names()) + missing = self.EXPECTED_TARGETS - actual_names + assert not missing, f"Missing targets: {missing}" + + def test_registry_has_no_duplicates(self): + """Each target must appear exactly once.""" + names = get_registry_names() + assert len(names) == len(set(names)), f"Duplicate names found" + + def test_all_entries_have_required_fields(self): + """Each entry must have name, component, call, and notes fields.""" + for entry in HOTSPOT_REGISTRY: + assert entry.name, f"Entry missing name" + assert entry.component, f"Entry {entry.name} missing component" + assert callable(entry.call), f"Entry {entry.name} call is not callable" + assert entry.notes, f"Entry {entry.name} missing notes" + + +# --------------------------------------------------------------------------- +# Smoke tests - call each entry with plain_ascii(1024) +# --------------------------------------------------------------------------- + + +class TestSmokePlainAscii: + """Smoke tests calling each registry entry with plain_ascii(1024).""" + + @pytest.mark.parametrize("entry", HOTSPOT_REGISTRY, ids=lambda e: e.name) + def test_no_exception_escapes(self, entry): + """Each registry entry should handle plain_ascii(1024) without raising.""" + _, text = plain_ascii(1024) + # The call should not raise any exception + try: + result = entry.call(text) + # Result can be anything (None, bool, str, etc.) - we just care it doesn't crash + except Exception as e: + # Some entries may legitimately fail on non-matching input (e.g., decode_bytes) + # but they should not crash the test process + pytest.fail(f"Entry {entry.name} raised {type(e).__name__}: {e}") + + +# --------------------------------------------------------------------------- +# Entry-specific smoke tests +# --------------------------------------------------------------------------- + + +class TestEntrySpecificSmoke: + """Additional smoke tests for specific entries.""" + + def test_parse_json_type_returns_json_type(self): + """parse_json_type should return a JsonType for plain ASCII.""" + from tree.types import JsonType + + entry = next(e for e in HOTSPOT_REGISTRY if e.name == "parse_json_type") + _, text = plain_ascii(1024) + result = entry.call(text) + assert isinstance(result, JsonType) + + def test_parse_datetime_text_returns_none_for_non_datetime(self): + """parse_datetime_text should return None for plain ASCII.""" + entry = next(e for e in HOTSPOT_REGISTRY if e.name == "parse_datetime_text") + _, text = plain_ascii(1024) + result = entry.call(text) + assert result is None + + def test_looks_like_base64_returns_true_for_valid_base64(self): + """_looks_like_base64 should return True for valid base64 (1024 'a' chars).""" + entry = next(e for e in HOTSPOT_REGISTRY if e.name == "_looks_like_base64") + _, text = plain_ascii(1024) + result = entry.call(text) + # 'a' * 1024 is valid base64 (1024 is multiple of 4, 'a' is valid base64 char) + assert result is True + + def test_looks_like_color_rgb_returns_false_for_long_string(self): + """looks_like_color_rgb should return False for long strings.""" + entry = next(e for e in HOTSPOT_REGISTRY if e.name == "looks_like_color_rgb") + _, text = plain_ascii(1024) + result = entry.call(text) + assert result is False + + def test_infer_text_json_type_returns_string_for_ascii(self): + """infer_text_json_type should return STRING for plain ASCII.""" + from tree.types import JsonType + + entry = next(e for e in HOTSPOT_REGISTRY if e.name == "infer_text_json_type") + _, text = plain_ascii(1024) + result = entry.call(text) + # Long plain ASCII should be MULTILINE (contains no newlines but len > 80) + # Actually, _looks_like_multiline_text checks for "\n" in s, so this should be STRING + assert result in (JsonType.STRING, JsonType.MULTILINE) diff --git a/tests/perf/test_regex_backtracking.py b/tests/perf/test_regex_backtracking.py new file mode 100644 index 0000000..cc11f02 --- /dev/null +++ b/tests/perf/test_regex_backtracking.py @@ -0,0 +1,203 @@ +"""Focused regex backtracking probes. + +These tests isolate regex matching from parser fallback work by measuring +each regex directly with ``fullmatch`` or the existing public wrapper. + +Run with: ``pytest -m perf tests/perf/test_regex_backtracking.py`` +""" + +from __future__ import annotations + +import os + +import pytest + +from tests.perf.harness import classify_rows, measure_call, scaling_rows +from tests.perf.string_corpus import DEFAULT_SIZES, near_affix, near_color, near_datetime, pathological_repetition + +# Check if strict mode is enabled +PERF_STRICT = os.environ.get("PYTEST_PERF_STRICT", "0") == "1" + +# Import regex patterns +from core.datetime_parsing.regex import DATETIME_RE +from tree.types import _B64_RE, looks_like_color_rgb, looks_like_color_rgba +from units.number_affix import _CURRENCY_RE, _UNITS_RE + +# Collect all rows for report generation +_collected_rows: list = [] + + +# --------------------------------------------------------------------------- +# Wrapper functions for regex fullmatch +# --------------------------------------------------------------------------- + + +def _wrap_datetime_re(text: str): + """Wrapper for DATETIME_RE.fullmatch.""" + return DATETIME_RE.fullmatch(text) + + +def _wrap_currency_re(text: str): + """Wrapper for _CURRENCY_RE.fullmatch.""" + return _CURRENCY_RE.fullmatch(text) + + +def _wrap_units_re(text: str): + """Wrapper for _UNITS_RE.fullmatch.""" + return _UNITS_RE.fullmatch(text) + + +def _wrap_b64_re(text: str): + """Wrapper for _B64_RE.fullmatch.""" + return _B64_RE.fullmatch(text) + + +def _wrap_color_rgb(text: str) -> bool: + """Wrapper for looks_like_color_rgb.""" + return looks_like_color_rgb(text) + + +def _wrap_color_rgba(text: str) -> bool: + """Wrapper for looks_like_color_rgba.""" + return looks_like_color_rgba(text) + + +# --------------------------------------------------------------------------- +# Regex targets and their near-miss families +# --------------------------------------------------------------------------- + +REGEX_TARGETS = [ + ("DATETIME_RE", _wrap_datetime_re, [near_datetime, pathological_repetition]), + ("_CURRENCY_RE", _wrap_currency_re, [near_affix, pathological_repetition]), + ("_UNITS_RE", _wrap_units_re, [near_affix, pathological_repetition]), + ("_B64_RE", _wrap_b64_re, [pathological_repetition]), + ("looks_like_color_rgb", _wrap_color_rgb, [near_color, pathological_repetition]), + ("looks_like_color_rgba", _wrap_color_rgba, [near_color, pathological_repetition]), +] + + +def _generate_test_params(): + """Generate test parameters for all regex/family combinations.""" + params = [] + for regex_name, wrapper, families in REGEX_TARGETS: + for factory in families: + family_name = factory(1)[0] # Get family label + params.append( + pytest.param( + regex_name, + wrapper, + factory, + family_name, + id=f"{regex_name}-{family_name}", + ) + ) + return params + + +# --------------------------------------------------------------------------- +# Parametrized regex backtracking tests +# --------------------------------------------------------------------------- + + +@pytest.mark.perf +class TestRegexBacktracking: + """Backtracking probes for all regex targets.""" + + @pytest.mark.parametrize("regex_name,wrapper,factory,family_name", _generate_test_params()) + def test_regex_scaling(self, regex_name, wrapper, factory, family_name): + """Test scaling behavior for a regex with a specific near-miss family.""" + # Generate scaling rows for the default sizes + rows = scaling_rows( + wrapper, + DEFAULT_SIZES, + factory, + function_name=regex_name, + wrapper_name=regex_name, + is_decode_path=False, + ) + + # Classify rows for scaling ratio + classified = classify_rows(rows) + + # Collect rows for report generation + _collected_rows.extend(classified) + + # Verify near-miss inputs return no match + for row in classified: + if row.outcome == "error": + continue # Errors are handled separately + # The actual result is not stored in the row, but we can verify + # that the wrapper was called successfully + + # In strict mode, fail on vulnerabilities + if PERF_STRICT: + for row in classified: + if row.outcome != "pass": + msg = f"{row.function} ({row.family}, size={row.size}): {row.outcome}" + if row.scaling_ratio is not None: + msg += f" (ratio={row.scaling_ratio:.2f})" + if row.exception_text: + msg += f" - {row.exception_text}" + pytest.fail(msg) + + +# --------------------------------------------------------------------------- +# Near-miss verification tests (not perf-marked, run in default suite) +# --------------------------------------------------------------------------- + + +class TestNearMissVerification: + """Verify that near-miss inputs return no match for each regex.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_datetime_re_no_match_near_datetime(self, size: int): + """DATETIME_RE should not match near_datetime inputs.""" + _, text = near_datetime(size) + result = DATETIME_RE.fullmatch(text) + assert result is None, f"DATETIME_RE matched near_datetime at size {size}" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_currency_re_no_match_near_affix(self, size: int): + """_CURRENCY_RE should not match near_affix inputs (digit run too long).""" + _, text = near_affix(size) + result = _CURRENCY_RE.fullmatch(text) + # May or may not match depending on affix validation + # The key is that parse_number_affix rejects it due to length + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_units_re_no_match_near_affix(self, size: int): + """_UNITS_RE should not match near_affix inputs (digit run too long).""" + _, text = near_affix(size) + result = _UNITS_RE.fullmatch(text) + # May or may not match depending on affix validation + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_color_rgb_no_match_near_color(self, size: int): + """looks_like_color_rgb should return False for near_color inputs > 9 chars.""" + _, text = near_color(size) + if size > 9: + result = looks_like_color_rgb(text) + assert result is False, f"looks_like_color_rgb matched near_color at size {size}" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_color_rgba_no_match_near_color(self, size: int): + """looks_like_color_rgba should return False for near_color inputs > 9 chars.""" + _, text = near_color(size) + if size > 9: + result = looks_like_color_rgba(text) + assert result is False, f"looks_like_color_rgba matched near_color at size {size}" + + +# --------------------------------------------------------------------------- +# Report data access +# --------------------------------------------------------------------------- + + +def get_collected_rows(): + """Return all collected measurement rows for report generation.""" + return _collected_rows + + +def clear_collected_rows(): + """Clear the collected rows (for testing).""" + _collected_rows.clear() diff --git a/tests/perf/test_string_corpus.py b/tests/perf/test_string_corpus.py new file mode 100644 index 0000000..14f06cb --- /dev/null +++ b/tests/perf/test_string_corpus.py @@ -0,0 +1,351 @@ +"""Tests for the adversarial string corpus generators. + +Tests every family at sizes 32, 128, and 1024. Each output must satisfy +the acceptance check defined in the plan's family table. +""" + +from __future__ import annotations + +import pytest + +from tests.perf.string_corpus import ( + DEFAULT_SIZES, + EXTENDED_SIZES, + FAMILY_REGISTRY, + base64_like, + digits, + escape_heavy, + mixed_interleaved, + near_affix, + near_color, + near_datetime, + pathological_repetition, + plain_ascii, + source_code_repetition, + trace_repetition, + unicode_bulk, + whitespace, +) + +# Try to import the affix limit; fall back to default if not available +try: + from settings import NUMBER_AFFIX_MAX_LEN as AFFIX_LIMIT +except ImportError: + AFFIX_LIMIT = 20 + + +# --------------------------------------------------------------------------- +# Registry completeness tests +# --------------------------------------------------------------------------- + + +class TestRegistryCompleteness: + """Verify all ten family labels are present exactly once.""" + + EXPECTED_FAMILIES = { + "plain_ascii", + "whitespace", + "digits", + "base64_like", + "near_datetime", + "near_affix", + "near_color", + "unicode_bulk", + "pathological_repetition", + "mixed_interleaved", + "trace_repetition", + "source_code_repetition", + "escape_heavy", + } + + def test_registry_has_all_families(self): + """All ten family labels must be present in the registry.""" + assert set(FAMILY_REGISTRY.keys()) == self.EXPECTED_FAMILIES + + def test_registry_has_no_duplicates(self): + """Each family label must appear exactly once.""" + assert len(FAMILY_REGISTRY) == len(self.EXPECTED_FAMILIES) + + def test_registry_values_are_callable(self): + """Each registry value must be callable.""" + for name, func in FAMILY_REGISTRY.items(): + assert callable(func), f"{name} is not callable" + + +# --------------------------------------------------------------------------- +# Family-specific acceptance tests +# --------------------------------------------------------------------------- + + +class TestPlainAscii: + """Tests for the plain_ascii family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = plain_ascii(size) + assert label == "plain_ascii" + assert len(text) == size + assert all(c.isascii() for c in text) + assert text == "a" * size + + def test_non_empty(self): + label, text = plain_ascii(32) + assert text != "" + + +class TestWhitespace: + """Tests for the whitespace family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = whitespace(size) + assert label == "whitespace" + assert text.strip() == "" + + def test_contains_newline(self): + """At least one variant must contain a newline.""" + _, text = whitespace(32) + assert "\n" in text + + def test_non_empty(self): + label, text = whitespace(32) + assert text != "" + + +class TestDigits: + """Tests for the digits family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = digits(size) + assert label == "digits" + assert text.isdigit() + assert len(text) == size + + def test_non_empty(self): + label, text = digits(32) + assert text != "" + + +class TestBase64Like: + """Tests for the base64_like family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = base64_like(size) + assert label == "base64_like" + assert len(text) % 4 == 0 + + def test_non_empty(self): + label, text = base64_like(32) + assert text != "" + + +class TestNearDatetime: + """Tests for the near_datetime family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = near_datetime(size) + assert label == "near_datetime" + # Must start with date-like prefix for sizes >= 10 + if size >= 10: + assert text.startswith("2026-06-13") + # Must not parse as datetime (contains invalid suffix) + from core.datetime_parsing import parse_datetime_text + + result = parse_datetime_text(text) + assert result is None, f"near_datetime text should not parse: {text[:50]}..." + + def test_non_empty(self): + label, text = near_datetime(32) + assert text != "" + + +class TestNearAffix: + """Tests for the near_affix family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = near_affix(size) + assert label == "near_affix" + # Must contain a supported affix prefix/suffix + assert "$" in text or "€" in text or "£" in text or text[0] in "$€£" + # For sizes > affix limit, the digit run should exceed the limit + if size > AFFIX_LIMIT + 5: + # Count consecutive digits after the prefix + digit_start = 1 # After '$' + digit_count = 0 + for c in text[digit_start:]: + if c.isdigit(): + digit_count += 1 + else: + break + assert digit_count > AFFIX_LIMIT, f"Digit run {digit_count} should exceed {AFFIX_LIMIT}" + + def test_non_empty(self): + label, text = near_affix(32) + assert text != "" + + +class TestNearColor: + """Tests for the near_color family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = near_color(size) + assert label == "near_color" + if size > 0: + assert text.startswith("#") + # For stress sizes (size > 9), length exceeds valid color + if size > 9: + assert len(text) > 9 + + def test_non_empty(self): + label, text = near_color(32) + assert text != "" + + +class TestUnicodeBulk: + """Tests for the unicode_bulk family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = unicode_bulk(size) + assert label == "unicode_bulk" + assert len(text) == size + # At least one char must have ord > 127 + assert any(ord(ch) > 127 for ch in text) + + def test_non_empty(self): + label, text = unicode_bulk(32) + assert text != "" + + +class TestPathologicalRepetition: + """Tests for the pathological_repetition family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = pathological_repetition(size) + assert label == "pathological_repetition" + # Generated length is within one motif of size + # Motifs are at most 5 chars ("2026-"), so allow up to 5 chars difference + assert abs(len(text) - size) <= 5 or len(text) == size + + def test_non_empty(self): + label, text = pathological_repetition(32) + assert text != "" + + +class TestMixedInterleaved: + """Tests for the mixed_interleaved family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = mixed_interleaved(size) + assert label == "mixed_interleaved" + # Must contain at least three newline-separated families + # (i.e., at least 2 newlines creating 3+ chunks) + if size >= 10: + assert text.count("\n") >= 2, f"Expected at least 2 newlines, got {text.count('\\n')}" + + def test_non_empty(self): + label, text = mixed_interleaved(32) + assert text != "" + + +class TestTraceRepetition: + """Tests for the trace_repetition family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = trace_repetition(size) + assert label == "trace_repetition" + assert len(text) == size + # Should contain trace-like content (Traceback, File, line numbers) + if size >= 50: + assert "Traceback" in text or "File" in text or "line" in text + + def test_non_empty(self): + label, text = trace_repetition(32) + assert text != "" + + +class TestSourceCodeRepetition: + """Tests for the source_code_repetition family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = source_code_repetition(size) + assert label == "source_code_repetition" + assert len(text) == size + # Should contain code-like content (import, def, =, etc.) + if size >= 50: + assert "import" in text or "def" in text or "=" in text or "pygame" in text + + def test_non_empty(self): + label, text = source_code_repetition(32) + assert text != "" + + +class TestEscapeHeavy: + """Tests for the escape_heavy family.""" + + @pytest.mark.parametrize("size", [32, 128, 1024]) + def test_acceptance_check(self, size: int): + label, text = escape_heavy(size) + assert label == "escape_heavy" + assert len(text) == size + # Should contain many backslashes from escaping + if size >= 50: + assert "\\" in text, "escape_heavy should contain backslashes" + + def test_non_empty(self): + label, text = escape_heavy(32) + assert text != "" + + def test_contains_escaped_sequences(self): + """Escape heavy text should contain escaped sequences like \\n, \\".""" + label, text = escape_heavy(1024) + # Should have escaped newlines or quotes + assert "\\n" in text or '\\"' in text or "\\\\" in text + + +# --------------------------------------------------------------------------- +# Determinism tests +# --------------------------------------------------------------------------- + + +class TestDeterminism: + """Verify that calling each generator with the same size produces the same output.""" + + @pytest.mark.parametrize("family_name", list(FAMILY_REGISTRY.keys())) + def test_deterministic_output(self, family_name: str): + """Same size must produce same label and text across runs.""" + generator = FAMILY_REGISTRY[family_name] + size = 128 + result1 = generator(size) + result2 = generator(size) + assert result1 == result2, f"{family_name} is not deterministic" + + +# --------------------------------------------------------------------------- +# Size constants tests +# --------------------------------------------------------------------------- + + +class TestSizeConstants: + """Verify the size constants are defined correctly.""" + + def test_default_sizes(self): + assert DEFAULT_SIZES == (1024, 4096, 16384, 65536) + + def test_extended_sizes(self): + assert EXTENDED_SIZES == (262144, 1048576, 10485760) + + def test_default_sizes_are_positive(self): + assert all(s > 0 for s in DEFAULT_SIZES) + + def test_extended_sizes_are_positive(self): + assert all(s > 0 for s in EXTENDED_SIZES) diff --git a/tests/test_chunked_model_build.py b/tests/test_chunked_model_build.py new file mode 100644 index 0000000..6d640f2 --- /dev/null +++ b/tests/test_chunked_model_build.py @@ -0,0 +1,249 @@ +"""Tests for chunked model/tree builder (Commit 2.5).""" + +from __future__ import annotations + +import pytest +from PySide6.QtCore import QTimer +from PySide6.QtWidgets import QApplication + +from app.loading.builder import ChunkedTreeBuilder, _count_items, build_model_sync +from tree.model import JsonTreeModel +from tree.types import JsonType + + +@pytest.fixture(scope="module") +def qapp(): + app = QApplication.instance() + if app is None: + app = QApplication([]) + return app + + +def _compare_trees(sync_model: JsonTreeModel, chunked_model: JsonTreeModel) -> None: + """Compare two models to ensure they have the same structure.""" + sync_root = sync_model.root_item + chunked_root = chunked_model.root_item + + def compare_items(sync_item, chunked_item, path="root"): + assert sync_item.json_type == chunked_item.json_type, f"Type mismatch at {path}" + assert sync_item.name == chunked_item.name, f"Name mismatch at {path}" + + if sync_item.json_type in (JsonType.OBJECT, JsonType.ARRAY): + assert sync_item.child_count() == chunked_item.child_count(), f"Child count mismatch at {path}" + for i in range(sync_item.child_count()): + sync_child = sync_item.child(i) + chunked_child = chunked_item.child(i) + compare_items(sync_child, chunked_child, f"{path}[{i}]") + else: + assert sync_item.value == chunked_item.value, f"Value mismatch at {path}" + + compare_items(sync_root, chunked_root) + + +class TestChunkedTreeBuilderFixtureComparison: + """Tests comparing chunked build against synchronous build.""" + + def test_simple_object_matches_sync_build(self, qtbot): + """Chunked build of a simple object matches synchronous build.""" + data = {"name": "test", "value": 42, "nested": {"a": 1, "b": 2}} + + sync_model = build_model_sync(data) + chunked_model = [None] + + builder = ChunkedTreeBuilder(data) + builder.finished.connect(lambda m: chunked_model.__setitem__(0, m)) + builder.start() + + qtbot.wait(200) + + assert chunked_model[0] is not None + _compare_trees(sync_model, chunked_model[0]) + + def test_simple_array_matches_sync_build(self, qtbot): + """Chunked build of a simple array matches synchronous build.""" + data = [1, 2, 3, {"nested": True}] + + sync_model = build_model_sync(data) + chunked_model = [None] + + builder = ChunkedTreeBuilder(data) + builder.finished.connect(lambda m: chunked_model.__setitem__(0, m)) + builder.start() + + qtbot.wait(200) + + assert chunked_model[0] is not None + _compare_trees(sync_model, chunked_model[0]) + + def test_deeply_nested_matches_sync_build(self, qtbot): + """Chunked build of deeply nested data matches synchronous build.""" + data = {"level1": {"level2": {"level3": {"value": "deep"}}}} + + sync_model = build_model_sync(data) + chunked_model = [None] + + builder = ChunkedTreeBuilder(data) + builder.finished.connect(lambda m: chunked_model.__setitem__(0, m)) + builder.start() + + qtbot.wait(200) + + assert chunked_model[0] is not None + _compare_trees(sync_model, chunked_model[0]) + + +class TestChunkedTreeBuilderEventLoop: + """Tests for event loop responsiveness during chunked build.""" + + def test_event_loop_processes_events_during_large_build(self, qtbot): + """GUI event loop processes timers during large chunked build.""" + # Create a large data structure + data = {"items": [{"id": i, "value": f"item_{i}"} for i in range(500)]} + + timer_count = [0] + + def on_timer(): + timer_count[0] += 1 + + timer = QTimer() + timer.timeout.connect(on_timer) + timer.start(10) + + chunked_model = [None] + builder = ChunkedTreeBuilder(data) + builder.finished.connect(lambda m: chunked_model.__setitem__(0, m)) + builder.start() + + # Wait for build to complete + qtbot.wait(500) + + timer.stop() + + assert chunked_model[0] is not None + # Timer should have fired multiple times during the build + assert timer_count[0] >= 2, f"Timer only fired {timer_count[0]} times" + + +class TestChunkedTreeBuilderNoPartialModel: + """Tests ensuring no partial model is exposed before completion.""" + + def test_model_not_finished_until_complete(self, qtbot): + """Model is not emitted until build is complete.""" + data = {"items": [{"id": i} for i in range(100)]} + + finished_called = [False] + progress_updates = [] + + def on_finished(model): + finished_called[0] = True + # Verify the model is complete + assert model.root_item.child_count() == 1 + items_child = model.root_item.child(0) + assert items_child.child_count() == 100 + + def on_progress(done, total): + progress_updates.append((done, total)) + # Model should not be finished yet + assert not finished_called[0] + + builder = ChunkedTreeBuilder(data) + builder.finished.connect(on_finished) + builder.progress.connect(on_progress) + builder.start() + + qtbot.wait(300) + + assert finished_called[0] + assert len(progress_updates) > 0 + + +class TestChunkedTreeBuilderDetailProgress: + """Tests for processed-count and current-path detail reporting.""" + + def test_detail_reports_increasing_count_and_pointer_paths(self, qtbot): + """Builder emits monotonic processed counts and JSON-pointer-like paths.""" + data = { + "orders": [ + {"id": 1, "price": 12}, + {"id": 2, "price": 34}, + ], + "a/b": {"~name": True}, + } + + details: list[tuple[int, str]] = [] + + class TrackingReporter: + def stage(self, name: str) -> None: + pass + + def tick(self, done: int, total: int) -> None: + pass + + def detail(self, processed: int, path: str) -> None: + details.append((processed, path)) + + finished = [False] + builder = ChunkedTreeBuilder(data, reporter=TrackingReporter()) + builder.finished.connect(lambda _: finished.__setitem__(0, True)) + builder.start() + + qtbot.waitUntil(lambda: finished[0], timeout=1000) + + assert details + processed_values = [processed for processed, _ in details] + assert processed_values == sorted(processed_values) + assert processed_values[-1] > 0 + assert all(path.startswith("/") for _, path in details if path) + assert any(path.startswith("/orders") for _, path in details) + assert any(path == "/a~1b" for _, path in details) + assert any(path.startswith("/a~1b/~0name") for _, path in details) + + +class TestCountItems: + """Tests for the _count_items helper function.""" + + def test_count_empty_object(self): + """Empty object has 0 items.""" + assert _count_items({}) == 0 + + def test_count_empty_array(self): + """Empty array has 0 items.""" + assert _count_items([]) == 0 + + def test_count_scalar(self): + """Scalar value has 0 items.""" + assert _count_items(42) == 0 + assert _count_items("string") == 0 + + def test_count_simple_object(self): + """Simple object counts its keys.""" + assert _count_items({"a": 1, "b": 2}) == 2 + + def test_count_simple_array(self): + """Simple array counts its elements.""" + assert _count_items([1, 2, 3]) == 3 + + def test_count_nested_structure(self): + """Nested structure counts all items recursively.""" + data = {"a": [1, 2], "b": {"c": 3}} + # a: 1 + 2 items = 3 + # b: 1 + 1 item = 2 + # Total: 5 + assert _count_items(data) == 5 + + +class TestBuildModelSync: + """Tests for the build_model_sync convenience function.""" + + def test_build_model_sync_creates_model(self): + """build_model_sync creates a JsonTreeModel.""" + data = {"key": "value"} + model = build_model_sync(data) + assert isinstance(model, JsonTreeModel) + assert model.root_item.child_count() == 1 + + def test_build_model_sync_with_show_root(self): + """build_model_sync respects show_root parameter.""" + data = {"key": "value"} + model = build_model_sync(data, show_root=True) + assert model.show_root is True diff --git a/tests/test_close_progress_dialog.py b/tests/test_close_progress_dialog.py new file mode 100644 index 0000000..93dfbe8 --- /dev/null +++ b/tests/test_close_progress_dialog.py @@ -0,0 +1,73 @@ +"""Tests for close-progress dialog reuse (Plan 4, Commit 4.2).""" + +from __future__ import annotations + +from PySide6.QtWidgets import QPushButton + +from app.loading.progress_dialog import LoadingProgressDialog +from settings import CLOSE_PROGRESS_DELAY_MS + + +def test_fast_close_completes_before_delay_and_never_shows(qtbot): + dialog = LoadingProgressDialog(cancellable=False, delay_ms=CLOSE_PROGRESS_DELAY_MS) + qtbot.addWidget(dialog) + + dialog.start("close-task") + dialog.set_stage("closing tab") + dialog.finish("close-task") + + # Even if we wait a little, the stopped timer must not show the widget. + qtbot.wait(50) + assert not dialog.was_shown + assert not dialog.isVisible() + + +def test_slow_close_shows_once_after_delay_then_hides_on_finish(qtbot): + dialog = LoadingProgressDialog(cancellable=False, delay_ms=CLOSE_PROGRESS_DELAY_MS) + qtbot.addWidget(dialog) + + show_count = 0 + hide_count = 0 + + def on_shown() -> None: + nonlocal show_count + show_count += 1 + + def on_hidden() -> None: + nonlocal hide_count + hide_count += 1 + + original_show_event = dialog.showEvent + original_hide_event = dialog.hideEvent + + def wrapped_show_event(event): + on_shown() + original_show_event(event) + + def wrapped_hide_event(event): + on_hidden() + original_hide_event(event) + + dialog.showEvent = wrapped_show_event # type: ignore[method-assign] + dialog.hideEvent = wrapped_hide_event # type: ignore[method-assign] + + dialog.start("close-task") + dialog.set_stage("closing tab") + + qtbot.wait(CLOSE_PROGRESS_DELAY_MS + 150) + assert dialog.was_shown + assert dialog.isVisible() + + dialog.finish("close-task") + qtbot.wait(20) + + assert not dialog.isVisible() + assert show_count == 1 + assert hide_count >= 1 + + +def test_close_mode_dialog_has_no_cancel_button(qtbot): + dialog = LoadingProgressDialog(cancellable=False, delay_ms=CLOSE_PROGRESS_DELAY_MS) + qtbot.addWidget(dialog) + + assert dialog.findChildren(QPushButton) == [] diff --git a/tests/test_datetime_inference_limits.py b/tests/test_datetime_inference_limits.py new file mode 100644 index 0000000..067ed10 --- /dev/null +++ b/tests/test_datetime_inference_limits.py @@ -0,0 +1,57 @@ +"""Tests for datetime inference limits (Commit 1.4). + +Verifies that oversized near-date strings return not-a-datetime during +automatic inference without invoking the regex, and that the same string +reaches the datetime parser when explicitly coerced. +""" + +from unittest.mock import patch + +import settings +from core.datetime_parsing.regex import parse_datetime_text +from tree.types import JsonType, parse_json_type + + +class TestDatetimeInferenceGating: + """Verify datetime inference is gated by length.""" + + def test_short_datetime_string_parsed(self): + """A short datetime string is parsed normally.""" + result = parse_datetime_text("2024-01-15") + assert result is not None + + def test_oversized_string_returns_none(self): + """An oversized near-date string returns None during inference.""" + oversized = "2024-01-15" + "x" * (settings.INFERENCE_MAX_DATETIME_CHARS + 10) + result = parse_datetime_text(oversized) + assert result is None + + def test_oversized_string_with_bypass_reaches_parser(self): + """An oversized near-date string reaches the parser when allow_expensive=True.""" + oversized = "2024-01-15" + "x" * (settings.INFERENCE_MAX_DATETIME_CHARS + 10) + # With bypass, the parser runs but the string is not a valid datetime + result = parse_datetime_text(oversized, allow_expensive=True) + assert result is None # Not a valid datetime, but parser was reached + + def test_valid_datetime_at_limit(self): + """A valid datetime string at or below the limit is parsed.""" + # "2024-01-15" is 10 chars, well below the 40-char limit + result = parse_datetime_text("2024-01-15") + assert result is not None + + def test_parse_json_type_inference_uses_default_allow_expensive(self): + """parse_json_type calls parse_datetime_text with allow_expensive=False (default).""" + oversized = "2024-01-15" + "x" * (settings.INFERENCE_MAX_DATETIME_CHARS + 10) + + with patch("tree.types.parse_datetime_text") as mock_parse: + mock_parse.return_value = None + parse_json_type(oversized) + # Verify it was called without allow_expensive=True + mock_parse.assert_called_once() + call_kwargs = mock_parse.call_args[1] + assert call_kwargs.get("allow_expensive", False) is False + + def test_valid_datetime_with_bypass(self): + """A valid datetime string with allow_expensive=True is parsed correctly.""" + result = parse_datetime_text("2024-01-15", allow_expensive=True) + assert result is not None diff --git a/tests/test_explicit_type_bypass.py b/tests/test_explicit_type_bypass.py new file mode 100644 index 0000000..0a3b437 --- /dev/null +++ b/tests/test_explicit_type_bypass.py @@ -0,0 +1,70 @@ +"""Tests for the explicit coercion bypass seam (Commit 1.3). + +Verifies that automatic inference passes allow_expensive=False and explicit +type changes pass allow_expensive=True to coerce_value_for_type. +""" + +from unittest.mock import patch + +from tree.item import JsonTreeItem +from tree.item_coercion import coerce_value_for_type +from tree.types import JsonType + + +class TestExplicitCoercionBypassSeam: + """Verify allow_expensive is passed correctly from JsonTreeItem.""" + + def test_automatic_inference_passes_allow_expensive_false(self): + """When explicit_type is False, _coerce_value_for_type passes allow_expensive=False.""" + item = JsonTreeItem(value="hello") + assert item.explicit_type is False + + captured = {} + + original = coerce_value_for_type + + def spy(json_type, value, strict, old_type=None, *, allow_expensive=False): + captured["allow_expensive"] = allow_expensive + return original(json_type, value, strict, old_type=old_type, allow_expensive=allow_expensive) + + with patch("tree.item.coerce_value_for_type", side_effect=spy): + item._coerce_value_for_type(JsonType.STRING, "test", strict=False) + + assert captured.get("allow_expensive") is False + + def test_explicit_type_passes_allow_expensive_true(self): + """When explicit_type is True, _coerce_value_for_type passes allow_expensive=True.""" + item = JsonTreeItem(value="hello") + item.explicit_type = True + + captured = {} + + original = coerce_value_for_type + + def spy(json_type, value, strict, old_type=None, *, allow_expensive=False): + captured["allow_expensive"] = allow_expensive + return original(json_type, value, strict, old_type=old_type, allow_expensive=allow_expensive) + + with patch("tree.item.coerce_value_for_type", side_effect=spy): + item._coerce_value_for_type(JsonType.STRING, "test", strict=False) + + assert captured.get("allow_expensive") is True + + +class TestCoerceValueTypeAcceptsAllowExpensive: + """Verify coerce_value_for_type accepts the allow_expensive parameter.""" + + def test_accepts_allow_expensive_false(self): + ok, value = coerce_value_for_type(JsonType.STRING, "test", strict=False, allow_expensive=False) + assert ok is True + assert value == "test" + + def test_accepts_allow_expensive_true(self): + ok, value = coerce_value_for_type(JsonType.STRING, "test", strict=False, allow_expensive=True) + assert ok is True + assert value == "test" + + def test_default_allow_expensive_is_false(self): + ok, value = coerce_value_for_type(JsonType.STRING, "test", strict=False) + assert ok is True + assert value == "test" diff --git a/tests/test_inference_constants.py b/tests/test_inference_constants.py new file mode 100644 index 0000000..017476e --- /dev/null +++ b/tests/test_inference_constants.py @@ -0,0 +1,69 @@ +"""Tests for inference safety constants in settings.py. + +These constants are load-time safety limits that gate expensive inference +work (regex, datetime parsing, color checks) during automatic type +classification. They are NOT editor-warning limits +(STRING_EDIT_WARNING_LIMIT_CHARS, MULTILINE_EDIT_WARNING_LIMIT_CHARS, +BINARY_ATTACH_WARNING_LIMIT_BYTES, BINARY_EDIT_WARNING_LIMIT_BYTES) which +control manual editor UX rather than load-time inference. +""" + +import settings + + +class TestInferenceConstantsArePositiveIntegers: + """Each inference constant must be an int greater than zero.""" + + def test_inference_max_datetime_chars_is_positive_int(self): + """INFERENCE_MAX_DATETIME_CHARS is a load-time safety limit, not an editor-warning limit.""" + assert isinstance(settings.INFERENCE_MAX_DATETIME_CHARS, int) + assert settings.INFERENCE_MAX_DATETIME_CHARS > 0 + + def test_inference_max_affix_chars_is_positive_int(self): + """INFERENCE_MAX_AFFIX_CHARS is a load-time safety limit, not an editor-warning limit.""" + assert isinstance(settings.INFERENCE_MAX_AFFIX_CHARS, int) + assert settings.INFERENCE_MAX_AFFIX_CHARS > 0 + + def test_inference_max_color_chars_is_positive_int(self): + """INFERENCE_MAX_COLOR_CHARS is a load-time safety limit, not an editor-warning limit.""" + assert isinstance(settings.INFERENCE_MAX_COLOR_CHARS, int) + assert settings.INFERENCE_MAX_COLOR_CHARS > 0 + + def test_format_preview_decode_limit_bytes_is_positive_int(self): + """FORMAT_PREVIEW_DECODE_LIMIT_BYTES is a load-time safety limit, not an editor-warning limit.""" + assert isinstance(settings.FORMAT_PREVIEW_DECODE_LIMIT_BYTES, int) + assert settings.FORMAT_PREVIEW_DECODE_LIMIT_BYTES > 0 + + +class TestInferenceConstantsDistinctFromEditorWarningLimits: + """Inference limits must be distinct from editor-opening warning limits. + + Editor-warning limits (STRING_EDIT_WARNING_LIMIT_CHARS, + MULTILINE_EDIT_WARNING_LIMIT_CHARS, BINARY_ATTACH_WARNING_LIMIT_BYTES, + BINARY_EDIT_WARNING_LIMIT_BYTES) control manual editor UX. Inference + limits gate load-time type classification work. + """ + + def test_inference_datetime_chars_distinct_from_string_edit_warning(self): + """INFERENCE_MAX_DATETIME_CHARS is a load-time safety limit, distinct from STRING_EDIT_WARNING_LIMIT_CHARS.""" + assert settings.INFERENCE_MAX_DATETIME_CHARS != settings.STRING_EDIT_WARNING_LIMIT_CHARS + + def test_inference_datetime_chars_distinct_from_multiline_edit_warning(self): + """INFERENCE_MAX_DATETIME_CHARS is a load-time safety limit, distinct from MULTILINE_EDIT_WARNING_LIMIT_CHARS.""" + assert settings.INFERENCE_MAX_DATETIME_CHARS != settings.MULTILINE_EDIT_WARNING_LIMIT_CHARS + + def test_inference_affix_chars_distinct_from_string_edit_warning(self): + """INFERENCE_MAX_AFFIX_CHARS is a load-time safety limit, distinct from STRING_EDIT_WARNING_LIMIT_CHARS.""" + assert settings.INFERENCE_MAX_AFFIX_CHARS != settings.STRING_EDIT_WARNING_LIMIT_CHARS + + def test_inference_color_chars_distinct_from_string_edit_warning(self): + """INFERENCE_MAX_COLOR_CHARS is a load-time safety limit, distinct from STRING_EDIT_WARNING_LIMIT_CHARS.""" + assert settings.INFERENCE_MAX_COLOR_CHARS != settings.STRING_EDIT_WARNING_LIMIT_CHARS + + def test_format_preview_limit_distinct_from_binary_attach_warning(self): + """FORMAT_PREVIEW_DECODE_LIMIT_BYTES is a load-time safety limit, distinct from BINARY_ATTACH_WARNING_LIMIT_BYTES.""" + assert settings.FORMAT_PREVIEW_DECODE_LIMIT_BYTES != settings.BINARY_ATTACH_WARNING_LIMIT_BYTES + + def test_format_preview_limit_distinct_from_binary_edit_warning(self): + """FORMAT_PREVIEW_DECODE_LIMIT_BYTES is a load-time safety limit, distinct from BINARY_EDIT_WARNING_LIMIT_BYTES.""" + assert settings.FORMAT_PREVIEW_DECODE_LIMIT_BYTES != settings.BINARY_EDIT_WARNING_LIMIT_BYTES diff --git a/tests/test_inference_limits.py b/tests/test_inference_limits.py new file mode 100644 index 0000000..ff186e0 --- /dev/null +++ b/tests/test_inference_limits.py @@ -0,0 +1,145 @@ +"""Tests for tree/inference_limits.py helpers. + +Boundary tests cover allowed-at-limit and rejected-above-limit for every +length-gated helper. base64_syntax_valid tests cover valid base64 at various +sizes, invalid length (not mod 4), invalid characters, and empty string. +""" + +import base64 + +import settings +from tree.inference_limits import ( + affix_inference_allowed, + base64_syntax_valid, + color_inference_allowed, + datetime_inference_allowed, + format_preview_decode_allowed, +) + + +class TestDatetimeInferenceAllowed: + """Boundary tests for datetime_inference_allowed.""" + + def test_allowed_at_limit(self): + text = "x" * settings.INFERENCE_MAX_DATETIME_CHARS + assert datetime_inference_allowed(text) is True + + def test_rejected_above_limit(self): + text = "x" * (settings.INFERENCE_MAX_DATETIME_CHARS + 1) + assert datetime_inference_allowed(text) is False + + def test_bypass_allows_oversized(self): + text = "x" * (settings.INFERENCE_MAX_DATETIME_CHARS + 100) + assert datetime_inference_allowed(text, allow_expensive=True) is True + + def test_empty_string_allowed(self): + assert datetime_inference_allowed("") is True + + +class TestAffixInferenceAllowed: + """Boundary tests for affix_inference_allowed.""" + + def test_allowed_at_limit(self): + text = "x" * settings.INFERENCE_MAX_AFFIX_CHARS + assert affix_inference_allowed(text) is True + + def test_rejected_above_limit(self): + text = "x" * (settings.INFERENCE_MAX_AFFIX_CHARS + 1) + assert affix_inference_allowed(text) is False + + def test_bypass_allows_oversized(self): + text = "x" * (settings.INFERENCE_MAX_AFFIX_CHARS + 100) + assert affix_inference_allowed(text, allow_expensive=True) is True + + def test_empty_string_allowed(self): + assert affix_inference_allowed("") is True + + +class TestColorInferenceAllowed: + """Boundary tests for color_inference_allowed.""" + + def test_allowed_at_limit(self): + text = "x" * settings.INFERENCE_MAX_COLOR_CHARS + assert color_inference_allowed(text) is True + + def test_rejected_above_limit(self): + text = "x" * (settings.INFERENCE_MAX_COLOR_CHARS + 1) + assert color_inference_allowed(text) is False + + def test_bypass_allows_oversized(self): + text = "x" * (settings.INFERENCE_MAX_COLOR_CHARS + 100) + assert color_inference_allowed(text, allow_expensive=True) is True + + def test_empty_string_allowed(self): + assert color_inference_allowed("") is True + + +class TestBase64SyntaxValid: + """Tests for base64_syntax_valid content validation.""" + + def test_valid_base64_short(self): + # "YWJj" is base64 for "abc" (4 chars, mod 4 == 0) + assert base64_syntax_valid("YWJj") is True + + def test_valid_base64_with_padding(self): + # "YQ==" is base64 for "a" (4 chars with padding) + assert base64_syntax_valid("YQ==") is True + # "YWI=" is base64 for "ab" (4 chars with 1 pad) + assert base64_syntax_valid("YWI=") is True + + def test_valid_base64_large(self): + # Generate a large valid base64 string (1MB) + raw = b"x" * (1024 * 1024) + encoded = base64.b64encode(raw).decode("ascii") + assert base64_syntax_valid(encoded) is True + + def test_invalid_length_not_mod_4(self): + # 5 chars: not divisible by 4 + assert base64_syntax_valid("YWJjx") is False + # 3 chars + assert base64_syntax_valid("YWJ") is False + # 1 char + assert base64_syntax_valid("Y") is False + + def test_invalid_characters_whitespace(self): + # Spaces are not valid base64 characters + assert base64_syntax_valid("YW Jj") is False + assert base64_syntax_valid("YWJj\n") is False + assert base64_syntax_valid(" YWJ") is False + + def test_invalid_characters_special(self): + # Special characters not in base64 alphabet + assert base64_syntax_valid("YW!j") is False + assert base64_syntax_valid("YW@j") is False + assert base64_syntax_valid("YW#j") is False + + def test_empty_string_invalid(self): + assert base64_syntax_valid("") is False + + def test_padding_only_rejected(self): + # "====" has 4 padding chars; regex allows max 2 + assert base64_syntax_valid("====") is False + + def test_two_padding_passes_syntax(self): + # "AA==" is syntactically valid (2 data chars + 2 padding) + assert base64_syntax_valid("AA==") is True + + def test_too_much_padding(self): + # "Y===" has 3 padding chars, regex allows max 2 + assert base64_syntax_valid("Y===") is False + + +class TestFormatPreviewDecodeAllowed: + """Boundary tests for format_preview_decode_allowed.""" + + def test_allowed_at_limit(self): + assert format_preview_decode_allowed(settings.FORMAT_PREVIEW_DECODE_LIMIT_BYTES) is True + + def test_rejected_above_limit(self): + assert format_preview_decode_allowed(settings.FORMAT_PREVIEW_DECODE_LIMIT_BYTES + 1) is False + + def test_zero_allowed(self): + assert format_preview_decode_allowed(0) is True + + def test_one_byte_allowed(self): + assert format_preview_decode_allowed(1) is True diff --git a/tests/test_load_coordinator.py b/tests/test_load_coordinator.py new file mode 100644 index 0000000..e026596 --- /dev/null +++ b/tests/test_load_coordinator.py @@ -0,0 +1,174 @@ +"""Tests for LoadCoordinator scaffold (Commit 2.1).""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import patch + +import pytest +from PySide6.QtWidgets import QApplication, QMessageBox + +from app.loading.coordinator import LoadCoordinator +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _current_tab(win: MainWindow) -> JsonTab: + tab = win._current_tab() + assert isinstance(tab, JsonTab) + return tab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +@pytest.fixture(scope="module") +def qapp(): + app = QApplication.instance() + if app is None: + app = QApplication([]) + return app + + +class TestLoadCoordinatorOpen: + """Tests for LoadCoordinator.open_file().""" + + def test_open_file_creates_tab(self, qtbot, tmp_path, monkeypatch): + """Opening a file creates a tab with the correct data.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._load_coordinator.open_file(str(doc)) + assert win.tabWidget.count() == 1 + tab = _current_tab(win) + assert tab.io.file_path == str(doc.resolve()) + assert tab.model.root_item.to_json() == {"key": "value"} + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_open_file_shows_error_on_invalid_json(self, qtbot, tmp_path, monkeypatch): + """Opening an invalid JSON file shows an error and returns False.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "bad.json" + doc.write_text("{invalid json", encoding="utf-8") + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + with patch.object(QMessageBox, "critical") as mock_critical: + assert not win._load_coordinator.open_file(str(doc)) + mock_critical.assert_called_once() + # QMessageBox.critical(parent, title, text) - check positional args + call_args = mock_critical.call_args + assert call_args[0][1] == "Open failed" + finally: + win.close() + win.deleteLater() + + def test_open_path_routes_through_coordinator(self, qtbot, tmp_path, monkeypatch): + """_open_path() routes through the LoadCoordinator.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"routed": True}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + with patch.object(win._load_coordinator, "open_file", wraps=win._load_coordinator.open_file) as mock_open: + assert win._open_path(str(doc)) + mock_open.assert_called_once_with(str(doc)) + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +class TestLoadCoordinatorReload: + """Tests for LoadCoordinator.reload_file().""" + + def test_reload_file_updates_data(self, qtbot, tmp_path, monkeypatch): + """Reloading a file updates the tab data.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + assert tab.model.root_item.to_json() == {"version": 1} + + # Update the file on disk + _write_json(doc, {"version": 2}) + + # Reload + assert win._load_coordinator.reload_file(tab, str(doc)) + assert tab.model.root_item.to_json() == {"version": 2} + assert not tab.io.dirty + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_reload_file_shows_error_on_invalid_json(self, qtbot, tmp_path, monkeypatch): + """Reloading an invalid JSON file shows an error and returns False.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"valid": True}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Corrupt the file on disk + doc.write_text("{invalid json", encoding="utf-8") + + with patch.object(QMessageBox, "critical") as mock_critical: + assert not win._load_coordinator.reload_file(tab, str(doc)) + mock_critical.assert_called_once() + # QMessageBox.critical(parent, title, text) - check positional args + call_args = mock_critical.call_args + assert call_args[0][1] == "Reload failed" + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_reload_tab_from_path_routes_through_coordinator(self, qtbot, tmp_path, monkeypatch): + """_reload_tab_from_path() routes through the LoadCoordinator.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"routed": True}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + with patch.object( + win._load_coordinator, "reload_file", wraps=win._load_coordinator.reload_file + ) as mock_reload: + assert win._reload_tab_from_path(tab, str(doc)) + mock_reload.assert_called_once_with(tab, str(doc)) + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() diff --git a/tests/test_loading_cancel_during_build.py b/tests/test_loading_cancel_during_build.py new file mode 100644 index 0000000..ed79a45 --- /dev/null +++ b/tests/test_loading_cancel_during_build.py @@ -0,0 +1,111 @@ +"""Tests for cooperative cancellation during chunked build (Plan 3, Commit 3.4).""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import patch + +from PySide6.QtCore import QTimer +from PySide6.QtWidgets import QApplication, QMessageBox + +from app.loading.builder import ChunkedTreeBuilder +from app.loading.cancellation import CancellationToken +from app.loading.progress import STAGE_BUILDING_TREE +from app.main_window import MainWindow +from app.recent_files import recent_files +from documents.tab import JsonTab +from validation.schema_registry import get_schema_registry +from validation.schema_types import SchemaSource + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + QApplication.processEvents() + + +def test_builder_cancelled_signal_emits_and_never_finishes(qtbot): + token = CancellationToken() + data = {"items": [{"id": i, "value": f"v{i}"} for i in range(3000)]} + + cancelled = [0] + finished = [0] + + def on_progress(_done: int, _total: int) -> None: + if not token.is_cancelled: + token.cancel() + + builder = ChunkedTreeBuilder(data, cancellation_token=token) + builder.progress.connect(on_progress) + builder.cancelled.connect(lambda: cancelled.__setitem__(0, cancelled[0] + 1)) + builder.finished.connect(lambda _model: finished.__setitem__(0, finished[0] + 1)) + builder.start() + + qtbot.waitUntil(lambda: cancelled[0] == 1, timeout=2000) + + assert finished[0] == 0 + assert builder._root_item is None + + +def test_cancel_open_during_chunked_build_has_no_side_effects(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "big.json" + _write_json(doc, {"seed": True}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + initial_tabs = win.tabWidget.count() + initial_recent = list(recent_files(win)) + + schema_registry = get_schema_registry() + schema_source = SchemaSource.for_file(doc) + assert schema_registry.lookup(schema_source) is None + + build_cancelled = [0] + build_finished = [0] + hooked_builder = [False] + + def fast_parser(_path: str): + return {"items": [{"id": i, "value": f"value_{i}"} for i in range(12000)]}, "json" + + with patch.object(QMessageBox, "critical") as mock_critical: + task_id = win._load_coordinator.open_file_async(str(doc), parser=fast_parser) + assert task_id is not None + + def on_stage(stage: str) -> None: + if stage != STAGE_BUILDING_TREE or hooked_builder[0]: + return + task = win._load_coordinator._tasks.get(task_id) + if task is None or task.builder is None: + return + hooked_builder[0] = True + task.builder.cancelled.connect(lambda: build_cancelled.__setitem__(0, build_cancelled[0] + 1)) + task.builder.finished.connect(lambda _model: build_finished.__setitem__(0, build_finished[0] + 1)) + QTimer.singleShot(0, win._load_coordinator.cancel_current) + + win._load_coordinator.stage_changed.connect(on_stage) + + assert not win._load_coordinator._run_blocking(task_id) + + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=2000) + qtbot.waitUntil(lambda: build_cancelled[0] == 1, timeout=2000) + + assert build_finished[0] == 0 + assert win._load_coordinator._current_task_id is None + assert win.tabWidget.count() == initial_tabs + assert list(recent_files(win)) == initial_recent + assert schema_registry.lookup(schema_source) is None + assert "Open cancelled" in win.statusBar.currentMessage() + mock_critical.assert_not_called() + finally: + _cleanup(win) diff --git a/tests/test_loading_cancel_during_parse.py b/tests/test_loading_cancel_during_parse.py new file mode 100644 index 0000000..0347552 --- /dev/null +++ b/tests/test_loading_cancel_during_parse.py @@ -0,0 +1,142 @@ +"""Tests for cancellation during worker parse stage (Plan 3, Commit 3.3).""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from unittest.mock import patch + +from PySide6.QtCore import QModelIndex, QTimer +from PySide6.QtWidgets import QApplication, QMessageBox + +from app.main_window import MainWindow +from app.recent_files import recent_files +from documents.tab import JsonTab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +def _current_tab(win: MainWindow) -> JsonTab: + tab = win._current_tab() + assert isinstance(tab, JsonTab) + return tab + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + QApplication.processEvents() + + +def test_cancel_open_during_parse_drops_late_success(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + initial_tabs = win.tabWidget.count() + initial_recent = list(recent_files(win)) + + def slow_parser(_path: str): + time.sleep(0.2) + return {"version": 2, "late": True}, "json" + + task_id = win._load_coordinator.open_file_async(str(doc), parser=slow_parser) + assert task_id is not None + + QTimer.singleShot(10, win._load_coordinator.cancel_current) + assert not win._load_coordinator._run_blocking(task_id) + + assert win._load_coordinator._current_task_id is None + assert win.tabWidget.count() == initial_tabs + assert list(recent_files(win)) == initial_recent + assert "Open cancelled" in win.statusBar.currentMessage() + + qtbot.wait(350) + assert win.tabWidget.count() == initial_tabs + assert list(recent_files(win)) == initial_recent + assert task_id not in win._load_coordinator._tasks + finally: + _cleanup(win) + + +def test_cancel_reload_during_parse_preserves_pre_reload_state(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"a": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + a_name = tab.model.index(0, 0, tab.model.index(0, 0, QModelIndex())) + a_value = a_name.siblingAtColumn(2) + assert tab.editing.commands.push_edit_value(a_value, 99, label="dirty") + + pre_data = tab.model.root_item.to_json() + pre_dirty = tab.io.dirty + pre_undo_count = tab.undo_stack.count() + + _write_json(doc, {"a": 2, "b": 3}) + + def slow_parser(_path: str): + time.sleep(0.2) + return {"a": 2, "b": 3}, "json" + + task_id = win._load_coordinator.reload_file_async(tab, str(doc), parser=slow_parser) + assert task_id is not None + + QTimer.singleShot(10, win._load_coordinator.cancel_current) + assert not win._load_coordinator._run_blocking(task_id) + + assert win._load_coordinator._current_task_id is None + assert tab.model.root_item.to_json() == pre_data + assert tab.io.dirty == pre_dirty + assert tab.undo_stack.count() == pre_undo_count + assert "Reload cancelled" in win.statusBar.currentMessage() + + qtbot.wait(350) + assert tab.model.root_item.to_json() == pre_data + assert tab.io.dirty == pre_dirty + assert tab.undo_stack.count() == pre_undo_count + assert task_id not in win._load_coordinator._tasks + finally: + _cleanup(win) + + +def test_late_parse_failure_after_cancel_shows_no_error_dialog(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + + def slow_failing_parser(_path: str): + time.sleep(0.2) + raise ValueError("late parse failure") + + with patch.object(QMessageBox, "critical") as mock_critical: + task_id = win._load_coordinator.open_file_async(str(doc), parser=slow_failing_parser) + assert task_id is not None + QTimer.singleShot(10, win._load_coordinator.cancel_current) + + assert not win._load_coordinator._run_blocking(task_id) + qtbot.wait(350) + + mock_critical.assert_not_called() + assert task_id not in win._load_coordinator._tasks + finally: + _cleanup(win) diff --git a/tests/test_loading_cancel_invariants.py b/tests/test_loading_cancel_invariants.py new file mode 100644 index 0000000..2c6f32d --- /dev/null +++ b/tests/test_loading_cancel_invariants.py @@ -0,0 +1,247 @@ +"""Regression invariants for loading cancellation (Plan 3, Commit 3.6).""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from unittest.mock import patch + +from PySide6.QtCore import QItemSelectionModel, QModelIndex, QTimer +from PySide6.QtWidgets import QApplication, QMessageBox + +import state.view_state as view_state +from app.loading.progress import STAGE_BUILDING_TREE +from app.main_window import MainWindow +from app.recent_files import recent_files +from documents.tab import JsonTab +from validation.schema_registry import get_schema_registry + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +def _current_tab(win: MainWindow) -> JsonTab: + tab = win._current_tab() + assert isinstance(tab, JsonTab) + return tab + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + QApplication.processEvents() + + +def _capture_reload_snapshot(tab: JsonTab) -> dict[str, object]: + return { + "model_identity": tab.model, + "root_identity": tab.model.root_item, + "data": tab.model.root_item.to_json(), + "dirty": tab.io.dirty, + "undo_count": tab.undo_stack.count(), + "clean_index": tab.undo_stack.cleanIndex(), + "view": view_state.capture_runtime_state(tab), + "schema_ref": tab.validation.schema_ref, + "schema_source": tab.validation.schema_source, + "issue_count": len(tab.validation.issue_index), + } + + +def test_open_cancel_during_build_has_no_recent_schema_or_validation_side_effects(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "open-cancel.json" + _write_json(doc, {"seed": True}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + initial_tabs = win.tabWidget.count() + initial_recent = list(recent_files(win)) + initial_schema_entries = len(get_schema_registry().all_entries()) + initial_schema_pool = len(win._schema_tab_pool._tabs_by_source) + + hook_done = [False] + + def parser(_path: str): + return {"items": [{"id": i, "value": i} for i in range(12000)]}, "json" + + def _on_stage(stage: str) -> None: + if stage != STAGE_BUILDING_TREE or hook_done[0]: + return + hook_done[0] = True + QTimer.singleShot(0, win._load_coordinator.cancel_current) + + win._load_coordinator.stage_changed.connect(_on_stage) + + with patch.object(QMessageBox, "critical") as mock_critical: + task_id = win._load_coordinator.open_file_async(str(doc), parser=parser) + assert task_id is not None + assert not win._load_coordinator._run_blocking(task_id) + + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=4000) + assert win._load_coordinator._current_task_id is None + assert win.tabWidget.count() == initial_tabs + assert list(recent_files(win)) == initial_recent + assert len(get_schema_registry().all_entries()) == initial_schema_entries + assert len(win._schema_tab_pool._tabs_by_source) == initial_schema_pool + assert all( + not isinstance(win.tabWidget.widget(i), JsonTab) + or win.tabWidget.widget(i).io.file_path != str(doc.resolve()) + for i in range(win.tabWidget.count()) + ) + assert "Open cancelled" in win.statusBar.currentMessage() + mock_critical.assert_not_called() + finally: + _cleanup(win) + + +def test_reload_cancel_preserves_dirty_undo_validation_and_view_state(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "reload-cancel.json" + _write_json(doc, {"a": 1, "items": [{"id": i, "value": i} for i in range(600)]}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + root_index = tab.model.index(0, 0, QModelIndex()) + a_name = tab.model.index(0, 0, root_index) + a_value = a_name.siblingAtColumn(2) + assert tab.editing.commands.push_edit_value(a_value, 99, label="dirty") + + items_index = tab.model.index(1, 0, root_index) + if items_index.isValid(): + tab.view.expand(items_index) + tab.view.selectionModel().setCurrentIndex( + a_value, + QItemSelectionModel.SelectionFlag.ClearAndSelect | QItemSelectionModel.SelectionFlag.Rows, + ) + QApplication.processEvents() + + before = _capture_reload_snapshot(tab) + _write_json(doc, {"a": 2, "items": [{"id": i, "value": -i} for i in range(12000)]}) + + hook_done = [False] + + def parser(_path: str): + return {"a": 2, "items": [{"id": i, "value": -i} for i in range(12000)]}, "json" + + def _on_stage(stage: str) -> None: + if stage != STAGE_BUILDING_TREE or hook_done[0]: + return + hook_done[0] = True + QTimer.singleShot(0, win._load_coordinator.cancel_current) + + win._load_coordinator.stage_changed.connect(_on_stage) + + with patch.object(QMessageBox, "critical") as mock_critical: + task_id = win._load_coordinator.reload_file_async(tab, str(doc), parser=parser) + assert task_id is not None + assert not win._load_coordinator._run_blocking(task_id) + + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=4000) + after = _capture_reload_snapshot(tab) + assert after == before + assert "Reload cancelled" in win.statusBar.currentMessage() + mock_critical.assert_not_called() + finally: + _cleanup(win) + + +def test_late_parse_success_after_cancel_is_discarded(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "late-success.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + initial_tabs = win.tabWidget.count() + initial_recent = list(recent_files(win)) + + def slow_parser(_path: str): + time.sleep(0.2) + return {"version": 2, "late": True}, "json" + + task_id = win._load_coordinator.open_file_async(str(doc), parser=slow_parser) + assert task_id is not None + + QTimer.singleShot(10, win._load_coordinator.cancel_current) + assert not win._load_coordinator._run_blocking(task_id) + + qtbot.wait(350) + assert win.tabWidget.count() == initial_tabs + assert list(recent_files(win)) == initial_recent + assert task_id not in win._load_coordinator._tasks + finally: + _cleanup(win) + + +def test_late_parse_failure_after_cancel_is_silent(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "late-failure.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + with patch.object(QMessageBox, "critical") as mock_critical: + + def slow_failing_parser(_path: str): + time.sleep(0.2) + raise ValueError("late parse failure") + + task_id = win._load_coordinator.open_file_async(str(doc), parser=slow_failing_parser) + assert task_id is not None + + QTimer.singleShot(10, win._load_coordinator.cancel_current) + assert not win._load_coordinator._run_blocking(task_id) + + qtbot.wait(350) + mock_critical.assert_not_called() + assert task_id not in win._load_coordinator._tasks + finally: + _cleanup(win) + + +def test_cancelled_task_releases_single_task_guard_for_next_open(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + first = tmp_path / "first.json" + second = tmp_path / "second.json" + _write_json(first, {"id": 1}) + _write_json(second, {"id": 2}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + + def slow_parser(_path: str): + time.sleep(0.2) + return {"id": 1}, "json" + + first_task_id = win._load_coordinator.open_file_async(str(first), parser=slow_parser) + assert first_task_id is not None + + QTimer.singleShot(10, win._load_coordinator.cancel_current) + assert not win._load_coordinator._run_blocking(first_task_id) + assert win._load_coordinator._current_task_id is None + + second_task_id = win._load_coordinator.open_file_async( + str(second), + parser=lambda _path: ({"id": 2, "ok": True}, "json"), + ) + assert second_task_id is not None + assert win._load_coordinator._run_blocking(second_task_id) + + tab = _current_tab(win) + assert tab.model.root_item.to_json() == {"id": 2, "ok": True} + finally: + _cleanup(win) diff --git a/tests/test_loading_cancel_reload_atomic.py b/tests/test_loading_cancel_reload_atomic.py new file mode 100644 index 0000000..aa0c6ec --- /dev/null +++ b/tests/test_loading_cancel_reload_atomic.py @@ -0,0 +1,211 @@ +"""Tests for atomic cancel-safe reload behavior (Plan 3, Commit 3.5).""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from unittest.mock import patch + +import pytest +from PySide6.QtCore import QItemSelectionModel, QModelIndex, QTimer +from PySide6.QtWidgets import QApplication, QMessageBox + +import state.view_state as view_state +from app.loading.progress import STAGE_BUILDING_TREE +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +def _current_tab(win: MainWindow) -> JsonTab: + tab = win._current_tab() + assert isinstance(tab, JsonTab) + return tab + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + QApplication.processEvents() + + +def _capture_reload_snapshot(tab: JsonTab) -> dict[str, object]: + return { + "model_identity": tab.model, + "root_identity": tab.model.root_item, + "data": tab.model.root_item.to_json(), + "dirty": tab.io.dirty, + "undo_count": tab.undo_stack.count(), + "clean_index": tab.undo_stack.cleanIndex(), + "view": view_state.capture_runtime_state(tab), + "schema_ref": tab.validation.schema_ref, + "schema_source": tab.validation.schema_source, + "issue_count": len(tab.validation.issue_index), + } + + +def _make_dirty_and_snapshot(tab: JsonTab) -> dict[str, object]: + root_index = tab.model.index(0, 0, QModelIndex()) + a_name = tab.model.index(0, 0, root_index) + a_value = a_name.siblingAtColumn(2) + assert tab.editing.commands.push_edit_value(a_value, 99, label="dirty") + + items_index = tab.model.index(1, 0, root_index) + if items_index.isValid(): + tab.view.expand(items_index) + + tab.view.selectionModel().setCurrentIndex( + a_value, + QItemSelectionModel.SelectionFlag.ClearAndSelect | QItemSelectionModel.SelectionFlag.Rows, + ) + + v_scroll = tab.view.verticalScrollBar() + if v_scroll.maximum() > 0: + v_scroll.setValue(min(20, v_scroll.maximum())) + + h_scroll = tab.view.horizontalScrollBar() + if h_scroll.maximum() > 0: + h_scroll.setValue(min(10, h_scroll.maximum())) + + QApplication.processEvents() + return _capture_reload_snapshot(tab) + + +@pytest.mark.parametrize("checkpoint", ["parse", "build", "before_swap"]) +def test_cancelled_reload_preserves_pre_reload_snapshot(qtbot, tmp_path, monkeypatch, checkpoint): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"a": 1, "items": [{"id": i, "value": i} for i in range(300)]}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + before = _make_dirty_and_snapshot(tab) + + _write_json(doc, {"a": 2, "items": [{"id": i, "value": -i} for i in range(1200)]}) + + hooked = [False] + + if checkpoint == "parse": + + def parser(_path: str): + time.sleep(0.2) + return {"a": 2, "items": [{"id": i, "value": -i} for i in range(1200)]}, "json" + + schedule_cancel = lambda: QTimer.singleShot(10, win._load_coordinator.cancel_current) + + elif checkpoint == "build": + + def parser(_path: str): + return {"a": 2, "items": [{"id": i, "value": -i} for i in range(12000)]}, "json" + + def _on_stage(stage: str) -> None: + if stage != STAGE_BUILDING_TREE or hooked[0]: + return + hooked[0] = True + QTimer.singleShot(0, win._load_coordinator.cancel_current) + + win._load_coordinator.stage_changed.connect(_on_stage) + schedule_cancel = lambda: None + + else: + + def parser(_path: str): + return {"a": 2, "items": [{"id": i, "value": -i} for i in range(1200)]}, "json" + + original_apply = win._load_coordinator._apply_reload + + def _cancel_before_swap(task, model): + task.token.cancel() + return original_apply(task, model) + + monkeypatch.setattr(win._load_coordinator, "_apply_reload", _cancel_before_swap) + schedule_cancel = lambda: None + + with patch.object(QMessageBox, "critical") as mock_critical: + task_id = win._load_coordinator.reload_file_async(tab, str(doc), parser=parser) + assert task_id is not None + schedule_cancel() + + assert not win._load_coordinator._run_blocking(task_id) + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=4000) + + after = _capture_reload_snapshot(tab) + assert after["model_identity"] is before["model_identity"] + assert after["root_identity"] is before["root_identity"] + assert after["data"] == before["data"] + assert after["dirty"] == before["dirty"] + assert after["undo_count"] == before["undo_count"] + assert after["clean_index"] == before["clean_index"] + assert after["schema_ref"] == before["schema_ref"] + assert after["schema_source"] == before["schema_source"] + assert after["issue_count"] == before["issue_count"] + assert after["view"] == before["view"] + assert "Reload cancelled" in win.statusBar.currentMessage() + mock_critical.assert_not_called() + finally: + _cleanup(win) + + +def test_reload_without_cancellation_preserves_existing_reload_behavior(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"a": 1, "items": [1, 2, 3]}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + model_identity = tab.model + old_root = tab.model.root_item + + _write_json(doc, {"a": 2, "items": [4, 5, 6, 7]}) + + assert win._load_coordinator.reload_file(tab, str(doc)) + assert tab.model is model_identity + assert tab.model.root_item is not old_root + assert tab.model.root_item.to_json() == {"a": 2, "items": [4, 5, 6, 7]} + assert not tab.io.dirty + finally: + _cleanup(win) + + +def test_reload_does_not_recheck_token_after_swap_begins(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + _write_json(doc, {"version": 2}) + + original_replace = tab.model.replace_root_item + + def _replace_and_cancel(root_item, *, estimated_item_count=None): + active_task = next(iter(win._load_coordinator._tasks.values())) + active_task.token.cancel() + return original_replace(root_item, estimated_item_count=estimated_item_count) + + monkeypatch.setattr(tab.model, "replace_root_item", _replace_and_cancel) + + assert win._load_coordinator.reload_file(tab, str(doc)) + assert tab.model.root_item.to_json() == {"version": 2} + assert "Reloaded" in win.statusBar.currentMessage() + finally: + _cleanup(win) diff --git a/tests/test_loading_cancellation.py b/tests/test_loading_cancellation.py new file mode 100644 index 0000000..604b452 --- /dev/null +++ b/tests/test_loading_cancellation.py @@ -0,0 +1,57 @@ +"""Tests for loading cancellation primitives (Plan 3, Commit 3.1).""" + +from __future__ import annotations + +import threading +import time + +import pytest + +from app.loading.cancellation import CancellationToken, CancelledError + + +def test_token_starts_not_cancelled() -> None: + token = CancellationToken() + assert not token.is_cancelled + token.raise_if_cancelled() + + +def test_cancel_is_idempotent() -> None: + token = CancellationToken() + token.cancel() + token.cancel() + assert token.is_cancelled + + +def test_raise_if_cancelled_raises_after_cancel() -> None: + token = CancellationToken() + token.cancel() + with pytest.raises(CancelledError): + token.raise_if_cancelled() + + +def test_cross_thread_observer_sees_cancellation() -> None: + token = CancellationToken() + observer_saw_cancel = threading.Event() + + def observe() -> None: + deadline = time.monotonic() + 1.0 + while time.monotonic() < deadline: + if token.is_cancelled: + observer_saw_cancel.set() + return + time.sleep(0.001) + + def trigger_cancel() -> None: + time.sleep(0.01) + token.cancel() + + observer = threading.Thread(target=observe) + canceller = threading.Thread(target=trigger_cancel) + observer.start() + canceller.start() + observer.join(timeout=2.0) + canceller.join(timeout=2.0) + + assert observer_saw_cancel.is_set() + assert token.is_cancelled diff --git a/tests/test_loading_first_presentation_responsive.py b/tests/test_loading_first_presentation_responsive.py new file mode 100644 index 0000000..ba55ecd --- /dev/null +++ b/tests/test_loading_first_presentation_responsive.py @@ -0,0 +1,63 @@ +from __future__ import annotations + +import settings +from app.main_window import MainWindow +from documents.controllers.view import ViewController +from documents.tab import JsonTab +from tree.model import JsonTreeModel + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +def test_above_threshold_initial_presentation_does_not_select_root(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 1) + calls = {"count": 0} + original_apply_select = ViewController._apply_select + + def _spy_apply_select(self, payload): + calls["count"] += 1 + original_apply_select(self, payload) + + monkeypatch.setattr(ViewController, "_apply_select", _spy_apply_select) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = {f"k{i}": i for i in range(300)} + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=100_000) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + qtbot.wait(50) + assert calls["count"] == 0 + finally: + _cleanup(win) + + +def test_below_threshold_initial_presentation_keeps_selection(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 100_000) + calls = {"count": 0} + original_apply_select = ViewController._apply_select + + def _spy_apply_select(self, payload): + calls["count"] += 1 + original_apply_select(self, payload) + + monkeypatch.setattr(ViewController, "_apply_select", _spy_apply_select) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = {"k": 1} + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=2) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + assert calls["count"] >= 1 + finally: + _cleanup(win) diff --git a/tests/test_loading_post_build_responsiveness.py b/tests/test_loading_post_build_responsiveness.py new file mode 100644 index 0000000..89033cd --- /dev/null +++ b/tests/test_loading_post_build_responsiveness.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +import json +from pathlib import Path + +from PySide6.QtCore import QTimer + +from app.loading.progress import ( + STAGE_APPLYING_RELOAD, + STAGE_BINDING_UI, + STAGE_COMPLETE, + STAGE_DISCOVERING_SCHEMA, + STAGE_VALIDATING_DOCUMENT, +) +from app.main_window import MainWindow +from documents.controllers.validation import TabValidationController +from documents.tab import JsonTab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +def test_large_open_keeps_event_loop_alive_after_build(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + monkeypatch.setattr(TabValidationController, "_REPAINT_BATCH_SIZE", 5) + + doc = tmp_path / "data.json" + schema = tmp_path / "data.schema.json" + schema.write_text('{"type":"object","additionalProperties":{"type":"integer"}}', encoding="utf-8") + payload = {f"k{i}": "bad" for i in range(300)} + _write_json(doc, payload) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + probe = QTimer(win) + try: + stages: list[str] = [] + ticks_post_build = {"count": 0} + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + def _on_probe() -> None: + if STAGE_BINDING_UI in stages and STAGE_COMPLETE not in stages: + ticks_post_build["count"] += 1 + + probe.timeout.connect(_on_probe) + probe.start(0) + + task_id = win._load_coordinator.open_file_async(str(doc)) + assert task_id is not None + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=6000) + + assert ticks_post_build["count"] >= 1 + assert STAGE_BINDING_UI in stages + assert STAGE_DISCOVERING_SCHEMA in stages + assert STAGE_VALIDATING_DOCUMENT in stages + assert STAGE_COMPLETE in stages + finally: + probe.stop() + _cleanup(win) + + +def test_large_reload_keeps_event_loop_alive_after_build(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + monkeypatch.setattr(TabValidationController, "_REPAINT_BATCH_SIZE", 5) + + doc = tmp_path / "data.json" + schema = tmp_path / "data.schema.json" + schema.write_text('{"type":"object","additionalProperties":{"type":"integer"}}', encoding="utf-8") + _write_json(doc, {"ok": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + probe = QTimer(win) + try: + assert win._open_path(str(doc)) + tab = win._current_tab() + assert isinstance(tab, JsonTab) + + _write_json(doc, {f"k{i}": "bad" for i in range(400)}) + + stages: list[str] = [] + ticks_post_build = {"count": 0} + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + def _on_probe() -> None: + if STAGE_APPLYING_RELOAD in stages and STAGE_COMPLETE not in stages: + ticks_post_build["count"] += 1 + + probe.timeout.connect(_on_probe) + probe.start(0) + + task_id = win._load_coordinator.reload_file_async(tab, str(doc)) + assert task_id is not None + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=6000) + + assert ticks_post_build["count"] >= 1 + assert STAGE_APPLYING_RELOAD in stages + assert STAGE_DISCOVERING_SCHEMA in stages + assert STAGE_VALIDATING_DOCUMENT in stages + assert STAGE_COMPLETE in stages + finally: + probe.stop() + _cleanup(win) diff --git a/tests/test_loading_progress_cancel_button.py b/tests/test_loading_progress_cancel_button.py new file mode 100644 index 0000000..00e05cb --- /dev/null +++ b/tests/test_loading_progress_cancel_button.py @@ -0,0 +1,84 @@ +"""Tests for loading progress cancellation button wiring (Plan 3, Commit 3.2).""" + +from __future__ import annotations + +from PySide6.QtCore import Qt +from PySide6.QtWidgets import QApplication, QPushButton + +from app.loading.cancellation import CancellationToken +from app.loading.progress_dialog import LoadingProgressDialog +from app.main_window import MainWindow + + +def test_cancel_button_sets_token_and_disables(qtbot): + dialog = LoadingProgressDialog(cancellable=True, delay_ms=0) + qtbot.addWidget(dialog) + + token = CancellationToken() + dialog.start("task-1", cancellation_token=token) + qtbot.wait(20) + QApplication.processEvents() + + button = dialog.findChildren(QPushButton)[0] + qtbot.mouseClick(button, Qt.MouseButton.LeftButton) + + assert token.is_cancelled + assert not button.isEnabled() + assert dialog._stage_label.text() == "Cancelling…" + assert dialog.isVisible() + + dialog.finish("task-1") + assert not dialog.isVisible() + + +def test_second_click_does_not_trigger_second_cancel_action(qtbot): + dialog = LoadingProgressDialog(cancellable=True, delay_ms=0) + qtbot.addWidget(dialog) + + token = CancellationToken() + cancel_count = {"value": 0} + + def on_cancel() -> None: + cancel_count["value"] += 1 + + dialog.start("task-1", cancellation_token=token, on_cancel=on_cancel) + qtbot.wait(20) + + button = dialog.findChildren(QPushButton)[0] + button.click() + button.click() + + assert token.is_cancelled + assert cancel_count["value"] == 1 + + dialog.finish("task-1") + + +def test_non_cancellable_dialog_has_no_cancel_button(qtbot): + dialog = LoadingProgressDialog(cancellable=False) + qtbot.addWidget(dialog) + assert dialog.findChildren(QPushButton) == [] + + +def test_coordinator_starts_cancellable_dialog_with_task_token(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + monkeypatch.setattr(win._load_coordinator, "_start_parse_worker", lambda *args, **kwargs: None) + task_id = win._load_coordinator.open_file_async(str(doc)) + assert task_id is not None + + task = win._load_coordinator._tasks[task_id] + dialog = win._load_coordinator._progress_dialog + assert dialog is not None + assert dialog._cancellable is True + assert dialog._cancel_token is task.token + finally: + if task_id is not None and task_id in win._load_coordinator._tasks: + win._load_coordinator._finish_progress(task_id) + win._load_coordinator._tasks.pop(task_id, None) + win.close() + win.deleteLater() diff --git a/tests/test_loading_progress_dialog.py b/tests/test_loading_progress_dialog.py new file mode 100644 index 0000000..c3dad06 --- /dev/null +++ b/tests/test_loading_progress_dialog.py @@ -0,0 +1,233 @@ +"""Tests for LoadingProgressDialog (Commit 2.2).""" + +from __future__ import annotations + +import pytest +from PySide6.QtCore import QTimer +from PySide6.QtWidgets import QApplication, QPushButton + +from app.loading.progress_dialog import LoadingProgressDialog + + +@pytest.fixture(scope="module") +def qapp(): + app = QApplication.instance() + if app is None: + app = QApplication([]) + return app + + +class TestLoadingProgressDialogFastTask: + """Tests for tasks that complete before the delay.""" + + def test_fast_task_never_shows_widget(self, qtbot): + """A task that finishes before the delay never shows the widget.""" + dialog = LoadingProgressDialog(delay_ms=100) + qtbot.addWidget(dialog) + + dialog.start("task-1") + assert not dialog.isVisible() + assert not dialog.was_shown + + # Finish before the timer fires + dialog.finish("task-1") + assert not dialog.isVisible() + assert not dialog.was_shown + + def test_fast_task_error_never_shows_widget(self, qtbot): + """A task that errors before the delay never shows the widget.""" + dialog = LoadingProgressDialog(delay_ms=100) + qtbot.addWidget(dialog) + + dialog.start("task-1") + assert not dialog.isVisible() + + # Error before the timer fires + dialog.error("task-1") + assert not dialog.isVisible() + assert not dialog.was_shown + + +class TestLoadingProgressDialogSlowTask: + """Tests for tasks that take longer than the delay.""" + + def test_slow_task_shows_widget_after_delay(self, qtbot): + """A task that takes longer than the delay shows the widget.""" + dialog = LoadingProgressDialog(delay_ms=50) + qtbot.addWidget(dialog) + + dialog.start("task-1") + assert not dialog.isVisible() + + # Wait for the timer to fire + qtbot.wait(100) + QApplication.processEvents() + + assert dialog.isVisible() + assert dialog.was_shown + + # Finish hides the widget + dialog.finish("task-1") + assert not dialog.isVisible() + + def test_slow_task_error_hides_widget(self, qtbot): + """A task that errors after showing hides the widget.""" + dialog = LoadingProgressDialog(delay_ms=50) + qtbot.addWidget(dialog) + + dialog.start("task-1") + qtbot.wait(100) + QApplication.processEvents() + + assert dialog.isVisible() + + # Error hides the widget + dialog.error("task-1") + assert not dialog.isVisible() + + +class TestLoadingProgressDialogCancelButton: + """Tests for the Cancel button presence.""" + + def test_no_cancel_button_when_not_cancellable(self, qtbot): + """No Cancel button exists when cancellable=False.""" + dialog = LoadingProgressDialog(cancellable=False) + qtbot.addWidget(dialog) + + # Find all QPushButton children + buttons = dialog.findChildren(QPushButton) + assert len(buttons) == 0 + + def test_cancel_button_exists_when_cancellable(self, qtbot): + """A Cancel button exists when cancellable=True.""" + dialog = LoadingProgressDialog(cancellable=True) + qtbot.addWidget(dialog) + + # Find all QPushButton children + buttons = dialog.findChildren(QPushButton) + assert len(buttons) == 1 + assert buttons[0].text() == "Cancel" + + +class TestLoadingProgressDialogStageAndProgress: + """Tests for stage text and progress bar updates.""" + + def test_set_stage_updates_label(self, qtbot): + """set_stage() updates the displayed stage text.""" + dialog = LoadingProgressDialog(delay_ms=50) + qtbot.addWidget(dialog) + + dialog.start("task-1") + dialog.set_stage("reading/parsing file") + + # Wait for widget to show + qtbot.wait(100) + QApplication.processEvents() + + assert dialog._stage_label.text() == "reading/parsing file" + + def test_set_progress_indeterminate(self, qtbot): + """set_progress(0, 0) sets indeterminate progress.""" + dialog = LoadingProgressDialog() + qtbot.addWidget(dialog) + + dialog.start("task-1") + dialog.set_progress(0, 0) + + assert dialog._progress_bar.maximum() == 0 + + def test_set_progress_determinate(self, qtbot): + """set_progress(done, total) sets determinate progress.""" + dialog = LoadingProgressDialog() + qtbot.addWidget(dialog) + + dialog.start("task-1") + dialog.set_progress(5, 10) + + assert dialog._progress_bar.maximum() == 10 + assert dialog._progress_bar.value() == 5 + + +class TestLoadingProgressDialogDetailRefresh: + """Tests for detail-label rendering and throttling.""" + + def test_detail_updates_only_on_refresh_tick(self, qtbot): + """Many detail updates are collapsed into timer-driven repaint.""" + dialog = LoadingProgressDialog(delay_ms=10, detail_refresh_ms=40) + qtbot.addWidget(dialog) + + dialog.start("task-1") + dialog.set_detail(1, "/a") + dialog.set_detail(50, "/b") + dialog.set_detail(1234, "/orders/1000/price") + + # No repaint before the refresh tick. + qtbot.wait(10) + assert dialog._detail_label.text() == "" + + # One refresh tick applies the latest values. + qtbot.wait(60) + assert "1,234" in dialog._detail_label.text() + assert "/orders/1000/price" in dialog._detail_label.text() + + def test_detail_resets_on_start_and_freezes_after_finish(self, qtbot): + """Detail text resets when task starts and no longer updates after finish.""" + dialog = LoadingProgressDialog(delay_ms=10, detail_refresh_ms=25) + qtbot.addWidget(dialog) + + dialog.start("task-1") + dialog.set_detail(42, "/x") + qtbot.wait(40) + assert "/x" in dialog._detail_label.text() + + dialog.finish("task-1") + frozen = dialog._detail_label.text() + + dialog.set_detail(999, "/y") + qtbot.wait(40) + assert dialog._detail_label.text() == frozen + + dialog.start("task-2") + assert dialog._detail_label.text() == "" + + +class TestLoadingProgressDialogTaskIdMatching: + """Tests for task ID matching on finish/error.""" + + def test_finish_with_wrong_task_id_is_noop(self, qtbot): + """finish() with a non-matching task_id does not hide the widget.""" + dialog = LoadingProgressDialog(delay_ms=50) + qtbot.addWidget(dialog) + + dialog.start("task-1") + qtbot.wait(100) + QApplication.processEvents() + + assert dialog.isVisible() + + # Finish with wrong task_id + dialog.finish("task-2") + assert dialog.isVisible() + + # Finish with correct task_id + dialog.finish("task-1") + assert not dialog.isVisible() + + def test_error_with_wrong_task_id_is_noop(self, qtbot): + """error() with a non-matching task_id does not hide the widget.""" + dialog = LoadingProgressDialog(delay_ms=50) + qtbot.addWidget(dialog) + + dialog.start("task-1") + qtbot.wait(100) + QApplication.processEvents() + + assert dialog.isVisible() + + # Error with wrong task_id + dialog.error("task-2") + assert dialog.isVisible() + + # Error with correct task_id + dialog.error("task-1") + assert not dialog.isVisible() diff --git a/tests/test_loading_progress_end_to_end.py b/tests/test_loading_progress_end_to_end.py new file mode 100644 index 0000000..f3904f6 --- /dev/null +++ b/tests/test_loading_progress_end_to_end.py @@ -0,0 +1,254 @@ +"""End-to-end tests for loading progress widget (Commit 2.7).""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from unittest.mock import patch + +import pytest +from PySide6.QtCore import QTimer +from PySide6.QtWidgets import QApplication, QFileDialog, QMessageBox + +from app.loading.progress import ( + STAGE_APPLYING_RELOAD, + STAGE_BINDING_UI, + STAGE_COMPLETE, + STAGE_DISCOVERING_SCHEMA, + STAGE_READING_PARSING, + STAGE_VALIDATING_DOCUMENT, +) +from app.loading.progress_dialog import LoadingProgressDialog +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _current_tab(win: MainWindow) -> JsonTab: + tab = win._current_tab() + assert isinstance(tab, JsonTab) + return tab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +@pytest.fixture(scope="module") +def qapp(): + app = QApplication.instance() + if app is None: + app = QApplication([]) + return app + + +class TestLoadingProgressEndToEnd: + """End-to-end tests for loading progress widget.""" + + def test_fast_open_shows_no_widget(self, qtbot, tmp_path, monkeypatch): + """A fast open operation does not show the progress widget.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + # Open the file + assert win._open_path(str(doc)) + + # The progress dialog should not have been shown + # (it's created but never made visible for fast operations) + if win._load_coordinator._progress_dialog is not None: + assert not win._load_coordinator._progress_dialog.was_shown + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_open_emits_all_stages(self, qtbot, tmp_path, monkeypatch): + """Opening a file emits all expected stages.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + stages = [] + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + assert win._open_path(str(doc)) + + # Verify all stages were emitted + assert STAGE_READING_PARSING in stages + assert STAGE_BINDING_UI in stages + assert STAGE_DISCOVERING_SCHEMA in stages + assert STAGE_VALIDATING_DOCUMENT in stages + assert STAGE_COMPLETE in stages + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_reload_emits_all_stages(self, qtbot, tmp_path, monkeypatch): + """Reloading a file emits all expected stages.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Update the file + _write_json(doc, {"version": 2}) + + stages = [] + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + assert win._load_coordinator.reload_file(tab, str(doc)) + + # Verify all stages were emitted (reload uses APPLYING_RELOAD instead of BINDING_UI) + assert STAGE_READING_PARSING in stages + assert STAGE_APPLYING_RELOAD in stages + assert STAGE_DISCOVERING_SCHEMA in stages + assert STAGE_VALIDATING_DOCUMENT in stages + assert STAGE_COMPLETE in stages + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_open_error_hides_widget(self, qtbot, tmp_path, monkeypatch): + """An error during open hides the progress widget.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "bad.json" + doc.write_text("{invalid json", encoding="utf-8") + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + with patch.object(QMessageBox, "critical"): + assert not win._open_path(str(doc)) + + # The progress dialog should not be visible after error + if win._load_coordinator._progress_dialog is not None: + assert not win._load_coordinator._progress_dialog.isVisible() + finally: + win.close() + win.deleteLater() + + def test_reload_error_hides_widget(self, qtbot, tmp_path, monkeypatch): + """An error during reload hides the progress widget.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"valid": True}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Corrupt the file + doc.write_text("{invalid json", encoding="utf-8") + + with patch.object(QMessageBox, "critical"): + assert not win._load_coordinator.reload_file(tab, str(doc)) + + # The progress dialog should not be visible after error + if win._load_coordinator._progress_dialog is not None: + assert not win._load_coordinator._progress_dialog.isVisible() + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_async_open_returns_immediately_and_shows_progress(self, qtbot, tmp_path, monkeypatch): + """A slow async open keeps the event loop alive and shows progress.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + progress = LoadingProgressDialog(win, delay_ms=50, detail_refresh_ms=20) + qtbot.addWidget(progress) + win._load_coordinator._progress_dialog = progress + + timer_count = [0] + + def on_timer(): + timer_count[0] += 1 + + timer = QTimer(win) + timer.timeout.connect(on_timer) + timer.start(20) + + payload = { + "items": [ + { + "id": i, + "value": f"item_{i}", + } + for i in range(300) + ] + } + + def slow_parser(_path: str): + time.sleep(0.2) + return payload, "json" + + try: + task_id = win._load_coordinator.open_file_async(str(doc), parser=slow_parser) + assert task_id is not None + assert win.tabWidget.count() == 0 + + qtbot.waitUntil(lambda: progress.was_shown, timeout=1000) + assert timer_count[0] >= 2 + + qtbot.waitUntil(lambda: progress._detail_label.text() != "", timeout=2000) + assert "Processed" in progress._detail_label.text() + assert "/" in progress._detail_label.text() + + qtbot.waitUntil(lambda: win.tabWidget.count() == 1, timeout=2000) + qtbot.waitUntil(lambda: not progress.isVisible(), timeout=1000) + tab = _current_tab(win) + loaded = tab.model.root_item.to_json() + assert isinstance(loaded, dict) + assert "items" in loaded + assert len(loaded["items"]) == 300 + finally: + timer.stop() + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_open_file_dialog_starts_async_open(self, qtbot, tmp_path, monkeypatch): + """The native open dialog returns before loading starts.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + with patch.object(QFileDialog, "getOpenFileName", return_value=(str(doc), "")): + with patch.object(win._load_coordinator, "open_file_async") as mock_open_async: + win.open_file_dialog() + mock_open_async.assert_called_once_with(str(doc)) + finally: + win.close() + win.deleteLater() diff --git a/tests/test_loading_progress_events.py b/tests/test_loading_progress_events.py new file mode 100644 index 0000000..929a79f --- /dev/null +++ b/tests/test_loading_progress_events.py @@ -0,0 +1,185 @@ +"""Tests for progress reporting protocol (Commit 2.4).""" + +from __future__ import annotations + +import pytest + +from app.loading.progress import ( + OPEN_STAGES, + RELOAD_STAGES, + STAGE_APPLYING_RELOAD, + STAGE_BINDING_UI, + STAGE_BUILDING_TREE, + STAGE_COMPLETE, + STAGE_DECODING_AFFIXES, + STAGE_DISCOVERING_SCHEMA, + STAGE_READING_PARSING, + STAGE_VALIDATING_DOCUMENT, + NullProgressReporter, + ProgressEvent, + ProgressReporter, +) + + +class TestProgressEvent: + """Tests for ProgressEvent dataclass.""" + + def test_progress_event_creation(self): + """ProgressEvent can be created with all fields.""" + event = ProgressEvent(task_id="task-1", stage="reading/parsing file", done=5, total=10) + assert event.task_id == "task-1" + assert event.stage == "reading/parsing file" + assert event.done == 5 + assert event.total == 10 + + def test_progress_event_defaults(self): + """ProgressEvent defaults done and total to 0.""" + event = ProgressEvent(task_id="task-1", stage="reading/parsing file") + assert event.done == 0 + assert event.total == 0 + assert event.processed == 0 + assert event.path == "" + + def test_progress_event_detail_fields(self): + """ProgressEvent carries detail payload fields.""" + event = ProgressEvent( + task_id="task-1", + stage="building item tree", + done=0, + total=0, + processed=1234, + path="/orders/1000/price", + ) + assert event.processed == 1234 + assert event.path == "/orders/1000/price" + + def test_progress_event_is_indeterminate(self): + """is_indeterminate returns True when total is 0.""" + event = ProgressEvent(task_id="task-1", stage="reading/parsing file", done=0, total=0) + assert event.is_indeterminate + + def test_progress_event_is_determinate(self): + """is_indeterminate returns False when total > 0.""" + event = ProgressEvent(task_id="task-1", stage="reading/parsing file", done=5, total=10) + assert not event.is_indeterminate + + def test_progress_event_is_frozen(self): + """ProgressEvent is immutable (frozen dataclass).""" + event = ProgressEvent(task_id="task-1", stage="reading/parsing file") + with pytest.raises(AttributeError): # allow: testing frozen dataclass immutability + event.stage = "new stage" # type: ignore + + +class TestStageConstants: + """Tests for stage constants.""" + + def test_open_stages_order(self): + """OPEN_STAGES contains stages in the correct order.""" + assert OPEN_STAGES == ( + STAGE_READING_PARSING, + STAGE_DECODING_AFFIXES, + STAGE_BUILDING_TREE, + STAGE_BINDING_UI, + STAGE_DISCOVERING_SCHEMA, + STAGE_VALIDATING_DOCUMENT, + STAGE_COMPLETE, + ) + + def test_reload_stages_order(self): + """RELOAD_STAGES contains stages in the correct order.""" + assert RELOAD_STAGES == ( + STAGE_READING_PARSING, + STAGE_DECODING_AFFIXES, + STAGE_BUILDING_TREE, + STAGE_APPLYING_RELOAD, + STAGE_DISCOVERING_SCHEMA, + STAGE_VALIDATING_DOCUMENT, + STAGE_COMPLETE, + ) + + def test_reload_uses_applying_reload_not_binding_ui(self): + """RELOAD_STAGES uses 'applying reload' instead of 'binding UI'.""" + assert STAGE_BINDING_UI not in RELOAD_STAGES + assert STAGE_APPLYING_RELOAD in RELOAD_STAGES + + +class TestProgressReporterProtocol: + """Tests for ProgressReporter protocol.""" + + def test_null_reporter_implements_protocol(self): + """NullProgressReporter implements ProgressReporter protocol.""" + reporter: ProgressReporter = NullProgressReporter() + assert isinstance(reporter, ProgressReporter) + + def test_null_reporter_accepts_stage(self): + """NullProgressReporter.stage() does not raise.""" + reporter = NullProgressReporter() + reporter.stage("reading/parsing file") + + def test_null_reporter_accepts_tick(self): + """NullProgressReporter.tick() does not raise.""" + reporter = NullProgressReporter() + reporter.tick(5, 10) + reporter.tick(0, 0) + + def test_null_reporter_accepts_detail(self): + """NullProgressReporter.detail() does not raise.""" + reporter = NullProgressReporter() + reporter.detail(321, "/a/1") + reporter.detail(0, "") + + +class TestStageTracking: + """Tests for tracking stage progression.""" + + def test_stages_emitted_in_order(self): + """Stages can be tracked in the expected order.""" + observed_stages: list[str] = [] + observed_details: list[tuple[int, str]] = [] + + class TrackingReporter: + def stage(self, name: str) -> None: + observed_stages.append(name) + + def tick(self, done: int, total: int) -> None: + pass + + def detail(self, processed: int, path: str) -> None: + observed_details.append((processed, path)) + + reporter = TrackingReporter() + + # Simulate emitting stages in order + for stage_name in OPEN_STAGES: + reporter.stage(stage_name) + reporter.detail(1, "/example") + + assert observed_stages == list(OPEN_STAGES) + assert len(observed_details) == len(OPEN_STAGES) + + def test_tick_values_are_valid(self): + """tick(done, total) values satisfy 0 <= done <= total.""" + tick_calls: list[tuple[int, int]] = [] + + class TrackingReporter: + def stage(self, name: str) -> None: + pass + + def tick(self, done: int, total: int) -> None: + tick_calls.append((done, total)) + + def detail(self, processed: int, path: str) -> None: + pass + + reporter = TrackingReporter() + + # Valid tick calls + reporter.tick(0, 0) # Indeterminate + reporter.tick(0, 10) + reporter.tick(5, 10) + reporter.tick(10, 10) + + for done, total in tick_calls: + assert 0 <= done + if total > 0: + assert done <= total diff --git a/tests/test_loading_reload_post_build_responsive.py b/tests/test_loading_reload_post_build_responsive.py new file mode 100644 index 0000000..f53fdd7 --- /dev/null +++ b/tests/test_loading_reload_post_build_responsive.py @@ -0,0 +1,81 @@ +from __future__ import annotations + +import json +from pathlib import Path + +from PySide6.QtCore import QTimer + +from app.loading.progress import STAGE_APPLYING_RELOAD, STAGE_COMPLETE +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +def test_reload_does_not_call_root_data_before_apply(qtbot, tmp_path, monkeypatch): + doc = tmp_path / "data.json" + _write_json(doc, {"a": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = win._current_tab() + assert isinstance(tab, JsonTab) + + monkeypatch.setattr( + tab, + "root_data", + lambda: (_ for _ in ()).throw(AssertionError("reload must not snapshot root_data() before apply")), + ) + + _write_json(doc, {"a": 2, "b": 3}) + assert win._load_coordinator.reload_file(tab, str(doc)) + assert tab.model.root_item.to_json() == {"a": 2, "b": 3} + finally: + _cleanup(win) + + +def test_reload_apply_and_finalize_yields_event_loop_turn(qtbot, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"a": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + probe = QTimer(win) + try: + assert win._open_path(str(doc)) + tab = win._current_tab() + assert isinstance(tab, JsonTab) + _write_json(doc, {f"k{i}": "x" for i in range(300)}) + + stages: list[str] = [] + ticks_after_apply = {"count": 0} + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + def _on_probe() -> None: + if STAGE_APPLYING_RELOAD in stages and STAGE_COMPLETE not in stages: + ticks_after_apply["count"] += 1 + + probe.timeout.connect(_on_probe) + probe.start(0) + + task_id = win._load_coordinator.reload_file_async(tab, str(doc)) + assert task_id is not None + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=5000) + assert ticks_after_apply["count"] >= 1 + finally: + probe.stop() + _cleanup(win) diff --git a/tests/test_loading_reload_swap.py b/tests/test_loading_reload_swap.py new file mode 100644 index 0000000..9ec9340 --- /dev/null +++ b/tests/test_loading_reload_swap.py @@ -0,0 +1,320 @@ +"""Tests for reload build-then-swap behavior (Commit 2.8).""" + +from __future__ import annotations + +import json +import time +from pathlib import Path + +import pytest +from PySide6.QtCore import QModelIndex, QTimer +from PySide6.QtWidgets import QApplication + +from app.loading.progress import ( + STAGE_APPLYING_RELOAD, + STAGE_BINDING_UI, + STAGE_BUILDING_TREE, + STAGE_COMPLETE, + STAGE_DISCOVERING_SCHEMA, + STAGE_READING_PARSING, + STAGE_VALIDATING_DOCUMENT, +) +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _current_tab(win: MainWindow) -> JsonTab: + tab = win._current_tab() + assert isinstance(tab, JsonTab) + return tab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +@pytest.fixture(scope="module") +def qapp(): + app = QApplication.instance() + if app is None: + app = QApplication([]) + return app + + +class TestReloadSwapStages: + """Tests for reload stage emission.""" + + def test_reload_uses_applying_reload_stage(self, qtbot, tmp_path, monkeypatch): + """Reload emits 'applying reload' instead of 'binding UI'.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Update the file + _write_json(doc, {"version": 2}) + + stages = [] + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + assert win._load_coordinator.reload_file(tab, str(doc)) + + # Verify 'applying reload' is used instead of 'binding UI' + assert STAGE_APPLYING_RELOAD in stages + assert STAGE_BINDING_UI not in stages + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_reload_emits_all_stages_in_order(self, qtbot, tmp_path, monkeypatch): + """Reload emits all stages in the correct order.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + _write_json(doc, {"version": 2}) + + stages = [] + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + assert win._load_coordinator.reload_file(tab, str(doc)) + + # Verify order + assert stages.index(STAGE_READING_PARSING) < stages.index(STAGE_APPLYING_RELOAD) + assert stages.index(STAGE_APPLYING_RELOAD) < stages.index(STAGE_DISCOVERING_SCHEMA) + assert stages.index(STAGE_DISCOVERING_SCHEMA) < stages.index(STAGE_VALIDATING_DOCUMENT) + assert stages.index(STAGE_VALIDATING_DOCUMENT) < stages.index(STAGE_COMPLETE) + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +class TestReloadSwapDataIntegrity: + """Tests for reload data integrity.""" + + def test_reload_updates_data(self, qtbot, tmp_path, monkeypatch): + """Reload updates the tab data correctly.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1, "items": [1, 2, 3]}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Verify initial data + assert tab.model.root_item.to_json() == {"version": 1, "items": [1, 2, 3]} + + # Update the file + _write_json(doc, {"version": 2, "items": [4, 5, 6, 7]}) + + assert win._load_coordinator.reload_file(tab, str(doc)) + + # Verify updated data + assert tab.model.root_item.to_json() == {"version": 2, "items": [4, 5, 6, 7]} + assert tab.model.estimated_item_count is not None + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_reload_preserves_model_identity_and_replaces_root(self, qtbot, tmp_path, monkeypatch): + """Reload keeps the same model object but swaps in a new root item.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1, "nested": {"value": 1}}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + model_identity = tab.model + old_root = tab.model.root_item + + _write_json(doc, {"version": 2, "nested": {"value": 2}, "new": [1, 2, 3]}) + + assert win._load_coordinator.reload_file(tab, str(doc)) + + assert tab.model is model_identity + assert tab.model.root_item is not old_root + assert tab.model.root_item.to_json() == {"version": 2, "nested": {"value": 2}, "new": [1, 2, 3]} + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_async_reload_keeps_old_state_until_applying_stage(self, qtbot, tmp_path, monkeypatch): + """Before `applying reload`, data/dirty/undo remain unchanged while event loop keeps ticking.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + a_name = tab.model.index(0, 0, tab.model.index(0, 0, QModelIndex())) + a_value = a_name.siblingAtColumn(2) + assert tab.editing.commands.push_edit_value(a_value, 99, label="dirty") + dirty_snapshot = tab.model.root_item.to_json() + undo_snapshot = tab.undo_stack.count() + assert tab.io.dirty + + _write_json(doc, {"version": 2, "items": list(range(200))}) + + stages: list[str] = [] + win._load_coordinator.stage_changed.connect(lambda stage: stages.append(stage)) + + sampled_data: list[dict] = [] + sampled_undo: list[int] = [] + sampled_dirty: list[bool] = [] + tick_count = [0] + + probe = QTimer(win) + + def _probe() -> None: + tick_count[0] += 1 + if STAGE_APPLYING_RELOAD in stages: + return + sampled_data.append(tab.model.root_item.to_json()) + sampled_undo.append(tab.undo_stack.count()) + sampled_dirty.append(tab.io.dirty) + + probe.timeout.connect(_probe) + probe.start(10) + + def slow_parser(_path: str): + time.sleep(0.2) + return {"version": 2, "items": list(range(200))}, "json" + + task_id = win._load_coordinator.reload_file_async(tab, str(doc), parser=slow_parser) + assert task_id is not None + + qtbot.waitUntil(lambda: STAGE_BUILDING_TREE in stages, timeout=2000) + qtbot.waitUntil(lambda: STAGE_APPLYING_RELOAD in stages, timeout=4000) + qtbot.waitUntil(lambda: task_id not in win._load_coordinator._tasks, timeout=4000) + + assert sampled_data + assert sampled_undo + assert sampled_dirty + assert all(sample == dirty_snapshot for sample in sampled_data) + assert all(sample == undo_snapshot for sample in sampled_undo) + assert all(sample is True for sample in sampled_dirty) + assert tick_count[0] >= 2 + + assert tab.model.root_item.to_json() == {"version": 2, "items": list(range(200))} + assert tab.undo_stack.count() == 0 + assert not tab.io.dirty + finally: + probe.stop() + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_reload_clears_undo_stack_on_change(self, qtbot, tmp_path, monkeypatch): + """Reload clears the undo stack when data changes.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Make an edit to add to undo stack + a_name = tab.model.index(0, 0, tab.model.index(0, 0, QModelIndex())) + a_value = a_name.siblingAtColumn(2) + assert tab.editing.commands.push_edit_value(a_value, 99, label="edit") + assert tab.undo_stack.count() > 0 + + # Update the file + _write_json(doc, {"version": 2}) + + assert win._load_coordinator.reload_file(tab, str(doc)) + + # Undo stack should be cleared + assert tab.undo_stack.count() == 0 + assert not tab.io.dirty + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_reload_preserves_clean_state_on_no_change(self, qtbot, tmp_path, monkeypatch): + """Reload preserves clean state when data doesn't change.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Don't change the file + assert win._load_coordinator.reload_file(tab, str(doc)) + + # Should still be clean + assert not tab.io.dirty + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +class TestReloadSwapErrorHandling: + """Tests for reload error handling.""" + + def test_reload_error_preserves_old_data(self, qtbot, tmp_path, monkeypatch): + """Reload error preserves the old tab data.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + doc = tmp_path / "data.json" + _write_json(doc, {"version": 1}) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Verify initial data + old_data = tab.model.root_item.to_json() + assert old_data == {"version": 1} + + # Corrupt the file + doc.write_text("{invalid json", encoding="utf-8") + + from unittest.mock import patch + + from PySide6.QtWidgets import QMessageBox + + with patch.object(QMessageBox, "critical"): + assert not win._load_coordinator.reload_file(tab, str(doc)) + + # Old data should be preserved + assert tab.model.root_item.to_json() == old_data + finally: + tab.undo_stack.setClean() + win.close() + win.deleteLater() diff --git a/tests/test_loading_validation_data_path.py b/tests/test_loading_validation_data_path.py new file mode 100644 index 0000000..004a7ba --- /dev/null +++ b/tests/test_loading_validation_data_path.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from documents.tab import JsonTab +from validation.schema_source import SchemaRef + + +def test_loading_validation_uses_supplied_parsed_data_not_tree_snapshot(qtbot, monkeypatch): + tab = JsonTab(lambda *_: None, data={"value": "oops"}, show_root=True) + qtbot.addWidget(tab) + + schema = { + "type": "object", + "properties": {"value": {"type": "integer"}}, + "required": ["value"], + } + tab.validation.set_schema(SchemaRef(path=None, inline=schema, origin="manual"), revalidate=False) + + monkeypatch.setattr( + tab.model.root_item, + "to_json", + lambda: (_ for _ in ()).throw(AssertionError("loading validation must not call model snapshot")), + ) + + tab.validation.revalidate_loading_data({"value": "oops"}) + assert len(tab.validation.issue_index) == 1 diff --git a/tests/test_loading_validation_progress.py b/tests/test_loading_validation_progress.py new file mode 100644 index 0000000..b038a70 --- /dev/null +++ b/tests/test_loading_validation_progress.py @@ -0,0 +1,167 @@ +"""Tests for schema/validation progress stages (Commit 2.6).""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from PySide6.QtWidgets import QApplication + +from app.loading.coordinator import LoadCoordinator +from app.loading.progress import ( + STAGE_COMPLETE, + STAGE_DISCOVERING_SCHEMA, + STAGE_VALIDATING_DOCUMENT, + NullProgressReporter, +) +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _current_tab(win: MainWindow) -> JsonTab: + tab = win._current_tab() + assert isinstance(tab, JsonTab) + return tab + + +def _write_json(path: Path, payload) -> None: + path.write_text(json.dumps(payload), encoding="utf-8") + + +@pytest.fixture(scope="module") +def qapp(): + app = QApplication.instance() + if app is None: + app = QApplication([]) + return app + + +class TestCoordinatorStageSignals: + """Tests for coordinator stage signal emission.""" + + def test_coordinator_emits_stage_changed_signal(self, qtbot, tmp_path, monkeypatch): + """Coordinator emits stage_changed signal when stages change.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + stages = [] + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + # Open a file to get a tab + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Run schema discovery and validation + win._load_coordinator.run_schema_discovery_and_validation(tab) + + # Verify stages were emitted + assert STAGE_DISCOVERING_SCHEMA in stages + assert STAGE_VALIDATING_DOCUMENT in stages + assert STAGE_COMPLETE in stages + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_coordinator_stages_in_correct_order(self, qtbot, tmp_path, monkeypatch): + """Coordinator emits stages in the correct order.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + stages = [] + win._load_coordinator.stage_changed.connect(lambda s: stages.append(s)) + + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + assert win._open_path(str(doc)) + tab = _current_tab(win) + + win._load_coordinator.run_schema_discovery_and_validation(tab) + + # Verify order + assert stages.index(STAGE_DISCOVERING_SCHEMA) < stages.index(STAGE_VALIDATING_DOCUMENT) + assert stages.index(STAGE_VALIDATING_DOCUMENT) < stages.index(STAGE_COMPLETE) + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +class TestCoordinatorProgressReporter: + """Tests for coordinator progress reporter integration.""" + + def test_coordinator_notifies_reporter(self, qtbot, tmp_path, monkeypatch): + """Coordinator notifies the progress reporter of stages.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + reported_stages = [] + + class TrackingReporter: + def stage(self, name: str) -> None: + reported_stages.append(name) + + def tick(self, done: int, total: int) -> None: + pass + + reporter = TrackingReporter() + win._load_coordinator.set_reporter(reporter) + + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + assert win._open_path(str(doc)) + tab = _current_tab(win) + + win._load_coordinator.run_schema_discovery_and_validation(tab) + + # Verify reporter received stages + assert STAGE_DISCOVERING_SCHEMA in reported_stages + assert STAGE_VALIDATING_DOCUMENT in reported_stages + assert STAGE_COMPLETE in reported_stages + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + def test_coordinator_works_without_reporter(self, qtbot, tmp_path, monkeypatch): + """Coordinator works correctly without a reporter set.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "xdg")) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + # Don't set a reporter + assert win._load_coordinator._reporter is None + + doc = tmp_path / "data.json" + _write_json(doc, {"key": "value"}) + assert win._open_path(str(doc)) + tab = _current_tab(win) + + # Should not raise + win._load_coordinator.run_schema_discovery_and_validation(tab) + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() diff --git a/tests/test_loading_worker_thread.py b/tests/test_loading_worker_thread.py new file mode 100644 index 0000000..2b1cd1b --- /dev/null +++ b/tests/test_loading_worker_thread.py @@ -0,0 +1,237 @@ +"""Tests for ParseWorker thread (Commit 2.3).""" + +from __future__ import annotations + +import time +from typing import Any + +import pytest +from PySide6.QtCore import QObject, Qt, QThread, QTimer, Slot +from PySide6.QtWidgets import QApplication + +from app.loading.worker import ParseWorker, start_parse_worker +from io_formats.load import _AFFIX_JSON_TYPES, _decode_number_affixes +from settings import NUMBER_AFFIX_MAX_LEN +from tree.types import parse_json_type +from units.number_affix import parse_number_affix + + +@pytest.fixture(scope="module") +def qapp(): + app = QApplication.instance() + if app is None: + app = QApplication([]) + return app + + +def _slow_parser(delay_ms: int, result: tuple[Any, str]): + """Create a parser that delays for the specified milliseconds.""" + + def parser(path: str) -> tuple[Any, str]: + time.sleep(delay_ms / 1000.0) + return result + + return parser + + +def _failing_parser(error_msg: str): + """Create a parser that raises an exception.""" + + def parser(path: str) -> tuple[Any, str]: + raise ValueError(error_msg) + + return parser + + +def _decode_number_affixes_recursive_legacy(value: Any) -> Any: + if isinstance(value, str): + if parse_json_type(value) in _AFFIX_JSON_TYPES: + parsed = parse_number_affix(value, max_affix_len=NUMBER_AFFIX_MAX_LEN) + if parsed is not None: + return parsed + return value + if isinstance(value, list): + return [_decode_number_affixes_recursive_legacy(v) for v in value] + if isinstance(value, dict): + return {k: _decode_number_affixes_recursive_legacy(v) for k, v in value.items()} + return value + + +class TestParseWorkerSuccess: + """Tests for successful parsing.""" + + def test_worker_emits_finished_with_result(self, qtbot): + """Worker emits finished signal with parse result.""" + expected_result = ({"key": "value"}, "json") + parser = _slow_parser(10, expected_result) + + thread, worker = start_parse_worker("/fake/path.json", parser=parser) + + results = [] + worker.finished.connect(lambda r: results.append(r)) + + # Wait for thread to finish + thread.wait(2000) + + assert len(results) == 1 + assert results[0] == expected_result + + # Cleanup + thread.deleteLater() + worker.deleteLater() + + def test_worker_emits_stage_signals(self, qtbot): + """Worker emits stage signals during parsing.""" + expected_result = ({"key": "value"}, "json") + parser = _slow_parser(10, expected_result) + + thread, worker = start_parse_worker("/fake/path.json", parser=parser) + + stages = [] + worker.stage.connect(lambda s: stages.append(s)) + + thread.wait(2000) + + assert "reading/parsing file" in stages + assert "decoding number affixes" in stages + + thread.deleteLater() + worker.deleteLater() + + def test_worker_emits_decode_detail_on_gui_thread(self, qtbot, tmp_path): + """Decode detail events are delivered on the GUI thread via queued connection.""" + doc = tmp_path / "data.json" + doc.write_text('{"orders":[{"price":"1kg"},{"price":"2kg"}],"meta":{"x":"3kg"}}', encoding="utf-8") + + thread, worker = start_parse_worker(str(doc), parser=None) + + details: list[tuple[int, str]] = [] + callback_threads: list[QThread] = [] + + class Collector(QObject): + @Slot(int, str) + def on_detail(self, processed: int, path: str) -> None: + details.append((processed, path)) + callback_threads.append(QThread.currentThread()) + + collector = Collector() + gui_thread = QApplication.instance().thread() + worker.detail.connect(collector.on_detail, Qt.ConnectionType.QueuedConnection) + + qtbot.waitUntil(lambda: not thread.isRunning(), timeout=2000) + qtbot.wait(20) + + assert details + assert any(path.startswith("/orders") for _, path in details) + assert all(t is gui_thread for t in callback_threads) + + thread.deleteLater() + worker.deleteLater() + + +class TestParseWorkerFailure: + """Tests for failed parsing.""" + + def test_worker_emits_failed_on_exception(self, qtbot): + """Worker emits failed signal when parser raises.""" + parser = _failing_parser("Test error message") + + thread, worker = start_parse_worker("/fake/path.json", parser=parser) + + errors = [] + worker.failed.connect(lambda e: errors.append(e)) + + thread.wait(2000) + + assert len(errors) == 1 + error_type, error_msg = errors[0] + assert error_type == "ValueError" + assert "Test error message" in error_msg + + thread.deleteLater() + worker.deleteLater() + + +class TestParseWorkerEventLoop: + """Tests for GUI event loop responsiveness during parsing.""" + + def test_event_loop_processes_events_during_slow_parse(self, qtbot): + """GUI event loop processes timers during slow parsing.""" + # Use a slow parser that takes 200ms + expected_result = ({"key": "value"}, "json") + parser = _slow_parser(200, expected_result) + + # Track timer firings + timer_count = [0] + + def on_timer(): + timer_count[0] += 1 + + # Create a timer that fires every 50ms + timer = QTimer() + timer.timeout.connect(on_timer) + timer.start(50) + + thread, worker = start_parse_worker("/fake/path.json", parser=parser) + + # Use qtbot.wait which processes events, unlike thread.wait + qtbot.wait(300) + + timer.stop() + + # Timer should have fired at least twice during the 200ms parse + assert timer_count[0] >= 2, f"Timer only fired {timer_count[0]} times" + + # Ensure thread finished + thread.wait(1000) + + thread.deleteLater() + worker.deleteLater() + + +class TestParseWorkerThreadCleanup: + """Tests for thread cleanup after parsing.""" + + def test_thread_quits_after_success(self, qtbot): + """Thread quits after successful parsing.""" + expected_result = ({"key": "value"}, "json") + parser = _slow_parser(10, expected_result) + + thread, worker = start_parse_worker("/fake/path.json", parser=parser) + + # Use qtbot.wait to process events so queued signals are delivered + qtbot.wait(100) + + assert not thread.isRunning() + + thread.deleteLater() + worker.deleteLater() + + +class TestDecodeOutputParity: + """Tests ensuring iterative decode matches legacy recursive semantics.""" + + def test_iterative_decode_matches_legacy_recursive_output(self): + """Decode output remains byte-for-byte equivalent for nested structures.""" + data = { + "root": ["1kg", "plain", {"money": "$12", "nested": ["2m", {"v": "3kg"}], "slash/key": "4kg"}], + "other": {"a": "b", "arr": [1, "5kg", False]}, + } + + actual = _decode_number_affixes(data) + expected = _decode_number_affixes_recursive_legacy(data) + assert actual == expected + + def test_thread_quits_after_failure(self, qtbot): + """Thread quits after failed parsing.""" + parser = _failing_parser("Test error") + + thread, worker = start_parse_worker("/fake/path.json", parser=parser) + + # Use qtbot.wait to process events so queued signals are delivered + qtbot.wait(100) + + assert not thread.isRunning() + + thread.deleteLater() + worker.deleteLater() diff --git a/tests/test_mpq2py.py b/tests/test_mpq2py.py index fb669ed..922a8a7 100644 --- a/tests/test_mpq2py.py +++ b/tests/test_mpq2py.py @@ -2,6 +2,7 @@ import yaml from gmpy2 import mpq +from core.raw_numeric import REASON_NON_FINITE, RawNumericValue from mpq2py import MpqSafeDumper, MpqSafeLoader, mpq_json_default json_floats = """ @@ -39,3 +40,16 @@ def test_mpq_with_yaml(): res = yaml.dump(data, Dumper=MpqSafeDumper, sort_keys=False) assert res == yaml_floats + + +def test_yaml_special_floats_become_raw_numeric_values() -> None: + data = yaml.load("pinf: .inf\nninf: -.inf\nnanv: .nan\n", Loader=MpqSafeLoader) + + for key in ("pinf", "ninf", "nanv"): + assert isinstance(data[key], RawNumericValue) + assert data[key].reason == REASON_NON_FINITE + + # The original literal is preserved exactly. + assert data["pinf"].raw == ".inf" + assert data["ninf"].raw == "-.inf" + assert data["nanv"].raw == ".nan" diff --git a/tests/test_mpq_overflow_protection.py b/tests/test_mpq_overflow_protection.py new file mode 100644 index 0000000..25a918f --- /dev/null +++ b/tests/test_mpq_overflow_protection.py @@ -0,0 +1,95 @@ +from PySide6.QtCore import QModelIndex, Qt + +from core.raw_numeric import REASON_INVALID_FORMAT, RawNumericValue +from io_formats import SAVE_FORMAT_JSON, dump_text, load_file_with_format +from io_formats.load import _safe_parse_float +from tree.model import JsonTreeModel +from tree.types import JsonType, parse_json_type +from units.number_affix import parse_number_affix + +_UNSAFE_NUMERIC = "31e-327018450730" + + +def test_affix_parser_rejects_unsafe_exponent_literal() -> None: + assert parse_number_affix(f"$ {_UNSAFE_NUMERIC}") is None + assert parse_number_affix(f"{_UNSAFE_NUMERIC} m/s") is None + + +def test_json_loader_preserves_unsafe_float_literal_as_raw(tmp_path) -> None: + path = tmp_path / "unsafe.json" + path.write_text( + '{"float": 31e-327018450730, "string": "$ 31e-327018450730"}\n', + encoding="utf-8", + ) + + loaded, fmt = load_file_with_format(str(path)) + + assert fmt == SAVE_FORMAT_JSON + assert isinstance(loaded["float"], RawNumericValue) + assert loaded["float"].raw == _UNSAFE_NUMERIC + assert loaded["string"] == "$ 31e-327018450730" + assert parse_json_type(loaded["float"]) is JsonType.RAW_FLOAT + + # The raw token is a valid JSON number, so it round-trips byte-for-byte. + dumped = dump_text(str(path), loaded, save_format=SAVE_FORMAT_JSON) + assert '"float": 31e-327018450730' in dumped + + +def test_raw_numeric_value_is_inline_editable() -> None: + model = JsonTreeModel({"float": RawNumericValue(_UNSAFE_NUMERIC)}) + + item = model.get_item(model.index(0, 0, QModelIndex())) + value_index = model.index(0, 2, QModelIndex()) + + assert item.json_type is JsonType.RAW_FLOAT + assert item.editable is True + assert (model.flags(value_index) & Qt.ItemFlag.ItemIsEditable) == Qt.ItemFlag.ItemIsEditable + + +def test_editing_raw_numeric_into_supported_number_converts_to_float() -> None: + model = JsonTreeModel({"float": RawNumericValue(_UNSAFE_NUMERIC)}) + item = model.get_item(model.index(0, 0, QModelIndex())) + value_index = model.index(0, 2, QModelIndex()) + + assert model.setData(value_index, "1.5", Qt.ItemDataRole.EditRole) is True + assert item.json_type is JsonType.FLOAT + assert not isinstance(item.value, RawNumericValue) + + +def test_editing_raw_numeric_unchanged_preserves_raw() -> None: + model = JsonTreeModel({"float": RawNumericValue(_UNSAFE_NUMERIC)}) + item = model.get_item(model.index(0, 0, QModelIndex())) + value_index = model.index(0, 2, QModelIndex()) + + assert model.setData(value_index, _UNSAFE_NUMERIC, Qt.ItemDataRole.EditRole) is True + assert item.json_type is JsonType.RAW_FLOAT + assert isinstance(item.value, RawNumericValue) + assert item.value.raw == _UNSAFE_NUMERIC + + +def test_editing_raw_numeric_into_unsupported_but_valid_shape_stays_raw() -> None: + model = JsonTreeModel({"float": RawNumericValue(_UNSAFE_NUMERIC)}) + item = model.get_item(model.index(0, 0, QModelIndex())) + value_index = model.index(0, 2, QModelIndex()) + + # Still underflowing, but matches the narrow edit grammar → kept raw. + assert model.setData(value_index, "9e-99999", Qt.ItemDataRole.EditRole) is True + assert item.json_type is JsonType.RAW_FLOAT + assert isinstance(item.value, RawNumericValue) + assert item.value.raw == "9e-99999" + + +def test_editing_raw_numeric_with_invalid_text_is_rejected() -> None: + model = JsonTreeModel({"float": RawNumericValue(_UNSAFE_NUMERIC)}) + item = model.get_item(model.index(0, 0, QModelIndex())) + value_index = model.index(0, 2, QModelIndex()) + + assert model.setData(value_index, "not-a-number", Qt.ItemDataRole.EditRole) is False + assert item.json_type is JsonType.RAW_FLOAT + assert item.value.raw == _UNSAFE_NUMERIC + + +def test_safe_parse_float_rejects_invalid_literal_before_mpq() -> None: + parsed = _safe_parse_float("not-a-float") + assert isinstance(parsed, RawNumericValue) + assert parsed.reason == REASON_INVALID_FORMAT diff --git a/tests/test_raw_numeric_values.py b/tests/test_raw_numeric_values.py new file mode 100644 index 0000000..f273fa9 --- /dev/null +++ b/tests/test_raw_numeric_values.py @@ -0,0 +1,127 @@ +"""Tests for unsupported raw numeric values (RawNumericValue / RAW_FLOAT).""" + +from __future__ import annotations + +import pytest +import yaml +from PySide6.QtCore import QModelIndex + +from core.raw_numeric import REASON_NON_FINITE, REASON_UNDERFLOW, RawNumericValue +from io_formats import SAVE_FORMAT_JSON, SAVE_FORMAT_YAML, dump_text +from mpq2py import MpqSafeLoader +from tree.model import JsonTreeModel +from tree.types import USER_SELECTABLE_TYPES, JsonType, canonical_type, parse_json_type +from validation._sanitize import to_jsonschema_input + +_UNSAFE = "31e-327018450730" + + +# --------------------------------------------------------------------------- +# Type metadata +# --------------------------------------------------------------------------- + + +def test_raw_numeric_infers_raw_float_pseudo_type() -> None: + assert parse_json_type(RawNumericValue(_UNSAFE)) is JsonType.RAW_FLOAT + + +def test_raw_float_is_not_user_selectable() -> None: + assert JsonType.RAW_FLOAT not in USER_SELECTABLE_TYPES + + +def test_raw_float_canonical_parent_is_float() -> None: + assert canonical_type(JsonType.RAW_FLOAT) is JsonType.FLOAT + + +# --------------------------------------------------------------------------- +# Saving / serialization +# --------------------------------------------------------------------------- + + +def test_yaml_roundtrip_preserves_raw_numeric_exactly() -> None: + data = {"a": RawNumericValue(_UNSAFE), "b": RawNumericValue(".inf")} + text = dump_text("f.yaml", data, save_format=SAVE_FORMAT_YAML) + + reloaded = yaml.load(text, Loader=MpqSafeLoader) + assert isinstance(reloaded["a"], RawNumericValue) + assert reloaded["a"].raw == _UNSAFE + assert isinstance(reloaded["b"], RawNumericValue) + assert reloaded["b"].raw == ".inf" + + +def test_json_dump_of_json_safe_raw_roundtrips() -> None: + data = {"x": RawNumericValue(_UNSAFE, source_syntax="json")} + text = dump_text("f.json", data, save_format=SAVE_FORMAT_JSON) + assert f'"x": {_UNSAFE}' in text + + +def test_json_dump_of_non_json_safe_raw_raises_controlled_error() -> None: + data = {"x": RawNumericValue(".inf", reason=REASON_NON_FINITE, source_syntax="yaml")} + with pytest.raises(ValueError): + dump_text("f.json", data, save_format=SAVE_FORMAT_JSON) + + +# --------------------------------------------------------------------------- +# Clipboard / drag-drop metadata fidelity +# --------------------------------------------------------------------------- + + +def test_clipboard_mime_roundtrip_preserves_raw_numeric(qtbot) -> None: + from tree_actions.clipboard import build_tree_mime, entries_from_mime + + model = JsonTreeModel({"x": RawNumericValue(".inf", reason=REASON_NON_FINITE, source_syntax="yaml")}) + row0 = model.index(0, 0, QModelIndex()) + + mime = build_tree_mime(model, [row0]) + assert mime is not None + + entries = entries_from_mime(mime) + assert entries is not None + value = entries[0]["value"] + assert isinstance(value, RawNumericValue) + assert value.raw == ".inf" + assert value.reason == REASON_NON_FINITE + + +# --------------------------------------------------------------------------- +# Validation +# --------------------------------------------------------------------------- + + +def test_validation_sanitize_converts_raw_numeric_to_string() -> None: + out = to_jsonschema_input({"x": RawNumericValue(_UNSAFE)}) + assert out["x"] == _UNSAFE + + +# --------------------------------------------------------------------------- +# Editor +# --------------------------------------------------------------------------- + + +def test_raw_float_editor_is_text_and_warns_once(qtbot) -> None: + from PySide6.QtWidgets import QStyleOptionViewItem, QWidget + + from delegates.edit_context import DefaultEditContext + from delegates.value import ValueDelegate + from editors.inline.raw_numeric_line import RawNumericLineEdit + + class _RecordingContext(DefaultEditContext): + def __init__(self) -> None: + super().__init__() + self.warn_reasons: list[str] = [] + + def warn_raw_numeric_edit(self, parent, *, reason: str) -> None: # type: ignore[override] + self.warn_reasons.append(reason) + + model = JsonTreeModel({"x": RawNumericValue(_UNSAFE, reason=REASON_UNDERFLOW)}) + ctx = _RecordingContext() + delegate = ValueDelegate(edit_context=ctx) + + parent = QWidget() + qtbot.addWidget(parent) + idx = model.index(0, 2, QModelIndex()) + + editor = delegate.createEditor(parent, QStyleOptionViewItem(), idx) + + assert isinstance(editor, RawNumericLineEdit) + assert ctx.warn_reasons == [REASON_UNDERFLOW] diff --git a/tests/test_safe_mpq.py b/tests/test_safe_mpq.py new file mode 100644 index 0000000..7ace4b6 --- /dev/null +++ b/tests/test_safe_mpq.py @@ -0,0 +1,80 @@ +from gmpy2 import mpq + +from core.raw_numeric import ( + REASON_INVALID_FORMAT, + REASON_NON_FINITE, + REASON_OVERFLOW, + REASON_PRECISION_LIMIT, + REASON_UNDERFLOW, +) +from core.safe_mpq import mpq_literal_is_safe, parse_mpq, safe_decimal_from_text, safe_mpq_from_text + + +def test_safe_mpq_accepts_regular_decimal_and_rational_literals() -> None: + assert safe_mpq_from_text("3.14") == mpq("3.14") + assert safe_mpq_from_text("10/4") == mpq("5/2") + assert safe_mpq_from_text("1e309") == mpq("1e309") + + +def test_safe_mpq_rejects_underflow_and_overflow_exponents() -> None: + assert safe_mpq_from_text("31e-327018450730") is None + assert safe_mpq_from_text("1e327018450730") is None + + +def test_safe_mpq_rejects_too_many_significant_digits_but_allows_trailing_zeros() -> None: + assert safe_mpq_from_text("9" * 4300) == mpq("9" * 4300) + assert safe_mpq_from_text("9" * 4301) is None + assert safe_mpq_from_text("1." + "0" * 5000) == mpq(1) + + +def test_safe_mpq_rational_requires_integer_sides_and_nonzero_denominator() -> None: + assert safe_mpq_from_text("1.5/2") is None + assert safe_mpq_from_text("1/0") is None + assert safe_mpq_from_text("1/2/3") is None + + +def test_safe_decimal_and_literal_is_safe_helpers() -> None: + assert safe_decimal_from_text("3.14") is not None + assert safe_decimal_from_text("not-a-number") is None + assert safe_decimal_from_text("nan") is None + assert mpq_literal_is_safe("7/3") + assert not mpq_literal_is_safe("31e-327018450730") + + +def test_parse_mpq_reports_success_and_value() -> None: + result = parse_mpq("3.14") + assert result.ok is True + assert result.value == mpq("3.14") + assert result.reason is None + + +def test_parse_mpq_reports_overflow_reason() -> None: + result = parse_mpq("1e327018450730") + assert result.value is None + assert result.reason == REASON_OVERFLOW + + +def test_parse_mpq_reports_underflow_reason() -> None: + result = parse_mpq("31e-327018450730") + assert result.value is None + assert result.reason == REASON_UNDERFLOW + + +def test_parse_mpq_reports_non_finite_reason() -> None: + for literal in (".inf", "-.inf", ".nan", "inf", "nan", "Infinity"): + result = parse_mpq(literal) + assert result.value is None + assert result.reason == REASON_NON_FINITE + + +def test_parse_mpq_reports_precision_limit_reason() -> None: + result = parse_mpq("9" * 4301) + assert result.value is None + assert result.reason == REASON_PRECISION_LIMIT + + +def test_parse_mpq_reports_invalid_format_reason() -> None: + for literal in ("not-a-number", "", "1/0", "1.5/2"): + result = parse_mpq(literal) + assert result.value is None + assert result.reason == REASON_INVALID_FORMAT diff --git a/tests/test_tab_close_progress.py b/tests/test_tab_close_progress.py new file mode 100644 index 0000000..17f045a --- /dev/null +++ b/tests/test_tab_close_progress.py @@ -0,0 +1,184 @@ +"""Tests for tab-close progress ownership (Plan 4, Commit 4.3).""" + +from __future__ import annotations + +import time + +import pytest +from PySide6.QtCore import QTimer +from PySide6.QtWidgets import QApplication + +import settings +import state.view_state as view_state +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def test_normal_close_finishes_without_showing_progress(qtbot, monkeypatch): + monkeypatch.setattr(settings, "CLOSE_PROGRESS_DELAY_MS", 200) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + win.create_new_file() + assert win.tabWidget.count() == 1 + + win.close_tab(0) + + dialog = win._tab_lifecycle._close_progress_dialog + assert dialog is not None + assert not dialog.was_shown + assert not dialog.isVisible() + assert win.tabWidget.count() == 0 + finally: + win.close() + win.deleteLater() + + +def test_slow_close_shows_progress_with_stage_text(qtbot, monkeypatch): + monkeypatch.setattr(settings, "CLOSE_PROGRESS_DELAY_MS", 20) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + win.create_new_file() + assert win.tabWidget.count() == 1 + tab = win.tabWidget.widget(0) + + original_root_data = tab.root_data + + def slow_root_data(): + end = time.monotonic() + 0.15 + while time.monotonic() < end: + QApplication.processEvents() + time.sleep(0.005) + return original_root_data() + + monkeypatch.setattr(tab, "root_data", slow_root_data) + + observed_stages: list[str] = [] + original_set_stage = win._tab_lifecycle._set_close_stage + + def record_stage(dialog, stage: str) -> None: + observed_stages.append(stage) + original_set_stage(dialog, stage) + + monkeypatch.setattr(win._tab_lifecycle, "_set_close_stage", record_stage) + + QTimer.singleShot(0, lambda: win.close_tab(0)) + qtbot.waitUntil(lambda: win.tabWidget.count() == 0, timeout=2000) + + dialog = win._tab_lifecycle._close_progress_dialog + assert dialog is not None + assert dialog.was_shown + assert "snapshot" in observed_stages + assert "saving view state" in observed_stages + assert "removing tab" in observed_stages + assert "destroying tab" in observed_stages + assert not dialog.isVisible() + finally: + win.close() + win.deleteLater() + + +def test_close_error_hides_progress_and_restores_cursor(qtbot, monkeypatch): + monkeypatch.setattr(settings, "CLOSE_PROGRESS_DELAY_MS", 0) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + win.create_new_file() + assert win.tabWidget.count() == 1 + + original_save = view_state.save + + def boom(_tab): + raise RuntimeError("close failed") + + monkeypatch.setattr(view_state, "save", boom) + + with pytest.raises(RuntimeError, match="close failed"): + win.close_tab(0) + + dialog = win._tab_lifecycle._close_progress_dialog + assert dialog is not None + assert not dialog.isVisible() + assert QApplication.overrideCursor() is None + finally: + monkeypatch.setattr(view_state, "save", original_save) + while QApplication.overrideCursor() is not None: + QApplication.restoreOverrideCursor() + win.close() + win.deleteLater() + + +def test_reopen_preserves_snapshot_for_normal_size_tab(qtbot, monkeypatch): + monkeypatch.setattr(settings, "CLOSE_PROGRESS_DELAY_MS", 0) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + monkeypatch.setattr(win, "_confirm_close", lambda _widget: True) + try: + tab = win._add_tab(data={"alpha": 1, "beta": {"x": True}}) + assert isinstance(tab, JsonTab) + assert win.tabWidget.count() == 1 + + win.close_tab(0) + assert win.tabWidget.count() == 0 + + win.reopen_closed_tab() + assert win.tabWidget.count() == 1 + + reopened = win._current_tab() + assert isinstance(reopened, JsonTab) + assert reopened.model.root_item.to_json() == {"alpha": 1, "beta": {"x": True}} + + dialog = win._tab_lifecycle._close_progress_dialog + assert dialog is not None + assert not dialog.isVisible() + assert QApplication.overrideCursor() is None + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + while win.tabWidget.count() > 0: + win.close_tab(0) + while QApplication.overrideCursor() is not None: + QApplication.restoreOverrideCursor() + win.close() + win.deleteLater() + + +def test_repeated_close_reopen_cycles_leave_no_orphan_progress(qtbot, monkeypatch): + monkeypatch.setattr(settings, "CLOSE_PROGRESS_DELAY_MS", 0) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + monkeypatch.setattr(win, "_confirm_close", lambda _widget: True) + try: + tab = win._add_tab(data={"seed": 1}) + assert isinstance(tab, JsonTab) + assert win.tabWidget.count() == 1 + + for _ in range(3): + win.close_tab(0) + assert win.tabWidget.count() == 0 + win.reopen_closed_tab() + assert win.tabWidget.count() == 1 + + dialog = win._tab_lifecycle._close_progress_dialog + assert dialog is not None + assert not dialog.isVisible() + assert QApplication.overrideCursor() is None + finally: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + while win.tabWidget.count() > 0: + win.close_tab(0) + while QApplication.overrideCursor() is not None: + QApplication.restoreOverrideCursor() + win.close() + win.deleteLater() diff --git a/tests/test_tab_close_progress_responsive.py b/tests/test_tab_close_progress_responsive.py new file mode 100644 index 0000000..a28d6b9 --- /dev/null +++ b/tests/test_tab_close_progress_responsive.py @@ -0,0 +1,54 @@ +"""Responsiveness tests for large tab close progress (Plan 4, Commit 4.4).""" + +from __future__ import annotations + +import time + +from PySide6.QtCore import QTimer +from PySide6.QtWidgets import QApplication + +import settings +from app.main_window import MainWindow + + +def test_large_close_processes_events_while_progress_visible(qtbot, monkeypatch): + monkeypatch.setattr(settings, "CLOSE_PROGRESS_DELAY_MS", 10) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + win.create_new_file() + assert win.tabWidget.count() == 1 + tab = win.tabWidget.widget(0) + tab.io.file_path = "/tmp/close-progress-responsive.json" + + def slow_expanded_paths_iter(): + for i in range(3000): + if i % 64 == 0: + time.sleep(0.001) + yield (i,) + + monkeypatch.setattr(tab.editing.move, "iter_expanded_paths", slow_expanded_paths_iter) + + tick_count = 0 + + def on_tick() -> None: + nonlocal tick_count + tick_count += 1 + + timer = QTimer(win) + timer.setInterval(5) + timer.timeout.connect(on_tick) + timer.start() + + QTimer.singleShot(0, lambda: win.close_tab(0)) + qtbot.waitUntil(lambda: win.tabWidget.count() == 0, timeout=4000) + + dialog = win._tab_lifecycle._close_progress_dialog + assert dialog is not None + assert dialog.was_shown + assert not dialog.isVisible() + assert tick_count > 0 + finally: + win.close() + win.deleteLater() diff --git a/tests/test_tab_lifecycle.py b/tests/test_tab_lifecycle.py index 8e4a5a3..8b2680b 100644 --- a/tests/test_tab_lifecycle.py +++ b/tests/test_tab_lifecycle.py @@ -4,11 +4,16 @@ import json -from PySide6.QtCore import QModelIndex +from PySide6.QtCore import QModelIndex, QTimer from PySide6.QtWidgets import QApplication, QMessageBox +import documents.composition.init as tab_init +import settings from app.main_window import MainWindow +from documents.controllers.view import ViewController from documents.tab import JsonTab +from tree.model import JsonTreeModel +from units.number_affix import AffixKind, NumberAffix def _current_tab(win: MainWindow) -> JsonTab | None: @@ -279,3 +284,174 @@ def test_reopen_after_discard_does_not_resurrect_dirty_data(qtbot, monkeypatch, assert data == {"original": True} # NOT the dirty value finally: _cleanup(win) + + +def test_deferred_first_presentation_yields_event_loop_turn(qtbot): + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + turn_fired = {"value": False} + presented = {"value": False} + + QTimer.singleShot(0, lambda: turn_fired.__setitem__("value", True)) + tab = win._tab_lifecycle.add_tab( + data={"k": 1}, + defer_first_presentation=True, + on_presentation_complete=lambda _tab: presented.__setitem__("value", True), + ) + assert tab is not None + assert not presented["value"] + + qtbot.waitUntil(lambda: presented["value"], timeout=1000) + assert turn_fired["value"] + finally: + _cleanup(win) + + +def test_large_open_skips_expand_all_and_expands_root_only(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 1) + calls_expand_all: list[int] = [] + calls_expand: list[tuple[int, ...]] = [] + monkeypatch.setattr(ViewController, "request_expand_all", lambda self: calls_expand_all.append(1)) + monkeypatch.setattr(ViewController, "request_expand", lambda self, path: calls_expand.append(path)) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = {"k": 1} + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=100) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + assert calls_expand_all == [] + assert () in calls_expand + finally: + _cleanup(win) + + +def test_small_open_keeps_expand_all_behavior(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 1000) + calls_expand_all: list[int] = [] + monkeypatch.setattr(ViewController, "request_expand_all", lambda self: calls_expand_all.append(1)) + + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = {"k": 1} + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=2) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + assert calls_expand_all + finally: + _cleanup(win) + + +def test_large_open_uses_bounded_column_sizing(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 1) + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = {"k": 1} + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=100) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + + calls: list[int] = [] + monkeypatch.setattr(tab.view, "resizeColumnToContents", lambda col: calls.append(col)) + tab.appearance.resize_key_columns(force=True) + assert calls == [] + finally: + _cleanup(win) + + +def test_small_open_keeps_content_based_column_sizing(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 1000) + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = {"k": 1} + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=2) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + + calls: list[int] = [] + monkeypatch.setattr(tab.view, "resizeColumnToContents", lambda col: calls.append(col)) + tab.appearance.resize_key_columns(force=True) + assert 0 in calls and 1 in calls + finally: + _cleanup(win) + + +def test_large_open_defers_affix_mru_bootstrap_but_populates(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 1) + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = { + "price": NumberAffix(kind=AffixKind.CURRENCY, affix="$", space=False, number=7), + } + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=100) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + + assert tab.affix_mru.items(AffixKind.CURRENCY) == [] + qtbot.waitUntil(lambda: "$" in tab.affix_mru.items(AffixKind.CURRENCY), timeout=1000) + finally: + _cleanup(win) + + +def test_small_open_bootstraps_affix_mru_inline(qtbot, monkeypatch): + monkeypatch.setattr(settings, "LOADING_AUTO_EXPAND_MAX_NODES", 1000) + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = { + "price": NumberAffix(kind=AffixKind.CURRENCY, affix="$", space=False, number=7), + } + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=2) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt) + assert tab is not None + assert "$" in tab.affix_mru.items(AffixKind.CURRENCY) + finally: + _cleanup(win) + + +def test_regular_tab_creation_initializes_validation_inline(qtbot, monkeypatch): + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + calls = {"count": 0} + + original_init = tab_init.init_validation_state + + def _spy_init_validation_state(tab, model_data): + calls["count"] += 1 + original_init(tab, model_data) + + monkeypatch.setattr(tab_init, "init_validation_state", _spy_init_validation_state) + try: + tab = win._add_tab(data={"k": 1}) + assert tab is not None + assert calls["count"] == 1 + finally: + _cleanup(win) + + +def test_loading_owned_add_tab_defers_bootstrap_validation_init(qtbot, monkeypatch): + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + calls = {"count": 0} + + original_init = tab_init.init_validation_state + + def _spy_init_validation_state(tab, model_data): + calls["count"] += 1 + original_init(tab, model_data) + + monkeypatch.setattr(tab_init, "init_validation_state", _spy_init_validation_state) + try: + payload = {"k": 1} + prebuilt = JsonTreeModel(payload, show_root=True, estimated_item_count=100) + tab = win._add_tab(data=payload, prebuilt_model=prebuilt, defer_validation_init=True) + assert tab is not None + assert calls["count"] == 0 + finally: + _cleanup(win) diff --git a/tests/test_validation_no_schema_fast_path.py b/tests/test_validation_no_schema_fast_path.py new file mode 100644 index 0000000..4a4a365 --- /dev/null +++ b/tests/test_validation_no_schema_fast_path.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +from app.main_window import MainWindow +from documents.tab import JsonTab + + +def _cleanup(win: MainWindow) -> None: + for i in range(win.tabWidget.count()): + maybe_tab = win.tabWidget.widget(i) + if isinstance(maybe_tab, JsonTab): + maybe_tab.undo_stack.setClean() + win.close() + win.deleteLater() + + +def test_loading_validation_with_no_schema_does_not_snapshot_tree(qtbot, monkeypatch): + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + payload = {f"k{i}": i for i in range(500)} + tab = win._add_tab(data=payload, defer_validation_init=True) + assert tab is not None + + monkeypatch.setattr( + tab.model.root_item, + "to_json", + lambda: (_ for _ in ()).throw(AssertionError("to_json() must not run for no-schema loading validation")), + ) + + win._load_coordinator.run_schema_discovery_and_validation(tab, parsed_data=payload) + assert len(tab.validation.issue_index) == 0 + finally: + _cleanup(win) + + +def test_clearing_schema_from_empty_state_emits_no_recursive_repaint(qtbot): + win = MainWindow(yaml_filename="") + qtbot.addWidget(win) + try: + tab = win._add_tab(data={"a": 1}) + assert tab is not None + + emitted = {"count": 0} + tab.model.dataChanged.connect(lambda *_args: emitted.__setitem__("count", emitted["count"] + 1)) + + tab.validation.clear_schema() + assert emitted["count"] == 0 + finally: + _cleanup(win) diff --git a/tests/test_validation_repaint_bounds.py b/tests/test_validation_repaint_bounds.py new file mode 100644 index 0000000..27aba62 --- /dev/null +++ b/tests/test_validation_repaint_bounds.py @@ -0,0 +1,78 @@ +from __future__ import annotations + +from PySide6.QtCore import QModelIndex, QTimer + +from documents.tab import JsonTab +from tree.model_roles import VALIDATION_SEVERITY_ROLE +from validation.schema_source import SchemaRef + + +def _make_tab(qtbot, data: dict, *, show_root: bool = True) -> JsonTab: + tab = JsonTab(lambda *_: None, data=data, show_root=show_root) + qtbot.addWidget(tab) + return tab + + +def test_empty_validation_emits_no_data_changed(qtbot): + tab = _make_tab(qtbot, {"a": 1}) + calls = {"count": 0} + + def _on_changed(_top, _bottom, roles): + if VALIDATION_SEVERITY_ROLE in roles: + calls["count"] += 1 + + tab.model.dataChanged.connect(_on_changed) + tab.validation.clear_schema() + assert calls["count"] == 0 + + +def test_single_issue_repaints_exact_path_and_ancestors(qtbot): + tab = _make_tab(qtbot, {"parent": {"leaf": "bad"}}, show_root=True) + repainted_paths: set[tuple[int, ...]] = set() + + def _on_changed(top, _bottom, roles): + if VALIDATION_SEVERITY_ROLE not in roles: + return + source = top.siblingAtColumn(0) + repainted_paths.add(tab.model._index_path(source)) + + tab.model.dataChanged.connect(_on_changed) + + schema = { + "type": "object", + "properties": { + "parent": { + "type": "object", + "properties": {"leaf": {"type": "integer"}}, + } + }, + } + tab.validation.set_schema(SchemaRef(path=None, inline=schema, origin="manual")) + + assert (0, 0) in repainted_paths + assert (0,) in repainted_paths + + +def test_large_issue_sets_emit_repaints_in_batches_with_event_turn(qtbot, monkeypatch): + data = {f"k{i}": "x" for i in range(80)} + tab = _make_tab(qtbot, data) + monkeypatch.setattr(tab.validation, "_REPAINT_BATCH_SIZE", 5) + + timer_turn = {"fired": False} + emissions = {"count": 0} + + def _on_changed(_top, _bottom, roles): + if VALIDATION_SEVERITY_ROLE in roles: + emissions["count"] += 1 + + tab.model.dataChanged.connect(_on_changed) + QTimer.singleShot(0, lambda: timer_turn.__setitem__("fired", True)) + + schema = { + "type": "object", + "additionalProperties": {"type": "integer"}, + } + tab.validation.set_schema(SchemaRef(path=None, inline=schema, origin="manual")) + + qtbot.waitUntil(lambda: emissions["count"] >= 10, timeout=1500) + assert timer_turn["fired"] diff --git a/tests/test_value_delegate_theme.py b/tests/test_value_delegate_theme.py index 7486421..deb4071 100644 --- a/tests/test_value_delegate_theme.py +++ b/tests/test_value_delegate_theme.py @@ -5,6 +5,7 @@ from PySide6.QtCore import QModelIndex, Qt from PySide6.QtWidgets import QApplication, QStyle, QStyleOptionViewItem +from core.raw_numeric import RawNumericValue from delegates.type_delegate import JsonTypeDelegate from delegates.value import ValueDelegate from documents.tab import JsonTab @@ -36,13 +37,16 @@ def _index_for_type(json_type: JsonType): # user-selectable via the type combobox. To put an item into one of those # states we seed the value with content that text_pseudotype_for collapses # to the desired pseudo while the item is already in its parent shape. - pseudo_seeds: dict[JsonType, tuple[JsonType, str]] = { + pseudo_seeds: dict[JsonType, tuple[JsonType, object]] = { JsonType.EMPTY_STRING: (JsonType.STRING, ""), JsonType.EMPTY_MULTILINE: (JsonType.MULTILINE, ""), JsonType.WS_STRING: (JsonType.STRING, " "), JsonType.WS_UNICODE: (JsonType.STRING, " \u00a0 "), JsonType.WS_MULTILINE: (JsonType.MULTILINE, " \n "), JsonType.WS_TEXT: (JsonType.MULTILINE, " \u00a0\n\u3000 "), + # RAW_FLOAT is a content-derived pseudo numeric type: seed it by setting + # a raw, unsupported numeric literal on a FLOAT-shaped item. + JsonType.RAW_FLOAT: (JsonType.FLOAT, RawNumericValue("31e-327018450730")), } if json_type in pseudo_seeds: parent, seed = pseudo_seeds[json_type] diff --git a/themes/_defaults.py b/themes/_defaults.py index f84d952..c47441c 100644 --- a/themes/_defaults.py +++ b/themes/_defaults.py @@ -55,6 +55,7 @@ def _theme( _LIGHT_TYPES: dict[JsonType, TypeStyle] = { JsonType.INTEGER: TypeStyle(fg=_c("#268bd2")), JsonType.FLOAT: TypeStyle(fg=_c("#1f8f89")), + JsonType.RAW_FLOAT: TypeStyle(fg=_c("#cb4b16"), italic=True), JsonType.PERCENT: TypeStyle(fg=_c("#1f8f89"), italic=True), JsonType.INTEGER_CURRENCY: TypeStyle(fg=_c("#268bd2"), italic=True), JsonType.INTEGER_UNITS: TypeStyle(fg=_c("#268bd2"), italic=True), @@ -85,6 +86,7 @@ def _theme( _DARK_TYPES: dict[JsonType, TypeStyle] = { JsonType.INTEGER: TypeStyle(fg=_c("#7aa2f7")), JsonType.FLOAT: TypeStyle(fg=_c("#73daca")), + JsonType.RAW_FLOAT: TypeStyle(fg=_c("#ff9e64"), italic=True), JsonType.PERCENT: TypeStyle(fg=_c("#73daca"), italic=True), JsonType.INTEGER_CURRENCY: TypeStyle(fg=_c("#7aa2f7"), italic=True), JsonType.INTEGER_UNITS: TypeStyle(fg=_c("#7aa2f7"), italic=True), diff --git a/themes/loader.py b/themes/loader.py index 1f9e6a9..7e21864 100644 --- a/themes/loader.py +++ b/themes/loader.py @@ -17,6 +17,7 @@ _TYPE_KEYS: dict[str, JsonType] = { "integer": JsonType.INTEGER, "float": JsonType.FLOAT, + "raw_float": JsonType.RAW_FLOAT, "percent": JsonType.PERCENT, "int_currency": JsonType.INTEGER_CURRENCY, "int_units": JsonType.INTEGER_UNITS, diff --git a/tree/filter_proxy.py b/tree/filter_proxy.py index c7f4f3e..2ee8990 100644 --- a/tree/filter_proxy.py +++ b/tree/filter_proxy.py @@ -18,6 +18,7 @@ def __init__(self, parent=None): def set_filter_text(self, text: str) -> None: # Keep a normalized plain-string needle; no regex semantics. self._needle = (text or "").strip().casefold() + self.setRecursiveFilteringEnabled(bool(self._needle)) self.invalidate() def filterAcceptsRow(self, src_row: int, src_parent: QModelIndex) -> bool: # type: ignore[override] diff --git a/tree/inference_limits.py b/tree/inference_limits.py new file mode 100644 index 0000000..ab192e0 --- /dev/null +++ b/tree/inference_limits.py @@ -0,0 +1,89 @@ +"""Length-gate helpers for expensive type inference (Plan 1). + +Each ``*_inference_allowed`` helper returns ``True`` when the text is short +enough to run the corresponding expensive branch, or when ``allow_expensive`` +is ``True`` (explicit user type change bypasses the gate). + +``base64_syntax_valid`` is a content-based syntax check (not a length gate): +it returns ``True`` when the text has valid base64 structure (length mod 4 +and alphabet-only characters). No bypass flag is needed because this is a +correctness check, not a performance gate. + +``format_preview_decode_allowed`` caps paint-time decode work for display +previews. No bypass flag: previews are always capped. + +This module imports only ``settings`` and standard-library modules. +""" + +import re + +from settings import ( + FORMAT_PREVIEW_DECODE_LIMIT_BYTES, + INFERENCE_MAX_AFFIX_CHARS, + INFERENCE_MAX_COLOR_CHARS, + INFERENCE_MAX_DATETIME_CHARS, +) + +# Base64 alphabet regex: A-Z, a-z, 0-9, +, /, with optional = padding. +# The * quantifier allows empty strings (handled by len % 4 check). +_B64_ALPHABET_RE = re.compile(r"^[A-Za-z0-9+/]*={0,2}$") + + +def datetime_inference_allowed(text: str, *, allow_expensive: bool = False) -> bool: + """Return True if datetime regex work is allowed for *text*. + + Returns True when ``len(text) <= INFERENCE_MAX_DATETIME_CHARS`` or + ``allow_expensive`` is True. + """ + if allow_expensive: + return True + return len(text) <= INFERENCE_MAX_DATETIME_CHARS + + +def affix_inference_allowed(text: str, *, allow_expensive: bool = False) -> bool: + """Return True if number-affix regex work is allowed for *text*. + + Returns True when ``len(text) <= INFERENCE_MAX_AFFIX_CHARS`` or + ``allow_expensive`` is True. + """ + if allow_expensive: + return True + return len(text) <= INFERENCE_MAX_AFFIX_CHARS + + +def color_inference_allowed(text: str, *, allow_expensive: bool = False) -> bool: + """Return True if color regex work is allowed for *text*. + + Returns True when ``len(text) <= INFERENCE_MAX_COLOR_CHARS`` or + ``allow_expensive`` is True. + """ + if allow_expensive: + return True + return len(text) <= INFERENCE_MAX_COLOR_CHARS + + +def base64_syntax_valid(text: str) -> bool: + """Return True if *text* has valid base64 syntax. + + Checks: + 1. ``len(text) % 4 == 0`` (base64 encoding always produces length + divisible by 4). + 2. Text matches the base64 alphabet regex + ``^[A-Za-z0-9+/]*={0,2}$`` (no whitespace or other invalid chars). + + This is a content validation, not a length gate. No bypass flag. + """ + if not text: + return False + if len(text) % 4 != 0: + return False + return _B64_ALPHABET_RE.fullmatch(text) is not None + + +def format_preview_decode_allowed(byte_count: int) -> bool: + """Return True if decode work is allowed for a display preview. + + Returns True when ``byte_count <= FORMAT_PREVIEW_DECODE_LIMIT_BYTES``. + No bypass flag: previews are always capped. + """ + return byte_count <= FORMAT_PREVIEW_DECODE_LIMIT_BYTES diff --git a/tree/item.py b/tree/item.py index be42d02..5a38aae 100644 --- a/tree/item.py +++ b/tree/item.py @@ -9,6 +9,8 @@ from core.datetime_parsing.enums import DateTimeCategory from core.datetime_parsing.nano_time import NanoTime from core.datetime_parsing.regex import parse_datetime_text +from core.raw_numeric import REASON_UNKNOWN, RawNumericValue, raw_numeric_text_is_acceptable +from core.safe_mpq import parse_mpq from tree.item_coercion import coerce_value_for_type, compute_editable, normalize_value_for_type from tree.item_names import unique_child_name, validated_child_name from tree.types import DATETIME_FAMILY, SECRET_FAMILY, TEXT_FAMILY, JsonType, parse_json_type, text_pseudotype_for @@ -170,6 +172,17 @@ def set_data(self, column: int, value: Any) -> bool: return True if column == 2: + # Internal paste / programmatic set of a raw numeric literal: keep + # it raw regardless of the current type. + if isinstance(value, RawNumericValue): + self.explicit_type = False + self._apply_typed_value(JsonType.RAW_FLOAT, value) + return True + # Editing an existing raw numeric value goes through the dedicated + # raw-text recovery path. + if self.json_type is JsonType.RAW_FLOAT: + return self._set_raw_numeric_value(value) + if self.explicit_type: ok, coerced = self._coerce_value_for_type(self.json_type, value, strict=True) if not ok: @@ -250,7 +263,54 @@ def _unique_child_name(self, base: str = "new_key", used_names: set[str] | None def _normalize_value_for_type(self, value: Any) -> Any: return normalize_value_for_type(self.json_type, value) + def _set_raw_numeric_value(self, value: Any) -> bool: + """Apply an edit to a value currently held as a raw numeric literal. + + - If the edited text parses safely, store it as a normal number. + Whole numbers (denominator 1) are stored as INTEGER, not FLOAT, + so they display as "42" instead of "42.0". + - If the text is unchanged, preserve the original raw value exactly. + - If the text matches the narrow raw-numeric edit grammar but is still + unsupported, keep it as a raw numeric value with the updated reason. + - Otherwise reject the edit. + """ + original = self.value if isinstance(self.value, RawNumericValue) else RawNumericValue(str(self.value)) + text = value if isinstance(value, str) else str(value) + + result = parse_mpq(text) + if result.value is not None: + self.explicit_type = False + # If the parsed value is a whole number (denominator 1), convert to int + # so it's stored as INTEGER instead of FLOAT (displaying as "42" not "42.0") + parsed = result.value + if parsed.denominator == 1: + parsed = int(parsed) + self._apply_typed_value(parse_json_type(parsed), parsed) + return True + + if text == original.raw: + self._apply_typed_value(JsonType.RAW_FLOAT, original) + return True + + if raw_numeric_text_is_acceptable(text): + self._apply_typed_value( + JsonType.RAW_FLOAT, + RawNumericValue( + raw=text, + reason=result.reason or REASON_UNKNOWN, + source_syntax=original.source_syntax, + ), + ) + return True + + return False + def _apply_typed_value(self, json_type: JsonType, value: Any) -> None: + # A raw numeric literal can only live under the RAW_FLOAT pseudo-type; + # never let it be stored under a regular numeric (or other) type. + if isinstance(value, RawNumericValue) and json_type not in TEXT_FAMILY and json_type not in SECRET_FAMILY: + json_type = JsonType.RAW_FLOAT + self.json_type = json_type self.child_items = [] self._children_dirty = True @@ -328,7 +388,7 @@ def _morph_container(self, new_type: JsonType) -> bool: def _coerce_value_for_type( self, json_type: JsonType, value: Any, strict: bool, old_type: JsonType | None = None ) -> tuple[bool, Any]: - return coerce_value_for_type(json_type, value, strict, old_type=old_type) + return coerce_value_for_type(json_type, value, strict, old_type=old_type, allow_expensive=self.explicit_type) def _compute_editable(self) -> bool: return compute_editable(self.json_type, self.value, self.EDITABLE_BLOB_LIMIT) diff --git a/tree/item_coercion.py b/tree/item_coercion.py index b0ecd46..d83e9bd 100644 --- a/tree/item_coercion.py +++ b/tree/item_coercion.py @@ -12,6 +12,8 @@ from core.datetime_parsing.enums import DateTimeCategory from core.datetime_parsing.nano_time import NanoTime from core.datetime_parsing.regex import parse_datetime_text +from core.raw_numeric import RawNumericValue +from core.safe_mpq import safe_mpq_from_any from settings import NUMBER_AFFIX_MAX_LEN from tree.codecs.bytes_codec import decode_bytes, encode_bytes from tree.codecs.color_codec import normalize_color_string @@ -106,7 +108,9 @@ def _is_temporal_type(json_type: JsonType | None) -> bool: return json_type in (JsonType.DATE, JsonType.TIME, JsonType.DATETIME, JsonType.DATETIMEZONE, JsonType.DATETIMEUTC) -def _epoch_seconds_from_temporal(value: Any, hinted_type: JsonType | None = None) -> mpq | None: +def _epoch_seconds_from_temporal( + value: Any, hinted_type: JsonType | None = None, *, allow_expensive: bool = False +) -> mpq | None: """Convert temporal-like input to epoch seconds. DATETIME/DATE are Unix epoch seconds (UTC for naive values). @@ -151,7 +155,7 @@ def _seconds_since_midnight(t: NanoTime) -> mpq: ) for category in categories: - parsed = parse_datetime_text(raw, category) + parsed = parse_datetime_text(raw, category, allow_expensive=allow_expensive) if parsed is None: continue if isinstance(parsed, Timestamp): @@ -167,7 +171,7 @@ def _seconds_since_midnight(t: NanoTime) -> mpq: return None -def _try_parse_temporal(json_type: JsonType, value: Any) -> str | None: +def _try_parse_temporal(json_type: JsonType, value: Any, *, allow_expensive: bool = False) -> str | None: """Convert *value* to a canonical ISO string for *json_type*. Handles Python date/Timestamp/NanoTime objects, int epoch seconds (≥ 10^12 → @@ -265,12 +269,12 @@ def _try_parse_temporal(json_type: JsonType, value: Any) -> str | None: hour, rem = divmod(whole_seconds, 3600) minute, second = divmod(rem, 60) parsed_time = NanoTime(hour=hour, minute=minute, second=second, nanosecond=nanos) - return _try_parse_temporal(json_type, parsed_time) + return _try_parse_temporal(json_type, parsed_time, allow_expensive=allow_expensive) ts = numeric / 1000.0 if isinstance(value, int) and abs(value) >= 10**12 else numeric try: dt = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc) - return _try_parse_temporal(json_type, dt) + return _try_parse_temporal(json_type, dt, allow_expensive=allow_expensive) except (ValueError, OSError, OverflowError): pass return None @@ -279,21 +283,21 @@ def _try_parse_temporal(json_type: JsonType, value: Any) -> str | None: if isinstance(value, str) and value: raw = value.strip() category = _category_for_temporal_type(json_type) - if category is not None and parse_datetime_text(raw, category) is not None: + if category is not None and parse_datetime_text(raw, category, allow_expensive=allow_expensive) is not None: # Keep exactly what user entered so optional parts (seconds/microseconds) # can be added/removed dynamically without being rewritten away. return raw # Try time first (time strings are not valid datetime strings) try: - return _try_parse_temporal(json_type, NanoTime.fromisoformat(raw)) + return _try_parse_temporal(json_type, NanoTime.fromisoformat(raw), allow_expensive=allow_expensive) except ValueError: pass # dateutil handles full datetime/date/tz strings try: from dateutil.parser import isoparse - return _try_parse_temporal(json_type, isoparse(raw)) + return _try_parse_temporal(json_type, isoparse(raw), allow_expensive=allow_expensive) except Exception: pass @@ -321,6 +325,8 @@ def _looks_valid_for(json_type: JsonType, value: str) -> bool: def normalize_value_for_type(json_type: JsonType, value: Any) -> Any: + if isinstance(value, RawNumericValue): + return value if (json_type in TEXT_FAMILY or json_type in (JsonType.SECRET_LINE, JsonType.SECRET_TEXT)) and not isinstance( value, str ): @@ -333,14 +339,11 @@ def coerce_value_for_type( value: Any, strict: bool, old_type: JsonType | None = None, + *, + allow_expensive: bool = False, ) -> tuple[bool, Any]: def _to_mpq_or_none(raw: Any) -> mpq | None: - if isinstance(raw, mpq): - return raw - try: - return mpq(str(raw)) - except (ValueError, TypeError): - return None + return safe_mpq_from_any(raw) def _int_from_exact(raw: Any) -> int | None: if isinstance(raw, int): @@ -366,6 +369,27 @@ def _affix_kind_for(target: JsonType) -> AffixKind: return AffixKind.CURRENCY return AffixKind.UNITS + # Targeting the RAW_FLOAT pseudo-type: always keep the value as a raw + # numeric literal (this path is only reachable programmatically; the type + # is not user-selectable). + if json_type is JsonType.RAW_FLOAT: + if isinstance(value, RawNumericValue): + return True, value + return True, RawNumericValue(raw="" if value is None else str(value)) + + # Raw, unsupported numeric literals: keep them as raw text on numeric + # targets when still unparseable (the model redirects to RAW_FLOAT). For + # any other target, fall back to the exact raw string so conversion never + # silently substitutes a numeric stub for the user's data. + if isinstance(value, RawNumericValue): + recovered = safe_mpq_from_any(value.raw) + if recovered is not None: + value = recovered + elif json_type in (JsonType.FLOAT, JsonType.PERCENT): + return True, value + else: + value = value.raw + match json_type: case JsonType.NULL: return True, None @@ -389,7 +413,7 @@ def _affix_kind_for(target: JsonType) -> AffixKind: if truncated is None: return False, None return True, truncated - temporal_epoch = _epoch_seconds_from_temporal(value, hinted_type=old_type) + temporal_epoch = _epoch_seconds_from_temporal(value, hinted_type=old_type, allow_expensive=allow_expensive) if temporal_epoch is not None: return True, int(temporal_epoch) if _is_temporal_type(old_type): @@ -420,31 +444,27 @@ def _affix_kind_for(target: JsonType) -> AffixKind: if q is None: return False, None return True, q - temporal_epoch = _epoch_seconds_from_temporal(value, hinted_type=old_type) + temporal_epoch = _epoch_seconds_from_temporal(value, hinted_type=old_type, allow_expensive=allow_expensive) if temporal_epoch is not None: return True, temporal_epoch if _is_temporal_type(old_type): # Same safety rule as INTEGER for non-applicable temporal transitions. return (False, None) if strict else (True, stub_float()) - try: - return True, mpq(str(value)) - except (ValueError, TypeError): - pass + q = _to_mpq_or_none(value) + if q is not None: + return True, q return (False, None) if strict else (True, stub_float()) case JsonType.PERCENT: - try: - v = mpq(str(value)) - if 0 <= v <= 1: - return True, v - except (ValueError, TypeError): - pass + v = _to_mpq_or_none(value) + if v is not None and 0 <= v <= 1: + return True, v return (False, None) if strict else (True, stub_percent()) case JsonType.INTEGER_CURRENCY | JsonType.INTEGER_UNITS: kind = _affix_kind_for(json_type) if isinstance(value, str): - parsed = parse_number_affix(value, max_affix_len=NUMBER_AFFIX_MAX_LEN) + parsed = parse_number_affix(value, max_affix_len=NUMBER_AFFIX_MAX_LEN, allow_expensive=allow_expensive) if parsed is not None: truncated = _int_from_truncated(parsed.number) if truncated is None: @@ -467,7 +487,7 @@ def _affix_kind_for(target: JsonType) -> AffixKind: case JsonType.FLOAT_CURRENCY | JsonType.FLOAT_UNITS: kind = _affix_kind_for(json_type) if isinstance(value, str): - parsed = parse_number_affix(value, max_affix_len=NUMBER_AFFIX_MAX_LEN) + parsed = parse_number_affix(value, max_affix_len=NUMBER_AFFIX_MAX_LEN, allow_expensive=allow_expensive) if parsed is not None: q = _to_mpq_or_none(parsed.number) if q is None: @@ -525,7 +545,7 @@ def _affix_kind_for(target: JsonType) -> AffixKind: # 3.2: fall back to "now" instead of epoch-zero when value is unparseable case JsonType.DATE | JsonType.TIME | JsonType.DATETIME | JsonType.DATETIMEZONE | JsonType.DATETIMEUTC: - parsed = _try_parse_temporal(json_type, value) + parsed = _try_parse_temporal(json_type, value, allow_expensive=allow_expensive) if parsed is not None: return True, parsed return True, _now_for_type(json_type) @@ -576,6 +596,10 @@ def _affix_kind_for(target: JsonType) -> AffixKind: def compute_editable(json_type: JsonType, value: Any, editable_blob_limit: int) -> bool: + if isinstance(value, RawNumericValue): + # Unsupported numeric literals are editable as plain raw text. + return True + if json_type in (JsonType.NULL, JsonType.ARRAY, JsonType.OBJECT): return False diff --git a/tree/model.py b/tree/model.py index dd6ac64..bbaa308 100644 --- a/tree/model.py +++ b/tree/model.py @@ -30,11 +30,16 @@ def __init__( *, show_root: bool = False, icon_provider: IconProvider | None = None, + root_item: JsonTreeItem | None = None, + estimated_item_count: int | None = None, ) -> None: super().__init__(parent) - self.root_item = JsonTreeItem(None, data) + self.root_item = root_item if root_item is not None else JsonTreeItem(None, data) self.show_root = show_root self._icon_provider: IconProvider = icon_provider or StubIconProvider() + self.estimated_item_count: int | None = ( + int(estimated_item_count) if isinstance(estimated_item_count, int) and estimated_item_count > 0 else None + ) self._attached_view = None self._drag_source_rows: list[QModelIndex] = [] self._issue_index_provider: Callable[[tuple[int, ...]], str | None] | None = None @@ -86,6 +91,19 @@ def set_issue_index_provider( """ self._issue_index_provider = provider + def replace_root_item(self, root_item: JsonTreeItem, *, estimated_item_count: int | None = None) -> None: + """Atomically replace ``root_item`` between reset signals.""" + self.beginResetModel() + try: + root_item.parent_item = None + self.root_item = root_item + if isinstance(estimated_item_count, int) and estimated_item_count > 0: + self.estimated_item_count = int(estimated_item_count) + else: + self.estimated_item_count = None + finally: + self.endResetModel() + def _root_index(self) -> QModelIndex: if not self.show_root: return QModelIndex() diff --git a/tree/model_roles.py b/tree/model_roles.py index 6d2ba97..ed8cbf9 100644 --- a/tree/model_roles.py +++ b/tree/model_roles.py @@ -4,6 +4,8 @@ from PySide6.QtCore import Qt from PySide6.QtGui import QFont +from core.raw_numeric import RawNumericValue, describe_reason +from core.safe_mpq import safe_mpq_from_any from mpq2py import mpq_serialization from settings import SECRET_MASK_CHAR, SECRET_MASK_GLYPHS from tree.types import SECRET_FAMILY, JsonType @@ -26,6 +28,14 @@ def tooltip_role_for_value(item) -> str | None: if item.json_type in SECRET_FAMILY: return SECRET_MASK_CHAR * SECRET_MASK_GLYPHS + if item.json_type is JsonType.RAW_FLOAT and isinstance(item.value, RawNumericValue): + return ( + "Unsupported as a regular number: " + f"{describe_reason(item.value.reason)}.\n" + "Edit it into a normal number, or leave it unchanged to preserve " + "the raw value for external software that accepts it." + ) + raw = item.data(2) text = "" if raw is None else str(raw) if len(text) <= 80: @@ -45,11 +55,9 @@ def display_role_value(item, column: int, is_root_item: bool) -> str: data = item.data(column) if column == 2 and item.json_type is JsonType.PERCENT: - try: - q = data if isinstance(data, gmpy2.mpq) else gmpy2.mpq(str(data)) + q = data if isinstance(data, gmpy2.mpq) else safe_mpq_from_any(data) + if q is not None: return f"{float(q * 100):g}%" - except (TypeError, ValueError): - pass match data: case bool(): diff --git a/tree/types.py b/tree/types.py index 0546edc..c5ddf83 100644 --- a/tree/types.py +++ b/tree/types.py @@ -12,7 +12,9 @@ from core.datetime_parsing import parse_datetime_text from core.datetime_parsing.nano_time import NanoTime -from settings import NUMBER_AFFIX_MAX_LEN +from core.raw_numeric import RawNumericValue +from settings import INFERENCE_MAX_COLOR_CHARS, NUMBER_AFFIX_MAX_LEN +from tree.inference_limits import base64_syntax_valid from units.number_affix import AffixKind, NumberAffix, parse_number_affix LOGGER = logging.getLogger(__name__) @@ -21,25 +23,34 @@ _COLOR_RGBA_RE = re.compile(r"^#(?:[0-9a-fA-F]{4}|[0-9a-fA-F]{8})$") -def looks_like_color_rgb(s: str) -> bool: +def looks_like_color_rgb(s: str, *, allow_expensive: bool = False) -> bool: + if not allow_expensive and len(s) > INFERENCE_MAX_COLOR_CHARS: + return False return bool(_COLOR_RGB_RE.fullmatch(s)) -def looks_like_color_rgba(s: str) -> bool: +def looks_like_color_rgba(s: str, *, allow_expensive: bool = False) -> bool: + if not allow_expensive and len(s) > INFERENCE_MAX_COLOR_CHARS: + return False return bool(_COLOR_RGBA_RE.fullmatch(s)) def _looks_like_base64(s: str) -> bool: """Return True iff *s* is a syntactically valid, non-empty base64 string. + Uses ``base64_syntax_valid`` as a cheap pre-check (len mod 4 + alphabet + regex) before attempting the expensive ``base64.b64decode``. A minimum + length of 20 chars is required to avoid false positives on short strings. + No content heuristics are applied: any string that decodes cleanly under strict base64 rules is treated as ``BYTES``. Callers that need to discriminate against short / human-readable strings (e.g. ``"abcd"``) must pin the type explicitly via the type editor. """ - if not s or len(s) % 4 != 0: + if not base64_syntax_valid(s): return False - if _B64_RE.fullmatch(s) is None: + # Require minimum 20 chars to avoid false positives on short strings + if len(s) < 20: return False try: base64.b64decode(s, validate=True) @@ -148,6 +159,9 @@ def parse_json_type(value: Any) -> "JsonType": return JsonType.PERCENT return JsonType.FLOAT + case RawNumericValue(): + return JsonType.RAW_FLOAT + case str(s): if s == "": return JsonType.EMPTY_STRING @@ -230,6 +244,9 @@ class JsonType(StrEnum): # Extra Number PERCENT = "percent" + # Raw, unsupported numeric literal preserved as editable text. Pseudo-type: + # derived from value content, never user-selectable. + RAW_FLOAT = "raw float" INTEGER_UNITS = "int units" FLOAT_UNITS = "float units" INTEGER_CURRENCY = "int currency" @@ -323,9 +340,27 @@ def canonical_text_type(json_type: JsonType) -> JsonType: return PSEUDO_TEXT_PARENT.get(json_type, json_type) -# Types the user can pick from the type combobox. Pseudo text types are -# excluded because they are derived purely from content. -USER_SELECTABLE_TYPES: tuple[JsonType, ...] = tuple(t for t in JsonType if t not in PSEUDO_TEXT_FAMILY) +# Parent (canonical, user-selectable) type for each non-text pseudo type. +# RAW_FLOAT collapses to FLOAT in the type combo so the user sees the type they +# could pick, while editing/coercion keep treating it as raw text. +PSEUDO_NUMERIC_PARENT: dict[JsonType, JsonType] = { + JsonType.RAW_FLOAT: JsonType.FLOAT, +} + + +def canonical_type(json_type: JsonType) -> JsonType: + """Return the user-selectable parent for any pseudo type, else *json_type*.""" + if json_type in PSEUDO_NUMERIC_PARENT: + return PSEUDO_NUMERIC_PARENT[json_type] + return PSEUDO_TEXT_PARENT.get(json_type, json_type) + + +# Non-user-selectable pseudo types (derived purely from content). +PSEUDO_FAMILY: frozenset[JsonType] = PSEUDO_TEXT_FAMILY | frozenset({JsonType.RAW_FLOAT}) + +# Types the user can pick from the type combobox. Pseudo types are excluded +# because they are derived purely from content. +USER_SELECTABLE_TYPES: tuple[JsonType, ...] = tuple(t for t in JsonType if t not in PSEUDO_FAMILY) SECRET_FAMILY: frozenset[JsonType] = frozenset({JsonType.SECRET_LINE, JsonType.SECRET_TEXT}) COLOR_FAMILY: frozenset[JsonType] = frozenset({JsonType.COLOR_RGB, JsonType.COLOR_RGBA}) DATETIME_FAMILY: frozenset[JsonType] = frozenset( diff --git a/tree_actions/clipboard.py b/tree_actions/clipboard.py index 9612ca3..b3c2e9b 100644 --- a/tree_actions/clipboard.py +++ b/tree_actions/clipboard.py @@ -6,7 +6,8 @@ from PySide6.QtCore import QMimeData from PySide6.QtWidgets import QApplication, QTreeView -from mpq2py import MpqSafeDumper, mpq_json_default +from core.raw_numeric import REASON_UNKNOWN, RawNumericValue +from mpq2py import MpqSafeDumper, mpq_json_default, raw_numeric_is_json_safe from tree.model import JsonTreeModel from tree.types import JsonType from tree_actions.selection import _index_path, _is_ancestor, _resolve_model, _selected_rows, selection_shape @@ -15,6 +16,51 @@ MIME_JSON_TREE = "application/x-json-tree" +def _clipboard_text_default(obj: Any): + """Lenient default for *human-readable* clipboard text. + + Raw numeric values are emitted as JSON number tokens when valid, otherwise + as a quoted string so copying never produces invalid JSON. High-fidelity + round-tripping is handled separately by the tagged MIME metadata. + """ + if isinstance(obj, RawNumericValue): + if raw_numeric_is_json_safe(obj.raw): + return simplejson.RawJSON(obj.raw.strip()) + return obj.raw + return mpq_json_default(obj) + + +def _mime_meta_default(obj: Any): + """Strict, lossless default for app-internal MIME metadata. + + Raw numeric values are encoded as a tagged object so paste can reconstruct + the exact :class:`RawNumericValue` (and keep it as RAW_FLOAT). + """ + if isinstance(obj, RawNumericValue): + return { + "__raw_numeric__": True, + "raw": obj.raw, + "reason": obj.reason, + "source_syntax": obj.source_syntax, + } + return mpq_json_default(obj) + + +def _revive_raw_numeric(value: Any) -> Any: + """Reconstruct tagged raw numeric objects produced by ``_mime_meta_default``.""" + if isinstance(value, dict): + if value.get("__raw_numeric__") is True and isinstance(value.get("raw"), str): + return RawNumericValue( + raw=value["raw"], + reason=value.get("reason") or REASON_UNKNOWN, + source_syntax=value.get("source_syntax") or "", + ) + return {k: _revive_raw_numeric(v) for k, v in value.items()} + if isinstance(value, list): + return [_revive_raw_numeric(v) for v in value] + return value + + def _dump_text(payload: Any) -> str: """Serialize *payload* to a string using the configured clipboard text format.""" from state.clipboard_settings import CLIPBOARD_TEXT_FORMAT_YAML, get_clipboard_text_format @@ -25,9 +71,9 @@ def _dump_text(payload: Any) -> str: except yaml.representer.RepresenterError: # Some app-native values (for example NumberAffix) are JSON-serializable # via mpq_json_default but do not have direct YAML representers. - normalized = json.loads(simplejson.dumps(payload, default=mpq_json_default)) + normalized = json.loads(simplejson.dumps(payload, default=_clipboard_text_default)) return yaml.dump(normalized, Dumper=MpqSafeDumper, allow_unicode=True, default_flow_style=False).rstrip() - return simplejson.dumps(payload, default=mpq_json_default, indent=2) + return simplejson.dumps(payload, default=_clipboard_text_default, indent=2) def _get_val_str(item) -> str: @@ -193,7 +239,7 @@ def build_tree_mime(model: JsonTreeModel, source_rows) -> QMimeData | None: text_payload = _entries_text_payload(model, rows, entries) source_paths = [list(_path_relative_to_root(model, idx)) for idx in rows] - metadata = simplejson.dumps({"entries": entries, "source_paths": source_paths}, default=mpq_json_default) + metadata = simplejson.dumps({"entries": entries, "source_paths": source_paths}, default=_mime_meta_default) text = _dump_text(text_payload) mime = QMimeData() @@ -225,13 +271,18 @@ def entries_from_mime(mime: QMimeData) -> list[dict[str, Any]] | None: if not isinstance(entry, dict) or "value" not in entry: continue name = entry.get("name") - normalized.append({"name": name if isinstance(name, str) else None, "value": entry["value"]}) + normalized.append( + { + "name": name if isinstance(name, str) else None, + "value": _revive_raw_numeric(entry["value"]), + } + ) if normalized: return normalized items = parsed.get("items") if isinstance(items, list): - return [{"name": None, "value": value} for value in items] + return [{"name": None, "value": _revive_raw_numeric(value)} for value in items] except Exception: pass @@ -376,13 +427,11 @@ def copy_selection_with_name(tree_view: QTreeView) -> bool: if not rows: return False - entries = _build_copy_entries(model, rows) - entries = _build_copy_entries(model, rows) text_payload = _entries_text_payload(model, rows, entries) text = _dump_text(text_payload) - metadata = simplejson.dumps({"entries": entries}, default=mpq_json_default) + metadata = simplejson.dumps({"entries": entries}, default=_mime_meta_default) mime = QMimeData() mime.setData(MIME_JSON_TREE, metadata.encode("utf-8")) mime.setText(text) diff --git a/ui/dialogs/attach_schema_dialog.py b/ui/dialogs/attach_schema_dialog.py index 307280e..c88c209 100644 --- a/ui/dialogs/attach_schema_dialog.py +++ b/ui/dialogs/attach_schema_dialog.py @@ -8,79 +8,118 @@ ## WARNING! All changes made in this file will be lost when recompiling UI file! ################################################################################ -from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale, - QMetaObject, QObject, QPoint, QRect, - QSize, QTime, QUrl, Qt) -from PySide6.QtGui import (QBrush, QColor, QConicalGradient, QCursor, - QFont, QFontDatabase, QGradient, QIcon, - QImage, QKeySequence, QLinearGradient, QPainter, - QPalette, QPixmap, QRadialGradient, QTransform) -from PySide6.QtWidgets import (QAbstractButton, QApplication, QComboBox, QDialog, - QDialogButtonBox, QHBoxLayout, QLabel, QLineEdit, - QPushButton, QSizePolicy, QVBoxLayout, QWidget) +from PySide6.QtCore import ( + QCoreApplication, + QDate, + QDateTime, + QLocale, + QMetaObject, + QObject, + QPoint, + QRect, + QSize, + Qt, + QTime, + QUrl, +) +from PySide6.QtGui import ( + QBrush, + QColor, + QConicalGradient, + QCursor, + QFont, + QFontDatabase, + QGradient, + QIcon, + QImage, + QKeySequence, + QLinearGradient, + QPainter, + QPalette, + QPixmap, + QRadialGradient, + QTransform, +) +from PySide6.QtWidgets import ( + QAbstractButton, + QApplication, + QComboBox, + QDialog, + QDialogButtonBox, + QHBoxLayout, + QLabel, + QLineEdit, + QPushButton, + QSizePolicy, + QVBoxLayout, + QWidget, +) + class Ui_AttachSchemaDialog(object): def setupUi(self, AttachSchemaDialog): if not AttachSchemaDialog.objectName(): - AttachSchemaDialog.setObjectName(u"AttachSchemaDialog") + AttachSchemaDialog.setObjectName("AttachSchemaDialog") AttachSchemaDialog.resize(540, 110) self.verticalLayout = QVBoxLayout(AttachSchemaDialog) - self.verticalLayout.setObjectName(u"verticalLayout") + self.verticalLayout.setObjectName("verticalLayout") self.recentRowWidget = QWidget(AttachSchemaDialog) - self.recentRowWidget.setObjectName(u"recentRowWidget") + self.recentRowWidget.setObjectName("recentRowWidget") self.recentRowLayout = QHBoxLayout(self.recentRowWidget) - self.recentRowLayout.setObjectName(u"recentRowLayout") + self.recentRowLayout.setObjectName("recentRowLayout") self.recentRowLayout.setContentsMargins(0, 0, 0, 0) self.recentLabel = QLabel(self.recentRowWidget) - self.recentLabel.setObjectName(u"recentLabel") + self.recentLabel.setObjectName("recentLabel") self.recentRowLayout.addWidget(self.recentLabel) self.recentComboBox = QComboBox(self.recentRowWidget) - self.recentComboBox.setObjectName(u"recentComboBox") + self.recentComboBox.setObjectName("recentComboBox") self.recentRowLayout.addWidget(self.recentComboBox) - self.verticalLayout.addWidget(self.recentRowWidget) self.pathLabel = QLabel(AttachSchemaDialog) - self.pathLabel.setObjectName(u"pathLabel") + self.pathLabel.setObjectName("pathLabel") self.verticalLayout.addWidget(self.pathLabel) self.pathRowLayout = QHBoxLayout() - self.pathRowLayout.setObjectName(u"pathRowLayout") + self.pathRowLayout.setObjectName("pathRowLayout") self.pathLineEdit = QLineEdit(AttachSchemaDialog) - self.pathLineEdit.setObjectName(u"pathLineEdit") + self.pathLineEdit.setObjectName("pathLineEdit") self.pathRowLayout.addWidget(self.pathLineEdit) self.browseButton = QPushButton(AttachSchemaDialog) - self.browseButton.setObjectName(u"browseButton") + self.browseButton.setObjectName("browseButton") self.pathRowLayout.addWidget(self.browseButton) - self.verticalLayout.addLayout(self.pathRowLayout) self.buttonBox = QDialogButtonBox(AttachSchemaDialog) - self.buttonBox.setObjectName(u"buttonBox") - self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel|QDialogButtonBox.Ok) + self.buttonBox.setObjectName("buttonBox") + self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel | QDialogButtonBox.Ok) self.verticalLayout.addWidget(self.buttonBox) - self.retranslateUi(AttachSchemaDialog) QMetaObject.connectSlotsByName(AttachSchemaDialog) + # setupUi def retranslateUi(self, AttachSchemaDialog): - AttachSchemaDialog.setWindowTitle(QCoreApplication.translate("AttachSchemaDialog", u"Attach JSON Schema", None)) - self.recentLabel.setText(QCoreApplication.translate("AttachSchemaDialog", u"Recent schemas:", None)) - self.pathLabel.setText(QCoreApplication.translate("AttachSchemaDialog", u"Schema file path or URL (http/https):", None)) - self.pathLineEdit.setPlaceholderText(QCoreApplication.translate("AttachSchemaDialog", u"https://... or /path/to/schema.json", None)) - self.browseButton.setText(QCoreApplication.translate("AttachSchemaDialog", u"Browse...", None)) - # retranslateUi + AttachSchemaDialog.setWindowTitle(QCoreApplication.translate("AttachSchemaDialog", "Attach JSON Schema", None)) + self.recentLabel.setText(QCoreApplication.translate("AttachSchemaDialog", "Recent schemas:", None)) + self.pathLabel.setText( + QCoreApplication.translate("AttachSchemaDialog", "Schema file path or URL (http/https):", None) + ) + self.pathLineEdit.setPlaceholderText( + QCoreApplication.translate("AttachSchemaDialog", "https://... or /path/to/schema.json", None) + ) + self.browseButton.setText(QCoreApplication.translate("AttachSchemaDialog", "Browse...", None)) + # retranslateUi diff --git a/ui/dialogs/qhex_dialog.py b/ui/dialogs/qhex_dialog.py index 42c60c9..b69e661 100644 --- a/ui/dialogs/qhex_dialog.py +++ b/ui/dialogs/qhex_dialog.py @@ -8,43 +8,79 @@ ## WARNING! All changes made in this file will be lost when recompiling UI file! ################################################################################ -from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale, - QMetaObject, QObject, QPoint, QRect, - QSize, QTime, QUrl, Qt) -from PySide6.QtGui import (QBrush, QColor, QConicalGradient, QCursor, - QFont, QFontDatabase, QGradient, QIcon, - QImage, QKeySequence, QLinearGradient, QPainter, - QPalette, QPixmap, QRadialGradient, QTransform) -from PySide6.QtWidgets import (QAbstractButton, QApplication, QCheckBox, QDialog, - QDialogButtonBox, QHBoxLayout, QSizePolicy, QSpacerItem, - QStatusBar, QVBoxLayout, QWidget) +from PySide6.QtCore import ( + QCoreApplication, + QDate, + QDateTime, + QLocale, + QMetaObject, + QObject, + QPoint, + QRect, + QSize, + Qt, + QTime, + QUrl, +) +from PySide6.QtGui import ( + QBrush, + QColor, + QConicalGradient, + QCursor, + QFont, + QFontDatabase, + QGradient, + QIcon, + QImage, + QKeySequence, + QLinearGradient, + QPainter, + QPalette, + QPixmap, + QRadialGradient, + QTransform, +) +from PySide6.QtWidgets import ( + QAbstractButton, + QApplication, + QCheckBox, + QDialog, + QDialogButtonBox, + QHBoxLayout, + QSizePolicy, + QSpacerItem, + QStatusBar, + QVBoxLayout, + QWidget, +) + class Ui_QHexDialog(object): def setupUi(self, QHexDialog): if not QHexDialog.objectName(): - QHexDialog.setObjectName(u"QHexDialog") + QHexDialog.setObjectName("QHexDialog") QHexDialog.resize(600, 440) self.verticalLayout = QVBoxLayout(QHexDialog) - self.verticalLayout.setObjectName(u"verticalLayout") + self.verticalLayout.setObjectName("verticalLayout") self.controlsLayout = QHBoxLayout() - self.controlsLayout.setObjectName(u"controlsLayout") + self.controlsLayout.setObjectName("controlsLayout") self.addressCheckBox = QCheckBox(QHexDialog) - self.addressCheckBox.setObjectName(u"addressCheckBox") + self.addressCheckBox.setObjectName("addressCheckBox") self.controlsLayout.addWidget(self.addressCheckBox) self.asciiCheckBox = QCheckBox(QHexDialog) - self.asciiCheckBox.setObjectName(u"asciiCheckBox") + self.asciiCheckBox.setObjectName("asciiCheckBox") self.controlsLayout.addWidget(self.asciiCheckBox) self.highlightingCheckBox = QCheckBox(QHexDialog) - self.highlightingCheckBox.setObjectName(u"highlightingCheckBox") + self.highlightingCheckBox.setObjectName("highlightingCheckBox") self.controlsLayout.addWidget(self.highlightingCheckBox) self.capsCheckBox = QCheckBox(QHexDialog) - self.capsCheckBox.setObjectName(u"capsCheckBox") + self.capsCheckBox.setObjectName("capsCheckBox") self.controlsLayout.addWidget(self.capsCheckBox) @@ -52,36 +88,35 @@ def setupUi(self, QHexDialog): self.controlsLayout.addItem(self.horizontalSpacer) - self.verticalLayout.addLayout(self.controlsLayout) self.editorHost = QWidget(QHexDialog) - self.editorHost.setObjectName(u"editorHost") + self.editorHost.setObjectName("editorHost") self.verticalLayout.addWidget(self.editorHost) self.statusBar = QStatusBar(QHexDialog) - self.statusBar.setObjectName(u"statusBar") + self.statusBar.setObjectName("statusBar") self.verticalLayout.addWidget(self.statusBar) self.buttonBox = QDialogButtonBox(QHexDialog) - self.buttonBox.setObjectName(u"buttonBox") - self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel|QDialogButtonBox.Ok) + self.buttonBox.setObjectName("buttonBox") + self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel | QDialogButtonBox.Ok) self.verticalLayout.addWidget(self.buttonBox) - self.retranslateUi(QHexDialog) QMetaObject.connectSlotsByName(QHexDialog) + # setupUi def retranslateUi(self, QHexDialog): - QHexDialog.setWindowTitle(QCoreApplication.translate("QHexDialog", u"Edit Binary Data", None)) - self.addressCheckBox.setText(QCoreApplication.translate("QHexDialog", u"Address area", None)) - self.asciiCheckBox.setText(QCoreApplication.translate("QHexDialog", u"ASCII area", None)) - self.highlightingCheckBox.setText(QCoreApplication.translate("QHexDialog", u"Modified bytes", None)) - self.capsCheckBox.setText(QCoreApplication.translate("QHexDialog", u"CAPS", None)) - # retranslateUi + QHexDialog.setWindowTitle(QCoreApplication.translate("QHexDialog", "Edit Binary Data", None)) + self.addressCheckBox.setText(QCoreApplication.translate("QHexDialog", "Address area", None)) + self.asciiCheckBox.setText(QCoreApplication.translate("QHexDialog", "ASCII area", None)) + self.highlightingCheckBox.setText(QCoreApplication.translate("QHexDialog", "Modified bytes", None)) + self.capsCheckBox.setText(QCoreApplication.translate("QHexDialog", "CAPS", None)) + # retranslateUi diff --git a/ui/dialogs/qmultiline_dialog.py b/ui/dialogs/qmultiline_dialog.py index 27f8739..f563ca5 100644 --- a/ui/dialogs/qmultiline_dialog.py +++ b/ui/dialogs/qmultiline_dialog.py @@ -8,38 +8,73 @@ ## WARNING! All changes made in this file will be lost when recompiling UI file! ################################################################################ -from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale, - QMetaObject, QObject, QPoint, QRect, - QSize, QTime, QUrl, Qt) -from PySide6.QtGui import (QBrush, QColor, QConicalGradient, QCursor, - QFont, QFontDatabase, QGradient, QIcon, - QImage, QKeySequence, QLinearGradient, QPainter, - QPalette, QPixmap, QRadialGradient, QTransform) -from PySide6.QtWidgets import (QAbstractButton, QApplication, QCheckBox, QDialog, - QDialogButtonBox, QHBoxLayout, QSizePolicy, QSpacerItem, - QVBoxLayout, QWidget) +from PySide6.QtCore import ( + QCoreApplication, + QDate, + QDateTime, + QLocale, + QMetaObject, + QObject, + QPoint, + QRect, + QSize, + Qt, + QTime, + QUrl, +) +from PySide6.QtGui import ( + QBrush, + QColor, + QConicalGradient, + QCursor, + QFont, + QFontDatabase, + QGradient, + QIcon, + QImage, + QKeySequence, + QLinearGradient, + QPainter, + QPalette, + QPixmap, + QRadialGradient, + QTransform, +) +from PySide6.QtWidgets import ( + QAbstractButton, + QApplication, + QCheckBox, + QDialog, + QDialogButtonBox, + QHBoxLayout, + QSizePolicy, + QSpacerItem, + QVBoxLayout, + QWidget, +) + class Ui_QMultilineDialog(object): def setupUi(self, QMultilineDialog): if not QMultilineDialog.objectName(): - QMultilineDialog.setObjectName(u"QMultilineDialog") + QMultilineDialog.setObjectName("QMultilineDialog") QMultilineDialog.resize(600, 440) self.verticalLayout = QVBoxLayout(QMultilineDialog) - self.verticalLayout.setObjectName(u"verticalLayout") + self.verticalLayout.setObjectName("verticalLayout") self.controlsLayout = QHBoxLayout() - self.controlsLayout.setObjectName(u"controlsLayout") + self.controlsLayout.setObjectName("controlsLayout") self.wrapCheckBox = QCheckBox(QMultilineDialog) - self.wrapCheckBox.setObjectName(u"wrapCheckBox") + self.wrapCheckBox.setObjectName("wrapCheckBox") self.controlsLayout.addWidget(self.wrapCheckBox) self.lineNumbersCheckBox = QCheckBox(QMultilineDialog) - self.lineNumbersCheckBox.setObjectName(u"lineNumbersCheckBox") + self.lineNumbersCheckBox.setObjectName("lineNumbersCheckBox") self.controlsLayout.addWidget(self.lineNumbersCheckBox) self.monospacedCheckBox = QCheckBox(QMultilineDialog) - self.monospacedCheckBox.setObjectName(u"monospacedCheckBox") + self.monospacedCheckBox.setObjectName("monospacedCheckBox") self.controlsLayout.addWidget(self.monospacedCheckBox) @@ -47,30 +82,29 @@ def setupUi(self, QMultilineDialog): self.controlsLayout.addItem(self.horizontalSpacer) - self.verticalLayout.addLayout(self.controlsLayout) self.editorHost = QWidget(QMultilineDialog) - self.editorHost.setObjectName(u"editorHost") + self.editorHost.setObjectName("editorHost") self.verticalLayout.addWidget(self.editorHost) self.buttonBox = QDialogButtonBox(QMultilineDialog) - self.buttonBox.setObjectName(u"buttonBox") - self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel|QDialogButtonBox.Ok) + self.buttonBox.setObjectName("buttonBox") + self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel | QDialogButtonBox.Ok) self.verticalLayout.addWidget(self.buttonBox) - self.retranslateUi(QMultilineDialog) QMetaObject.connectSlotsByName(QMultilineDialog) + # setupUi def retranslateUi(self, QMultilineDialog): - QMultilineDialog.setWindowTitle(QCoreApplication.translate("QMultilineDialog", u"Edit Multiline Text", None)) - self.wrapCheckBox.setText(QCoreApplication.translate("QMultilineDialog", u"Word wrap", None)) - self.lineNumbersCheckBox.setText(QCoreApplication.translate("QMultilineDialog", u"Line numbers", None)) - self.monospacedCheckBox.setText(QCoreApplication.translate("QMultilineDialog", u"Monospaced", None)) - # retranslateUi + QMultilineDialog.setWindowTitle(QCoreApplication.translate("QMultilineDialog", "Edit Multiline Text", None)) + self.wrapCheckBox.setText(QCoreApplication.translate("QMultilineDialog", "Word wrap", None)) + self.lineNumbersCheckBox.setText(QCoreApplication.translate("QMultilineDialog", "Line numbers", None)) + self.monospacedCheckBox.setText(QCoreApplication.translate("QMultilineDialog", "Monospaced", None)) + # retranslateUi diff --git a/ui/dialogs/secret_prefixes_dialog.py b/ui/dialogs/secret_prefixes_dialog.py index d916dc6..af15e70 100644 --- a/ui/dialogs/secret_prefixes_dialog.py +++ b/ui/dialogs/secret_prefixes_dialog.py @@ -8,65 +8,102 @@ ## WARNING! All changes made in this file will be lost when recompiling UI file! ################################################################################ -from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale, - QMetaObject, QObject, QPoint, QRect, - QSize, QTime, QUrl, Qt) -from PySide6.QtGui import (QBrush, QColor, QConicalGradient, QCursor, - QFont, QFontDatabase, QGradient, QIcon, - QImage, QKeySequence, QLinearGradient, QPainter, - QPalette, QPixmap, QRadialGradient, QTransform) -from PySide6.QtWidgets import (QAbstractButton, QApplication, QDialog, QDialogButtonBox, - QHBoxLayout, QListWidget, QListWidgetItem, QPushButton, - QSizePolicy, QVBoxLayout, QWidget) +from PySide6.QtCore import ( + QCoreApplication, + QDate, + QDateTime, + QLocale, + QMetaObject, + QObject, + QPoint, + QRect, + QSize, + Qt, + QTime, + QUrl, +) +from PySide6.QtGui import ( + QBrush, + QColor, + QConicalGradient, + QCursor, + QFont, + QFontDatabase, + QGradient, + QIcon, + QImage, + QKeySequence, + QLinearGradient, + QPainter, + QPalette, + QPixmap, + QRadialGradient, + QTransform, +) +from PySide6.QtWidgets import ( + QAbstractButton, + QApplication, + QDialog, + QDialogButtonBox, + QHBoxLayout, + QListWidget, + QListWidgetItem, + QPushButton, + QSizePolicy, + QVBoxLayout, + QWidget, +) + class Ui_SecretPrefixesDialog(object): def setupUi(self, SecretPrefixesDialog): if not SecretPrefixesDialog.objectName(): - SecretPrefixesDialog.setObjectName(u"SecretPrefixesDialog") + SecretPrefixesDialog.setObjectName("SecretPrefixesDialog") SecretPrefixesDialog.resize(420, 320) self.verticalLayout = QVBoxLayout(SecretPrefixesDialog) - self.verticalLayout.setObjectName(u"verticalLayout") + self.verticalLayout.setObjectName("verticalLayout") self.listWidget = QListWidget(SecretPrefixesDialog) - self.listWidget.setObjectName(u"listWidget") + self.listWidget.setObjectName("listWidget") self.verticalLayout.addWidget(self.listWidget) self.buttonsRowLayout = QHBoxLayout() - self.buttonsRowLayout.setObjectName(u"buttonsRowLayout") + self.buttonsRowLayout.setObjectName("buttonsRowLayout") self.addButton = QPushButton(SecretPrefixesDialog) - self.addButton.setObjectName(u"addButton") + self.addButton.setObjectName("addButton") self.buttonsRowLayout.addWidget(self.addButton) self.editButton = QPushButton(SecretPrefixesDialog) - self.editButton.setObjectName(u"editButton") + self.editButton.setObjectName("editButton") self.buttonsRowLayout.addWidget(self.editButton) self.removeButton = QPushButton(SecretPrefixesDialog) - self.removeButton.setObjectName(u"removeButton") + self.removeButton.setObjectName("removeButton") self.buttonsRowLayout.addWidget(self.removeButton) - self.verticalLayout.addLayout(self.buttonsRowLayout) self.buttonBox = QDialogButtonBox(SecretPrefixesDialog) - self.buttonBox.setObjectName(u"buttonBox") - self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel|QDialogButtonBox.Ok) + self.buttonBox.setObjectName("buttonBox") + self.buttonBox.setStandardButtons(QDialogButtonBox.Cancel | QDialogButtonBox.Ok) self.verticalLayout.addWidget(self.buttonBox) - self.retranslateUi(SecretPrefixesDialog) QMetaObject.connectSlotsByName(SecretPrefixesDialog) + # setupUi def retranslateUi(self, SecretPrefixesDialog): - SecretPrefixesDialog.setWindowTitle(QCoreApplication.translate("SecretPrefixesDialog", u"Secret word prefixes", None)) - self.addButton.setText(QCoreApplication.translate("SecretPrefixesDialog", u"Add", None)) - self.editButton.setText(QCoreApplication.translate("SecretPrefixesDialog", u"Edit", None)) - self.removeButton.setText(QCoreApplication.translate("SecretPrefixesDialog", u"Remove", None)) - # retranslateUi + SecretPrefixesDialog.setWindowTitle( + QCoreApplication.translate("SecretPrefixesDialog", "Secret word prefixes", None) + ) + self.addButton.setText(QCoreApplication.translate("SecretPrefixesDialog", "Add", None)) + self.editButton.setText(QCoreApplication.translate("SecretPrefixesDialog", "Edit", None)) + self.removeButton.setText(QCoreApplication.translate("SecretPrefixesDialog", "Remove", None)) + # retranslateUi diff --git a/ui/json_tab_ui.py b/ui/json_tab_ui.py index c06a3f5..6ec8371 100644 --- a/ui/json_tab_ui.py +++ b/ui/json_tab_ui.py @@ -8,45 +8,70 @@ ## WARNING! All changes made in this file will be lost when recompiling UI file! ################################################################################ -from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale, - QMetaObject, QObject, QPoint, QRect, - QSize, QTime, QUrl, Qt) -from PySide6.QtGui import (QBrush, QColor, QConicalGradient, QCursor, - QFont, QFontDatabase, QGradient, QIcon, - QImage, QKeySequence, QLinearGradient, QPainter, - QPalette, QPixmap, QRadialGradient, QTransform) -from PySide6.QtWidgets import (QApplication, QHeaderView, QLineEdit, QSizePolicy, - QVBoxLayout, QWidget) +from PySide6.QtCore import ( + QCoreApplication, + QDate, + QDateTime, + QLocale, + QMetaObject, + QObject, + QPoint, + QRect, + QSize, + Qt, + QTime, + QUrl, +) +from PySide6.QtGui import ( + QBrush, + QColor, + QConicalGradient, + QCursor, + QFont, + QFontDatabase, + QGradient, + QIcon, + QImage, + QKeySequence, + QLinearGradient, + QPainter, + QPalette, + QPixmap, + QRadialGradient, + QTransform, +) +from PySide6.QtWidgets import QApplication, QHeaderView, QLineEdit, QSizePolicy, QVBoxLayout, QWidget from tree.view import JsonTreeView + class Ui_JsonTab(object): def setupUi(self, JsonTab): if not JsonTab.objectName(): - JsonTab.setObjectName(u"JsonTab") + JsonTab.setObjectName("JsonTab") JsonTab.resize(480, 360) self.verticalLayout = QVBoxLayout(JsonTab) self.verticalLayout.setSpacing(0) - self.verticalLayout.setObjectName(u"verticalLayout") + self.verticalLayout.setObjectName("verticalLayout") self.verticalLayout.setContentsMargins(0, 0, 0, 0) self.searchEdit = QLineEdit(JsonTab) - self.searchEdit.setObjectName(u"searchEdit") + self.searchEdit.setObjectName("searchEdit") self.verticalLayout.addWidget(self.searchEdit) self.treeView = JsonTreeView(JsonTab) - self.treeView.setObjectName(u"treeView") + self.treeView.setObjectName("treeView") self.verticalLayout.addWidget(self.treeView) - self.retranslateUi(JsonTab) QMetaObject.connectSlotsByName(JsonTab) + # setupUi def retranslateUi(self, JsonTab): - self.searchEdit.setPlaceholderText(QCoreApplication.translate("JsonTab", u"Filter (Ctrl+F)", None)) + self.searchEdit.setPlaceholderText(QCoreApplication.translate("JsonTab", "Filter (Ctrl+F)", None)) pass - # retranslateUi + # retranslateUi diff --git a/ui/mainwindow.py b/ui/mainwindow.py index 96349d9..66a06c5 100644 --- a/ui/mainwindow.py +++ b/ui/mainwindow.py @@ -8,59 +8,93 @@ ## WARNING! All changes made in this file will be lost when recompiling UI file! ################################################################################ -from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale, - QMetaObject, QObject, QPoint, QRect, - QSize, QTime, QUrl, Qt) -from PySide6.QtGui import (QAction, QBrush, QColor, QConicalGradient, - QCursor, QFont, QFontDatabase, QGradient, - QIcon, QImage, QKeySequence, QLinearGradient, - QPainter, QPalette, QPixmap, QRadialGradient, - QTransform) -from PySide6.QtWidgets import (QApplication, QMainWindow, QMenu, QMenuBar, - QSizePolicy, QStatusBar, QTabWidget, QVBoxLayout, - QWidget) +from PySide6.QtCore import ( + QCoreApplication, + QDate, + QDateTime, + QLocale, + QMetaObject, + QObject, + QPoint, + QRect, + QSize, + Qt, + QTime, + QUrl, +) +from PySide6.QtGui import ( + QAction, + QBrush, + QColor, + QConicalGradient, + QCursor, + QFont, + QFontDatabase, + QGradient, + QIcon, + QImage, + QKeySequence, + QLinearGradient, + QPainter, + QPalette, + QPixmap, + QRadialGradient, + QTransform, +) +from PySide6.QtWidgets import ( + QApplication, + QMainWindow, + QMenu, + QMenuBar, + QSizePolicy, + QStatusBar, + QTabWidget, + QVBoxLayout, + QWidget, +) + class Ui_MainWindow(object): def setupUi(self, MainWindow): if not MainWindow.objectName(): - MainWindow.setObjectName(u"MainWindow") + MainWindow.setObjectName("MainWindow") MainWindow.resize(573, 468) self.appExitAction = QAction(MainWindow) - self.appExitAction.setObjectName(u"appExitAction") + self.appExitAction.setObjectName("appExitAction") self.rowInsertAction = QAction(MainWindow) - self.rowInsertAction.setObjectName(u"rowInsertAction") + self.rowInsertAction.setObjectName("rowInsertAction") self.rowRemoveAction = QAction(MainWindow) - self.rowRemoveAction.setObjectName(u"rowRemoveAction") + self.rowRemoveAction.setObjectName("rowRemoveAction") self.fileCreateNewAction = QAction(MainWindow) - self.fileCreateNewAction.setObjectName(u"fileCreateNewAction") + self.fileCreateNewAction.setObjectName("fileCreateNewAction") self.fileOpenAction = QAction(MainWindow) - self.fileOpenAction.setObjectName(u"fileOpenAction") + self.fileOpenAction.setObjectName("fileOpenAction") self.fileSaveAction = QAction(MainWindow) - self.fileSaveAction.setObjectName(u"fileSaveAction") + self.fileSaveAction.setObjectName("fileSaveAction") self.fileSaveAsAction = QAction(MainWindow) - self.fileSaveAsAction.setObjectName(u"fileSaveAsAction") + self.fileSaveAsAction.setObjectName("fileSaveAsAction") self.rowInsertAfterAction = QAction(MainWindow) - self.rowInsertAfterAction.setObjectName(u"rowInsertAfterAction") + self.rowInsertAfterAction.setObjectName("rowInsertAfterAction") self.viewExpandAllAction = QAction(MainWindow) - self.viewExpandAllAction.setObjectName(u"viewExpandAllAction") + self.viewExpandAllAction.setObjectName("viewExpandAllAction") self.viewCollapseAllAction = QAction(MainWindow) - self.viewCollapseAllAction.setObjectName(u"viewCollapseAllAction") + self.viewCollapseAllAction.setObjectName("viewCollapseAllAction") self.viewZoomInAction = QAction(MainWindow) - self.viewZoomInAction.setObjectName(u"viewZoomInAction") + self.viewZoomInAction.setObjectName("viewZoomInAction") self.viewZoomOutAction = QAction(MainWindow) - self.viewZoomOutAction.setObjectName(u"viewZoomOutAction") + self.viewZoomOutAction.setObjectName("viewZoomOutAction") self.viewResetZoomAction = QAction(MainWindow) - self.viewResetZoomAction.setObjectName(u"viewResetZoomAction") + self.viewResetZoomAction.setObjectName("viewResetZoomAction") self.fileCopyPathAction = QAction(MainWindow) - self.fileCopyPathAction.setObjectName(u"fileCopyPathAction") + self.fileCopyPathAction.setObjectName("fileCopyPathAction") self.centralWidget = QWidget(MainWindow) - self.centralWidget.setObjectName(u"centralWidget") + self.centralWidget.setObjectName("centralWidget") self.vboxLayout = QVBoxLayout(self.centralWidget) self.vboxLayout.setSpacing(0) self.vboxLayout.setContentsMargins(0, 0, 0, 0) - self.vboxLayout.setObjectName(u"vboxLayout") + self.vboxLayout.setObjectName("vboxLayout") self.tabWidget = QTabWidget(self.centralWidget) - self.tabWidget.setObjectName(u"tabWidget") + self.tabWidget.setObjectName("tabWidget") self.tabWidget.setDocumentMode(True) self.tabWidget.setTabsClosable(True) self.tabWidget.setMovable(True) @@ -69,17 +103,17 @@ def setupUi(self, MainWindow): MainWindow.setCentralWidget(self.centralWidget) self.menuBar = QMenuBar(MainWindow) - self.menuBar.setObjectName(u"menuBar") + self.menuBar.setObjectName("menuBar") self.menuBar.setGeometry(QRect(0, 0, 573, 33)) self.fileMenu = QMenu(self.menuBar) - self.fileMenu.setObjectName(u"fileMenu") + self.fileMenu.setObjectName("fileMenu") self.actionsMenu = QMenu(self.menuBar) - self.actionsMenu.setObjectName(u"actionsMenu") + self.actionsMenu.setObjectName("actionsMenu") self.viewMenu = QMenu(self.menuBar) - self.viewMenu.setObjectName(u"viewMenu") + self.viewMenu.setObjectName("viewMenu") MainWindow.setMenuBar(self.menuBar) self.statusBar = QStatusBar(MainWindow) - self.statusBar.setObjectName(u"statusBar") + self.statusBar.setObjectName("statusBar") MainWindow.setStatusBar(self.statusBar) self.menuBar.addAction(self.fileMenu.menuAction()) @@ -109,67 +143,69 @@ def setupUi(self, MainWindow): self.tabWidget.setCurrentIndex(-1) - QMetaObject.connectSlotsByName(MainWindow) + # setupUi def retranslateUi(self, MainWindow): - MainWindow.setWindowTitle(QCoreApplication.translate("MainWindow", u"Editable Tree Model", None)) - self.appExitAction.setText(QCoreApplication.translate("MainWindow", u"&Exit", None)) -#if QT_CONFIG(shortcut) - self.appExitAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+Q", None)) -#endif // QT_CONFIG(shortcut) - self.rowInsertAction.setText(QCoreApplication.translate("MainWindow", u"&Insert Row", None)) -#if QT_CONFIG(shortcut) - self.rowInsertAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+I", None)) -#endif // QT_CONFIG(shortcut) - self.rowRemoveAction.setText(QCoreApplication.translate("MainWindow", u"&Remove Row", None)) -#if QT_CONFIG(shortcut) - self.rowRemoveAction.setShortcut(QCoreApplication.translate("MainWindow", u"Del", None)) -#endif // QT_CONFIG(shortcut) - self.fileCreateNewAction.setText(QCoreApplication.translate("MainWindow", u"Create &New", None)) -#if QT_CONFIG(shortcut) - self.fileCreateNewAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+N", None)) -#endif // QT_CONFIG(shortcut) - self.fileOpenAction.setText(QCoreApplication.translate("MainWindow", u"&Open File", None)) -#if QT_CONFIG(shortcut) - self.fileOpenAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+O", None)) -#endif // QT_CONFIG(shortcut) - self.fileSaveAction.setText(QCoreApplication.translate("MainWindow", u"&Save File", None)) -#if QT_CONFIG(shortcut) - self.fileSaveAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+S", None)) -#endif // QT_CONFIG(shortcut) - self.fileSaveAsAction.setText(QCoreApplication.translate("MainWindow", u"Save File &as ...", None)) -#if QT_CONFIG(shortcut) - self.fileSaveAsAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+Shift+S", None)) -#endif // QT_CONFIG(shortcut) - self.rowInsertAfterAction.setText(QCoreApplication.translate("MainWindow", u"Insert Row &after", None)) -#if QT_CONFIG(shortcut) - self.rowInsertAfterAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+Shift+I", None)) -#endif // QT_CONFIG(shortcut) - self.viewExpandAllAction.setText(QCoreApplication.translate("MainWindow", u"Expand All", None)) - self.viewCollapseAllAction.setText(QCoreApplication.translate("MainWindow", u"Collapse All", None)) - self.viewZoomInAction.setText(QCoreApplication.translate("MainWindow", u"Zoom In", None)) -#if QT_CONFIG(shortcut) - self.viewZoomInAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl++", None)) -#endif // QT_CONFIG(shortcut) - self.viewZoomOutAction.setText(QCoreApplication.translate("MainWindow", u"Zoom Out", None)) -#if QT_CONFIG(shortcut) - self.viewZoomOutAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+-", None)) -#endif // QT_CONFIG(shortcut) - self.viewResetZoomAction.setText(QCoreApplication.translate("MainWindow", u"Reset Zoom", None)) -#if QT_CONFIG(shortcut) - self.viewResetZoomAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+0", None)) -#endif // QT_CONFIG(shortcut) - self.fileCopyPathAction.setText(QCoreApplication.translate("MainWindow", u"Copy Full File &Path", None)) -#if QT_CONFIG(tooltip) - self.fileCopyPathAction.setToolTip(QCoreApplication.translate("MainWindow", u"Copy absolute path of the current document", None)) -#endif // QT_CONFIG(tooltip) -#if QT_CONFIG(shortcut) - self.fileCopyPathAction.setShortcut(QCoreApplication.translate("MainWindow", u"Ctrl+Alt+C", None)) -#endif // QT_CONFIG(shortcut) - self.fileMenu.setTitle(QCoreApplication.translate("MainWindow", u"&File", None)) - self.actionsMenu.setTitle(QCoreApplication.translate("MainWindow", u"&Actions", None)) - self.viewMenu.setTitle(QCoreApplication.translate("MainWindow", u"&View", None)) - # retranslateUi + MainWindow.setWindowTitle(QCoreApplication.translate("MainWindow", "Editable Tree Model", None)) + self.appExitAction.setText(QCoreApplication.translate("MainWindow", "&Exit", None)) + # if QT_CONFIG(shortcut) + self.appExitAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+Q", None)) + # endif // QT_CONFIG(shortcut) + self.rowInsertAction.setText(QCoreApplication.translate("MainWindow", "&Insert Row", None)) + # if QT_CONFIG(shortcut) + self.rowInsertAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+I", None)) + # endif // QT_CONFIG(shortcut) + self.rowRemoveAction.setText(QCoreApplication.translate("MainWindow", "&Remove Row", None)) + # if QT_CONFIG(shortcut) + self.rowRemoveAction.setShortcut(QCoreApplication.translate("MainWindow", "Del", None)) + # endif // QT_CONFIG(shortcut) + self.fileCreateNewAction.setText(QCoreApplication.translate("MainWindow", "Create &New", None)) + # if QT_CONFIG(shortcut) + self.fileCreateNewAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+N", None)) + # endif // QT_CONFIG(shortcut) + self.fileOpenAction.setText(QCoreApplication.translate("MainWindow", "&Open File", None)) + # if QT_CONFIG(shortcut) + self.fileOpenAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+O", None)) + # endif // QT_CONFIG(shortcut) + self.fileSaveAction.setText(QCoreApplication.translate("MainWindow", "&Save File", None)) + # if QT_CONFIG(shortcut) + self.fileSaveAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+S", None)) + # endif // QT_CONFIG(shortcut) + self.fileSaveAsAction.setText(QCoreApplication.translate("MainWindow", "Save File &as ...", None)) + # if QT_CONFIG(shortcut) + self.fileSaveAsAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+Shift+S", None)) + # endif // QT_CONFIG(shortcut) + self.rowInsertAfterAction.setText(QCoreApplication.translate("MainWindow", "Insert Row &after", None)) + # if QT_CONFIG(shortcut) + self.rowInsertAfterAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+Shift+I", None)) + # endif // QT_CONFIG(shortcut) + self.viewExpandAllAction.setText(QCoreApplication.translate("MainWindow", "Expand All", None)) + self.viewCollapseAllAction.setText(QCoreApplication.translate("MainWindow", "Collapse All", None)) + self.viewZoomInAction.setText(QCoreApplication.translate("MainWindow", "Zoom In", None)) + # if QT_CONFIG(shortcut) + self.viewZoomInAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl++", None)) + # endif // QT_CONFIG(shortcut) + self.viewZoomOutAction.setText(QCoreApplication.translate("MainWindow", "Zoom Out", None)) + # if QT_CONFIG(shortcut) + self.viewZoomOutAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+-", None)) + # endif // QT_CONFIG(shortcut) + self.viewResetZoomAction.setText(QCoreApplication.translate("MainWindow", "Reset Zoom", None)) + # if QT_CONFIG(shortcut) + self.viewResetZoomAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+0", None)) + # endif // QT_CONFIG(shortcut) + self.fileCopyPathAction.setText(QCoreApplication.translate("MainWindow", "Copy Full File &Path", None)) + # if QT_CONFIG(tooltip) + self.fileCopyPathAction.setToolTip( + QCoreApplication.translate("MainWindow", "Copy absolute path of the current document", None) + ) + # endif // QT_CONFIG(tooltip) + # if QT_CONFIG(shortcut) + self.fileCopyPathAction.setShortcut(QCoreApplication.translate("MainWindow", "Ctrl+Alt+C", None)) + # endif // QT_CONFIG(shortcut) + self.fileMenu.setTitle(QCoreApplication.translate("MainWindow", "&File", None)) + self.actionsMenu.setTitle(QCoreApplication.translate("MainWindow", "&Actions", None)) + self.viewMenu.setTitle(QCoreApplication.translate("MainWindow", "&View", None)) + # retranslateUi diff --git a/undo/diff.py b/undo/diff.py index 0c86489..71c7464 100644 --- a/undo/diff.py +++ b/undo/diff.py @@ -26,6 +26,14 @@ def apply(self, item: JsonTreeItem, target: Any, item_index: QModelIndex) -> boo self.convert_to_leaf(item, item_index, target) return True + # Raw numeric edits: route through the dedicated recovery path so + # string edits from RawNumericLineEdit are properly parsed / kept raw. + if item.json_type is JsonType.RAW_FLOAT: + if item._set_raw_numeric_value(target): + self.emit_row_changed(item_index) + return True + return False + if item.value == target: return True if type(item.value) is type(target) and not isinstance(target, str): diff --git a/units/number_affix.py b/units/number_affix.py index aebf865..aede4a1 100644 --- a/units/number_affix.py +++ b/units/number_affix.py @@ -6,6 +6,9 @@ from gmpy2 import mpq +from core.safe_mpq import safe_mpq_from_text +from settings import INFERENCE_MAX_AFFIX_CHARS + _AFFIX_FORBIDDEN_TOUCH_CHARS = set("+-.") _NUMBER_RE = r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?" @@ -25,6 +28,18 @@ class NumberAffix: space: bool number: int | mpq + def __str__(self) -> str: + try: + return format_number_affix(self) + except ValueError as e: + return str(e) + + def __repr__(self) -> str: + try: + return format_number_affix(self) + except ValueError as e: + return str(e) + def _is_valid_affix(affix: str, *, kind: AffixKind, max_affix_len: int) -> bool: if not affix or len(affix) > max_affix_len: @@ -41,7 +56,10 @@ def _is_valid_affix(affix: str, *, kind: AffixKind, max_affix_len: int) -> bool: def _parse_number(num_text: str) -> int | mpq: if re.fullmatch(r"[+-]?\d+", num_text): return int(num_text) - return mpq(num_text) + parsed = safe_mpq_from_text(num_text) + if parsed is None: + raise ValueError("Unsafe numeric literal") + return parsed def _format_mpq_decimal(value: mpq) -> str: @@ -76,17 +94,26 @@ def _format_mpq_decimal(value: mpq) -> str: return f"{sign}{digits[:i]}.{digits[i:]}" -def parse_number_affix(s: str, *, max_affix_len: int = 16) -> NumberAffix | None: +def parse_number_affix(s: str, *, max_affix_len: int = 16, allow_expensive: bool = False) -> NumberAffix | None: + # Gate expensive regex work for oversized strings during automatic inference. + # Explicit coercion passes allow_expensive=True to bypass this gate. + if not allow_expensive and len(s) > INFERENCE_MAX_AFFIX_CHARS: + return None + m = _CURRENCY_RE.fullmatch(s) if m is not None: affix = m.group("affix") if not _is_valid_affix(affix, kind=AffixKind.CURRENCY, max_affix_len=max_affix_len): return None + try: + number = _parse_number(m.group("num")) + except ValueError: + return None return NumberAffix( kind=AffixKind.CURRENCY, affix=affix, space=(m.group("sp") == " "), - number=_parse_number(m.group("num")), + number=number, ) m = _UNITS_RE.fullmatch(s) @@ -94,11 +121,15 @@ def parse_number_affix(s: str, *, max_affix_len: int = 16) -> NumberAffix | None affix = m.group("affix") if not _is_valid_affix(affix, kind=AffixKind.UNITS, max_affix_len=max_affix_len): return None + try: + number = _parse_number(m.group("num")) + except ValueError: + return None return NumberAffix( kind=AffixKind.UNITS, affix=affix, space=(m.group("sp") == " "), - number=_parse_number(m.group("num")), + number=number, ) return None diff --git a/validation/_sanitize.py b/validation/_sanitize.py index 77b3f5f..7d5ec03 100644 --- a/validation/_sanitize.py +++ b/validation/_sanitize.py @@ -16,6 +16,7 @@ from pandas import Timestamp from core.datetime_parsing.nano_time import NanoTime +from core.raw_numeric import RawNumericValue _log = logging.getLogger(__name__) @@ -61,6 +62,12 @@ def to_jsonschema_input(value: Any, *, _lossy: list[bool] | None = None) -> Any: def _coerce(value: Any, lossy: list[bool]) -> Any: # noqa: PLR0911 + # ── raw, unsupported numeric literals → raw string ──────────────────── + # Validation sees the literal as text so schemas expecting a number report + # a normal type mismatch instead of crashing. Stored data is untouched. + if isinstance(value, RawNumericValue): + return value.raw + # ── gmpy2 exact numerics ────────────────────────────────────────────── try: import gmpy2 # noqa: PLC0415 diff --git a/validation/index.py b/validation/index.py index 27daed9..ff35dac 100644 --- a/validation/index.py +++ b/validation/index.py @@ -15,6 +15,7 @@ def __init__(self, issues: Iterable[ValidationIssue], root_data: Any): self._exact: dict[tuple[int, ...], list[ValidationIssue]] = defaultdict(list) self._severity: dict[tuple[int, ...], str] = {} self._ancestor: dict[tuple[int, ...], str] = {} + self._affected_paths: set[tuple[int, ...]] = set() self._issues: list[ValidationIssue] = [] self._count = 0 @@ -26,10 +27,12 @@ def __init__(self, issues: Iterable[ValidationIssue], root_data: Any): continue self._exact[model_path].append(issue) self._severity[model_path] = "error" + self._affected_paths.add(model_path) for i in range(len(model_path)): ancestor = model_path[:i] self._ancestor[ancestor] = "error" + self._affected_paths.add(ancestor) def severity_at(self, model_path: tuple[int, ...]) -> str | None: return self._severity.get(model_path) @@ -43,5 +46,11 @@ def ancestor_severity(self, model_path: tuple[int, ...]) -> str | None: def all_issues(self) -> list[ValidationIssue]: return list(self._issues) + def is_empty(self) -> bool: + return self._count == 0 + + def affected_paths(self) -> set[tuple[int, ...]]: + return set(self._affected_paths) + def __len__(self) -> int: return self._count