diff --git a/README.md b/README.md index ea5b13b..75f05c9 100644 --- a/README.md +++ b/README.md @@ -1,813 +1,177 @@ # Query Doctor -Last reviewed: 2026-08-10 +Last reviewed: 2026-08-11 Language: English | [Russian](README.ru.md) [![Safety CI](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/ci.yml) -[![Package CI](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/package.yml/badge.svg?branch=main)](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/package.yml) -[![Docs CI](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/docs.yml/badge.svg?branch=main)](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/docs.yml) [![CodeQL](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/codeql.yml/badge.svg?branch=main)](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/codeql.yml) [![PyPI](https://img.shields.io/pypi/v/query-doctor.svg?cacheSeconds=300)](https://pypi.org/project/query-doctor/) -Query Doctor is a local-first Big Data query diagnostic tool focused on Apache -Impala production triage, with bounded local Trino production lanes. It helps -operators rank suspicious Recent -queries, collect bounded profile context, derive deterministic evidence, and -generate validated reports without exposing raw SQL or raw profiles in trusted -browser/report surfaces. The Query Doctor 0.11.0 release adds a -supported container image, Kubernetes manifests, a Helm chart, raw-free Online -History, and bounded parsing of one already-provided Impala EXPLAIN artifact. +Find the Apache Impala queries worth investigating, and get a deterministic +answer about what to inspect and what to change — without your SQL or profiles +leaving the machine. -Core rule: +Query Doctor ranks suspicious Recent queries, collects bounded profile context, +derives evidence with plain Python rules, and generates validated reports. No +raw SQL and no raw profile text reaches the browser, the reports, or any remote +service. ```text Python owns facts. LLM owns wording only. ``` -Recent scan is the flagship workflow. Query ID diagnosis is secondary for one -known Impala query, with a local Trino lane for bounded retained-list Recent -diagnosis and one explicit Query ID when the required coordinator contracts are -configured. Query Optimizer is separate, read-only, and does not execute or echo -submitted SQL. +## Try It -## Quickstart +No cluster, no config, no credentials — synthetic data only: ```bash python -m pip install query-doctor -query-doctor-self-test -query-doctor-analyze \ - --profile-text ./exported-impala-profile.txt \ - --out cases/cm-corpus -query-doctor-web --corpus-dir cases/cm-corpus -``` - -Run `query-doctor-self-test` after installation to verify the installed console -scripts, synthetic demo generation, one-profile analysis, local web rendering, -Impala Web UI filename fallback, deterministic report generation, and corpus -smoke path. It uses synthetic local data only and does not contact Cloudera -Manager, impalad, Spark, Trino, Prometheus, Ollama, or external LLM services. -Package and release CI also run a README Quickstart smoke against a clean wheel -install: `query-doctor-self-test`, `query-doctor-analyze --profile-text -./exported-impala-profile.txt --out cases/cm-corpus`, and -`query-doctor-web --corpus-dir cases/cm-corpus`. - -The CLI profile-analysis path needs one exported Impala text profile and no -Cloudera Manager, Kerberos, local config, Prometheus, or LLM. A direct Impala -Web UI download named `profile__` can be used -as-is. The web UI opens staged cases from `--corpus-dir` automatically, and -local/private web sessions can also upload one exported text profile from the -Query Inbox page. See [Pick A First Path](#pick-a-first-path) for the demo and -Cloudera Manager options. - -## What It Is / Is Not - -Query Doctor is: - -- a local-first Impala production triage workbench with official bounded local - Trino production lanes; -- a deterministic evidence extractor; -- a Recent-query ranking workflow for operators and administrators; -- a safe report generator using validated facts; -- a practical tool for deciding what to inspect, change, and verify next; -- a containerized web application that can run as a read-only public demo or a - configured private operator service behind a trusted ingress/auth proxy; -- a Big Data SQL/lakehouse diagnostics wedge whose full production triage - engine is Apache Impala, with bounded raw-free local Trino production lanes - and future-engine preview seams. - -Query Doctor is not: - -- a generic AI chatbot over raw profiles; -- a replacement for the Impala Web UI; -- a tool that executes user SQL or optimizer draft SQL; -- a tool that sends raw SQL/profile data to remote services by default; -- a root-cause oracle; -- a broad live multi-engine query-history collector. - -## What It Does - -- Turns one exported Apache Impala text profile into a local deterministic - diagnosis through CLI staging, corpus browsing, or bounded local/private web - upload, without Cloudera Manager, Kerberos, metadata, Prometheus, or an LLM - provider. -- Scans completed Recent queries as the flagship production workflow, with - Running queries and one explicit Known Query ID as focused secondary modes. -- Works with Cloudera Manager when available, or with bounded direct Impala - daemon endpoints for non-Cloudera-Manager Impala clusters. -- Opens on Query Inbox: safe materialized Recent cases are shown immediately - when available, with an explicit empty/ready/running/partial/stale status - strip, safe source/window/time-range/query-type scope chips, - URL-driven source/window/time-range/workflow/query-type scope filters, - first-screen result presets, view-only owner/pool tag and opaque owner/pool - value filters, lifecycle, readiness, and action filters for owner-tagged - rows, pool-tagged rows, safe owner/pool values, clean analysis, status - follow-up, metadata availability, validated reports, optimizer guidance, and - recorded action outcomes, and New scan as the secondary control action. If - the selected scope filters do not - match the current - materialized snapshot, Query Inbox shows a safe filtered state and the New - scan form instead of showing stale rows; that form is prefilled from the - selected safe source/window/time-range/workflow/query-type filters when they - map to supported scan controls. When materialized results are open, New scan - is prefilled from safe source/window/time-range/workflow/query-type scope - defaults so a refresh does not require re-entering the same bounded scan - shape. Safe scope filters are preserved through New scan submit and job pages - without echoing arbitrary query parameters; owner/pool tag and opaque - owner/pool value filters, lifecycle/readiness/action result filters are - preserved only in result links, spill filtering, and pagination. - Window, UTC time-range, and query-type scope also have inline controls, so - operators can change the bounded lookback, exact finished-query range, and - short query type identifier directly from Query Inbox before materializing - that scope. -- Supports bounded local Trino production lanes when explicitly configured: - retained-list Recent, one explicit Query ID, raw-free materialized Details, - deterministic Python Report, and optimizer guidance over the same - server-owned case facts. -- Runs from a supported Docker image and Kubernetes manifests for read-only - synthetic demo or configured private web deployments. -- Optionally adds bounded Prometheus runtime summaries for direct Impala - workflows and bounded read-only Impala metadata through `impala-shell`. -- Ranks suspicious cases and action candidates from deterministic analyzer - facts, not LLM scoring. -- Presents Details as an analyst decision page: why the query matters, where to - inspect, what to try, how to verify a comparable rerun, and what evidence is - missing. -- Folds validated selected-case optimizer guidance into the same Recommended - change area when available. Report generation remains an explicit selected-case - action, and optimizer generation is offered only when deterministic rewrite - support marks the case safe to attempt. -- Generates trusted reports only after deterministic normalization, - sanitization, and validation. -- Provides a separate read-only Query Optimizer workflow for pasted SQL review, - plus explicit selected-case optimizer actions for server-owned analyzed - cases. -- Keeps raw SQL, raw profiles, raw metadata, local paths, secrets, subprocess - output, model/runtime internals, and raw artifact filenames out of browser and - trusted report surfaces. - -## Support Boundary - -| Surface | Current status | -| --- | --- | -| Query engine | Apache Impala is the full production triage engine. Trino has bounded local production support only for the raw-free lanes named below. | -| First-value intake | One local exported Impala text profile can be uploaded from a local/private web session or staged from CLI/manual inbox, redacted, analyzed, and opened from Known Query ID. | -| Recent scan | Cloudera Manager is the full Recent discovery/profile/metrics/events provider for Impala workflows. | -| Direct Impala | Bounded Recent scans, Running scans, and one Known Query ID through impalad daemon endpoints; no Cloudera Manager events and no SQL execution. | -| Runtime metrics | Optional bounded Prometheus summaries for configured direct Impala workflows; no arbitrary PromQL from users. | -| Metadata | Read-only allowlisted Impala metadata statements through `impala-shell`; no user SQL execution or unbounded metadata crawl. | -| Reports and optimizer | Python-owned facts and validation. Known Query ID prepares the deterministic Python report in its explicit submit job; LLM narratives remain explicit selected-case actions, and optimizer actions are shown only for cases with safe-to-attempt rewrite support. | -| Trusted SSO/auth proxy deployment | Query Doctor supports deployment behind a trusted SSO/auth proxy via `viewer_identity_header` for shared/non-local `owner_raw` access only after the raw-free D3 support-readiness gate passes. The proxy or ingress owns authentication, MFA, session lifecycle, token handling, and inbound-header stripping; Query Doctor only enforces the normalized viewer owner header against `query.user`. | -| Container/Kubernetes web deployment | Supported starting point through the official container image, `/healthz` and `/readyz` probes, raw-free deployment readiness summary, a read-only `public-demo` manifest, a configured private web manifest, a synthetic self-test Job, and the `deploy/helm/query-doctor` chart with a `helm test` hook. Kubernetes support does not add native auth, RBAC, sessions, multi-tenant isolation, an operator/CRD, arbitrary command running, SQL execution, or broader engine support. Shared configured deployments still require a trusted ingress/auth proxy and the same safety gates as any shared/non-local web bind. | -| Trino local | Local web Trino mode can read one bounded retained pruned coordinator query list for Recent diagnosis, then bounded pruned coordinator QueryInfo payloads for selected rows or one explicit Query ID, render deterministic compact diagnosis, materialize server-owned raw-free case artifacts, open a raw-free Details view, and generate deterministic Python Report plus optimizer guidance from those materialized case facts. `trino_support_mode=beta` keeps the legacy beta label; `trino_support_mode=production` marks the same bounded raw-free local lanes as local production support and removes that label. No Running scans, query-history crawling, metadata collection, LLM report output, Query Optimizer jobs, generated Trino SQL, SQL execution, or broader/shared Trino production triage support. | -| Spark | Bounded compact support surfaces only. Spark is not production engine support, live Recent scans, Details/trusted report output, optimizer behavior, raw event-log handling, Spark job execution, or Query Doctor-generated SQL. | - -The public GHCR release contains the Query Doctor web image. - -Trino compact/dev surfaces include offline or compact raw-free imports and checks: -sanitized evidence packages, bounded local compact imports, explicit -source-contract checks, a contract-gated local metadata CLI summary builder -and dev-only round-trip smoke gate that emit aggregate metadata coverage only, -and bounded pruned QueryInfo paths -documented in the engine docs. The only local production Trino product surfaces are local web retained-list -Recent diagnosis, One Query ID diagnosis, the raw-free Details view, -deterministic Python Report, and optimizer guidance for server-owned materialized cases from those -lanes. The diagnosis lanes require -`trino_support_mode=beta` or `trino_support_mode=production`, -`trino_coordinator_url`, and `trino_query_info_source_contract` in local config; -Recent also requires `trino_query_list_source_contract`. The legacy -`trino_beta_enabled=true` key remains beta-only for existing local setups and -must not be combined with `trino_support_mode=production`. Startup validation -checks local source contracts, safe coordinator URL shape, and optional auth -reference (`trino_auth_header_file` or local Kerberos/SPNEGO settings) before -the lane is marked configured. Configured beta sources are marked as -`Trino Beta Recent + One Query ID` or `Trino Beta One Query ID`; configured -production-mode sources are marked without the beta label. The Diagnose Engine -control narrows the Source cluster selector to Impala-capable sources or -Trino-ready sources before workflow selection, and stale or forged Trino submits -still fail closed before analysis or async job creation. -Coordinator URL, auth header references, raw QueryInfo, raw SQL, and local paths -stay out of the browser. Trino web case artifacts contain only the normalized -boundary, compact diagnosis, metadata-not-collected summary, typed analysis, and -safe analyzer facts view; Details opens only after those artifacts exist. -Python Report and optimizer guidance use the same raw-free facts and hide raw payloads, query IDs, -paths, LLM report output, Query Optimizer jobs, and generated SQL. Broader/shared Trino -live collection and broader Trino production triage remain unsupported. -Spark compact support surfaces are limited to bounded compact History Server -intake, compact evidence-package build/validation, and compact diagnosis; there -is no public Spark engine support. - -Future Big Data SQL/lakehouse live collectors, broader providers, prepared -event/log sources, and Cluster Doctor workflows remain roadmap seams, not -current support. For the detailed Trino and Spark preview command catalog, use -[docs/engines/README.md](docs/engines/README.md) and -[docs/engine-support-gap-matrix.md](docs/engine-support-gap-matrix.md). - -Direct Impala Recent and Running scans currently see only the query history -exposed by the configured coordinator daemon query-list endpoints. Upstream -Impala keeps the coordinator query log at `--query_log_size=200` entries by -default, further bounded by `--query_log_size_in_bytes`. Operators who need -deeper direct history can increase those Impala daemon settings on each -coordinator, while watching coordinator Web UI memory and `/queries` response -latency. Future deeper-history options are deliberately separate sources: -operator-managed read-only profile-log directory ingestion, or bounded external -history sources such as Loki or OpenSearch. They require explicit source -contracts, allowlists, byte/window bounds, and raw-free browser/report output; -the current product does not read coordinator filesystems, pod filesystems, or -external log indexes for direct Recent scans. - -Apache Impala also has upstream work around native AI query profile analysis. -Query Doctor aligns with that direction by staying focused on local-first -production triage across many queries, deterministic evidence, safe enrichment, -and validated raw-free reports. See -[docs/upstream-impala-ai-analyzer.md](docs/upstream-impala-ai-analyzer.md). - -## Install - -Install the current public package from PyPI: - -```bash -python3 -m venv .venv -. .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install query-doctor -query-doctor-self-test -``` - -`query-doctor-self-test` is the installed-package confidence check. It uses -synthetic local data to exercise packaged console scripts, one-profile analysis, -Impala Web UI filename fallback, local web rendering, deterministic reports, -and corpus smoke without contacting Cloudera Manager, impalad, Spark, Trino, -Prometheus, Ollama, or external LLM services. - -For local development from a checkout: - -```bash -python3 -m venv .venv -. .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install -e ".[dev]" -pre-commit install +query-doctor-web --public-demo ``` -Local JSON configuration is documented in [docs/configuration.md](docs/configuration.md). -The preferred workstation path is `~/.qdcreds/query-doctor-config.json`; -secrets stay in environment variables or local env files. Start from -`query-doctor-config.minimal.example.json` for a Cloudera Manager Impala -workflow, then use `query-doctor-config.example.json` only when you need the -advanced direct-Impala, Prometheus, metadata, or LLM routing fields. - -### Container And Kubernetes +Installing pulls zero third-party dependencies and takes a few seconds. The demo +is deterministic, local-only, read-only, and blocks every write action. -The release workflow will publish the 0.11.0 web image on GitHub Container -Registry after the `v0.11.0` release is approved: +![Synthetic Query Doctor Query Inbox status](docs/assets/demo_search.png) -```text -ghcr.io/alexandrefimov/query-doctor:0.11.0 -``` +![Synthetic Query Doctor finished queries results](docs/assets/demo_finished_queries.png) -After publication, the image defaults to the safe synthetic public demo: +## Diagnose A Real Query -```bash -docker run --rm -p 127.0.0.1:8765:8765 ghcr.io/alexandrefimov/query-doctor:0.11.0 -``` - -The image runs Query Doctor on supported Python 3.10 and also carries Kerberos -client tools and the isolated -`/opt/query-doctor/.venv-impala-shell/bin/impala-shell` runtime used by -configured Impala metadata collection. Metadata still requires explicit -coordinator, Kerberos cache, and redacted metadata settings in the private -deployment config. - -The Helm chart can optionally initialize and continuously refresh a Kerberos -ticket cache from an existing Kubernetes Secret in configured mode. A dedicated -sidecar owns refresh for the long-lived web pod, while finite collector and -worker Jobs obtain a fresh ticket at startup. The chart references only Secret -names and keys; it does not render keytabs, principals, ticket contents, or -inline credential values, and the web container never mounts the keytab. - -Configured non-local web deployments use conservative Recent defaults unless -private config explicitly overrides them: profile analysis limit `50`, overall -Recent parallelism `4`, metadata parallelism `2`, and metadata top limit `10`. -Raise those together with pod resources and a focused Kubernetes smoke. -Finished-query web Recent scans also reuse already analyzed cases from prior -local web batch outputs when the Query ID matches, the previous case completed -successfully, and the scan is still in the same safe redacted reuse contract. -The default local web cache lives under the system temp directory. Configured -Kubernetes and Helm examples set `recent_batch_root` to the dedicated -temp-backed case-PVC cache mount, so repeated finished Recent scans can reuse -analyzed profiles across pod restarts without broadening the raw-data boundary. -The Results coverage strip shows only the aggregate reused-profile count. A -second submit for the same running finished Recent scan redirects to the -existing job instead of starting duplicate collection. -For deployments that want a durable summary index, the batch CLI also accepts -`recent_history_db` / `--recent-history-db` to write raw-free Recent summary -history into a local SQLite database. That store records bounded summary -signals, selected/suspicion reason codes, profile status, and profile-budget -queue state without storing raw SQL or profile text. It also includes a -raw-free analyzed-result cache foundation keyed by Query ID, profile -fingerprint, and analyzer contract, plus raw-free profile-artifact metadata. -When `query-doctor-batch-recent --discover-only` runs with a history backend, -it also plans pending profile jobs for suspicious or selected summaries under -the existing Recent profile budget. The history database does not store raw -profile bytes, local paths, or artifact filenames. Explicit retention pruning -for old raw-free rows is opt-in through the `recent_history_*_retention_days` -config fields or matching `--recent-history-*-retention-days` CLI flags; -retention output stays aggregate-only. A separate -`query-doctor-recent-profile-remediation` maintenance CLI can dry-run or -explicitly `--apply` a bounded requeue of terminal failed profile jobs after an -operator fixes collection or materialization settings. Its output stays -aggregate-only and does not echo Query IDs, normalized error values, backend -paths, raw SQL, profile text, DSNs, or source filter values. -When the local web UI is configured with a Recent history backend, Query Inbox -can render retained raw-free summary rows as a read-only Online History view. -Those rows do not expose raw SQL or profile text and stay read-only until a -scan or profile worker materializes a safe Details snapshot. Online History -also shows the current raw-free profile worker state for retained rows, such as -pending, processing, retry-pending, analyzed, or failed, without exposing raw -profile bytes or resetting progressed states during rediscovery. The Query -Inbox status banner also summarizes retained versus shown rows, separates -profile states, rolls up normalized profile-worker error codes, and shows how -many analyzed rows have Details ready. It also shows safe Recent summary -collector freshness from the latest retained planning timestamp, so operators -can distinguish a healthy profile backlog from a stale producer without -exposing Query IDs, source keys, local paths, or raw collector payloads. -When configured with `recent_history_collector_summary_json`, the same banner -also projects the producer's last raw-free run summary as allowlisted status, -age, recorded-row, and planned-job counters. Invalid or unsafe retained -summaries degrade to blocked/unavailable status without rendering the -configured path, raw JSON, Query IDs, source keys, local paths, raw errors, or -retained free-form text. Safe next-step labels are derived from retained worker -states and aggregate counters so operators can distinguish retry, failed, -waiting, materialized backlog, and stale producer states without exposing raw -errors or trusting retained free-form text. If the web config points -`recent_history_operator_readiness_summary_json` at an already retained -`query_doctor_recent_history_operator_readiness_v1` summary, the same status -banner also shows only allowlisted operator readiness, evidence, reason-code -labels, schema, worker, worker-materialization, worker next-step, -profile-backlog health, backlog next-step, retention, and profile-remediation -counters and next-step text. It does not render the configured path, raw JSON, -raw SQL, profile text, local store paths, retained free-form remediation text, -or raw artifact names. An -analyzed row becomes an Online History Details link only when the history store -also has a ready raw-free analysis-cache payload linked through available -`fingerprint_only` profile-artifact metadata. Those Details snapshots can show -allowlisted score, recommendation, aggregate metadata coverage, and processing -timing fields, but still do not expose profile bytes, storage keys, local -paths, LLM reports, optimizer jobs, generated SQL, or SQL execution. -Configured deployments can instead set `recent_history_backend=postgres` and -provide the DSN through `QUERY_DOCTOR_RECENT_HISTORY_POSTGRES_DSN` or the -environment variable named by `recent_history_postgres_dsn_env`; the DSN value -must stay in the environment/Secret, not in Query Doctor JSON config. Container -images need the optional Postgres driver extra, for example -`QUERY_DOCTOR_INSTALL_EXTRAS=postgres scripts/build-image.sh query-doctor:dev`. -The Helm chart can reference that DSN from an existing Secret through -`recentHistory.postgres` in configured mode. It can also optionally render a -configured-mode CloudNativePG `Cluster`. When CNPG owns credential generation, -Query Doctor can consume the controller-generated application Secret through -its `uri` key; externally managed owner credentials continue to use a separate -operator-managed DSN Secret. The chart does not install CNPG or render Secret -objects. -Use `query-doctor-recent-history-postgres-readiness --json` inside the -configured image/pod environment to verify that the DSN env is present and the -history schema can initialize without printing the DSN, host, credentials, -paths, Query IDs, or raw payloads. Helm configured installs with -`recentHistory.postgres.enabled=true` run that raw-free readiness check as a -web pod initContainer by default. -Use `query-doctor-recent-history-operator-readiness` after retaining the -raw-free Postgres readiness and profile-worker summary JSON files, plus the -optional collector, retention, and profile-remediation summaries, to build one operator -handoff summary. The audit reads only those retained summaries, rejects unsafe -retained fields or values, does not print input paths, and does not contact -Postgres, Kubernetes, query engines, or profile collectors. The handoff summary -also includes accepted raw-free operation counters for schema readiness, -profile-worker jobs and materialized records, profile-backlog health, optional -collector producer status/freshness, optional retention deletes, and optional -remediation dry-run/apply counts. In Helm -configured mode, -`recentHistory.operatorReadiness.enabled=true` wires that handoff as a separate -CronJob after Postgres history, Postgres readiness, and -`recentProfileWorker.enabled=true` are enabled. The optional -`recentSummaryCollector.enabled=true` CronJob can run -`query-doctor-batch-recent --discover-only` on a schedule to write retained -summary rows and planned profile jobs to Postgres, plus a raw-free collector -run summary and progress JSONL stream to the case PVC, without collecting -profiles, running metadata SQL, LLM reports, or optimizer jobs. The chart -can pass that collector summary to operator readiness through -`recentHistory.operatorReadiness.collectorSummaryJson`. It writes the upstream -Postgres readiness, profile-worker, optional retention, and optional dry-run -profile-remediation summaries to the case PVC, then runs the -operator-readiness CronJob over only those raw-free files. The optional -`recentHistory.postgres.profileRemediation` CronJob runs only -`query-doctor-recent-profile-remediation --dry-run`; the chart does not render -an apply/requeue remediation job. That CronJob does not mount Query Doctor -config, collection credentials, Kerberos material, or source endpoint -configuration. The operator-readiness CronJob does not mount Query Doctor -config, collection credentials, the Postgres DSN Secret, Kerberos material, or -source endpoint configuration. -Use `query-doctor-recent-profile-worker` with the same batch/config flags to -claim jobs for the configured source, collect bounded profiles through the -existing Impala Recent collectors, run deterministic analysis only, and write -raw-free analysis-cache/profile-artifact metadata. The worker does not run LLM -reports, optimizer jobs, metadata SQL collection, or trusted browser raw-output -surfaces, and it cleans the worker-owned temporary `profile-worker-cases/job-*` -directory after each processed job. With `clusters[]` config and no explicit -source flags, the worker uses `active_cluster_key` or the only configured -cluster as its source. In Helm configured mode, -`recentProfileWorker.enabled=true` renders the same worker as a source-filtered -CronJob after Postgres history is enabled; it uses the configured Secret/env -DSN handoff, case PVC, and raw-free JSON output. Helm configured mode can also -render `recentHistory.postgres.retention` as a Postgres-only retention CronJob. -That job runs -`query-doctor-recent-history-retention` with only the DSN Secret env, deletes -old raw-free history rows by aggregate retention policy, and does not mount -Query Doctor config, collection credentials, Kerberos material, or case PVCs. -Profile-artifact metadata is `fingerprint_only`: storage kinds that would -retain profile bytes, local paths, object names, or external artifact -references are rejected until a bounded delete implementation exists. - -For an already installed configured release with metadata enabled, run the -bounded Kubernetes metadata smoke before a release handoff: +If you can export one Impala text profile from the Impala Web UI, that is the +whole setup — no Cloudera Manager, Kerberos, metadata, Prometheus, or LLM: ```bash -scripts/kubernetes-configured-metadata-smoke.sh +query-doctor-analyze --profile-text ./your-profile.txt --out cases/cm-corpus +query-doctor-web --corpus-dir cases/cm-corpus ``` -The smoke uses a local port-forward, submits a one-case Recent scan with -`metadata_top_limit=1`, and requires collected or partial metadata with table -context. It prints only aggregate job, status, and table-context counters. +A direct Impala Web UI download named `profile__` +works as-is. Local and private web sessions can also upload one exported profile +from the Query Inbox page. -For a configured Kubernetes release handoff after ingress/auth is wired, set -the expected external front-door values through environment variables and run: +Three entry paths, depending on the access you have: -```bash -scripts/kubernetes-configured-release-gate.sh -``` +| Door | Use when | +| --- | --- | +| One exported profile | You can get a text profile but cannot grant live access yet. | +| Synthetic demo | You want a read-only click-through with no real data. | +| Minimal CM scan | You have read-only Cloudera Manager access for an Impala service. | -The gate composes the configured metadata smoke, live unauthenticated auth -redirect smoke, and raw-free Kubernetes auth-front-door resource audit. +Full setup, options, and troubleshooting for each: [docs/first-path.md](docs/first-path.md). -From a checkout, build and smoke the same shape locally: +## Install ```bash -scripts/build-image.sh query-doctor:dev -scripts/image-smoke.sh query-doctor:dev +python3 -m venv .venv +. .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install query-doctor +query-doctor-self-test ``` -From an arm64 workstation, build an amd64 image before testing on amd64 -Kubernetes nodes: +`query-doctor-self-test` is the installed-package confidence check. It exercises +the packaged console scripts, one-profile analysis, local web rendering, +deterministic reports, and the corpus smoke path against synthetic data, without +contacting Cloudera Manager, impalad, Spark, Trino, Prometheus, Ollama, or any +external LLM service. -```bash -QUERY_DOCTOR_IMAGE_PLATFORM=linux/amd64 scripts/build-image.sh query-doctor:dev-amd64 -QUERY_DOCTOR_IMAGE_PLATFORM=linux/amd64 scripts/image-smoke.sh query-doctor:dev-amd64 -``` +Local JSON configuration is documented in +[docs/configuration.md](docs/configuration.md). The preferred workstation path is +`~/.qdcreds/query-doctor-config.json`, with secrets in environment variables or +local env files. Start from `query-doctor-config.minimal.example.json` for a +Cloudera Manager workflow; `query-doctor-config.example.json` adds the advanced +direct-Impala, Prometheus, metadata, and LLM fields. -Kubernetes manifests live in [deploy/kubernetes/](deploy/kubernetes/): - -- `public-demo.yaml`: read-only synthetic demo with no credentials and denied - pod egress. -- `configured-web.yaml`: private operator template with mounted config, - externally created credentials Secret, PVC-backed case storage, and probes. -- `self-test-job.yaml`: synthetic package confidence check that runs only - `query-doctor-self-test` without config, credentials, PVCs, live engine - access, optimizer jobs, metadata collection, or SQL. - -Web deployments expose `/healthz`, `/readyz`, and -`/deployment/readiness.json`. The companion -`query-doctor-deployment-readiness` CLI prints the same raw-free deployment -summary without starting the server. For configured or shared access, put Query -Doctor behind a trusted ingress/auth proxy; Kubernetes support does not add -native authentication, sessions, RBAC, tenant isolation, SQL execution, or -broader engine support inside Query Doctor. See -[docs/kubernetes-auth-front-door.md](docs/kubernetes-auth-front-door.md) and -`scripts/audit_kubernetes_auth_front_door.py` for the raw-free -oauth2-proxy/Keycloak-style front-door and NetworkPolicy acceptance check. Use -`scripts/kubernetes_auth_front_door_smoke.py` for a raw-free live -unauthenticated redirect smoke against the external ingress. -Configured private web pods should keep at least the packaged Recent-ready -resource baseline: `250m` CPU and `512Mi` memory requests, with `2` CPU and -`2Gi` memory limits. - -The Helm chart lives in [deploy/helm/query-doctor/](deploy/helm/query-doctor/). -It renders the same safe public-demo default and configured private mode, adds -values schema coverage, includes a synthetic `helm test` hook, and supports -generic user-provided pod labels and annotations without embedding a platform -controller contract. - -For disposable cluster-side checks, use `scripts/kubernetes-self-test-smoke.sh` -to install the chart in a temporary namespace, run the Helm self-test, capture -the synthetic Job logs, and clean up after the synthetic self-test. - -## Pick A First Path - -Use the smallest path that matches the access you have. - -| Door | Use when | Starts from | -| --- | --- | --- | -| One exported profile | You can get one Impala Web UI text profile, but cannot grant live access yet. | `query-doctor-analyze --profile-text`, `query-doctor-web` upload, or `query-doctor-web` with `manual_profile_dir` | -| Synthetic demo | You want a read-only local click-through with no real data. | `query-doctor-web --public-demo` | -| Minimal CM scan | You have read-only Cloudera Manager access for an Impala service. | `query-doctor-web` or `query-doctor-batch-recent` | - -### Door 1: Analyze One Exported Profile - -The lowest-setup path is one exported Apache Impala text profile to one local -diagnosis. This does not contact Cloudera Manager or impalad, does not require -Kerberos, metadata collection, Prometheus, or an LLM provider. +For development from a checkout: ```bash -query-doctor-analyze \ - --profile-text ./exported-impala-profile.txt \ - --out cases/cm-corpus -``` - -The command stages a collector-shaped local case under `cases/cm-corpus`, -redacts users, hosts, credentials, and common secret forms by default, writes -`analysis_facts.md` plus `analysis.json`, and prints the output case directory. -Use `--redact-identifiers` when the staged local artifacts may be shared. The -manual profile intake accepts exported text profiles only; JSON, Thrift, and -profile-v2 payloads remain outside this entry path. The CLI uses the Query ID -header from the exported profile, or the downloaded Impala Web UI filename when -it has the strict `profile__` shape. If neither is -readable, add `--query-id `; when multiple Query ID sources are -present, they must match before the local case is written. - -To inspect staged cases in the local UI, start `query-doctor-web --corpus-dir -cases/cm-corpus` from the same workspace. The Query Inbox page opens an Exported -Profiles results table from complete manual-profile cases in that corpus -without requiring Cloudera Manager settings, credentials, or default local -config. You can still choose `One Query ID` and enter the Query ID from a staged -profile to reopen that exact case. LLM narrative and optimizer actions remain -explicit buttons. - -For a local or private web session, you can also choose `One Query ID`, enter -the matching Impala Query ID in `Profile Query ID`, select one exported text -profile in `Exported profile`, and press `Upload`. The upload path is bounded by -`max_profile_bytes`, accepts exactly one multipart file, rejects JSON, Thrift, -and profile-v2 payloads in the same analyzer path, stages a server-owned case -under `corpus_dir`, and removes the temporary upload file after staging. The -public synthetic demo hides this form and blocks uploads before reading the -request body. - -You can also configure a local profile inbox for the web UI. Put the exported -text profile in `manual_profile_dir` using the Query ID slug as the file name -(for example, replace the Query ID separator with `_` and save -`.txt`), start `query-doctor-web`, choose `One Query ID`, and -enter the original Query ID. The web path stages and analyzes the local file -through the same text-only, bounded, redacted analyzer path. If the file -contains an embedded Query ID for a different query, staging fails closed before -replacing any existing case. For a self-contained one-profile workspace, set -both paths in an ignored local config file and keep generated cases outside the -source tree: - -```json -{ - "manual_profile_dir": "/path/to/profile-inbox", - "corpus_dir": "/path/to/query-doctor-cases", - "no_llm": true -} +python -m pip install -e ".[dev]" +pre-commit install ``` -Then start `query-doctor-web --config ./query-doctor-one-profile.json`. -Relative `corpus_dir` values in config resolve from the config file; the -`--corpus-dir` CLI flag resolves relative paths from the current directory. -When neither is set, the web UI stores generated Query ID cases under -`./cases/cm-corpus` from the directory where you started `query-doctor-web`. - -### Troubleshooting One Exported Profile - -- `Profile text does not include a Query ID`: keep the original Impala Web UI - download name when it has the strict - `profile__` shape, or pass - `--query-id `. Query Doctor also accepts a `Query ID:` header inside - the text export. If multiple Query ID sources are present, they must match. -- `Parsed operators: 0`: the case is still staged and can open in the UI, but - that text export did not include a parseable `ExecSummary`/operator table. - Use the preserved Impala text profile export when available; JSON, Thrift, - and profile-v2 payloads are outside this manual profile path. -- `query-doctor-web --corpus-dir cases/cm-corpus` asks for Cloudera Manager - settings: confirm that `query-doctor-analyze` wrote a complete case under the - same corpus directory you pass to web, and run web from the same workspace or - use an absolute `--corpus-dir`. - -### Door 2: Run The Synthetic Demo - -The synthetic demo is the fastest way to see the product. It is deterministic, -local-only, and contains no real SQL, profiles, metadata, hostnames, users, or -credentials. +## Main Commands -```bash -query-doctor-web --public-demo -``` +| Command | What it does | +| --- | --- | +| `query-doctor-web` | Local browser UI: Recent scan, Running now, one Known Query ID, Details, explicit report and optimizer actions | +| `query-doctor-analyze` | Deterministic analyzer over one staged exported profile or collected case files | +| `query-doctor-batch-recent` | Headless bounded Recent scan and ranking | +| `query-doctor-report` | Validated report generation from Python-owned facts | +| `query-doctor-optimize-query` | Read-only review of pasted SQL | +| `query-doctor-self-test` | Installed-package confidence check over synthetic data | -This one-command mode is documented in [docs/demo-mode.md](docs/demo-mode.md). -It generates the synthetic demo pack in a dedicated temp directory, forces -Python-only mode, ignores default local config, and blocks all POST actions. +Every packaged console script accepts `--help`. From an uninstalled checkout, +use `python -m query_doctor.cli.`. -If you need to inspect or reuse the generated pack manually, use the lower-level -commands: +## Container And Kubernetes ```bash -query-doctor-demo-preflight -DEMO_PACK="${TMPDIR:-/tmp}/query-doctor-demo-pack" -query-doctor-demo --out "$DEMO_PACK" --overwrite -QUERY_DOCTOR_ACTION_OUTCOMES_PATH="$DEMO_PACK/action_outcomes.jsonl" \ - query-doctor-web --host 127.0.0.1 --port 8766 --batch-summary "$DEMO_PACK/batch_summary.json" +docker run --rm -p 127.0.0.1:8765:8765 ghcr.io/alexandrefimov/query-doctor:0.11.0 ``` -Open the localhost URL printed by `query-doctor-web`. Start with -`/?query_group=workloads#scan-context` to show the compact Scan context -workload follow-up links and local synthetic action outcomes before opening -Workload Details. -When a Recent summary has repeated safe row-level workload fingerprints but no -materialized workload payload, the UI derives bounded repeated workload details -from the sanitized rows; those details have no baseline or regression claim -until local history evidence is available. - -The local web UI starts with Query Inbox status, safe -source/window/time-range/query-type scope, a compact Filters and views -disclosure for URL-driven source/window/time-range/workflow/query-type scope -filters, result presets, view-only owner/pool tag and opaque owner/pool value -filters, lifecycle, readiness, and action filters, plus synthetic Finished -Queries results when safe materialized cases are available. The -collapsed New scan form keeps safe refresh defaults from that materialized -scope without auto-running collection: - -The main results table is decision-focused: attention rows show one short -deterministic classification, priority, duration, owner context, and a clear -Details path. Repeated workloads keep priority, p95, total observed impact, and -top owner in the inbox; p50, pool, bottleneck, and supporting evidence remain -available in Workload Details. +The image defaults to the safe synthetic public demo. It runs on Python 3.10 and +carries the Kerberos client tools and isolated `impala-shell` runtime used by +configured metadata collection. -![Synthetic Query Doctor Query Inbox status](docs/assets/demo_search.png) - -![Synthetic Query Doctor finished queries results](docs/assets/demo_finished_queries.png) +Kubernetes manifests, probes, resource baselines, Recent history storage, and +the Helm chart are documented in +[deploy/kubernetes/README.md](deploy/kubernetes/README.md), +[deploy/helm/query-doctor/README.md](deploy/helm/query-doctor/README.md), and +[docs/recent-history-store.md](docs/recent-history-store.md). Shared deployments +require a trusted ingress/auth proxy; Kubernetes support adds no native auth, +RBAC, sessions, or tenant isolation inside Query Doctor. -The synthetic demo pack contains eleven sanitized Impala cases covering -workload follow-up, repeated patterns, trusted optimizer recommendations, -stats maintenance, storage/HDFS follow-up, frequent-short workloads, mixed -signals, unknown but useful limited evidence, and direct-Impala compatibility. -It also includes two read-only raw-free Trino Beta demo cases rendered from -static compact diagnosis facts, without contacting a Trino coordinator or -enabling Details, reports, optimizer guidance, generated SQL, or SQL execution. -See -[docs/demo-cases.md](docs/demo-cases.md) for the full scenario list and talk -track. +## Safety -### Door 3: Run A Minimal Cloudera Manager Scan +- Deterministic Python analysis is the only trusted source of diagnostic facts. + LLM output is untrusted until normalized, sanitized, and validated. +- Trusted browser and report surfaces never show raw SQL, raw profiles, raw + metadata, local paths, secrets, subprocess output, or raw artifact filenames. + The isolated owner-only source view is the one narrow, gated exception. +- External collection is explicit, bounded, read-only, and redacted by default. + Query Doctor never executes user SQL or optimizer draft SQL. -Use this when you have read-only Cloudera Manager access for an Impala service. -Keep secrets in the shell environment or a local env file, not in JSON config. -Create `~/.qdcreds/cm-ro.env` with `CM_USERNAME` plus `CM_PASSWORD` or -`CM_TOKEN` before sourcing it. +`privacy_mode` defaults to `true`; `no_llm=true` keeps reports and optimizer +output on deterministic Python facts alone. Full contract: +[docs/safety-contract.md](docs/safety-contract.md). Reviewer-oriented overview: +[docs/security-model.md](docs/security-model.md). -```bash -mkdir -p ~/.qdcreds -cp query-doctor-config.minimal.example.json ~/.qdcreds/query-doctor-config.json -# Edit ~/.qdcreds/query-doctor-config.json with CM URL, cluster, service, and CA bundle if needed. -set -a -source ~/.qdcreds/cm-ro.env -set +a -query-doctor-web \ - --config ~/.qdcreds/query-doctor-config.json \ - --host 127.0.0.1 \ - --port 8765 -``` +## Scope -For a headless bounded Recent scan without automatic LLM reports: +Apache Impala is the full production triage engine. Trino has bounded local +production support for retained-list Recent, one Query ID, raw-free Details, +deterministic reports, and optimizer guidance. Spark has compact support +surfaces only, and is not production engine support. -```bash -query-doctor-batch-recent \ - --config ~/.qdcreds/query-doctor-config.json \ - --recent-window-minutes 60 \ - --triage-profile-limit 10 \ - --top-reports 0 -``` +The complete surface-by-surface contract, including what is deliberately out of +scope, is in [docs/support-boundary.md](docs/support-boundary.md). -The minimal path uses Cloudera Manager for Impala Recent discovery and profile -collection. Add metadata, CM time-series, direct Impala, Prometheus, or LLM -settings only after this basic scan path works. See -[docs/configuration.md](docs/configuration.md) and -[docs/credentials.md](docs/credentials.md). -For repeated safe local runs, `--reuse-analyzed-profiles-from ` can -reuse completed analyzed cases from direct child `query-doctor-*` batch outputs -when the Query ID and explicit profile reuse contract match. - -## Main Workflows - -- `query-doctor-self-test --help`: local installed-package confidence check - over synthetic data and core offline user paths. -- `query-doctor-deployment-readiness --help`: raw-free deployment summary for - the same settings used by `query-doctor-web`. -- `query-doctor-web --help`: local browser UI for Recent scan, Running now, one - Known Query ID, Details pages, explicit report actions, and explicit - details-page optimizer actions. -- `query-doctor-batch-recent --help`: headless Recent scan workflow for bounded - local collection and ranking. -- `query-doctor-analyze --help`: deterministic analyzer over collected local - case files, or over one staged local exported Impala text profile. -- `query-doctor-report --help`: validated report generation from Python-owned - facts. -- `query-doctor-optimize-query --help`: read-only pasted-SQL optimizer review. - -Every packaged console script accepts `--help`. Root-level compatibility -launchers have been removed; use `query-doctor-*` commands or -`python -m query_doctor.cli.` from an uninstalled checkout. - -Query Doctor is supported as a single-user, local-first tool run by an operator -with their own local Cloudera Manager, Kerberos, Impala, Prometheus, and LLM -credentials. Use localhost or a tightly controlled local bind for the web UI. -Do not deploy ordinary local mode as a shared service without a separate design -for authentication, authorization, tenant/job isolation, audit logging, -TLS/reverse-proxy trust, and resource limits. Shared public demos should use -the read-only `query-doctor-web --public-demo` mode. Shared `owner_raw` source -access requires authenticated per-request viewer identity through an explicit -`viewer_identity_header` supplied only by a trusted SSO/auth proxy or ingress -that strips inbound copies of the same header and sets exactly one normalized -simple owner value. Query Doctor supports that deployment pattern after the -raw-free support-readiness gate in -[docs/owner-raw-d3-deployment.md](docs/owner-raw-d3-deployment.md). A -dev-only Keycloak/oauth2-proxy smoke is available in -[docs/dev-sso-keycloak.md](docs/dev-sso-keycloak.md) to test the front-door -viewer header contract locally; `scripts/dev_sso_keycloak_smoke.py` verifies -the running local compose path with raw-free output. The dev smoke is not -production SSO support evidence and does not add native SSO to Query Doctor. - -## Safety Model - -- Python/analyzer-owned facts are the only trusted diagnostic evidence. -- Raw LLM output is untrusted unless normalized, sanitized, and validated. -- Trusted browser/report surfaces must not expose raw SQL, raw profiles, raw - metadata, local paths, secrets, subprocess output, model/runtime internals, or - raw artifact filenames. The isolated owner-only selected-case source surface is - the narrow raw-SQL browser exception. -- External collection must be explicit, bounded, read-only, redacted, and safe - by default. -- Local config `privacy_mode` defaults to `true`; disabling it can relax local - artifact identifier/host masking, but trusted browser/report surfaces still do - not show raw SQL, profiles, or metadata. -- Local config `no_llm=true` keeps report and optimizer actions on deterministic - Python-owned output. -- SQL browser exceptions are selected-case and owner-gated: Details can show a - validated optimizer SQL draft for an explicit safe-to-attempt optimizer action - when `source_visibility=owner_raw`, and the isolated owner-only source view can - show read-only original SQL for an authorized query owner. On localhost, raw - viewer subjects come from local collectable owner users; on shared binds they - must come from authenticated per-request viewer identity. The original source - view can be disabled globally with `owner_raw_source_enabled=false` or - `--disable-owner-raw-source`, and each attempt writes a reason-coded raw-free - server audit line. The default `safe` mode shows trusted - recommendations/no-rewrite guidance instead. -- Query Optimizer accepts only a single safe read-only statement and never - executes pasted SQL. - -See [docs/safety-contract.md](docs/safety-contract.md) for the full trust and -redaction contract, including the narrow owner-only source exception. For -shared/non-local owner-raw deployment, use -[docs/owner-raw-d3-deployment.md](docs/owner-raw-d3-deployment.md). For a -reviewer-oriented overview, see -[docs/security-model.md](docs/security-model.md). +Query Doctor is supported as a single-user, local-first tool. Do not deploy +ordinary local mode as a shared service without the separate design described in +that document. ## Documentation -Start with [docs/README.md](docs/README.md). It separates current user docs, -operations guides, architecture contracts, audit docs, and supporting -references. - -High-value next reads: - -- [docs/demo-mode.md](docs/demo-mode.md): synthetic demo pack generation and - README screenshot refresh path. -- [docs/DEMO.md](docs/DEMO.md): localhost UI demo runbook and talk track. -- [docs/local-smoke.md](docs/local-smoke.md): local validation and smoke checks. -- [deploy/kubernetes/README.md](deploy/kubernetes/README.md): container image, - Kubernetes manifests, probes, and deployment boundaries. -- [deploy/helm/query-doctor/README.md](deploy/helm/query-doctor/README.md): - Helm chart modes, validation, and deployment boundaries. -- [docs/credentials.md](docs/credentials.md): local credentials layout. -- [docs/roadmap.md](docs/roadmap.md): implemented scope and planned seams. -- [docs/security-model.md](docs/security-model.md): public security, privacy, - and demo-sharing overview. -- [docs/query-optimizer-contract.md](docs/query-optimizer-contract.md): - optimizer trust boundary. -- [docs/release-checklist.md](docs/release-checklist.md): final tag, - package-index, and visibility-change checklist. - -The canonical documentation language is English. The Russian layer is limited -to [README.ru.md](README.ru.md) plus practical user/operator instructions under -[docs/i18n/ru/](docs/i18n/ru/); internal, agent, research, release, and engine -deep-dive docs stay English-only. - -## Development Checks - -For ordinary changes, run focused tests for the touched area and always run: +Start with [docs/README.md](docs/README.md), which separates user docs, +operations guides, architecture contracts, and references. High-value next +reads: [docs/first-path.md](docs/first-path.md), +[docs/demo-mode.md](docs/demo-mode.md), +[docs/configuration.md](docs/configuration.md), +[docs/credentials.md](docs/credentials.md), +[docs/roadmap.md](docs/roadmap.md). -```bash -git diff --check -``` +The canonical documentation language is English. The Russian layer is +[README.ru.md](README.ru.md) plus practical user and operator instructions under +[docs/i18n/ru/](docs/i18n/ru/). + +## Development -Use [docs/agent-quickstart.md](docs/agent-quickstart.md) and +For ordinary changes, run focused tests for the touched area and always run +`git diff --check`. Use [docs/agent-quickstart.md](docs/agent-quickstart.md) and [docs/test-matrix.md](docs/test-matrix.md) to choose focused validation. Before -release cleanup or public-sharing work, broaden to: +release or public-sharing work, broaden to: ```bash pre-commit run --all-files @@ -816,28 +180,17 @@ query-doctor-demo-preflight --public-release ``` Stage only explicit files. Do not commit generated cases, reports, local -configs, credentials, raw profiles, raw metadata, or temporary outputs. +configs, credentials, raw profiles, raw metadata, or temporary outputs. See +[CONTRIBUTING.md](CONTRIBUTING.md). -## Public Status +## Licensing And Status -This repository is public. Public source releases start at `v0.4.2`; -`v0.11.0` continues that public source release line. Older package-index -releases remain visible on -[query-doctor on PyPI](https://pypi.org/project/query-doctor/) where needed for -installed-artifact history. The public license is Apache-2.0. - -PyPI publishing uses GitHub OIDC Trusted Publishing. The repository-side -`testpypi` and `pypi` environments require maintainer approval and do not use -stored package-index API tokens. - -Query Doctor web container images are published to GitHub Container Registry as +Apache-2.0. See [LICENSE](LICENSE). Public source releases start at `v0.4.2`; +`v0.11.0` continues that line. PyPI publishing uses GitHub OIDC Trusted +Publishing with maintainer-approved environments and no stored API tokens. Web +container images are published to GitHub Container Registry as `ghcr.io/alexandrefimov/query-doctor:` from GitHub Releases. -## Licensing - -Query Doctor is licensed under the Apache License, Version 2.0 -(`Apache-2.0`). See [LICENSE](LICENSE). - Apache, Apache Impala, and Impala are trademarks of The Apache Software Foundation. Query Doctor is an independent project and is not endorsed by The Apache Software Foundation or the Apache Impala project. diff --git a/README.ru.md b/README.ru.md index c98e427..dfcea71 100644 --- a/README.ru.md +++ b/README.ru.md @@ -1,608 +1,180 @@ # Query Doctor -Last reviewed: 2026-08-10 +Last reviewed: 2026-08-11 Язык: [English](README.md) | Русский -Query Doctor - локальный диагностический инструмент для Big Data-запросов, -сфокусированный на production triage для Apache Impala и bounded local Trino -production lanes. Он помогает операторам ранжировать подозрительные Recent -queries, собирать ограниченный контекст профиля, извлекать детерминированные -evidence и генерировать -проверенные отчеты без показа raw SQL или raw profiles в trusted -browser/report surfaces. -В релизе 0.11.0 добавлены supported container image, Kubernetes -manifests, Helm chart, raw-free Online History и bounded parsing одного уже -предоставленного Impala EXPLAIN artifact. +[![Safety CI](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/ci.yml) +[![CodeQL](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/codeql.yml/badge.svg?branch=main)](https://github.com/alexandrefimov/Query-Doctor/actions/workflows/codeql.yml) +[![PyPI](https://img.shields.io/pypi/v/query-doctor.svg?cacheSeconds=300)](https://pypi.org/project/query-doctor/) -Главное правило: +Находит запросы Apache Impala, которые стоит разобрать, и даёт +детерминированный ответ, куда смотреть и что менять — так, что SQL и профили +не покидают машину. + +Query Doctor ранжирует подозрительные Recent-запросы, собирает ограниченный +контекст профиля, извлекает факты обычными правилами на Python и генерирует +проверенные отчёты. Ни raw SQL, ни текст профиля не попадают в браузер, в +отчёты и ни в один внешний сервис. ```text Python owns facts. LLM owns wording only. ``` -Recent scan - основной workflow. Диагностика по Query ID вторична и рассчитана -на один известный Impala query; есть локальный Trino lane для bounded -retained-list Recent diagnosis и одного explicit Query ID, если настроены -нужные coordinator contracts. Query Optimizer отдельный, read-only, не -выполняет SQL и не показывает отправленный SQL обратно. +## Попробовать -## Quickstart +Без кластера, конфигурации и доступов — только синтетические данные: ```bash python -m pip install query-doctor -query-doctor-self-test -query-doctor-analyze \ - --profile-text ./exported-impala-profile.txt \ - --out cases/cm-corpus -query-doctor-web --corpus-dir cases/cm-corpus +query-doctor-web --public-demo ``` -После установки запустите `query-doctor-self-test`: он проверяет installed -console scripts, synthetic demo generation, анализ одного профиля, fallback по -имени скачанного Impala Web UI профиля, local web rendering, deterministic -report generation и corpus smoke. Команда использует только synthetic local -data и не обращается к Cloudera Manager, impalad, Spark, Trino, Prometheus, -Ollama или external LLM services. -Package и release CI также запускают README Quickstart smoke против clean wheel -install: `query-doctor-self-test`, `query-doctor-analyze --profile-text -./exported-impala-profile.txt --out cases/cm-corpus` и -`query-doctor-web --corpus-dir cases/cm-corpus`. - -Для CLI-пути анализа профиля нужен один экспортированный Impala text profile; не -нужны Cloudera Manager, Kerberos, local config, Prometheus или LLM. Direct -Impala Web UI download с именем `profile__` можно -использовать как есть. Web UI автоматически открывает staged cases из -`--corpus-dir`, а local/private web sessions могут загрузить один exported text -profile прямо с Query Inbox page. Demo и Cloudera Manager варианты описаны ниже в -[Выберите первый путь](#выберите-первый-путь). - -## Что это / что это не - -Query Doctor это: - -- локальный рабочий инструмент для Impala production triage с официальными - bounded local Trino production lanes; -- извлекатель детерминированных diagnostic facts; -- workflow ранжирования Recent queries для операторов и администраторов; -- безопасный генератор отчетов на проверенных фактах; -- практический инструмент для решения, что смотреть, что менять и как - проверять; -- containerized web application для read-only public demo или configured - private operator service за trusted ingress/auth proxy; -- первый узкий слой диагностики Big Data SQL/lakehouse, где full production - triage engine - Apache Impala, с bounded raw-free local Trino production - lanes и preview seams для будущих движков. - -Query Doctor это не: - -- универсальный AI-чатбот поверх raw profiles; -- замена Impala Web UI; -- инструмент выполнения пользовательского SQL или чернового SQL из optimizer; -- инструмент, который по умолчанию отправляет сырой SQL или данные профилей во - внешние сервисы; -- оракул первопричин; -- broad live multi-engine query-history collector. - -## Что он делает - -- Превращает один экспортированный текстовый профиль Apache Impala в локальный - deterministic diagnosis через CLI staging, corpus browsing или bounded - local/private web upload, без Cloudera Manager, Kerberos, metadata, - Prometheus или LLM provider. -- Сканирует завершенные Recent queries как основной production workflow; - Running queries и один explicit Known Query ID остаются сфокусированными - вторичными режимами. -- Работает с Cloudera Manager, когда он доступен, или с ограниченными direct - Impala daemon endpoints для non-Cloudera-Manager Impala clusters. -- Открывается на Query Inbox: safe materialized Recent cases показываются - сразу, если они уже есть, со status strip для - empty/ready/running/partial/stale, safe source/window/time-range/query-type - scope chips, URL-driven source/window/time-range/workflow/query-type scope - filters, first-screen result presets, view-only owner/pool tag и opaque - owner/pool value filters, lifecycle, readiness и action filters для - owner-tagged rows, pool-tagged rows, safe owner/pool values, clean analysis, - status follow-up, metadata availability, validated reports, optimizer - guidance и recorded action outcomes, а также New scan как вторичным control - action. - Если выбранные scope filters не совпадают с текущим materialized snapshot, - Query Inbox показывает safe filtered state и New scan form вместо stale rows; - эта форма получает prefill из выбранных safe - source/window/time-range/workflow/query-type filters, когда они соответствуют - поддерживаемым scan controls. Когда materialized results открыты, New scan - получает safe source/window/time-range/workflow/query-type refresh defaults, - чтобы повторный bounded scan не требовал заново вводить тот же scope. Safe - scope filters сохраняются через New scan submit и job pages без echo - произвольных query parameters; owner/pool tag и opaque owner/pool value - filters, lifecycle/readiness/action result filters сохраняются только в - result links, spill filtering и pagination. Window, UTC time-range и - query-type scope теперь имеют inline controls, чтобы менять bounded lookback, exact - finished-query range и короткий query type identifier прямо из Query Inbox - перед materialization этого scope. -- Поддерживает bounded local Trino production lanes при явной local config: - retained-list Recent, один explicit Query ID, raw-free materialized Details, - deterministic Python Report и optimizer guidance over the same server-owned - case facts. -- Запускается из supported Docker image и Kubernetes manifests для read-only - synthetic demo или configured private web deployments. -- Опционально добавляет ограниченные Prometheus runtime summaries для direct - Impala workflows и ограниченные read-only метаданные Impala через - `impala-shell`. -- Ранжирует подозрительные cases и action candidates по детерминированным - analyzer facts, а не по LLM scoring. -- Показывает Details как analyst decision page: почему запрос важен, где - проверить, что попробовать, как проверить comparable rerun и каких evidence - не хватает. -- Встраивает validated selected-case optimizer guidance в ту же зону - Recommended change, когда оно доступно, но report и optimizer generation - остаются отдельными explicit actions. -- Генерирует доверенные отчеты только после детерминированной нормализации, - очистки и проверки. -- Дает отдельный read-only Query Optimizer workflow для разбора вставленного - SQL и explicit selected-case optimizer actions для уже разобранных сервером - случаев. -- Не показывает сырой SQL, сырые профили, сырые метаданные, локальные пути, - секреты, subprocess output, model/runtime internals и raw artifact filenames - в browser и trusted report surfaces. - -## Граница поддержки - -| Surface | Current status | -| --- | --- | -| Query engine | Apache Impala - full production triage engine. Trino имеет bounded local production support только для raw-free lanes ниже. | -| First-value intake | Один локальный exported Impala text profile можно загрузить из local/private web session или staged через CLI/manual inbox, затем redacted/analyzed и открыть через Known Query ID. | -| Recent scan | Cloudera Manager - полный Recent discovery/profile/metrics/events provider для Impala workflows. | -| Direct Impala | Bounded Recent scans, Running scans и один Known Query ID через impalad daemon endpoints; без Cloudera Manager events и без SQL execution. | -| Runtime metrics | Optional bounded Prometheus summaries для configured direct Impala workflows; без arbitrary PromQL from users. | -| Metadata | Read-only allowlisted Impala metadata statements через `impala-shell`; без user SQL execution и unbounded metadata crawl. | -| Reports and optimizer | Python-owned facts и validation. Known Query ID готовит deterministic Python report в explicit submit-job; LLM narratives и optimizer actions остаются explicit selected-case actions. | -| Container/Kubernetes web deployment | Supported starting point через official container image, `/healthz` и `/readyz` probes, raw-free deployment readiness summary, read-only `public-demo` manifest, configured private web manifest, synthetic self-test Job и `deploy/helm/query-doctor` chart с `helm test` hook. Kubernetes support не добавляет native auth, RBAC, sessions, multi-tenant isolation, operator/CRD, arbitrary command running, SQL execution или broader engine support. Shared configured deployments все равно требуют trusted ingress/auth proxy и те же safety gates, что любой shared/non-local web bind. | -| Trino local | Local web Trino mode может прочитать один bounded retained pruned coordinator query list для Recent diagnosis, затем bounded pruned coordinator QueryInfo payloads для выбранных rows или одного explicit Query ID, показать deterministic compact diagnosis, materialize server-owned raw-free case artifacts, открыть raw-free Details view и создать deterministic Python Report плюс optimizer guidance для этих materialized cases. `trino_support_mode=beta` сохраняет legacy beta label; `trino_support_mode=production` помечает те же bounded raw-free local lanes как local production support и убирает этот label. Без Running scans, query-history crawling, metadata collection, LLM report output, Query Optimizer jobs, generated Trino SQL, SQL execution и broader/shared Trino production triage support. | -| Spark | Только bounded compact support surfaces. Spark не является production engine support, live Recent scans, Details/trusted report output, optimizer behavior, raw event-log handling, Spark job execution или Query Doctor-generated SQL. | - -Публичный GHCR release содержит Query Doctor web image. - -Trino compact/dev surfaces включают offline/local raw-free imports and checks: -bounded local pruned QueryInfo import принимает one explicit compact sanitized -local pruned QueryInfo JSON через `query-doctor-trino-query-info-pruned-import` -после source-contract checks. `query-doctor-trino-coordinator-query-info-pruned-probe` -и `query-doctor-trino-coordinator-query-info-pruned-import` могут использовать -`--auth-header-file`, но safe output не печатает auth header paths или values. -Local production Trino product surfaces - local web retained-list Recent diagnosis, One -Query ID diagnosis, raw-free Details view, deterministic Python Report и optimizer guidance для -server-owned materialized cases из этих lanes. Diagnosis lanes требуют -`trino_support_mode=beta` или -`trino_support_mode=production`, `trino_coordinator_url` и -`trino_query_info_source_contract` в local config; Recent дополнительно требует -`trino_query_list_source_contract`. Legacy `trino_beta_enabled=true` остается -beta-only switch для existing local setups и не должен комбинироваться с -`trino_support_mode=production`. Startup validation проверяет local source -contracts, safe coordinator URL shape и optional auth reference -(`trino_auth_header_file` или local Kerberos/SPNEGO settings) до того, как lane -считается configured. Configured beta sources помечаются в source selector как -`Trino Beta Recent + One Query ID` или `Trino Beta One Query ID`; configured -production-mode sources используют labels без `Beta`. Diagnose Engine control -сужает Source cluster selector до Impala-capable sources или Trino-ready sources -до выбора workflow, а stale или forged Trino submits все равно fail closed до -analysis или async job creation. Этот lane не делает network read вне explicit -bounded probe/import, reject-ит raw QueryInfo fields вроде query text, session -fields, endpoint URLs, object names и stage/task detail. Details открывается -только после materialized artifacts. Python Report и optimizer guidance используют те же raw-free -facts и не показывают raw payloads, query IDs, paths, LLM report output, -Query Optimizer jobs или generated SQL; Running scans, Query Optimizer jobs и -metadata collection остаются unavailable. -Broader/shared Trino live collection и broader Trino production triage остаются unsupported. - -Spark compact support surfaces остаются только compact History Server intake, -compact evidence-package build/validation и compact diagnosis; no public Spark -engine support, без Recent scans, Details/trusted report output, optimizer -behavior, raw event logs или Spark job execution. - -Будущие Big Data SQL/lakehouse live collectors, более широкие providers, -подготовленные event/log sources и Cluster Doctor workflows остаются roadmap -seams, а не текущей поддержкой. Detailed Trino/Spark preview command catalog: -[docs/engines/README.md](docs/engines/README.md). Текущий support/research -boundary: [docs/engine-support-gap-matrix.md](docs/engine-support-gap-matrix.md). - -В Apache Impala также появилась upstream работа над native AI query profile -analysis. Query Doctor выравнивается с этим направлением и остается локальным -production triage по многим queries, с детерминированными доказательствами, -безопасным enrichment и проверенными отчетами без сырых данных. См. -[docs/upstream-impala-ai-analyzer.md](docs/upstream-impala-ai-analyzer.md). +Установка не тянет ни одной сторонней зависимости и занимает несколько секунд. +Демо детерминированное, локальное, read-only и блокирует любые действия записи. -## Установка - -Установка текущего публичного пакета из PyPI: +![Synthetic Query Doctor Query Inbox status](docs/assets/demo_search.png) -```bash -python3 -m venv .venv -. .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install query-doctor -query-doctor-self-test -``` +![Synthetic Query Doctor finished queries results](docs/assets/demo_finished_queries.png) -`query-doctor-self-test` - confidence check установленного пакета. Он использует -synthetic local data и проверяет packaged console scripts, анализ одного -профиля, fallback по имени скачанного Impala Web UI профиля, local web -rendering, deterministic reports и corpus smoke без доступа к Cloudera Manager, -impalad, Spark, Trino, Prometheus, Ollama или external LLM services. +## Разобрать настоящий запрос -Для локальной разработки из checkout: +Если можно выгрузить один текстовый профиль Impala из Impala Web UI — это вся +подготовка. Cloudera Manager, Kerberos, metadata, Prometheus и LLM не нужны: ```bash -python3 -m venv .venv -. .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install -e ".[dev]" -pre-commit install +query-doctor-analyze --profile-text ./your-profile.txt --out cases/cm-corpus +query-doctor-web --corpus-dir cases/cm-corpus ``` -Локальная JSON-конфигурация описана в -[docs/configuration.md](docs/configuration.md). Предпочтительный путь на -рабочей станции: `~/.qdcreds/query-doctor-config.json`; secrets остаются в -environment variables или local env files. - -### Container and Kubernetes - -После одобрения и публикации release `v0.11.0` workflow опубликует web image в -GitHub Container Registry: +Скачанный из Impala Web UI файл с именем вида +`profile__` подходит как есть. В локальной и +приватной веб-сессии профиль можно ещё и загрузить со страницы Query Inbox. -```text -ghcr.io/alexandrefimov/query-doctor:0.11.0 -``` +Три входа, по уровню доступа: -После публикации image по умолчанию стартует безопасный synthetic public demo: +| Вход | Когда подходит | +| --- | --- | +| Один выгруженный профиль | Профиль достать можно, живой доступ пока не дадут. | +| Синтетическое демо | Хочется покликать read-only без реальных данных. | +| Минимальный CM-скан | Есть read-only доступ к Cloudera Manager для сервиса Impala. | -```bash -docker run --rm -p 127.0.0.1:8765:8765 ghcr.io/alexandrefimov/query-doctor:0.11.0 -``` +Полная настройка, опции и разбор ошибок для каждого входа: +[docs/first-path.md](docs/first-path.md). -Из checkout можно собрать и проверить тот же shape локально: +## Установка ```bash -scripts/build-image.sh query-doctor:dev -scripts/image-smoke.sh query-doctor:dev +python3 -m venv .venv +. .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install query-doctor +query-doctor-self-test ``` -С arm64 workstation собирайте amd64 image перед проверкой на amd64 Kubernetes -nodes: +`query-doctor-self-test` — проверка установленного пакета. Прогоняет +консольные скрипты, анализ одного профиля, локальный рендер веб-интерфейса, +детерминированные отчёты и corpus smoke на синтетических данных, не обращаясь +к Cloudera Manager, impalad, Spark, Trino, Prometheus, Ollama и внешним +LLM-сервисам. -```bash -QUERY_DOCTOR_IMAGE_PLATFORM=linux/amd64 scripts/build-image.sh query-doctor:dev-amd64 -QUERY_DOCTOR_IMAGE_PLATFORM=linux/amd64 scripts/image-smoke.sh query-doctor:dev-amd64 -``` +Локальная JSON-конфигурация описана в +[docs/i18n/ru/configuration.md](docs/i18n/ru/configuration.md). Рекомендуемый +путь на рабочей станции — `~/.qdcreds/query-doctor-config.json`, секреты в +переменных окружения или локальных env-файлах. Для Cloudera Manager начинайте с +`query-doctor-config.minimal.example.json`; в +`query-doctor-config.example.json` лежат продвинутые поля для прямого доступа к +Impala, Prometheus, metadata и LLM. -Kubernetes manifests находятся в [deploy/kubernetes/](deploy/kubernetes/): - -- `public-demo.yaml`: read-only synthetic demo без credentials и с denied pod - egress. -- `configured-web.yaml`: private operator template с mounted config, внешне - созданным credentials Secret, PVC для cases, persistent Recent cache и - probes. -- `self-test-job.yaml`: synthetic package confidence check, который запускает - только `query-doctor-self-test` без config, credentials, PVC, live engine - access, optimizer jobs, metadata collection или SQL. - -Configured Kubernetes и Helm examples ставят `recent_batch_root` в dedicated -temp-backed case-PVC cache mount. Поэтому повторные finished Recent scans могут -переиспользовать уже проанализированные profiles после pod restart, если Query -ID и explicit profile reuse contract совпадают. Results coverage показывает -только aggregate reused-profile count, а второй submit того же running -finished Recent scan возвращает существующий job вместо duplicate collection. - -Web deployments expose `/healthz`, `/readyz` и -`/deployment/readiness.json`. Companion CLI -`query-doctor-deployment-readiness` печатает тот же raw-free deployment summary -без запуска сервера. Для configured или shared access ставьте Query Doctor за -trusted ingress/auth proxy; Kubernetes support не добавляет native -authentication, sessions, RBAC, tenant isolation, SQL execution или broader -engine support внутри Query Doctor. См. -[docs/kubernetes-auth-front-door.md](docs/kubernetes-auth-front-door.md) и -`scripts/audit_kubernetes_auth_front_door.py` для raw-free acceptance check -oauth2-proxy/Keycloak-style front door и NetworkPolicy isolation. - -Helm chart находится в [deploy/helm/query-doctor/](deploy/helm/query-doctor/). -Он рендерит тот же safe public-demo default и configured private mode, добавляет -values schema coverage, включает synthetic `helm test` hook и поддерживает -generic user-provided pod labels и annotations без встроенного platform -controller contract. - -Для disposable cluster-side checks используйте -`scripts/kubernetes-self-test-smoke.sh`: он ставит chart во временный namespace, -запускает `helm test --logs` и чистит ресурсы после synthetic self-test. - -## Выберите первый путь - -Берите самый короткий путь, который соответствует вашему уровню доступа. - -| Дверь | Когда подходит | С чего начать | -| --- | --- | --- | -| Один экспортированный профиль | Можно получить один текстовый профиль из Impala Web UI, но live access пока недоступен. | `query-doctor-analyze --profile-text`, `query-doctor-web` upload или `query-doctor-web` с `manual_profile_dir` | -| Synthetic demo | Нужно read-only local click-through без реальных данных. | `query-doctor-web --public-demo` | -| Minimal CM scan | Есть read-only Cloudera Manager access к Impala service. | `query-doctor-web` или `query-doctor-batch-recent` | - -### Дверь 1: анализ одного экспортированного профиля - -Самый простой путь - один экспортированный Apache Impala text profile в один -локальный diagnosis. Он не обращается к Cloudera Manager или impalad, не требует -Kerberos, metadata collection, Prometheus или LLM provider. +Для разработки из чекаута: ```bash -query-doctor-analyze \ - --profile-text ./exported-impala-profile.txt \ - --out cases/cm-corpus -``` - -Команда создает collector-shaped local case под `cases/cm-corpus`, по умолчанию -редактирует users, hosts, credentials и common secret forms, пишет -`analysis_facts.md` и `analysis.json`, затем печатает output case directory. -Используйте `--redact-identifiers`, если staged local artifacts могут быть -переданы наружу. Manual profile intake принимает только exported text profiles; -JSON, Thrift и profile-v2 payloads остаются вне этого entry path. CLI берет -Query ID из header внутри exported profile или из downloaded Impala Web UI -filename строгой формы `profile__`. Если ни один -источник не читается, добавьте `--query-id `; когда есть несколько -источников Query ID, они должны совпасть до записи local case. - -Чтобы открыть staged cases в local UI, запустите `query-doctor-web ---corpus-dir cases/cm-corpus` из того же workspace. Query Inbox page откроет -таблицу Exported Profiles из complete manual-profile cases в этом corpus без -Cloudera Manager settings, credentials или default local config. Вы все еще -можете выбрать `One Query ID` и ввести Query ID из staged profile, чтобы открыть -именно этот case. LLM narrative и optimizer actions остаются explicit buttons. - -В local или private web session можно также выбрать `One Query ID`, ввести -matching Impala Query ID в `Profile Query ID`, выбрать один exported text -profile в `Exported profile` и нажать `Upload`. Upload path bounded через -`max_profile_bytes`, принимает ровно один multipart file, отклоняет JSON, Thrift -и profile-v2 payloads тем же analyzer path, stages server-owned case под -`corpus_dir` и удаляет временный upload file после staging. Public synthetic -demo скрывает эту форму и блокирует uploads до чтения request body. - -Также можно настроить local profile inbox для web UI. Положите exported text -profile в `manual_profile_dir`, назвав файл slug-версией Query ID: замените -разделитель Query ID на `_` и сохраните как `.txt`. Затем -запустите `query-doctor-web`, выберите `One Query ID` и введите исходный Query -ID. Web path staged/analyzed этот local file через тот же text-only, bounded, -redacted analyzer path. Если файл содержит embedded Query ID другого запроса, -staging fail-closed до замены существующего case. -Для self-contained one-profile workspace задайте оба пути в ignored local -config и держите generated cases вне source tree: - -```json -{ - "manual_profile_dir": "/path/to/profile-inbox", - "corpus_dir": "/path/to/query-doctor-cases", - "no_llm": true -} +python -m pip install -e ".[dev]" +pre-commit install ``` -Затем запустите `query-doctor-web --config ./query-doctor-one-profile.json`. -Relative `corpus_dir` в config разрешается от файла config; CLI-флаг -`--corpus-dir` разрешает relative path от current directory. Если оба способа -не заданы, web UI хранит generated Query ID cases в `./cases/cm-corpus` от -директории, где запущен `query-doctor-web`. - -### Troubleshooting для одного экспортированного профиля - -- `Profile text does not include a Query ID`: сохраните исходное имя скачанного - из Impala Web UI файла, если оно имеет строгую форму - `profile__`, или передайте - `--query-id `. Query Doctor также принимает `Query ID:` header - внутри text export. Если есть несколько источников Query ID, они должны - совпасть. -- `Parsed operators: 0`: case все равно staged и может открыться в UI, но этот - text export не содержит parseable `ExecSummary`/operator table. По возможности - используйте сохраненный Impala text profile export; JSON, Thrift и profile-v2 - payloads остаются вне manual profile path. -- `query-doctor-web --corpus-dir cases/cm-corpus` просит Cloudera Manager - settings: проверьте, что `query-doctor-analyze` записал complete case в тот - же corpus directory, который передан web, и запускайте web из того же - workspace или используйте absolute `--corpus-dir`. - -### Дверь 2: synthetic demo - -Synthetic demo - самый быстрый способ увидеть продукт. Он deterministic, -local-only и не содержит real SQL, profiles, metadata, hostnames, users или -credentials. +## Основные команды -```bash -query-doctor-web --public-demo -``` +| Команда | Что делает | +| --- | --- | +| `query-doctor-web` | Локальный веб-интерфейс: Recent scan, Running now, один известный Query ID, Details, явные действия для отчётов и оптимизатора | +| `query-doctor-analyze` | Детерминированный анализ одного выгруженного профиля или собранных файлов кейса | +| `query-doctor-batch-recent` | Headless-скан Recent с ограничениями и ранжированием | +| `query-doctor-report` | Генерация проверенного отчёта из фактов, которыми владеет Python | +| `query-doctor-optimize-query` | Read-only разбор вставленного SQL | +| `query-doctor-self-test` | Проверка установленного пакета на синтетических данных | -Этот one-command mode описан в [docs/demo-mode.md](docs/demo-mode.md). Он сам -генерирует synthetic demo pack в dedicated temp directory, включает Python-only -mode, игнорирует default local config и блокирует все POST actions. +Каждый упакованный консольный скрипт принимает `--help`. Из неустановленного +чекаута — `python -m query_doctor.cli.`. -Если нужно вручную посмотреть или переиспользовать generated pack, используйте -lower-level commands: +## Контейнер и Kubernetes ```bash -query-doctor-demo-preflight -DEMO_PACK="${TMPDIR:-/tmp}/query-doctor-demo-pack" -query-doctor-demo --out "$DEMO_PACK" --overwrite -QUERY_DOCTOR_ACTION_OUTCOMES_PATH="$DEMO_PACK/action_outcomes.jsonl" \ - query-doctor-web --host 127.0.0.1 --port 8766 --batch-summary "$DEMO_PACK/batch_summary.json" +docker run --rm -p 127.0.0.1:8765:8765 ghcr.io/alexandrefimov/query-doctor:0.11.0 ``` -Откройте localhost URL, который напечатает `query-doctor-web`. Начните с -`/?query_group=workloads#scan-context`, чтобы показать компактный Scan context, -workload follow-up links и local synthetic action outcomes перед открытием -Workload Details. - -Локальный web UI начинается со status strip Query Inbox, safe -source/window/time-range/query-type scope, compact Filters and views disclosure -для URL-driven source/window/time-range/workflow/query-type scope filters, -result presets, view-only owner/pool tag и opaque owner/pool value filters, -lifecycle, readiness и action filters, а также synthetic Finished Queries -results, если safe materialized cases уже доступны. -Collapsed New scan form -сохраняет safe refresh defaults из этого materialized scope без automatic -collection: - -Основная results table сфокусирована на решении: строки attention показывают -одну короткую детерминированную классификацию, priority, duration, owner context -и явный переход в Details. Для repeated workloads в inbox остаются priority, -p95, суммарный observed impact и top owner; p50, pool, bottleneck и supporting -evidence доступны в Workload Details. +Образ по умолчанию поднимает безопасное синтетическое публичное демо. Работает +на Python 3.10 и несёт клиентские инструменты Kerberos и изолированный +`impala-shell`, который используется при настроенном сборе metadata. -![Synthetic Query Doctor Query Inbox status](docs/assets/demo_search.png) +Kubernetes-манифесты, пробы, базовые ресурсы, хранение истории Recent и Helm +chart описаны в [deploy/kubernetes/README.md](deploy/kubernetes/README.md), +[deploy/helm/query-doctor/README.md](deploy/helm/query-doctor/README.md) и +[docs/recent-history-store.md](docs/recent-history-store.md). Общие развёртывания +требуют доверенного ingress/auth-прокси; Kubernetes не добавляет внутрь Query +Doctor ни аутентификацию, ни RBAC, ни сессии, ни изоляцию арендаторов. -![Synthetic Query Doctor finished queries results](docs/assets/demo_finished_queries.png) - -Synthetic demo pack содержит eleven sanitized Impala cases: workload follow-up, -repeated patterns, trusted optimizer recommendations, stats maintenance, -storage/HDFS follow-up, frequent-short workloads, mixed signals, -unknown-but-useful limited evidence и direct-Impala compatibility. Также в нем -есть two read-only raw-free Trino Beta demo cases из static compact diagnosis -facts, без Trino coordinator, Details, reports, optimizer behavior, generated -SQL или SQL execution. Полный список scenarios: -[docs/demo-cases.md](docs/demo-cases.md). +## Безопасность -### Дверь 3: minimal Cloudera Manager scan +- Детерминированный анализ на Python — единственный доверенный источник + диагностических фактов. Вывод LLM недоверенный, пока не нормализован, + не санитизирован и не провалидирован. +- Доверенные поверхности браузера и отчётов никогда не показывают raw SQL, + raw-профили, raw metadata, локальные пути, секреты, вывод подпроцессов и + имена артефактов. Изолированный owner-only просмотр исходника — единственное + узкое исключение под явным гейтом. +- Внешний сбор всегда явный, ограниченный, read-only и с редактированием по + умолчанию. Query Doctor не выполняет ни пользовательский SQL, ни черновики + оптимизатора. -Используйте этот путь, если есть read-only Cloudera Manager access к Impala -service. Secrets держите в shell environment или local env file, не в JSON -config. Перед `source` создайте `~/.qdcreds/cm-ro.env` с `CM_USERNAME` плюс -`CM_PASSWORD` или `CM_TOKEN`. +`privacy_mode` по умолчанию `true`; `no_llm=true` оставляет отчёты и +оптимизатор на детерминированных фактах Python. Полный контракт: +[docs/i18n/ru/safety-contract.md](docs/i18n/ru/safety-contract.md). Обзор для +ревьюера: [docs/i18n/ru/security-model.md](docs/i18n/ru/security-model.md). -```bash -mkdir -p ~/.qdcreds -cp query-doctor-config.minimal.example.json ~/.qdcreds/query-doctor-config.json -# Отредактируйте CM URL, cluster, service и CA bundle при необходимости. -set -a -source ~/.qdcreds/cm-ro.env -set +a -query-doctor-web \ - --config ~/.qdcreds/query-doctor-config.json \ - --host 127.0.0.1 \ - --port 8765 -``` +## Границы поддержки -Headless bounded Recent scan без automatic LLM reports: +Apache Impala — полноценный production-движок для triage. У Trino есть +bounded local production support для retained-list Recent, одного Query ID, +raw-free Details, детерминированных отчётов и подсказок оптимизатора. У Spark +только compact-поверхности, это не поддержка движка в production. -```bash -query-doctor-batch-recent \ - --config ~/.qdcreds/query-doctor-config.json \ - --recent-window-minutes 60 \ - --triage-profile-limit 10 \ - --top-reports 0 -``` +Полный контракт по каждой поверхности, включая то, что сознательно вне +области, — в [docs/support-boundary.md](docs/support-boundary.md). -Minimal path использует Cloudera Manager для Impala Recent discovery и profile -collection. Metadata, CM time-series, direct Impala, Prometheus или LLM settings -лучше добавлять только после того, как базовый scan path заработал. См. -[docs/configuration.md](docs/configuration.md) и -[docs/credentials.md](docs/credentials.md). -Для повторных safe local runs `--reuse-analyzed-profiles-from ` -может переиспользовать completed analyzed cases из direct child -`query-doctor-*` batch outputs, когда Query ID и explicit profile reuse -contract совпадают. - -## Основные workflows - -- `query-doctor-self-test --help`: local installed-package confidence check по - synthetic data и core offline user paths. -- `query-doctor-deployment-readiness --help`: raw-free deployment summary для - тех же settings, что использует `query-doctor-web`. -- `query-doctor-web --help`: local browser UI для Recent scan, Running now, - одного Known Query ID, Details pages, explicit report actions и explicit - details-page optimizer actions. -- `query-doctor-batch-recent --help`: headless Recent scan для bounded local - collection и ranking. -- `query-doctor-analyze --help`: deterministic analyzer по collected local case - files или одному staged local exported Impala text profile. -- `query-doctor-report --help`: validated report generation из Python-owned - facts. -- `query-doctor-optimize-query --help`: read-only pasted-SQL optimizer review. - -Все packaged console scripts принимают `--help`. Root-level compatibility -launchers удалены; используйте `query-doctor-*` commands или -`python -m query_doctor.cli.` из checkout без установки. - -Query Doctor поддержан как single-user, local-first tool, запускаемый -оператором со своими local Cloudera Manager, Kerberos, Impala, Prometheus и LLM -credentials. Для web UI используйте localhost или tightly controlled local -bind. Не разворачивайте ordinary local mode как shared service без отдельного -дизайна authentication, authorization, tenant/job isolation, audit logging, -TLS/reverse-proxy trust и resource limits. Shared public demos должны -использовать read-only режим `query-doctor-web --public-demo`. Shared -`owner_raw` source access требует authenticated per-request viewer identity: -сейчас это явный `viewer_identity_header`, который выставляет только trusted -auth proxy или ingress после удаления входящих копий того же header. - -## Safety model - -- Python/analyzer-owned facts - единственное trusted diagnostic evidence. -- Raw LLM output недоверенный, пока не пройдет normalization, sanitization и - validation. -- Trusted browser/report surfaces не должны раскрывать raw SQL, raw profiles, - raw metadata, local paths, secrets, subprocess output, model/runtime internals - или raw artifact filenames. Isolated owner-only selected-case source surface - - узкое browser-исключение для raw SQL. -- External collection должен быть explicit, bounded, read-only, redacted и safe - by default. -- Local config `privacy_mode` по умолчанию `true`; отключение может ослабить - local artifact identifier/host masking, но trusted browser/report surfaces все - равно не показывают raw SQL, profiles или metadata. -- Local config `no_llm=true` оставляет report и optimizer actions на - deterministic Python-owned output. -- SQL browser exceptions остаются selected-case и owner-gated: Details может - показать validated optimizer SQL draft для explicit optimizer action при - `source_visibility=owner_raw`, а isolated owner-only source view может - показать read-only original SQL для authorized query owner. На localhost raw - viewer subjects берутся из local collectable owner users; на shared bind они - должны приходить из authenticated per-request viewer identity. Original - source view можно глобально отключить через - `owner_raw_source_enabled=false` или `--disable-owner-raw-source`; каждая - попытка пишет reason-coded raw-free server audit line. Default `safe` mode - показывает trusted recommendations/no-rewrite guidance вместо SQL draft. -- Query Optimizer принимает только один safe read-only statement и никогда не - выполняет pasted SQL. - -Полный trust/redaction contract: [docs/safety-contract.md](docs/safety-contract.md). -Reviewer-oriented обзор: [docs/security-model.md](docs/security-model.md). +Query Doctor поддерживается как однопользовательский локальный инструмент. Не +разворачивайте обычный локальный режим как общий сервис без отдельного +проектирования, описанного в том же документе. ## Документация -Начинайте с [docs/README.md](docs/README.md). Он разделяет current user docs, -operations guides, architecture contracts, audit docs и supporting references. - -Полезные следующие документы: - -- [docs/demo-mode.md](docs/demo-mode.md): synthetic demo pack и README - screenshot refresh path. -- [docs/DEMO.md](docs/DEMO.md): localhost UI demo runbook и talk track. -- [docs/local-smoke.md](docs/local-smoke.md): local validation и smoke checks. -- [deploy/kubernetes/README.md](deploy/kubernetes/README.md): container image, - Kubernetes manifests, probes и deployment boundaries. -- [deploy/helm/query-doctor/README.md](deploy/helm/query-doctor/README.md): - Helm chart modes, validation и deployment boundaries. -- [docs/credentials.md](docs/credentials.md): локальная раскладка credentials. -- [docs/roadmap.md](docs/roadmap.md): implemented scope и planned seams. -- [docs/query-optimizer-contract.md](docs/query-optimizer-contract.md): - optimizer trust boundary. -- [docs/release-checklist.md](docs/release-checklist.md): final tag, - package-index и visibility-change checklist. +Начинайте с [docs/i18n/ru/README.md](docs/i18n/ru/README.md). Дальше по +пользе: [docs/first-path.md](docs/first-path.md), +[docs/i18n/ru/demo-mode.md](docs/i18n/ru/demo-mode.md), +[docs/i18n/ru/configuration.md](docs/i18n/ru/configuration.md), +[docs/i18n/ru/credentials.md](docs/i18n/ru/credentials.md), +[docs/i18n/ru/roadmap.md](docs/i18n/ru/roadmap.md). -Английская документация является канонической. Русский слой ограничен этим -README и практическими user/operator инструкциями в -[docs/i18n/ru/](docs/i18n/ru/); internal, agent, research, release и engine -deep-dive docs остаются English-only. +Канонический язык документации — английский. Русский слой ограничен этим файлом +и практическими инструкциями для пользователей и операторов в +[docs/i18n/ru/](docs/i18n/ru/). -## Development checks +## Разработка -Для обычных изменений запускайте focused tests для touched area и всегда: - -```bash -git diff --check -``` - -Для выбора focused validation используйте +Для обычных изменений прогоняйте точечные тесты по затронутой области и всегда +`git diff --check`. Выбрать проверки помогают [docs/agent-quickstart.md](docs/agent-quickstart.md) и -[docs/test-matrix.md](docs/test-matrix.md). Перед release cleanup или -public-sharing work расширяйте gate до: +[docs/test-matrix.md](docs/test-matrix.md). Перед релизом или публикацией: ```bash pre-commit run --all-files @@ -610,29 +182,19 @@ scripts/local_gate.sh query-doctor-demo-preflight --public-release ``` -Stage only explicit files. Не commit generated cases, reports, local configs, -credentials, raw profiles, raw metadata или temporary outputs. +Добавляйте в коммит только явные файлы. Не коммитьте сгенерированные кейсы, +отчёты, локальные конфиги, учётные данные, raw-профили, raw metadata и +временные выводы. См. [CONTRIBUTING.md](CONTRIBUTING.md). -## Public status +## Лицензия и статус -Репозиторий публичный. Public source releases начинаются с `v0.4.2`; release -candidate `v0.11.0` продолжит эту public source release line после одобрения и -публикации. Older package-index releases остаются видимыми на -[query-doctor on PyPI](https://pypi.org/project/query-doctor/) для -installed-artifact history. Public license is Apache-2.0. - -PyPI publishing использует GitHub OIDC Trusted Publishing. Repository-side -`testpypi` и `pypi` environments требуют maintainer approval и не используют -stored package-index API tokens. - -Query Doctor web container images публикуются в GitHub Container Registry как +Apache-2.0, см. [LICENSE](LICENSE). Публичные релизы исходного кода начинаются +с `v0.4.2`; `v0.11.0` продолжает эту линию. Публикация на PyPI идёт через +GitHub OIDC Trusted Publishing с окружениями, требующими подтверждения +мейнтейнера, без хранимых API-токенов. Образы веб-контейнера публикуются в +GitHub Container Registry как `ghcr.io/alexandrefimov/query-doctor:` из GitHub Releases. -## Licensing - -Query Doctor лицензирован под Apache License, Version 2.0 (`Apache-2.0`). -См. [LICENSE](LICENSE). - -Apache, Apache Impala и Impala являются товарными знаками The Apache Software -Foundation. Query Doctor - независимый проект; он не одобрен The Apache -Software Foundation или проектом Apache Impala. +Apache, Apache Impala и Impala — товарные знаки The Apache Software Foundation. +Query Doctor — независимый проект, не одобренный The Apache Software Foundation +или проектом Apache Impala. diff --git a/docs/README.md b/docs/README.md index 9c0c5f6..e7ad1c5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -58,6 +58,12 @@ See [../AGENTS.md](../AGENTS.md) for the full agent hard-rules list. - [../README.ru.md](../README.ru.md): Russian companion for the public project README. Longer Russian pages are limited to practical user/operator instructions under [i18n/ru/](i18n/ru/). +- [first-path.md](first-path.md): the three entry paths in full — one exported + profile, synthetic demo, minimal Cloudera Manager scan — with setup options + and troubleshooting. +- [support-boundary.md](support-boundary.md): surface-by-surface support + contract, what is deliberately out of scope, and the supported deployment + shape. - [DEMO.md](DEMO.md): localhost UI demo runbook, main surfaces, safety rules, and public demo storyline. - [../deploy/kubernetes/README.md](../deploy/kubernetes/README.md): supported @@ -200,6 +206,8 @@ Status legend: | [upstream-impala-ai-analyzer.md](upstream-impala-ai-analyzer.md) | active | Upstream Impala AI analyzer alignment and Query Doctor differentiation. | | [impala-profile-counter-caveats.md](impala-profile-counter-caveats.md) | active | Impala profile dialect and counter evidence-tier caveats. | | [engine-expansion-plan.md](engine-expansion-plan.md) | active | Future source-provider, engine, metrics, and storage expansion order. | +| [support-boundary.md](support-boundary.md) | active | Surface-by-surface support contract, explicit non-scope, Trino/Spark detail, direct Impala history depth, and supported deployment shape. | +| [first-path.md](first-path.md) | active | The three entry paths in full, with setup options and troubleshooting. | | [engine-support-gap-matrix.md](engine-support-gap-matrix.md) | active | Current engine support status, normalized fact coverage, fixture/research boundaries, and second-engine support gaps. | | [customer-readiness-priorities.md](customer-readiness-priorities.md) | active | Near-term Impala-first customer-readiness backlog for demo, config, docs, UI, and validation focus. | | [trino-beta-ui-readiness.md](trino-beta-ui-readiness.md) | active | Acceptance gate for showing the local production Trino retained-list Recent, One Query ID, raw-free materialized Details, Python Report, and optimizer guidance UI surfaces while keeping Trino Beta as the legacy label. | diff --git a/docs/changelog.md b/docs/changelog.md index e5f8bde..2d5f0b3 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -24,7 +24,24 @@ release notes remain in [release-notes-0.10.0.md](release-notes-0.10.0.md), ## Unreleased -- The public distribution boundary now rejects platform-specific controller +- The README is now a landing page rather than the full contract. Its first + command runs without any input the reader has to obtain, the synthetic demo + leads instead of sitting several hundred lines down, and the one-profile path + follows it. The support contract, the What Is / Is Not scope lists, the Trino + and Spark detail, the direct-Impala history depth note, and the supported + deployment shape moved to `support-boundary.md` with a Russian companion under + `i18n/ru/`; the three entry paths moved to `first-path.md` with their setup + options and troubleshooting unchanged. No support claim changed. Tests that + pinned individual sentences to `README.md` now assert against the document + that carries the claim, so a landing page can stay short without dropping a + guarantee. +- Added a browser analyzer prototype under `web/`. It runs the deterministic + analyzer in the visitor's browser through Pyodide, so an exported Impala text + profile can be diagnosed without a server, an upload, or any request after the + page loads. The analyzer core needed no changes, and the page applies the same + default redaction `query-doctor-analyze` applies during case staging before + rendering facts. The prototype is not deployed, not linked from the README, + and not part of the published distribution. metadata across public documentation, source, deployment assets, packaging, tests, and release tooling. Its public-release mode scans every commit's metadata and UTF-8 text blobs after the configured public base using a diff --git a/docs/first-path.md b/docs/first-path.md new file mode 100644 index 0000000..faa5b84 --- /dev/null +++ b/docs/first-path.md @@ -0,0 +1,183 @@ +# Pick A First Path + +Last reviewed: 2026-08-11 + +Use the smallest path that matches the access you have. The README shows the +one-line version; this page has the full setup, options, and troubleshooting. + +| Door | Use when | Starts from | +| --- | --- | --- | +| One exported profile | You can get one Impala Web UI text profile, but cannot grant live access yet. | `query-doctor-analyze --profile-text`, `query-doctor-web` upload, or `query-doctor-web` with `manual_profile_dir` | +| Synthetic demo | You want a read-only local click-through with no real data. | `query-doctor-web --public-demo` | +| Minimal CM scan | You have read-only Cloudera Manager access for an Impala service. | `query-doctor-web` or `query-doctor-batch-recent` | + +## Door 1: Analyze One Exported Profile + +The lowest-setup path is one exported Apache Impala text profile to one local +diagnosis. This does not contact Cloudera Manager or impalad, and does not +require Kerberos, metadata collection, Prometheus, or an LLM provider. + +```bash +query-doctor-analyze \ + --profile-text ./your-profile.txt \ + --out cases/cm-corpus +``` + +The command stages a collector-shaped local case under `cases/cm-corpus`, +redacts users, hosts, credentials, and common secret forms by default, writes +`analysis_facts.md` plus `analysis.json`, and prints the output case directory. +Use `--redact-identifiers` when the staged local artifacts may be shared. + +The manual profile intake accepts exported text profiles only; JSON, Thrift, and +profile-v2 payloads remain outside this entry path. The CLI uses the Query ID +header from the exported profile, or the downloaded Impala Web UI filename when +it has the strict `profile__` shape. If neither is +readable, add `--query-id `; when multiple Query ID sources are +present, they must match before the local case is written. + +To inspect staged cases in the local UI, start `query-doctor-web --corpus-dir +cases/cm-corpus` from the same workspace. The Query Inbox page opens an Exported +Profiles results table from complete manual-profile cases in that corpus without +requiring Cloudera Manager settings, credentials, or default local config. You +can still choose `One Query ID` and enter the Query ID from a staged profile to +reopen that exact case. LLM narrative and optimizer actions remain explicit +buttons. + +### Upload from a local web session + +For a local or private web session, choose `One Query ID`, enter the matching +Impala Query ID in `Profile Query ID`, select one exported text profile in +`Exported profile`, and press `Upload`. The upload path is bounded by +`max_profile_bytes`, accepts exactly one multipart file, rejects JSON, Thrift, +and profile-v2 payloads in the same analyzer path, stages a server-owned case +under `corpus_dir`, and removes the temporary upload file after staging. The +public synthetic demo hides this form and blocks uploads before reading the +request body. + +### Local profile inbox + +Put the exported text profile in `manual_profile_dir` using the Query ID slug as +the file name (for example, replace the Query ID separator with `_` and save +`.txt`), start `query-doctor-web`, choose `One Query ID`, and +enter the original Query ID. The web path stages and analyzes the local file +through the same text-only, bounded, redacted analyzer path. If the file +contains an embedded Query ID for a different query, staging fails closed before +replacing any existing case. + +For a self-contained one-profile workspace, set both paths in an ignored local +config file and keep generated cases outside the source tree: + +```json +{ + "manual_profile_dir": "/path/to/profile-inbox", + "corpus_dir": "/path/to/query-doctor-cases", + "no_llm": true +} +``` + +Then start `query-doctor-web --config ./query-doctor-one-profile.json`. Relative +`corpus_dir` values in config resolve from the config file; the `--corpus-dir` +CLI flag resolves relative paths from the current directory. When neither is +set, the web UI stores generated Query ID cases under `./cases/cm-corpus` from +the directory where you started `query-doctor-web`. + +### Troubleshooting + +- `Profile text does not include a Query ID`: keep the original Impala Web UI + download name when it has the strict `profile__` + shape, or pass `--query-id `. Query Doctor also accepts a + `Query ID:` header inside the text export. If multiple Query ID sources are + present, they must match. +- `Parsed operators: 0`: the case is still staged and can open in the UI, but + that text export did not include a parseable `ExecSummary`/operator table. Use + the preserved Impala text profile export when available; JSON, Thrift, and + profile-v2 payloads are outside this manual profile path. +- `query-doctor-web --corpus-dir cases/cm-corpus` asks for Cloudera Manager + settings: confirm that `query-doctor-analyze` wrote a complete case under the + same corpus directory you pass to web, and run web from the same workspace or + use an absolute `--corpus-dir`. + +## Door 2: Run The Synthetic Demo + +The synthetic demo is deterministic, local-only, and contains no real SQL, +profiles, metadata, hostnames, users, or credentials. + +```bash +query-doctor-web --public-demo +``` + +This one-command mode is documented in [demo-mode.md](demo-mode.md). It +generates the synthetic demo pack in a dedicated temp directory, forces +Python-only mode, ignores default local config, and blocks all POST actions. + +To inspect or reuse the generated pack manually, use the lower-level commands: + +```bash +query-doctor-demo-preflight +DEMO_PACK="${TMPDIR:-/tmp}/query-doctor-demo-pack" +query-doctor-demo --out "$DEMO_PACK" --overwrite +QUERY_DOCTOR_ACTION_OUTCOMES_PATH="$DEMO_PACK/action_outcomes.jsonl" \ + query-doctor-web --host 127.0.0.1 --port 8766 --batch-summary "$DEMO_PACK/batch_summary.json" +``` + +Open the localhost URL printed by `query-doctor-web`. Start with +`/?query_group=workloads#scan-context` to show the compact Scan context workload +follow-up links and local synthetic action outcomes before opening Workload +Details. When a Recent summary has repeated safe row-level workload fingerprints +but no materialized workload payload, the UI derives bounded repeated workload +details from the sanitized rows; those details have no baseline or regression +claim until local history evidence is available. + +The main results table is decision-focused: attention rows show one short +deterministic classification, priority, duration, owner context, and a clear +Details path. Repeated workloads keep priority, p95, total observed impact, and +top owner in the inbox; p50, pool, bottleneck, and supporting evidence remain +available in Workload Details. + +The synthetic demo pack contains eleven sanitized Impala cases covering workload +follow-up, repeated patterns, trusted optimizer recommendations, stats +maintenance, storage/HDFS follow-up, frequent-short workloads, mixed signals, +unknown but useful limited evidence, and direct-Impala compatibility. It also +includes two read-only raw-free Trino Beta demo cases rendered from static +compact diagnosis facts, without contacting a Trino coordinator or enabling +Details, reports, optimizer guidance, generated SQL, or SQL execution. See +[demo-cases.md](demo-cases.md) for the full scenario list and talk track. + +## Door 3: Run A Minimal Cloudera Manager Scan + +Use this when you have read-only Cloudera Manager access for an Impala service. +Keep secrets in the shell environment or a local env file, not in JSON config. +Create `~/.qdcreds/cm-ro.env` with `CM_USERNAME` plus `CM_PASSWORD` or `CM_TOKEN` +before sourcing it. + +```bash +mkdir -p ~/.qdcreds +cp query-doctor-config.minimal.example.json ~/.qdcreds/query-doctor-config.json +# Edit with CM URL, cluster, service, and CA bundle if needed. +set -a +source ~/.qdcreds/cm-ro.env +set +a +query-doctor-web \ + --config ~/.qdcreds/query-doctor-config.json \ + --host 127.0.0.1 \ + --port 8765 +``` + +For a headless bounded Recent scan without automatic LLM reports: + +```bash +query-doctor-batch-recent \ + --config ~/.qdcreds/query-doctor-config.json \ + --recent-window-minutes 60 \ + --triage-profile-limit 10 \ + --top-reports 0 +``` + +The minimal path uses Cloudera Manager for Impala Recent discovery and profile +collection. Add metadata, CM time-series, direct Impala, Prometheus, or LLM +settings only after this basic scan path works. See +[configuration.md](configuration.md) and [credentials.md](credentials.md). + +For repeated safe local runs, `--reuse-analyzed-profiles-from ` can +reuse completed analyzed cases from direct child `query-doctor-*` batch outputs +when the Query ID and explicit profile reuse contract match. diff --git a/docs/i18n/ru/README.md b/docs/i18n/ru/README.md index 0c3c317..4625d64 100644 --- a/docs/i18n/ru/README.md +++ b/docs/i18n/ru/README.md @@ -32,6 +32,9 @@ architecture, audit, agent, contributor, release, research и engine deep-dive - [Локальные smoke-проверки](local-smoke.md): public-safe проверки package, analyzer, report, metadata и Recent scan; private targets остаются в ignored local notes. +- [Границы поддержки](support-boundary.md): поверхности поддержки целиком, + Trino/Spark boundary и то, что сознательно вне области. Английский оригинал — + [../../support-boundary.md](../../support-boundary.md). - [Security model](security-model.md): публичная модель безопасности и privacy overview. - [Safety contract](safety-contract.md): пользовательски важные trust/redaction diff --git a/docs/i18n/ru/support-boundary.md b/docs/i18n/ru/support-boundary.md new file mode 100644 index 0000000..3b9c03f --- /dev/null +++ b/docs/i18n/ru/support-boundary.md @@ -0,0 +1,68 @@ +# Границы поддержки + +Last reviewed: 2026-08-11 + +Русская версия контракта поддержки. Английский оригинал и полный текст, +включая supported deployment shape, — в +[../../support-boundary.md](../../support-boundary.md). README держит короткую +сводку из трёх строк, здесь — поверхности целиком. + +## Текущие поверхности + + +| Surface | Current status | +| --- | --- | +| Query engine | Apache Impala - full production triage engine. Trino имеет bounded local production support только для raw-free lanes ниже. | +| First-value intake | Один локальный exported Impala text profile можно загрузить из local/private web session или staged через CLI/manual inbox, затем redacted/analyzed и открыть через Known Query ID. | +| Recent scan | Cloudera Manager - полный Recent discovery/profile/metrics/events provider для Impala workflows. | +| Direct Impala | Bounded Recent scans, Running scans и один Known Query ID через impalad daemon endpoints; без Cloudera Manager events и без SQL execution. | +| Runtime metrics | Optional bounded Prometheus summaries для configured direct Impala workflows; без arbitrary PromQL from users. | +| Metadata | Read-only allowlisted Impala metadata statements через `impala-shell`; без user SQL execution и unbounded metadata crawl. | +| Reports and optimizer | Python-owned facts и validation. Known Query ID готовит deterministic Python report в explicit submit-job; LLM narratives и optimizer actions остаются explicit selected-case actions. | +| Container/Kubernetes web deployment | Supported starting point через official container image, `/healthz` и `/readyz` probes, raw-free deployment readiness summary, read-only `public-demo` manifest, configured private web manifest, synthetic self-test Job и `deploy/helm/query-doctor` chart с `helm test` hook. Kubernetes support не добавляет native auth, RBAC, sessions, multi-tenant isolation, operator/CRD, arbitrary command running, SQL execution или broader engine support. Shared configured deployments все равно требуют trusted ingress/auth proxy и те же safety gates, что любой shared/non-local web bind. | +| Trino local | Local web Trino mode может прочитать один bounded retained pruned coordinator query list для Recent diagnosis, затем bounded pruned coordinator QueryInfo payloads для выбранных rows или одного explicit Query ID, показать deterministic compact diagnosis, materialize server-owned raw-free case artifacts, открыть raw-free Details view и создать deterministic Python Report плюс optimizer guidance для этих materialized cases. `trino_support_mode=beta` сохраняет legacy beta label; `trino_support_mode=production` помечает те же bounded raw-free local lanes как local production support и убирает этот label. Без Running scans, query-history crawling, metadata collection, LLM report output, Query Optimizer jobs, generated Trino SQL, SQL execution и broader/shared Trino production triage support. | +| Spark | Только bounded compact support surfaces. Spark не является production engine support, live Recent scans, Details/trusted report output, optimizer behavior, raw event-log handling, Spark job execution или Query Doctor-generated SQL. | + +Публичный GHCR release содержит Query Doctor web image. + +Trino compact/dev surfaces включают offline/local raw-free imports and checks: +bounded local pruned QueryInfo import принимает one explicit compact sanitized +local pruned QueryInfo JSON через `query-doctor-trino-query-info-pruned-import` +после source-contract checks. `query-doctor-trino-coordinator-query-info-pruned-probe` +и `query-doctor-trino-coordinator-query-info-pruned-import` могут использовать +`--auth-header-file`, но safe output не печатает auth header paths или values. +Local production Trino product surfaces - local web retained-list Recent diagnosis, One +Query ID diagnosis, raw-free Details view, deterministic Python Report и optimizer guidance для +server-owned materialized cases из этих lanes. Diagnosis lanes требуют +`trino_support_mode=beta` или +`trino_support_mode=production`, `trino_coordinator_url` и +`trino_query_info_source_contract` в local config; Recent дополнительно требует +`trino_query_list_source_contract`. Legacy `trino_beta_enabled=true` остается +beta-only switch для existing local setups и не должен комбинироваться с +`trino_support_mode=production`. Startup validation проверяет local source +contracts, safe coordinator URL shape и optional auth reference +(`trino_auth_header_file` или local Kerberos/SPNEGO settings) до того, как lane +считается configured. Configured beta sources помечаются в source selector как +`Trino Beta Recent + One Query ID` или `Trino Beta One Query ID`; configured +production-mode sources используют labels без `Beta`. Diagnose Engine control +сужает Source cluster selector до Impala-capable sources или Trino-ready sources +до выбора workflow, а stale или forged Trino submits все равно fail closed до +analysis или async job creation. Этот lane не делает network read вне explicit +bounded probe/import, reject-ит raw QueryInfo fields вроде query text, session +fields, endpoint URLs, object names и stage/task detail. Details открывается +только после materialized artifacts. Python Report и optimizer guidance используют те же raw-free +facts и не показывают raw payloads, query IDs, paths, LLM report output, +Query Optimizer jobs или generated SQL; Running scans, Query Optimizer jobs и +metadata collection остаются unavailable. +Broader/shared Trino live collection и broader Trino production triage остаются unsupported. + +Spark compact support surfaces остаются только compact History Server intake, +compact evidence-package build/validation и compact diagnosis; no public Spark +engine support, без Recent scans, Details/trusted report output, optimizer +behavior, raw event logs или Spark job execution. + +Будущие Big Data SQL/lakehouse live collectors, более широкие providers, +подготовленные event/log sources и Cluster Doctor workflows остаются roadmap +seams, а не текущей поддержкой. Detailed Trino/Spark preview command catalog: +[engines/README.md](../../engines/README.md). Текущий support/research +boundary: [engine-support-gap-matrix.md](../../engine-support-gap-matrix.md). diff --git a/docs/support-boundary.md b/docs/support-boundary.md new file mode 100644 index 0000000..c9e768c --- /dev/null +++ b/docs/support-boundary.md @@ -0,0 +1,141 @@ +# Support Boundary + +Last reviewed: 2026-08-11 + +What Query Doctor supports today, what it deliberately does not, and the +deployment shape it is supported in. The README keeps a three-line summary; this +is the full contract. For the engine-by-engine gap detail see +[engine-support-gap-matrix.md](engine-support-gap-matrix.md). + +## What It Is + +- a local-first Impala production triage workbench with official bounded local + Trino production lanes; +- a deterministic evidence extractor; +- a Recent-query ranking workflow for operators and administrators; +- a safe report generator using validated facts; +- a practical tool for deciding what to inspect, change, and verify next; +- a containerized web application that can run as a read-only public demo or a + configured private operator service behind a trusted ingress/auth proxy; +- a Big Data SQL/lakehouse diagnostics wedge whose full production triage engine + is Apache Impala, with bounded raw-free local Trino production lanes and + future-engine preview seams. + +## What It Is Not + +- a generic AI chatbot over raw profiles; +- a replacement for the Impala Web UI; +- a tool that executes user SQL or optimizer draft SQL; +- a tool that sends raw SQL/profile data to remote services by default; +- a root-cause oracle; +- a broad live multi-engine query-history collector. + +## Current Surfaces + +| Surface | Current status | +| --- | --- | +| Query engine | Apache Impala is the full production triage engine. Trino has bounded local production support only for the raw-free lanes named below. | +| First-value intake | One local exported Impala text profile can be uploaded from a local/private web session or staged from CLI/manual inbox, redacted, analyzed, and opened from Known Query ID. | +| Recent scan | Cloudera Manager is the full Recent discovery/profile/metrics/events provider for Impala workflows. | +| Direct Impala | Bounded Recent scans, Running scans, and one Known Query ID through impalad daemon endpoints; no Cloudera Manager events and no SQL execution. | +| Runtime metrics | Optional bounded Prometheus summaries for configured direct Impala workflows; no arbitrary PromQL from users. | +| Metadata | Read-only allowlisted Impala metadata statements through `impala-shell`; no user SQL execution or unbounded metadata crawl. | +| Reports and optimizer | Python-owned facts and validation. Known Query ID prepares the deterministic Python report in its explicit submit job; LLM narratives remain explicit selected-case actions, and optimizer actions are shown only for cases with safe-to-attempt rewrite support. | +| Trusted SSO/auth proxy deployment | Query Doctor supports deployment behind a trusted SSO/auth proxy via `viewer_identity_header` for shared/non-local `owner_raw` access only after the raw-free D3 support-readiness gate passes. The proxy or ingress owns authentication, MFA, session lifecycle, token handling, and inbound-header stripping; Query Doctor only enforces the normalized viewer owner header against `query.user`. | +| Container/Kubernetes web deployment | Supported starting point through the official container image, `/healthz` and `/readyz` probes, raw-free deployment readiness summary, a read-only `public-demo` manifest, a configured private web manifest, a synthetic self-test Job, and the `deploy/helm/query-doctor` chart with a `helm test` hook. Kubernetes support does not add native auth, RBAC, sessions, multi-tenant isolation, an operator/CRD, arbitrary command running, SQL execution, or broader engine support. | +| Trino local | Local web Trino mode can read one bounded retained pruned coordinator query list for Recent diagnosis, then bounded pruned coordinator QueryInfo payloads for selected rows or one explicit Query ID, render deterministic compact diagnosis, materialize server-owned raw-free case artifacts, open a raw-free Details view, and generate deterministic Python Report plus optimizer guidance from those materialized case facts. No Running scans, query-history crawling, metadata collection, LLM report output, Query Optimizer jobs, generated Trino SQL, SQL execution, or broader/shared Trino production triage support. | +| Spark | Bounded compact support surfaces only. Not production engine support, live Recent scans, Details/trusted report output, optimizer behavior, raw event-log handling, Spark job execution, or Query Doctor-generated SQL. | + +## Trino Detail + +Trino compact/dev surfaces include offline or compact raw-free imports and +checks: sanitized evidence packages, bounded local compact imports, explicit +source-contract checks, a contract-gated local metadata CLI summary builder and +dev-only round-trip smoke gate that emit aggregate metadata coverage only, and +bounded pruned QueryInfo paths documented in the engine docs. + +The only local production Trino product surfaces are local web retained-list +Recent diagnosis, One Query ID diagnosis, the raw-free Details view, +deterministic Python Report, and optimizer guidance for server-owned +materialized cases from those lanes. + +Those lanes require `trino_support_mode=beta` or `trino_support_mode=production`, +`trino_coordinator_url`, and `trino_query_info_source_contract` in local config; +Recent also requires `trino_query_list_source_contract`. The legacy +`trino_beta_enabled=true` key remains beta-only for existing local setups and +must not be combined with `trino_support_mode=production`. Startup validation +checks local source contracts, safe coordinator URL shape, and optional auth +reference (`trino_auth_header_file` or local Kerberos/SPNEGO settings) before +the lane is marked configured. + +Configured beta sources are marked as `Trino Beta Recent + One Query ID` or +`Trino Beta One Query ID`; configured production-mode sources are marked without +the beta label. The Diagnose Engine control narrows the Source cluster selector +to Impala-capable or Trino-ready sources before workflow selection, and stale or +forged Trino submits still fail closed before analysis or async job creation. + +Coordinator URL, auth header references, raw QueryInfo, raw SQL, and local paths +stay out of the browser. Trino web case artifacts contain only the normalized +boundary, compact diagnosis, metadata-not-collected summary, typed analysis, and +safe analyzer facts view; Details opens only after those artifacts exist. Python +Report and optimizer guidance use the same raw-free facts and hide raw payloads, +query IDs, paths, LLM report output, Query Optimizer jobs, and generated SQL. + +Broader/shared Trino live collection and broader Trino production triage remain +unsupported. + +Spark compact support surfaces are limited to bounded compact History Server +intake, compact evidence-package build/validation, and compact diagnosis; there +is no public Spark engine support. + +Future Big Data SQL/lakehouse live collectors, broader providers, prepared +event/log sources, and Cluster Doctor workflows remain roadmap seams, not +current support. For the preview command catalog see +[engines/README.md](engines/README.md). + +## Direct Impala History Depth + +Direct Impala Recent and Running scans currently see only the query history +exposed by the configured coordinator daemon query-list endpoints. Upstream +Impala keeps the coordinator query log at `--query_log_size=200` entries by +default, further bounded by `--query_log_size_in_bytes`. Operators who need +deeper direct history can increase those Impala daemon settings on each +coordinator, while watching coordinator Web UI memory and `/queries` response +latency. + +Future deeper-history options are deliberately separate sources: +operator-managed read-only profile-log directory ingestion, or bounded external +history sources such as Loki or OpenSearch. They require explicit source +contracts, allowlists, byte/window bounds, and raw-free browser/report output; +the current product does not read coordinator filesystems, pod filesystems, or +external log indexes for direct Recent scans. + +Apache Impala also has upstream work around native AI query profile analysis. +Query Doctor aligns with that direction by staying focused on local-first +production triage across many queries, deterministic evidence, safe enrichment, +and validated raw-free reports. See +[upstream-impala-ai-analyzer.md](upstream-impala-ai-analyzer.md). + +## Supported Deployment Shape + +Query Doctor is supported as a single-user, local-first tool run by an operator +with their own local Cloudera Manager, Kerberos, Impala, Prometheus, and LLM +credentials. Use localhost or a tightly controlled local bind for the web UI. + +Do not deploy ordinary local mode as a shared service without a separate design +for authentication, authorization, tenant/job isolation, audit logging, +TLS/reverse-proxy trust, and resource limits. Shared public demos should use the +read-only `query-doctor-web --public-demo` mode. + +Shared `owner_raw` source access requires authenticated per-request viewer +identity through an explicit `viewer_identity_header` supplied only by a trusted +SSO/auth proxy or ingress that strips inbound copies of the same header and sets +exactly one normalized simple owner value. Query Doctor supports that deployment +pattern after the raw-free support-readiness gate in +[owner-raw-d3-deployment.md](owner-raw-d3-deployment.md). + +A dev-only Keycloak/oauth2-proxy smoke is available in +[dev-sso-keycloak.md](dev-sso-keycloak.md) to test the front-door viewer header +contract locally; `scripts/dev_sso_keycloak_smoke.py` verifies the running local +compose path with raw-free output. The dev smoke is not production SSO support +evidence and does not add native SSO to Query Doctor. diff --git a/pyproject.toml b/pyproject.toml index 8d78b08..20ed127 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "query-doctor" version = "0.11.0" -description = "Local-first Big Data query diagnostics for Apache Impala production triage, bounded local Trino lanes, and containerized web deployment." +description = "Rank slow Apache Impala queries and turn one exported profile into a deterministic diagnosis, without raw SQL or profiles leaving the machine." readme = "README.md" license = "Apache-2.0" requires-python = ">=3.9" @@ -30,7 +30,7 @@ keywords = [ "trino", ] classifiers = [ - "Development Status :: 3 - Alpha", + "Development Status :: 4 - Beta", "Environment :: Console", "Environment :: Web Environment", "Intended Audience :: Developers", diff --git a/scripts/audit_spark_support_boundary.py b/scripts/audit_spark_support_boundary.py index 52b7fc4..fbeff62 100644 --- a/scripts/audit_spark_support_boundary.py +++ b/scripts/audit_spark_support_boundary.py @@ -182,7 +182,14 @@ def count_forbidden_product_imports(paths: Iterable[Path]) -> int: def _audit_docs(result: SparkSupportBoundaryAuditResult, root: Path) -> None: - readme = _normalized_doc(root, "README.md") + # The public support statement moved out of the landing README into the + # support-boundary document; both count as the public claim surface. + readme = " ".join( + ( + _normalized_doc(root, "README.md"), + _normalized_doc(root, "docs/support-boundary.md"), + ) + ) matrix = _read_doc(root, "docs/engine-support-gap-matrix.md") spark_public_status = _normalized_matrix_spark_cell(matrix, "Public support status") spark_live_collection = _normalized_matrix_spark_cell(matrix, "Live query/profile collection") diff --git a/scripts/check_active_docs.py b/scripts/check_active_docs.py index b4639a9..b57a733 100644 --- a/scripts/check_active_docs.py +++ b/scripts/check_active_docs.py @@ -36,6 +36,8 @@ "docs/impala-profile-counter-caveats.md", "docs/engine-expansion-plan.md", "docs/engine-support-gap-matrix.md", + "docs/support-boundary.md", + "docs/first-path.md", "docs/recent-history-store.md", "docs/trino-beta-ui-readiness.md", "docs/trino-shared-deployment-hardening.md", diff --git a/scripts/installed_readme_quickstart_smoke.py b/scripts/installed_readme_quickstart_smoke.py index 4bd3a0c..a3bcd45 100644 --- a/scripts/installed_readme_quickstart_smoke.py +++ b/scripts/installed_readme_quickstart_smoke.py @@ -33,8 +33,8 @@ ROOT / "tests" / "fixtures" / "mixed_stats_runtime_case" / "profile_digest.md" ) INSTALLED_BIN_ENV = "QUERY_DOCTOR_INSTALLED_CLI_BIN" -README_PROFILE_NAME = "exported-impala-profile.txt" -README_PROFILE_ARG = "./exported-impala-profile.txt" +README_PROFILE_NAME = "your-profile.txt" +README_PROFILE_ARG = "./your-profile.txt" README_CORPUS_ARG = "cases/cm-corpus" README_CORPUS_DIR = Path(README_CORPUS_ARG) REQUIRED_SELF_TEST_CHECKS = frozenset( diff --git a/scripts/installed_user_paths_smoke.py b/scripts/installed_user_paths_smoke.py index 408411c..da4480d 100644 --- a/scripts/installed_user_paths_smoke.py +++ b/scripts/installed_user_paths_smoke.py @@ -318,7 +318,7 @@ def smoke_readme_quickstart( or summary.get("self_test_checked") is not True or summary.get("analyze_checked") is not True or summary.get("real_web_server") is not True - or summary.get("relative_profile_path_checked") != "./exported-impala-profile.txt" + or summary.get("relative_profile_path_checked") != "./your-profile.txt" or summary.get("relative_corpus_dir_checked") != "cases/cm-corpus" ): raise SystemExit("[installed-user-paths-smoke] README Quickstart smoke did not pass") diff --git a/setup.py b/setup.py index 28f4b1c..21588e4 100644 --- a/setup.py +++ b/setup.py @@ -47,8 +47,8 @@ def project_version(pyproject_path: Path = Path("pyproject.toml")) -> str: name="query-doctor", version=VERSION, description=( - "Local-first Big Data query diagnostics for Apache Impala production triage, " - "bounded local Trino lanes, and containerized web deployment." + "Rank slow Apache Impala queries and turn one exported profile into a " + "deterministic diagnosis, without raw SQL or profiles leaving the machine." ), author="Aleksandr Efimov", maintainer="Aleksandr Efimov", @@ -61,7 +61,7 @@ def project_version(pyproject_path: Path = Path("pyproject.toml")) -> str: }, license="Apache-2.0", classifiers=[ - "Development Status :: 3 - Alpha", + "Development Status :: 4 - Beta", "Environment :: Console", "Environment :: Web Environment", "Intended Audience :: Developers", diff --git a/tests/test_audit_owner_raw_sso_proxy_support_readiness.py b/tests/test_audit_owner_raw_sso_proxy_support_readiness.py index 59f569c..6c6bd4a 100644 --- a/tests/test_audit_owner_raw_sso_proxy_support_readiness.py +++ b/tests/test_audit_owner_raw_sso_proxy_support_readiness.py @@ -367,7 +367,9 @@ def test_sso_proxy_support_readiness_docs_mention_script() -> None: release_checklist = (readiness.ROOT / "docs" / "release-checklist.md").read_text( encoding="utf-8" ) - readme = (readiness.ROOT / "README.md").read_text(encoding="utf-8") + support_boundary = (readiness.ROOT / "docs" / "support-boundary.md").read_text( + encoding="utf-8" + ) assert "scripts/audit_owner_raw_sso_proxy_support_readiness.py" in deployment assert "owner_raw_sso_proxy_support_readiness_v1" in changelog @@ -378,4 +380,7 @@ def test_sso_proxy_support_readiness_docs_mention_script() -> None: assert "scripts/audit_owner_raw_sso_proxy_support_readiness.py" in release_checklist assert "native OIDC" in release_checklist assert "must not gate raw reveal on collection" in release_checklist - assert "deployment behind a trusted SSO/auth proxy via `viewer_identity_header`" in readme + assert ( + "deployment behind a trusted SSO/auth proxy via `viewer_identity_header`" + in support_boundary + ) diff --git a/tests/test_dev_sso_keycloak.py b/tests/test_dev_sso_keycloak.py index 940a4a6..da27c98 100644 --- a/tests/test_dev_sso_keycloak.py +++ b/tests/test_dev_sso_keycloak.py @@ -112,7 +112,9 @@ def test_dev_sso_docs_are_indexed_and_warn_against_production_claims(): docs_index = (REPO_DIR / "docs" / "README.md").read_text(encoding="utf-8") deployment = (REPO_DIR / "docs" / "owner-raw-d3-deployment.md").read_text(encoding="utf-8") test_matrix = (REPO_DIR / "docs" / "test-matrix.md").read_text(encoding="utf-8") - readme = (REPO_DIR / "README.md").read_text(encoding="utf-8") + support_boundary = (REPO_DIR / "docs" / "support-boundary.md").read_text( + encoding="utf-8" + ) assert "developer-only SSO front-door smoke" in doc assert "not production SSO support" in doc @@ -128,4 +130,4 @@ def test_dev_sso_docs_are_indexed_and_warn_against_production_claims(): assert "dev/sso/compose.yaml" in deployment assert "dev-sso-keycloak.md" in test_matrix assert "tests/test_dev_sso_keycloak*.py" in test_matrix - assert "dev-only Keycloak/oauth2-proxy smoke" in readme + assert "dev-only Keycloak/oauth2-proxy smoke" in support_boundary diff --git a/tests/test_dev_sso_keycloak_smoke.py b/tests/test_dev_sso_keycloak_smoke.py index 56e8cc8..1e75ca6 100644 --- a/tests/test_dev_sso_keycloak_smoke.py +++ b/tests/test_dev_sso_keycloak_smoke.py @@ -141,8 +141,10 @@ def test_dev_sso_keycloak_smoke_docs_mention_script(): deployment_doc = (smoke.ROOT / "docs" / "owner-raw-d3-deployment.md").read_text( encoding="utf-8" ) - readme = (smoke.ROOT / "README.md").read_text(encoding="utf-8") + support_boundary = (smoke.ROOT / "docs" / "support-boundary.md").read_text( + encoding="utf-8" + ) assert "scripts/dev_sso_keycloak_smoke.py --compact" in dev_doc assert "scripts/dev_sso_keycloak_smoke.py" in deployment_doc - assert "scripts/dev_sso_keycloak_smoke.py" in readme + assert "scripts/dev_sso_keycloak_smoke.py" in support_boundary diff --git a/tests/test_installed_readme_quickstart_smoke.py b/tests/test_installed_readme_quickstart_smoke.py index 51a3d39..5472177 100644 --- a/tests/test_installed_readme_quickstart_smoke.py +++ b/tests/test_installed_readme_quickstart_smoke.py @@ -37,7 +37,7 @@ def test_installed_readme_quickstart_smoke_uses_installed_cli_and_real_server() assert 'installed_executable(bin_dir, "query-doctor-analyze")' in text assert 'installed_executable(bin_dir, "query-doctor-web")' in text assert '"--profile-text"' in text - assert '"./exported-impala-profile.txt"' in text + assert '"./your-profile.txt"' in text assert '"--out"' in text assert '"cases/cm-corpus"' in text assert "subprocess.Popen" in text diff --git a/tests/test_kubernetes_auth_front_door_smoke.py b/tests/test_kubernetes_auth_front_door_smoke.py index 5633363..bbce837 100644 --- a/tests/test_kubernetes_auth_front_door_smoke.py +++ b/tests/test_kubernetes_auth_front_door_smoke.py @@ -135,9 +135,9 @@ def test_kubernetes_auth_front_door_smoke_docs_mention_script() -> None: root = Path(__file__).resolve().parents[1] auth_doc = (root / "docs" / "kubernetes-auth-front-door.md").read_text(encoding="utf-8") test_matrix = (root / "docs" / "test-matrix.md").read_text(encoding="utf-8") - readme = (root / "README.md").read_text(encoding="utf-8") + deploy_doc = (root / "deploy" / "kubernetes" / "README.md").read_text(encoding="utf-8") assert "scripts/kubernetes_auth_front_door_smoke.py --compact" in auth_doc assert "kubernetes-auth-front-door.md#live-external-smoke" in test_matrix assert "tests/test_kubernetes_auth_front_door_smoke.py" in test_matrix - assert "scripts/kubernetes_auth_front_door_smoke.py" in readme + assert "scripts/kubernetes_auth_front_door_smoke.py" in deploy_doc diff --git a/tests/test_pyproject.py b/tests/test_pyproject.py index e877481..3fcc6a1 100644 --- a/tests/test_pyproject.py +++ b/tests/test_pyproject.py @@ -87,7 +87,7 @@ def test_public_packaging_metadata_is_present(): "[project.urls]", 'Homepage = "https://github.com/alexandrefimov/Query-Doctor"', 'Issues = "https://github.com/alexandrefimov/Query-Doctor/issues"', - '"Development Status :: 3 - Alpha"', + '"Development Status :: 4 - Beta"', '"Programming Language :: Python :: 3.9"', '"Programming Language :: Python :: 3.11"', ): @@ -98,7 +98,7 @@ def test_public_packaging_metadata_is_present(): 'maintainer="Aleksandr Efimov"', 'url="https://github.com/alexandrefimov/Query-Doctor"', '"Homepage": "https://github.com/alexandrefimov/Query-Doctor"', - '"Development Status :: 3 - Alpha"', + '"Development Status :: 4 - Beta"', '"Programming Language :: Python :: 3.9"', '"Programming Language :: Python :: 3.11"', ): diff --git a/tests/test_spark_support_boundary_docs.py b/tests/test_spark_support_boundary_docs.py index 104f21b..0199ed8 100644 --- a/tests/test_spark_support_boundary_docs.py +++ b/tests/test_spark_support_boundary_docs.py @@ -42,7 +42,9 @@ def test_spark_support_matrix_records_package_handoff_without_support_claim() -> def test_spark_public_docs_index_package_code_under_experimental_boundary() -> None: code_map = _normalized(CODE_MAP) handoff = _normalized(CODEX_HANDOFF) - ru_readme = _normalized(README_RU) + ru_support_boundary = _normalized( + ROOT / "docs" / "i18n" / "ru" / "support-boundary.md" + ) ru_docs_index = _normalized(RU_DOCS_INDEX) assert "Spark compact History Server intake and evidence packages" in code_map @@ -93,8 +95,8 @@ def test_spark_public_docs_index_package_code_under_experimental_boundary() -> N in handoff ) - assert "compact evidence-package build/validation" in ru_readme - assert "no public Spark engine support" in ru_readme + assert "compact evidence-package build/validation" in ru_support_boundary + assert "no public Spark engine support" in ru_support_boundary assert "engine deep-dive документы остаются English-only" in ru_docs_index assert "engine-support-gap-matrix.md" in ru_docs_index diff --git a/tests/test_trino_private_preview_release_doc.py b/tests/test_trino_private_preview_release_doc.py index 10c43cb..3202354 100644 --- a/tests/test_trino_private_preview_release_doc.py +++ b/tests/test_trino_private_preview_release_doc.py @@ -7,6 +7,8 @@ RU_DOCS_INDEX = REPO_ROOT / "docs" / "i18n" / "ru" / "README.md" README = REPO_ROOT / "README.md" README_RU = REPO_ROOT / "README.ru.md" +SUPPORT_BOUNDARY = REPO_ROOT / "docs" / "support-boundary.md" +RU_SUPPORT_BOUNDARY = REPO_ROOT / "docs" / "i18n" / "ru" / "support-boundary.md" RELEASE_CHECKLIST = REPO_ROOT / "docs" / "release-checklist.md" PUBLIC_READINESS = REPO_ROOT / "docs" / "public-release-readiness.md" TRINO_LIVE_COLLECTION_DOC = REPO_ROOT / "docs" / "engines" / "trino-live-collection-design.md" @@ -251,7 +253,9 @@ def test_trino_private_preview_release_path_is_indexed_and_linked(): def test_readme_and_release_docs_keep_trino_limited_to_offline_import(): - for path in (README, README_RU, RELEASE_CHECKLIST, PUBLIC_READINESS): + # The full Trino boundary claim lives in the support-boundary document. The + # landing READMEs keep a short scope statement and must not overclaim. + for path in (SUPPORT_BOUNDARY, RELEASE_CHECKLIST, PUBLIC_READINESS): text = _normalized_doc_text(path) lower_text = text.lower() assert "Trino" in text @@ -264,6 +268,14 @@ def test_readme_and_release_docs_keep_trino_limited_to_offline_import(): assert "Query Doctor-generated" in text assert "SQL" in text + for path in (README, README_RU): + text = _normalized_doc_text(path) + lower_text = text.lower() + assert "Trino" in text + assert "Apache Impala" in text + assert "bounded local production support" in lower_text + assert "support-boundary.md" in text + def test_release_checklists_name_trino_beta_without_support_expansion(): text = _normalized_doc_text(RELEASE_CHECKLIST) @@ -285,7 +297,9 @@ def test_release_checklists_name_trino_beta_without_support_expansion(): def test_russian_readme_names_current_trino_pruned_query_info_surfaces(): - text = _normalized_doc_text(README_RU) + # Moved out of the landing README.ru.md; the Russian claim now lives in the + # Russian support-boundary document. + text = _normalized_doc_text(RU_SUPPORT_BOUNDARY) for required in ( "bounded local pruned QueryInfo import", diff --git a/web/.gitignore b/web/.gitignore new file mode 100644 index 0000000..849ddff --- /dev/null +++ b/web/.gitignore @@ -0,0 +1 @@ +dist/ diff --git a/web/README.md b/web/README.md new file mode 100644 index 0000000..693715b --- /dev/null +++ b/web/README.md @@ -0,0 +1,107 @@ +# Browser analyzer (prototype) + +A static page that runs the Query Doctor deterministic analyzer entirely in the +visitor's browser through Pyodide. Drop an exported Impala text profile, get the +same `analysis_facts` output `query-doctor-analyze` writes locally. + +The point is not convenience. The most common reason an operator cannot try +Query Doctor is that the profile must not leave their machine. On this page it +does not: there is no server, no upload, and no request to any host after the +page loads. That is checkable in DevTools in ten seconds, which the safety +documentation is not. + +Status: working prototype on a branch. Not deployed, not linked from the README. + +## Build and run + +```bash +web/build.sh +python3 -m http.server 8799 --directory web/dist +``` + +`build.sh` builds the wheel, fetches the Pyodide runtime from npm, generates the +synthetic sample profile, and writes everything into `web/dist`. The runtime is +fetched at build time rather than vendored so the repository does not carry +~13 MB of binaries; the built site still serves every asset from its own origin. + +## Why the analyzer runs unmodified + +The deterministic core imports no third-party package and no networking or +process stdlib. Measured on `origin/main` at 0.11.0: + +``` +query_doctor.analyzer.service + action_cards + facts_renderer + +runtime_diagnosis + scan_skew + case_bottleneck + data_movement + +memory_pressure + query_doctor.report + -> 90 modules, 0 third-party packages + -> no socket, ssl, subprocess, urllib, sqlite3, asyncio +``` + +Only the `query_doctor.cli` wrapper pulls sockets and subprocesses, because the +collectors live behind the same entry point. The page calls +`analyzer.service.analyze` directly with a `SimpleNamespace` of the seven +threshold arguments the analyzer reads, so no argparse and no CLI import. + +Calling the analyzer directly skips the CLI's case staging, which is where +`query-doctor-analyze` applies redaction. The page therefore calls +`safety.redaction.redact_profile_text` itself with the same defaults +(`redact_hosts=True`, `redact_identifiers=False`) before analyzing, so users, +pools, emails, secrets, local paths, and host identifiers do not reach the +rendered facts. Nothing is transmitted either way, but the rendered output is a +browser surface, and screenshots of it travel. + +## Measurements + +Analyzer throughput, best of three runs, synthetic profiles shaped like real +Impala text exports (`bench/make_profile.py`): + +| profile | CPython 3.13 | Pyodide | ratio | +| --- | --- | --- | --- | +| 103 KiB | 52 ms | 113 ms | 2.2x | +| 1013 KiB | 568 ms | 1101 ms | 1.9x | +| 4154 KiB | 2297 ms | 4263 ms | 1.9x | + +Roughly 1.1 s per megabyte, linear, with identical output to native (operator +counts and fact-group counts match exactly). + +Full page in headless Chromium (`bench/check_page.py`): + +``` +boot: 1196 ms (runtime + wheel + analyzer import) +run: 127 ms (103 KiB sample profile) +network: 8 requests, external hosts: none +JS errors: none +``` + +First-visit transfer, gzipped: Pyodide wasm 3.4 MB, Python stdlib 2.4 MB, +runtime glue 0.2 MB, query-doctor wheel 1.19 MB, page under 15 KB. About 7.2 MB, +cached afterwards. 5.8 MB of that is Python itself and cannot be reduced. + +The wheel could be reduced: the analyzer core is 587 KiB of source out of +5559 KiB in the package. It is deliberately not trimmed yet. An import trace +from one sample profile is not proof that other profile shapes take the same +path, and shipping a wheel that fails on an unusual export would be a worse +outcome than a megabyte of transfer. Trim only against a corpus that covers the +profile dialects in `analyzer/profile_counter_registry.py`. + +## Bench harness + +- `bench/make_profile.py ` — synthesize an Impala-shaped profile. + Synthetic only; no real cluster data, hosts, users, tables, or SQL. +- `bench/bench.py ` — time `analyze` on CPython. +- `bench/harness.mjs ` — the same timing under Pyodide + in Node. Needs `npm install pyodide` in its working directory. +- `bench/check_page.py [out.png]` — drive the served page in Chromium, assert no + external requests, screenshot the result. Needs `pip install playwright` and + `playwright install chromium`. + +## Open decisions + +- **Hosting.** Not deployed. A `gh-pages` branch or an `actions/deploy-pages` + workflow from `web/dist` both work; neither is set up. +- **Output language.** Settled: English. Page chrome and `render_md` output now + match. A Russian layer would follow the `docs/i18n/ru/` convention, not a + second page. +- **Sample profile.** Currently synthetic and generated at build time. A + hand-written sample with a real pathology would demo better than uniform + generated counters. diff --git a/web/bench/bench.py b/web/bench/bench.py new file mode 100644 index 0000000..e4a6429 --- /dev/null +++ b/web/bench/bench.py @@ -0,0 +1,49 @@ +"""Time query_doctor.analyzer.service.analyze on a profile text. + +Runs identically on CPython and inside Pyodide: no argparse, no I/O helpers, +no query_doctor.cli import. +""" +from __future__ import annotations + +import sys +import time +from types import SimpleNamespace + +from query_doctor.analyzer.action_cards import DEFAULT_LARGE_BYTES_THRESHOLD +from query_doctor.analyzer.service import analyze + +ARGS = SimpleNamespace( + top_n=10, + rows_ratio_threshold=10.0, + mem_ratio_threshold=4.0, + slow_operator_ms=10_000.0, + large_rows_threshold=1_000_000.0, + large_bytes_threshold=DEFAULT_LARGE_BYTES_THRESHOLD, + max_evidence_lines=30, +) + + +def run(text: str, repeats: int = 3) -> dict: + best = None + result = None + for _ in range(repeats): + t0 = time.perf_counter() + result = analyze(text, ARGS) + dt = time.perf_counter() - t0 + best = dt if best is None else min(best, dt) + return { + "seconds": best, + "kib": len(text.encode()) / 1024, + "operators": len(result.get("operators", []) or []), + "keys": len(result), + } + + +if __name__ == "__main__": + with open(sys.argv[1], encoding="utf-8") as fh: + text = fh.read() + out = run(text) + print( + f"{sys.argv[1]}: {out['kib']:.0f} KiB -> {out['seconds'] * 1000:.0f} ms " + f"(operators={out['operators']}, fact keys={out['keys']})" + ) diff --git a/web/bench/check_page.py b/web/bench/check_page.py new file mode 100644 index 0000000..03ba3e8 --- /dev/null +++ b/web/bench/check_page.py @@ -0,0 +1,66 @@ +"""Drive the prototype page in a real browser: measure load, run the sample, screenshot.""" +from __future__ import annotations + +import sys + +from playwright.sync_api import sync_playwright + +URL = "http://127.0.0.1:8799/" +OUT = sys.argv[1] if len(sys.argv) > 1 else "page.png" + +transferred = {"bytes": 0, "requests": 0, "external": []} + + +def main() -> int: + with sync_playwright() as p: + browser = p.chromium.launch() + page = browser.new_page(viewport={"width": 1000, "height": 1100}) + errors: list[str] = [] + page.on("pageerror", lambda e: errors.append(str(e))) + page.on("console", lambda m: errors.append(m.text) if m.type == "error" else None) + + def on_response(resp): + transferred["requests"] += 1 + if not resp.url.startswith("http://127.0.0.1:8799/"): + transferred["external"].append(resp.url) + try: + transferred["bytes"] += len(resp.body()) + except Exception: + pass + + page.on("response", on_response) + + page.goto(URL, wait_until="load") + page.wait_for_function( + "() => document.getElementById('status').textContent.includes('Ready in')", + timeout=120_000, + ) + boot_status = page.inner_text("#status") + print("boot:", boot_status) + + page.click("#sample") + page.wait_for_function( + "() => document.getElementById('status').textContent.includes('Analyzed in')", + timeout=120_000, + ) + print("run :", page.inner_text("#status")) + + headings = page.eval_on_selector_all("#out h1, #out h2", "els => els.map(e => e.textContent)") + tables = page.eval_on_selector_all("#out table", "els => els.length") + print(f"output: {len(headings)} sections, {tables} tables") + print("sections:", " | ".join(headings[:12])) + + print(f"network: {transferred['requests']} requests, " + f"{transferred['bytes'] / 1048576:.1f} MB total") + print("external hosts:", sorted({u.split('/')[2] for u in transferred['external']}) or "none") + + page.screenshot(path=OUT, full_page=False) + print("screenshot:", OUT) + if errors: + print("JS ERRORS:", errors[:5]) + browser.close() + return 1 if errors else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/web/bench/harness.mjs b/web/bench/harness.mjs new file mode 100644 index 0000000..783d07c --- /dev/null +++ b/web/bench/harness.mjs @@ -0,0 +1,50 @@ +import { loadPyodide } from "pyodide"; +import fs from "node:fs"; +import path from "node:path"; + +const SP = path.resolve(process.argv[2]); +const wheel = path.join(SP, "wheel", "query_doctor-0.11.0-py3-none-any.whl"); +const bench = fs.readFileSync(path.join(SP, "bench.py"), "utf8"); +const profiles = process.argv.slice(3); + +const mark = (label, t0) => + console.log(`${label}: ${(performance.now() - t0).toFixed(0)} ms`); + +let t = performance.now(); +const py = await loadPyodide(); +mark("pyodide boot", t); + +t = performance.now(); +try { + await py.loadPackage("micropip"); + py.FS.writeFile("/query_doctor-0.11.0-py3-none-any.whl", new Uint8Array(fs.readFileSync(wheel))); + const micropip = py.pyimport("micropip"); + await micropip.install("emfs:/query_doctor-0.11.0-py3-none-any.whl"); +} catch (e) { + console.error("INSTALL FAILED:", e.message?.slice(0, 4000) ?? String(e).slice(0, 4000)); + process.exit(1); +} +mark("wheel install", t); + +t = performance.now(); +py.FS.writeFile("/bench.py", bench); +await py.runPythonAsync(` +import sys +sys.path.insert(0, "/") +import bench +`); +mark("import analyzer core", t); + +for (const p of profiles) { + const text = fs.readFileSync(path.join(SP, p), "utf8"); + py.globals.set("profile_text", text); + const res = await py.runPythonAsync(` +import json, bench +json.dumps(bench.run(profile_text)) +`); + const out = JSON.parse(res); + console.log( + `${p}: ${out.kib.toFixed(0)} KiB -> ${(out.seconds * 1000).toFixed(0)} ms ` + + `(operators=${out.operators}, fact keys=${out.keys})`, + ); +} diff --git a/web/bench/make_profile.py b/web/bench/make_profile.py new file mode 100644 index 0000000..15fdc89 --- /dev/null +++ b/web/bench/make_profile.py @@ -0,0 +1,125 @@ +"""Generate synthetic Impala-shaped text profiles of a target size for timing tests. + +Synthetic only. No real cluster data, hostnames, users, tables, or SQL. +""" +from __future__ import annotations + +import sys + +HEADER = """Query Runtime Profile +Query (id=aa4d4a4a4a4a4a4a:bb5b5b5b5b5b5b5b) +User: synthetic_user +Request Pool: synthetic_pool +Start Time: 2026-06-14 11:00:00.000000000 +End Time: 2026-06-14 11:07:00.000000000 +Coordinator: synthetic-impalad.example.invalid:22000 +Query Type: QUERY +Query State: FINISHED +Impala Version: impalad version 4.4.0-SYNTHETIC + +Sql Statement: +SELECT /* synthetic */ a, b, count(*) FROM synthetic_db.synthetic_fact GROUP BY a, b + +Query Timeline: + Query submitted: 0ns + Planning finished: 1s200ms + Ready to start on 20 backends: 1s400ms + All 20 execution backends (40 fragment instances) started: 2s100ms + Rows available: 6m30s + First row fetched: 6m40s + Last row fetched: 6m55s + Query finished: 7m + +TotalTime: 7m +TotalBytesRead: 6.00 GiB +TotalBytesSent: 256.00 MiB +""" + + +def exec_summary(n_ops: int) -> str: + lines = [ + "ExecSummary:", + "Operator #Hosts Avg Time Max Time #Rows Est. #Rows " + "Peak Mem Est. Peak Mem Detail", + ] + for i in range(n_ops): + kind = ("SCAN HDFS", "AGGREGATE", "HASH JOIN", "EXCHANGE")[i % 4] + avg = 500 + (i * 37) % 4000 + mx = avg + 900 + rows = 100_000 * (i + 1) + est = 20_000 * (i + 1) + lines.append( + f"{i:02d}:{kind:<18} 20 {avg // 1000}s{avg % 1000:03d}ms " + f"{mx // 1000}s{mx % 1000:03d}ms {rows / 1e6:.2f}M {est / 1e3:.2f}K " + f"{128 + i}.00 MB {64 + i}.00 MB table=synthetic_db.synthetic_t{i % 7}" + ) + return "\n".join(lines) + "\n" + + +COUNTERS = [ + "BytesRead: {b}.00 GB", + "BytesReadLocal: {b}.00 GB", + "DecompressionTime: {t}ms", + "RowsRead: {r}", + "RowsReturned: {r}", + "ScannerThreadsTotalWallClockTime: {t}ms", + "ScannerThreadsSysTime: {t}ms", + "ScannerThreadsUserTime: {t}ms", + "PeakMemoryUsage: {m}.00 MB", + "TotalStorageWaitTime: {t}ms", + "TotalNetworkSendTime: {t}ms", + "TotalNetworkReceiveTime: {t}ms", + "InactiveTotalTime: {t}ms", + "TotalTime: {t}ms", + "BuildRows: {r}", + "BuildTime: {t}ms", + "ProbeRows: {r}", + "ProbeTime: {t}ms", + "SpilledPartitions: 0", + "NumHashTableBuildsSkipped: 0", + "RowsReturnedRate: {r}/sec", + "OpenTime: {t}ms", + "PrepareTime: {t}ms", + "ExecTreeExecTime: {t}ms", + "CodegenTime: {t}ms", +] + + +def execution_profile(n_frag: int, n_inst: int) -> str: + out = ["Execution Profile aa4d4a4a4a4a4a4a:bb5b5b5b5b5b5b5b:"] + for f in range(n_frag): + out.append(f" Fragment F{f:02d}:") + out.append(f" Instance aa4d4a4a4a4a4a4a:bb5b5b5b5b5b{f:04x} " + f"(host=synthetic-impalad-{f % 20}.example.invalid:22000):") + for inst in range(n_inst): + out.append(f" HDFS_SCAN_NODE (id={f * 10 + inst % 10}):") + for k, tmpl in enumerate(COUNTERS): + val = tmpl.format( + b=(f + inst + k) % 9 + 1, + t=(f * 131 + inst * 17 + k * 7) % 90000, + r=(f * 100003 + inst * 977 + k) % 5_000_000, + m=(f + inst * 3 + k) % 512 + 32, + ) + out.append(f" - {val}") + return "\n".join(out) + "\n" + + +def build(target_bytes: int) -> str: + n_ops = 40 + n_inst = 8 + n_frag = 1 + base = HEADER + exec_summary(n_ops) + while True: + text = base + execution_profile(n_frag, n_inst) + if len(text.encode()) >= target_bytes or n_frag > 4000: + return text + n_frag = max(n_frag + 1, int(n_frag * 1.6)) + + +if __name__ == "__main__": + target = int(sys.argv[1]) + out = sys.argv[2] + text = build(target) + with open(out, "w", encoding="utf-8") as fh: + fh.write(text) + print(f"{out}: {len(text.encode()) / 1024:.0f} KiB, {text.count(chr(10))} lines") diff --git a/web/build.sh b/web/build.sh new file mode 100755 index 0000000..b884828 --- /dev/null +++ b/web/build.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# Assemble the self-contained browser analyzer into web/dist. +# +# The Pyodide runtime is fetched at build time instead of being vendored in git, +# so the repository stays small while the built site still loads everything from +# its own origin. Nothing in dist/ reaches an external host at runtime. +set -euo pipefail + +WEB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(cd "${WEB_DIR}/.." && pwd)" +DIST_DIR="${WEB_DIR}/dist" +PYTHON_BIN="${PYTHON:-python3}" +PYODIDE_VERSION="${PYODIDE_VERSION:-0.28.3}" + +# Files the Pyodide loader actually requests. Keep this list tight: the npm +# package also ships type definitions, source maps, and a console app. +PYODIDE_FILES=( + pyodide.js + pyodide.mjs + pyodide.asm.js + pyodide.asm.mjs + pyodide.asm.wasm + python_stdlib.zip + pyodide-lock.json +) + +step() { printf '\n==> %s\n' "$*"; } + +step "Reset ${DIST_DIR}" +rm -rf "${DIST_DIR}" +mkdir -p "${DIST_DIR}/vendor" + +step "Build the query-doctor wheel" +# setuptools uses ROOT_DIR/build as scratch space. Leave the tree as we found it. +HAD_BUILD_DIR=0 +[ -d "${ROOT_DIR}/build" ] && HAD_BUILD_DIR=1 +"${PYTHON_BIN}" -m build --wheel --outdir "${DIST_DIR}" "${ROOT_DIR}" +[ "${HAD_BUILD_DIR}" -eq 0 ] && rm -rf "${ROOT_DIR}/build" +WHEEL_PATH="$(ls "${DIST_DIR}"/query_doctor-*-py3-none-any.whl)" +WHEEL_NAME="$(basename "${WHEEL_PATH}")" + +step "Fetch Pyodide ${PYODIDE_VERSION}" +NPM_TMP="$(mktemp -d)" +trap 'rm -rf "${NPM_TMP}"' EXIT +(cd "${NPM_TMP}" && npm install --silent --no-package-lock "pyodide@${PYODIDE_VERSION}") +for file in "${PYODIDE_FILES[@]}"; do + src="${NPM_TMP}/node_modules/pyodide/${file}" + if [ -f "${src}" ]; then + cp "${src}" "${DIST_DIR}/vendor/" + else + printf 'warning: pyodide file not present in this release: %s\n' "${file}" >&2 + fi +done + +step "Generate the synthetic sample profile" +"${PYTHON_BIN}" "${WEB_DIR}/bench/make_profile.py" 100000 "${DIST_DIR}/sample-profile.txt" + +step "Copy the page" +# The page loads the wheel by name; keep the built filename in sync. +sed "s|query_doctor-0\.11\.0-py3-none-any\.whl|${WHEEL_NAME}|g" \ + "${WEB_DIR}/index.html" > "${DIST_DIR}/index.html" + +step "Result" +du -sh "${DIST_DIR}" +printf 'Serve locally with: %s -m http.server 8799 --directory %s\n' "${PYTHON_BIN}" "${DIST_DIR}" diff --git a/web/index.html b/web/index.html new file mode 100644 index 0000000..d0156da --- /dev/null +++ b/web/index.html @@ -0,0 +1,256 @@ + + + + + +Query Doctor — analyze an Impala profile in your browser + + + +
+

Analyze an Impala query profile

+

Drop an exported text query profile and get a deterministic diagnosis.

+ +
+ The profile never leaves your browser. The Query Doctor analyzer runs here, + in WebAssembly. There is no server and no upload — open this page offline and it + keeps working. +
+ +
+

Drop a profile file here

+ or click to choose one — processed locally + +
+ +
+ + +
+ +
Loading the analysis runtime…
+
+ +
+ Synthetic demo data. Not performance evidence. + · GitHub + · Apache-2.0 +
+
+ + + + +