Anthropic MVP#126
Open
MichaelMcCulloch wants to merge 26 commits intomicrosoft:mainfrom
Open
Conversation
|
|
||
| dependencies = [ | ||
| "llmx>=0.0.21a", | ||
| "llmx@git+https://github.com/MichaelMcCulloch/llmx.git#egg=main", |
Author
There was a problem hiding this comment.
TODO: Revert this to upstream llmx when and if the above is merged into it
overhaul the backend and the UI See merge request MichaelMcCulloch/lida!1
Replace the single "Analyzing data..." spinner with a deterministic 5-stage tracker (compress / upload / decompress / analyze / visualize), real byte-level compress + upload progress bars, an LLM token counter fed by streaming deltas, and per-table sub-progress for sqlite/tar fan-out. - LiteLLMTextGenerator gains a token_sink ContextVar; when set, the generator switches to OpenAI streaming and routes deltas to the sink without changing adapter call sites. - New POST /api/v1/summarize/stream returns text/event-stream and emits stage/llm.token/table/chart events; final 'complete' payload mirrors the existing /summarize response shape so the App handler is unchanged. - Frontend useUploadPipeline hook drives chunked pako compression, XHR upload progress, and incremental SSE frame parsing into a reducer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SQLite BLOBs (and similar object-dtype byte columns) were being sampled and f-stringed into the summarizer prompt, where the LLM either echoed the raw bytes back or hallucinated against them — observed as little-endian float32 garbage in DeepSeek output for embedding tables. - Detect bytes/bytearray/memoryview columns and classify dtype="binary". - Replace samples with size summaries (<binary blob: N bytes>) and expose min/max/mean size so the model has shape context. - Add a recursive _scrub_summary() pass in enrich() as defense-in-depth for stragglers (mixed-type columns, numpy bytes_, nested structures). - Catch TypeError in pd.to_datetime so non-string object columns no longer crash detection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three things, one commit because they ride together: 1. Parallel table fan-out. SQLite/tar uploads now run summarize_and_visualize concurrently across tables via ThreadPoolExecutor (LIDA_TABLE_CONCURRENCY, default 4). Token sinks moved into summarize_and_visualize so each table tags its llm.token events with file_name. Per-table chart pools are sized to share CPUs (cpu_count // n_parallel) so we don't oversubscribe when 4 tables each spawn matplotlib worker pools. Original table order is preserved in the response. 2. SSE keepalive. Yield ': keepalive' comment every 5s when the queue is quiet so buffering proxies (Vite dev proxy, nginx, k8s ingress) don't hold bytes back during gaps between LLM streaming and the heuristic phases — the gap was masking summary.ready and later events on a deployed setup. 3. Pipeline logging at emit boundaries (summary.ready, goals.ready, viz.code.ready, chart count, complete) so the docker logs show real pipeline progress instead of going silent after the LLM call. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Goal generation now uses an SSE endpoint (/api/v1/goal/stream) so the user sees progress while the LLM is producing the array. Goals appear one at a time as their closing brace arrives — same cost as the existing single call (no per-goal prompt overhead, the LLM still balances diversity across the full set), with progressive UI. - Extracted _sse_streaming_response so /summarize/stream and /goal/stream share the queue/thread/keepalive plumbing. - _GoalStreamParser walks streamed tokens once, tracking brace depth and string state to emit each top-level object as it closes. State is preserved across feeds so we don't rescan from the start. - Final 'goals' event still emits the adapter's authoritative parse so anything the streaming tracker missed is reconciled. - Frontend: useSseFetch hook (XHR + incremental SSE parser, smaller cousin of useUploadPipeline). GoalGenerator now shows the LLM token counter while streaming and pops in goal cards as goal.ready events arrive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The upload pipeline is now end-to-end LLM-driven and pipelined: as each
goal closes in the streamed JSON array, an LLM viz-code task is
submitted to a thread pool without waiting for subsequent goals. Chart
execution runs in a shared process pool because matplotlib state is
not thread-safe. With N=10 goals you get up to N+1 concurrent LLM
calls (1 goal stream + N plot streams) all running while goals are
still being generated.
User-visible changes:
- Goal-count slider (1-10, default 5) on the upload screen.
- Goal generation is automatic — no button click after upload.
- New GoalsBoard renders goals progressively with attached chart
slots that show "generating code" -> "rendering chart" -> rendered,
driven by plot.started / plot.code.ready / chart.rendered events.
Backend:
- /summarize/stream takes n_goals as a query param (clamped 1..10).
- summarize_and_visualize splits behavior on `emit`: streaming flow
uses _llm_visualize_pipeline; JSON endpoint keeps heuristic flow.
- _GoalStreamParser feeds a goal token sink that submits plot tasks
the moment each '{...}' closes.
- _run_plot_task per goal: LLM viz call with token sink (tagged with
goal_index), then chart execution dispatched to ProcessPoolExecutor.
- New events: goal.ready, plot.started, plot.code.ready, plot.failed,
chart.rendered (now goal_index-keyed). Existing llm.token events
carry phase=summary|goals|plot for attribution.
Frontend:
- App.tsx now owns the pipeline hook and renders progressively from
pipeline state (summary / goals / charts appear during the run, not
just at the end). FileUploader becomes presentational.
- useUploadPipeline state grows summary, goals[], charts{}, plotStatuses{}
on each TableProgress entry, populated by the new event types.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per the user's clarification: analyzing tables is a fence (wait for all),
but goals are NOT per-table — one LLM call ingests every analysis and
returns n_goals globally, each tagged with the data_source it queries.
Plots stay 1:1 with goals.
Pipeline phases (replacing the old per-table goals/plots inside each
summarize_and_visualize call):
1. analyze per-table summaries in parallel; fence on completion.
2. goals single LLM call with all summaries; goals carry data_source.
3. visualize 1:1 plot tasks per goal; each picks its DataFrame via
data_source. LLM viz calls run in a thread pool, chart
execution in a shared process pool.
Backend:
- summarize_and_visualize in streaming mode now returns summary only
(and file_path so plots can re-open the dataframe). The orchestrator
in _run_streaming_pipeline drives goals + plots at the global level.
- New _generate_cross_dataset_goals composes a multi-summary prompt,
asks the model to emit a "data_source" field on every goal, and
parses the stream incrementally into goal.ready events.
- New _run_parallel_plots dispatches plot tasks 1:1 with goals, each
picking its (summary, data_df) by data_source.
- Stage list collapsed: dropped "decompress" (folded into analyze),
added explicit "goals" stage. Final order: compress / upload /
analyze / goals / visualize.
- JSON /summarize endpoint keeps its legacy heuristic per-table flow
(unchanged behavior for non-streaming clients).
Frontend:
- PipelineState moves goals/charts/plotStatuses to the top level — they
are no longer keyed by table because the backend produces them once
globally. TableProgress now holds only {name, status, summary}.
- GoalsBoard takes GoalWithSource[] and renders a data_source pill on
each card when the upload is multi-table.
- App.tsx selects the active table for summary display, but the goals
board shows the global list with data_source attribution. The
Visualizer pulls the right summary by matching selectedGoal.dataSource
against the table list.
- TableProgressList simplified — per-table chart counts are gone since
charts are global.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LIDA_MO_INSTANT_ANALYSIS=false in the deployed env was suppressing goals/visualize entirely — pipeline emitted "summary-only" right after the analyze fence. The env var was a relic from the heuristic auto-flow; in the new architecture an explicit n_goals slider implies the user wants goals and plots. Default visualize=True unconditionally and have the frontend pass it explicitly so server-side env changes can't bite us again. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plot tasks were calling lida.visualize(), which under the hood does both code generation AND execution — so every chart ran twice and the second pass got a ChartExecutorResponse where it expected a code string. The 'NoneType object is not subscriptable' errors in the docker logs were the LLM-generated chart code crashing during the FIRST execution, and 'ChartExecutorResponse object has no attribute replace' in the UI was preprocess_code() trying to .replace() the response object during the SECOND execution attempt. Switch to lida.vizgen.generate() which returns just the code strings, then submit to chart_pool exactly once. Halves the work and gives clean error attribution: failed user code now surfaces with the actual seaborn / pandas exception message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.