feat: implement multi-file workspace upload system with schema parsin…

[01Vishwa]

…g and centralized state management

[Copilot]

Pull request overview

Implements a workspace-scoped, multi-file upload pipeline that persists per-file schema metadata, infers join candidates server-side, and surfaces that context to the frontend via centralized state. In parallel, it removes the Eval dashboard surface area and includes several backend reliability/security improvements (MIME sniffing, parse warnings, disconnect handling, lifespan + eviction supervision).

Changes:

• Added workspace-aware upload + file listing/deletion endpoints backed by a new workspace_files table, plus a schema_merger service and MultiFileContext model for join-candidate inference.
• Updated the React upload flow/store to use workspace-scoped endpoints, track file-loading state, and store multiFileContext + join overrides.
• Removed Eval dashboard codepaths (frontend + backend) and tightened backend orchestration behavior (token budget guard, supervised tasks, async JWKS fetch, parse warnings).

Reviewed changes

Copilot reviewed 77 out of 77 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/stores/workspaceStore.js	Adds multi-file context + join override state/actions to the workspace store.
src/services/evalApi.js	Removes eval dashboard API client.
src/services/api.js	Routes uploads to workspace-scoped endpoint when workspaceId is present.
src/pages/ProjectsPage.jsx	UI/layout tweaks (pagination, footer, scroll behavior) and removes unused auth token accessor.
src/pages/HomePage.jsx	Wires filesLoading into upload panel; removes eval link/metrics props.
src/hooks/useFileUpload.js	Fetches workspace files from new endpoint; maps schema_json dtypes; stores multi_file_context from upload responses; adds filesLoading.
src/hooks/useAgentRun.js	Removes metrics state; switches DEFAULT_SETTINGS import to constants module.
src/contexts/AuthContext.jsx	Disables react-refresh export rule (lint workaround).
src/components/upload/FileUploadPanel.jsx	Adds loading empty-state (spinner) when files are being fetched.
src/components/shared/WorkspaceSelector.jsx	Simplifies create handler (drops useCallback).
src/components/OutputPanel.jsx	Adjusts bullet-cleaning regex.
src/components/eval/TracePanel.jsx	Removes eval trace UI.
src/components/eval/RunList.jsx	Removes eval run list UI.
src/components/eval/KpiCard.jsx	Removes eval KPI card UI.
src/components/eval/DebugChart.jsx	Removes eval debug chart UI.
src/components/eval/AgentTable.jsx	Removes eval agent table UI.
src/components/agent/HistoryPanel.jsx	Removes eval-metrics display from history items.
src/components/agent/CodeBlock.jsx	Changes CodeBlock props (drops onToggleCollapse).
src/components/agent/agentSettingsConstants.js	New constants module for DEFAULT_SETTINGS (Fast Refresh compatibility).
src/components/agent/AgentSettings.jsx	Imports DEFAULT_SETTINGS from new constants module.
src/components/agent/AgentProgressPanel.jsx	Removes metrics overlay usage and tweaks bullet-cleaning regex.
src/components/agent/AgentMetricsOverlay.jsx	Removes metrics overlay component.
src/App.jsx	Removes /eval route; minor hook usage adjustment.
scratch/test_supabase_schema.py	Removes scratch script.
scratch/test_planner.py	Removes scratch script.
scratch/test_planner_debug.py	Removes scratch script.
scratch/test_env.py	Removes scratch script.
scratch/test_dump.py	Removes scratch script.
scratch/test_db_data.py	Removes scratch script.
scratch/backfill_user_id.py	Removes scratch script.
README.md	Updates tech stack/docs and removes eval/API reference sections.
eslint.config.js	Expands unused-vars ignore patterns; adds Node globals for config files.
backend/utils/helpers.py	Removes unused UUID ID generator helper.
backend/tests/test_upload_controller.py	Adds controller tests for workspace upload/delete + legacy upload behavior.
backend/tests/test_schema_merger.py	Adds tests for join inference + prompt/header generation.
backend/services/upload_service.py	Adds magic-byte MIME validation and cache locking for thread safety.
backend/services/supabase_service.py	Fixes file listing scoping by user; adds workspace_files CRUD; updates timestamps to UTC-aware; removes eval-metrics persistence.
backend/services/schema_merger.py	New join-candidate inference service producing MultiFileContext.
backend/services/process_service.py	Captures near-empty parse warnings separately from hard failures.
backend/services/parsers/txt_parser.py	Raises ParseWarning for near-empty parsed text; improves decode/sanitize error handling.
backend/services/parsers/pdf_parser.py	Raises ParseWarning for near-empty extracted text; improves error specificity.
backend/services/parsers/parquet_parser.py	Uses JSON-safe record sanitization for sample rows.
backend/services/parsers/md_parser.py	Raises ParseWarning for near-empty parsed text; improves decode/parse error handling.
backend/services/parsers/excel_parser.py	Uses JSON-safe record sanitization for sample rows/previews.
backend/services/parsers/csv_parser.py	Uses JSON-safe record sanitization for sample rows/sample_data.
backend/services/parsers/init.py	Introduces ParseWarning and sanitize_records helper.
backend/requirements.txt	Adds kaleido for plotly static export.
backend/models/multi_file_context.py	Adds canonical Pydantic models and prompt/reader-header helpers for multi-file context.
backend/models/metrics_schema.py	Removes DS-STAR metrics schema (eval/metrics de-scoped).
backend/middleware/auth.py	Makes JWKS refresh async + cached with rotation retry; updates JWT decode flow.
backend/main.py	Migrates to lifespan; adds supervised session-eviction background task; centralizes config health logging.
backend/eval/schemas.py	Removes eval schema module.
backend/eval/eval_store.py	Removes eval persistence store.
backend/eval/eval_routes.py	Removes eval API routes.
backend/eval/eval_logger.py	Removes eval SSE sidecar logger.
backend/eval/eval_engine.py	Removes eval metrics computation engine.
backend/eval/init.py	Removes eval package marker.
backend/db/migrations/004_workspace_files_file_size.sql	Adds file_size column and widens file_type constraint (txt/pdf/etc).
backend/db/migrations/003_workspace_files.sql	Adds workspace_files table + RLS + index + backfill.
backend/db/create_eval_schema.sql	Removes eval schema SQL.
backend/db/add_eval_metrics_fk.sql	Removes eval FK migration SQL.
backend/core/planner/planner_agent.py	Adds step_type and join-hints injection; wraps LLM calls with timeouts.
backend/core/ds_star_orchestrator.py	Adds context budget guard + per-call max_rounds; removes eval metrics event emission.
backend/core/deep_research_orchestrator.py	Uses fresh orchestrator per subquestion (state isolation); forwards per-call max_rounds; improves event timing.
backend/core/debugger/debugger_agent.py	Injects multi-file join context into debugger on merge-related errors.
backend/core/config.py	Adds sandbox RLIMITs and context budget reserve; removes unused IDP formats constant.
backend/core/analyzer/file_analyzer.py	Adds analyze_multi and disk fallback for file content; multi-file prompt header.
backend/api/routes.py	Adds typed create-workspace request; adds workspace upload/list/delete endpoints; passes user_id to file list; removes eval router.
backend/api/controllers/research_controller.py	Adds disconnect monitoring; forwards max_rounds; emits parse warnings; workspace auto-hydration.
backend/api/controllers/process_controller.py	Adds get_schema_context helper for workspace multi-file schema context.
backend/api/controllers/agent_controller.py	Adds workspace auto-hydration; forwards per-call max_rounds/workspace_id; emits parse warnings; removes eval logger.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.