Add Docker support, dataset validation, and CLI enhancements - #33
Open
pradeepgudipati wants to merge 2 commits into
Open
Add Docker support, dataset validation, and CLI enhancements#33pradeepgudipati wants to merge 2 commits into
pradeepgudipati wants to merge 2 commits into
Conversation
Add Docker/Docker Compose and .env.example for containerised local runs; update README and add MIT LICENSE. Introduce a CLI entrypoint (src/cli.py) and package script, load env vars in app, and wire a Dockerfile to run the app. Implement dataset validation (src/data_tools/dataset_validation.py) and integrate validation into CSV/JSONL import/export flows; improve JSONL↔SQLite converters, add schema migration/ensure logic, and robust backup/restore utilities. Refactor utils (remove old inline validators), add samples and CSV/JSONL examples, and add unit tests covering startup, CSV→JSONL, DB backup/restore, and JSONL roundtrip. Update CI to use uv, run precommit and pytest, and refresh pyproject.toml (dependencies, optional 'ai' extras, scripts, build settings). Minor housekeeping: .gitignore/.dockerignore updates and various small bugfixes and robustness improvements (encoding, path handling, safer file ops).
Introduce dataset validation end-to-end and developer ergonomics: add /api/validate_dataset, a comprehensive dataset validation implementation (CSV/JSONL parsing, HF conversation/instruction/preference/tool/rag/eval normalizers, duplicate detection, PII/secret and size checks, and build_validation_report). Add scripts/quickstart.sh for bootstrapping and update README and CI to use scripts/precommit.sh and the quickstart flow; move precommit.sh into scripts/. Add src/config.py to load .env.example/.env into the runtime and require certain env vars; refactor app/cli/ai_api to use the new config loader and to resolve DB paths and ensure DB schema on startup. Update frontend upload UI and JS to show a Validate button and render validation reports before processing. Remove legacy requirements.txt and update .env.example, .gitignore, and docker-compose to surface the example env and DB path. Add tests covering validation, duplicate detection, quickstart, and validation API.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| http_status = 200 if report["status"] == "valid" else 422 | ||
| return jsonify(status=report["status"], report=report), http_status | ||
| except ValueError as error: | ||
| return jsonify(status="error", message=str(error)), 400 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a comprehensive modernization and refactor of the LLMTrainingTools project, focusing on improved developer experience, clearer onboarding, and enhanced local and containerized workflows. Key improvements include a streamlined quickstart script, Docker support, a modernized dependency and configuration system, and a rewritten README for clarity. Several sample datasets and validation examples are also added.
Project Setup & Developer Experience
quickstart.shscript for one-command local setup, dependency installation, and app launch, supporting dry-run and no-run modes for flexibility..dockerignorefile and aDockerfilefor efficient container builds, plus adocker-compose.ymlfor simple multi-service orchestration. [1] [2] [3]uvfor dependency management, runs lint and tests on push and PR, and supports multiple Python versions.Configuration & Dependency Management
.env.examplefor environment variable documentation and default config, and updated code to load configuration from this file..env.localis now ignored by default. [1] [2] [3] [4]pyproject.toml: core dependencies are minimal, AI/vector extras are optional, dev dependencies are grouped, and the CLI entrypoint is defined.Documentation & Usability
README.mdfor clarity and quick onboarding, including usage walkthroughs, dependency groups, configuration, Docker/Compose instructions, and validation guidance.sample_qa.csv,sample_chat.jsonl) to illustrate supported dataset shapes and validation. [1] [2]Licensing
LICENSEfile for clear open-source terms.App Initialization & Robustness
Project Setup & Developer Experience
quickstart.shfor streamlined local bootstrap, dependency install, and app launch, with dry-run and no-run options.Dockerfile,.dockerignore,docker-compose.yml) for containerized deployment and development. [1] [2] [3]uv, runs lint/tests on push and PR, and supports Python 3.10–3.12.Configuration & Dependency Management
.env.examplefor config, with code loading from this file;.env.localis now ignored. [1] [2] [3]pyproject.toml: minimal core deps, optional AI/vector extras, dev group, CLI entrypoint.Documentation & Usability
README.mdfor clear onboarding, usage, validation, and Docker/CI instructions.Licensing
LICENSEfile.App Initialization & Robustness