TyGen

An experimental, Dockerized research pipeline for observing TypeScript compiler behavior across supplied npm package artifacts.

TyGen takes package tarballs as input, extracts them into a controlled workspace, runs tsc --noEmit with a fixed configuration, and writes a CSV summary plus per-package compiler output. It is intended for reproducible exploration—not as a production package-quality score or a general-purpose sandbox.

Status: experimental research prototype. The checked-in results are raw artifacts from a small run and should not be generalized into a formal claim about package type safety.

What it measures

For each .tgz or .tar.gz artifact in tarballs/, the pipeline:

1. extracts the archive using Python's data-only tar filter;
2. finds the package and likely JavaScript code roots;
3. copies the repository's tsconfig.json into that code root;
4. invokes npx tsc --noEmit with a 60-second per-package timeout;
5. counts TypeScript diagnostic codes and records the result in results/results.csv;
6. saves combined compiler output to a per-package error log.

flowchart LR
  A["npm package tarballs"] --> B["safe archive extraction"]
  B --> C["package / code-root detection"]
  C --> D["fixed tsconfig + tsc --noEmit"]
  D --> E["CSV result row"]
  D --> F["per-package compiler log"]

Loading

Run with Docker

Docker is the most reproducible way to run the current experiment.

# Requires Docker. Create the input/output directories first.
mkdir -p tarballs results
# Place one or more .tgz / .tar.gz package artifacts in tarballs/ first.
docker build -t tygen .
docker run --rm -v "$PWD/results:/usr/src/app/results" tygen

The Dockerfile supplies Python 3.13, Node.js, npm, and TypeScript. Package artifacts are copied into the image at build time, so rebuild after changing tarballs/. Results are mounted back to results/ on the host. The Justfile contains equivalent just build and just run shortcuts for systems with just installed.

Run locally

The local path requires Python 3.13+, Node.js/npm, and TypeScript.

uv sync
npm install --global typescript
mkdir -p tarballs
# Add package archives to tarballs/, then:
uv run python main.py

Outputs

Artifact	Purpose
results/results.csv	One row per analyzed package: package, phase, status, message, compiler-help snapshot, and log path
results/*-errors.txt	Combined stdout/stderr from the TypeScript compiler
packages/	Extracted package workspaces created during analysis

The checked-in sample CSV is a legacy artifact from an earlier experiment and does not match the current six-column writer in main.py. Run the pipeline to generate results with the current schema.

Design and safety notes

• Compiler commands are invoked without a shell.
• Archive extraction uses Python's filter="data" mode.
• A TypeScript invocation that exceeds 60 seconds is recorded as a timeout rather than aborting the full run.
• Dependency installation and @types/node installation are disabled by default. If enabled for an experiment, installation uses --ignore-scripts and a bounded timeout.

TyGen is not a sandbox. npm package artifacts are executable code, and experiments involving untrusted packages should run in an isolation boundary appropriate to the threat model.

Limitations

• The current runner analyzes at most five artifacts per execution.
• The compiler configuration is intentionally fixed and may not match a package's supported TypeScript versions, dependencies, or build layout.
• Error-counting is a lightweight diagnostic heuristic, not a formal measure of type safety.
• There is no automated test suite, benchmark corpus, or published evaluation protocol yet.
• The project is not packaged or released as an npm/PyPI tool.

Provenance

This repository is a fork of dominictassio/tygen. The current branch contains Caleb Morse's experiment-specific pipeline work and results; preserve the upstream attribution when extending or presenting the project.

License

No license file is currently included. Do not assume permission to reuse the code until a license and any necessary upstream attribution are clarified.