Retransformer — Language-Agnostic Semantic Code Adapter

RETF (REtransformer Format) is a universal semantic intermediate representation that captures the logical skeleton of any program: what it does, not how it looks in a specific language.

Each language has an Encoder (source → RETF) and Decoder (RETF → source), enabling cross-language code migration with structural fidelity.

Quick Start

# Encode Python to RETF
cargo run -- encode --input app.py -o app.retf

# Decode RETF to any supported language
cargo run -- decode --coder Rust.red --input app.retf -o app.rs
cargo run -- decode --coder JavaScript.red --input app.retf -o app.js
cargo run -- decode --coder Go.red --input app.retf -o app.go

# Roundtrip (encode + decode + verify determinism)
cargo run -- roundtrip --coder Python.red --input app.py

Supported Languages

Language	Encoder	Decoder	AST Parser	.red Plugin
Python	✅	✅	python3 ast.parse	Python.red
Rust	✅	✅	syn crate (native)	Rust.red
JavaScript	✅	✅	Node.js + TS parser	JavaScript.red
TypeScript	✅	✅	Node.js + TS parser	TypeScript.red
Go	✅	✅	go/parser + go/ast	Go.red
Java	✅	✅	javac Tree API (JDK 26)	Java.red
C	✅	✅	clang -ast-dump=json	C.red
C++	✅	✅	clang++ -ast-dump=json	Cpp.red
Kotlin	✅	✅	kotlinc PSI API	Kotlin.red
C#	✅	✅	dotnet Roslyn API	CSharp.red
Swift	✅	✅	swiftc -dump-ast	Swift.red

How It Works

┌──────────────┐     ┌─────────────────┐     ┌──────────────┐
│  Python .py  │ ──→ │    RETF IR      │ ──→ │   Rust .rs   │
│              │     │  (semantic IR)  │     │              │
│  x: int = 42 │     │  {              │     │ let x: i64   │
│  def f():    │     │    "intent":    │     │   = 42;      │
│    return 1  │     │    "function",  │     │ fn f() ->    │
│              │     │    "name": "f"  │     │   i64 { 1 }  │
│              │     │  }              │     │              │
└──────────────┘     └─────────────────┘     └──────────────┘
     ENCODER                                  DECODER
  (Python.red)                              (Rust.red)

RETF captures 38 intent kinds covering all programming constructs: variables, functions, classes/structs, interfaces, enums, generics, conditionals, loops, iterators, try/catch, resource guards, yield, comprehensions, f-strings, lambdas, imports, assignments, and more.

Architecture

src/
├── retf/              ← RETF semantic IR (intents, types, flow graph)
│   ├── intent.rs       ← 38 intent kinds + expression types
│   ├── type.rs         ← Semantic type system (SemType)
│   ├── graph.rs        ← Flow graph (nodes, edges, roots)
│   ├── document.rs     ← RetfDocument (contains intents + flow graph)
│   └── ser.rs          ← JSON serialization
│
├── red/               ← Encoder/Decoder framework
│   ├── mod.rs          ← Encoder & Decoder traits
│   ├── plugin.rs       ← Dynamic .red plugin loader (C ABI)
│   ├── library.rs      ← Library manifest types
│   ├── registry.rs     ← PackageRegistry trait + CratesIoRegistry
│   ├── resolver.rs     ← LibraryResolver (coverage scoring)
│   ├── python/         ← Python encoder + decoder
│   ├── rust/           ← Rust encoder (syn) + decoder
│   ├── javascript/     ← JS encoder + decoder
│   ├── typescript/     ← TS encoder + decoder
│   ├── golang/         ← Go encoder + decoder
│   ├── java/           ← Java encoder + decoder
│   ├── clang/          ← C encoder + decoder
│   ├── cpp/            ← C++ encoder + decoder
│   ├── kotlin/         ← Kotlin encoder + decoder
│   ├── csharp/         ← C# encoder + decoder
│   └── swift/          ← Swift encoder + decoder
│
├── analysis/          ← Code analysis
│   └── infer.rs        ← FlowAnalyzer (type inference)
│
└── main.rs            ← CLI (encode, decode, roundtrip)

.red Plugin Format

.red plugins are shared libraries exposing a C ABI:

uint32_t red_version();         // ABI version (1)
const char* red_language();     // "python", "rust", ...
char* red_encode(const char*);  // source → RETF JSON
char* red_decode(const char*);  // RETF JSON → source
void red_free_string(char*);    // free allocated string

Pre-compiled plugins are in red_plugins/. To use:

retransformer encode --coder red_plugins/Python.red --input app.py
retransformer decode --coder red_plugins/Rust.red --input app.retf -o app.rs

Library Migration

retransformer can resolve library dependencies across ecosystems. When encountering import numpy as np, the encoder:

1. Extracts metadata (pip show numpy → version, description)
2. Tracks used functions (np.array, np.sum, np.dot, ...)
3. Stores a LibraryManifest in the RETF document

The decoder then:

1. Searches the target ecosystem registry (crates.io, npm, ...)
2. Maps known functions via static mappings (~40 built-in)
3. Ranks candidates by API coverage + popularity + category match
4. Emits recommendations as comments + generates mapped code

Python:  json.dumps(obj)          →  Rust:  serde_json::to_string(obj)
Python:  np.array(data)           →  Rust:  Array1::from_vec(data)
Python:  np.sum(arr)              →  Rust:  arr.sum()

Testing

cargo test    # 97 tests, 0 failures

Test suites:

• retf::ser — JSON serialization roundtrip
• roundtrip_tests — 28 Python roundtrip tests
• demo_test — Full pipeline: Python → RETF → Python + Rust
• lib_demo_test — Library resolution: numpy→ndarray, json→serde
• multilang_test — 1 RETF → 11 target languages
• skeleton_proof_test — 17 proof tests: 15 constructs × all decoders
• encoder_validation_test — 12 encoder validation tests
• rustgen_tests — Python → RETF → Rust generation

Requirements

• Rust 1.85+ (edition 2024)
• Python 3.8+ (for Python encoder)
• Node.js 22+ (for JavaScript/TypeScript encoders)
• Go 1.21+ (for Go encoder)
• JDK 23+ (for Java encoder)
• clang 18+ (for C/C++ encoders)
• .NET SDK 10+ (for C# encoder)
• Kotlin 2.0+ (for Kotlin encoder)
• Swift 6.0+ (for Swift encoder)