feat: next .af syntax surface — comments, .doc, .var modifiers, iteration, raw blocks

[Peyton-Spencer]

Context

With the v1 surface now stable (codegen, LSP, demo init, fmt, lint shipped through v0.10.0), the next milestone is filling out the .af template language itself. This issue tracks the syntax features that are either explicitly TODO in the codebase or natural follow-ups we've referenced repeatedly.

Proposed syntax additions

1. Comments (# line comments)

Reserved at pkg/token/kind/kind.go:56 (// Comment). No way to annotate templates today without committing the comment text into the rendered output.

.title System Prompt
# This prompt is used for the onboarding agent only.
You are a helpful assistant.

• New token kind: Comment
• Tokenizer recognizes # at column 0 (or after .title/before next directive) and consumes the rest of the line.
• Codegen and fmt strip comments from output but preserve them in the file.
• LSP: highlight as comment semantic token.

2. Doc strings on titles and vars

Explicit TODO at pkg/token/kind/kind.go:7 (KindDoc, KindVarDoc).

.title Greeting
.doc Greet the user by name. Used by /onboard.
Hello <!name string>!
.doc full legal name as it appears on the customer's account

• .doc directive attaches to the next title or variable declaration.
• Codegen emits the doc string as a Go doc comment on the struct or struct field.
• LSP hover shows the doc text.

3. .var predeclare directive with type modifiers

Explicit TODO at pkg/token/kind/kind.go:8-23. Lets users predeclare a variable's type and modifiers without inlining at first use.

.title say hello to your new friends
.var names string list join="\n"
Please say hello to:
<!names>

INPUT: ["Joe", "Mary", "Jane"]

OUTPUT:

Please say hello to:
Joe
Mary
Jane

Scope for v1:

• list modifier → field type becomes []T
• join="..." modifier → joiner string for list rendering
• default="..." modifier → renders if value is the zero value
• .var is a declaration only, no rendering at the directive line.

4. Iteration over lists (<#each>)

Once .var ... list exists, we need a way to loop with non-trivial body content.

.var users user list
<#each users>
- <!users.name> (<!users.email>)
</each>

• New directive: DirectiveEach (<#) with matching </each>.
• Inside the block, <!users.field> refers to the current iteration element.
• Codegen produces a for _, users := range input.Users { ... } over a struct slice.

5. Raw blocks (<raw>...</raw>)

Reserved at pkg/token/kind/kind.go:55 (// RawBlock). Today there is no way to emit literal <!foo> or <?cond> text — the tokenizer always interprets them.

.title docs
The variable syntax in agentflow looks like:
<raw>
<!variable_name>
<?conditional>...</conditional>
</raw>

• New token kind: RawBlock.
• Everything between <raw> and </raw> passes through verbatim, including angle brackets.
• Useful for self-documenting prompts that describe the syntax.

6. Comparison shorthand for booleans

Today: <?premium bool>...<else>...</premium> only checks truthiness. There's no clean way to express "false branch only" or to negate without the awkward <else> swap.

Proposal:

• <?!premium>...</premium> — negation form (inverts the conditional).
• <?premium eq false> already works but is verbose; <?!premium> is the ergonomic form.

Token additions: DirectiveCondNeg or treat ! as a modifier on DirectiveCond.

7. Includes / partials

Lower priority but commonly requested for any template engine.

.title Email
.include components/header.af
Body text here.
.include components/footer.af

• .include <path> directive at file or section level.
• Path resolution relative to the including file.
• Recursion guard for cycle detection.
• Out of scope for v1 — file as a follow-up if there's user demand.

Suggested ordering

Phase 1 (low-risk, no parser ambiguity):

1. Comments (#)
2. .doc strings
3. <raw>...</raw> blocks

Phase 2 (real language work):
4. .var predeclare with modifiers (list, join, default)
5. <#each> iteration
6. <?!cond> negation shorthand

Phase 3 (deferred):
7. Includes / partials

Out of scope

• Function calls / pipes (<!name | upper>) — too far from the "no logic in templates" stance.
• Math expressions in conditionals — current eq/ne/gt/lt/gte/lte is enough.
• New scalar types (time.Time, []byte) — file separately if needed.

Definition of done per phase

Each phase ships:

• Tokenizer + AST changes with golden file tests in pkg/gen/gogen/testdata/
• Codegen support
• fmt round-trip preservation
• lint rules for misuse (e.g. AF010 unclosed-raw-block, AF011 each-without-list-var)
• LSP semantic tokens + hover updates
• README syntax section update
• VS Code TextMate grammar update in omniaura/agentflow-vscode

References

• TODO comments: pkg/token/kind/kind.go:7-23, 55-56
• Existing token kinds: pkg/token/kind/kind.go
• Existing AST: pkg/ast/ast.go
• Codegen: pkg/gen/gogen/gogen.go
• Test fixtures: pkg/gen/gogen/testdata/