Skip to content

Reject a rule output that would overwrite a source file (fixes #194) - #198

Open
typeless wants to merge 1 commit into
mainfrom
fix/194-reject-shadowed-source
Open

Reject a rule output that would overwrite a source file (fixes #194)#198
typeless wants to merge 1 commit into
mainfrom
fix/194-reject-shadowed-source

Conversation

@typeless

Copy link
Copy Markdown
Owner

A rule whose output path was also a checked-in source file was accepted, and in an in-tree build putup overwrote the tracked file on disk:

a/Tupfile:  : gen.src |> cp %f %o |> x.dat      (a/x.dat is checked in)

before: FROMSRC
after:  FROMRULE      <- the tracked file is gone

ensure_file_node silently upgraded an existing File node to Generated (src/graph/dag.cpp:140-145), so the generated node simply won. Upstream tup rejects the project instead — in-tree and in variant builds alike — and the diagnostic here follows its wording, including both remedies it offers.

The bisect found no first-bad commit: this reproduces at the earliest commit where a subdirectory rule can run at all. Nothing to revert; the check has never existed.

What putup can decide, and what it can't

The previous index is what distinguishes a source file from our own output sitting in the source tree after an in-tree build — the distinction tup's persistent node types carry for free. So the check fires on exactly two answerable questions:

  • No previous index → nothing we produced can be on disk yet, so a file at an output's path is a source.
  • A previous index → a path it recorded as a source File is a source, whatever is on disk now.

A generated file the previous index does not mention is not decidable from here and is left alone. That case is real rather than hypothetical: a scoped build drops out-of-scope commands from the saved index, and the first draft of this check rejected the subsequent full build because of it. That is the gap, stated deliberately — a shadowing rule added to an established project, whose target no rule reads, will still slip through.

Consequence worth knowing

Deleting .pup while in-tree artifacts remain now errors until they are cleaned. That is not a regression against tup — deleting .tup in the same situation produces the same error and the same remedy, which I verified by running the tup binary:

tup error: Attempting to insert 'foo.o' as a generated node when it already exists
as a different type (normal file).

Config outputs are exempt

The two-stage configure design has a rule produce the tup.config that the same build then reads as configuration. That path is generated and read on purpose rather than shadowing anything, so it is skipped.

The [deviation] test from #193 now expects the rejection

That scenario asserted the old shadow-wins behaviour and said outright that it should be rewritten if #194 were fixed by rejecting the project. It now expects the error. A consequence the test records: the dedup #191 added for this shape is unreachable through a legal project, so it remains only as a defence.

Verification

make test: 142412 assertions, 639 cases, 32 e2e shards. make tidy, make iwyu clean.

By execution: rejection in-tree and out-of-tree with the source file intact; an in-tree rebuild still succeeds with its own output on disk; scoped-then-full in-tree still succeeds (the false positive that the first draft produced); GCC BSP parses (24 Tupfiles, 1853 commands); busybox parses (31 Tupfiles, 293 commands).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA

A rule whose output path was also a checked-in source file was accepted, and in an
in-tree build putup overwrote the tracked file on disk:

  a/Tupfile:  : gen.src |> cp %f %o |> x.dat     (a/x.dat is checked in)

  before: FROMSRC
  after:  FROMRULE          <- the tracked file is gone

ensure_file_node silently upgraded an existing File node to Generated
(src/graph/dag.cpp:140-145), so the generated node simply won. Upstream tup rejects
the project instead, in-tree and in variant builds alike, and the diagnostic here
follows its wording including both remedies.

The bisect on the issue found no first-bad commit: this reproduces at the earliest
commit where a subdirectory rule can run at all, so there is nothing to revert and
the check has simply never existed.

What putup can decide, and what it cannot. The previous index is what tells a
source file apart from our own output sitting in the source tree after an in-tree
build -- the distinction tup's persistent node types carry. So the check fires on
exactly two answerable questions: with no previous index nothing we produced can be
on disk yet, so a file at an output's path is a source; with one, a path recorded
as a source File is a source whatever is on disk now. A generated file the previous
index does not mention is not decidable from here and is left alone -- that case is
real, not hypothetical: a scoped build drops out-of-scope commands from the saved
index, and the first draft of this check rejected the subsequent full build because
of it.

Deleting .pup while in-tree artifacts remain therefore errors until they are
cleaned. That is not a regression against tup: deleting .tup in the same situation
produces the same error and the same remedy, verified against the tup binary.

Config outputs are exempt. The two-stage configure design has a rule produce the
tup.config that the same build then reads as configuration, so that path is
generated and read on purpose rather than shadowing anything.

The [deviation]-tagged scenario added in #193 asserted the old shadow-wins
behaviour and said it should be rewritten if this issue were fixed by rejecting the
project. It now expects the rejection. The dedup #191 added for this shape is
consequently unreachable through a legal project and stays only as a defence, which
the test records.

Verified: rejection in-tree and out-of-tree with the source file intact; an in-tree
rebuild still succeeds with its own output on disk; scoped-then-full in-tree still
succeeds; GCC BSP parses (24 Tupfiles, 1853 commands) and busybox parses (31
Tupfiles, 293 commands).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
@github-actions

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1076 M 0.51 s 14.2 k 0.8% (+0.2pp) 0.1% (+0.1pp) 0.51 s 31.2 MB (-0.1MB)
dry-run 1886 M (+0.3%) 0.61 s 16.6 k 0.9% (+0.2pp) 0.1% (+0.1pp) 0.639 s 40 MB (-0.2MB)

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5818
Files changed 0
Files in index 6087
Graph edges 367913
Index size (bytes) 7378282
Implicit deps 330624
Hash computations 0
Hashes skipped (stat cache) 5818
Stat calls 5869
Parse time (ms) 506.6
Total time (ms) 617.7
Runner CPU AMD EPYC 7763 64-Core Processor

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.
Timing deltas suppressed: baseline ran on different hardware (INTEL(R) XEON(R) PLATINUM 8573C).

Binary size (Linux)

Binary .text .data .bss File
putup 519.1 KB (+0.3%) 1.6 KB 98.7 KB 615.4 KB

Code churn (whole codebase, last 30d)

Files Lines written Still present Churned Churn rate
119 7581 6808 773 10.2%

Of the lines written across the codebase in the last 30 days, how many are already gone — work that was written and then discarded or rewritten inside the same window. This is the state of the tree including this PR, not a measure of the PR itself. Only code we write is counted: tests, examples, vendored and generated files, CI plumbing and prose are excluded. 3571 lines were deleted in the window in total, most of them older than it.

Where the churn is
File Lines written then discarded
src/cli/cmd_build.cpp 247
src/core/layout.cpp 69
src/core/new_delete.cpp 63
src/cli/context.cpp 55
src/graph/builder.cpp 44
src/core/string_pool.cpp 34
src/core/heap_buf.cpp 33
src/platform/vm-posix.cpp 22
src/cli/options.cpp 21
include/pup/core/layout.hpp 15

Test coverage (lines)

Overall Median file Min file Max file
86.7% 95% 0.0% src/platform/path-posix.cpp 100.0% include/pup/core/arena.hpp

103 files · 15195/17532 lines covered

Deltas vs main@a1277e1d4.

Updated for 7e9df2d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant