Problem
GDBSample.ground_truth stores references to input/mask/reference image assets
as host-absolute filesystem paths pointing into data/gdb-dataset/…. For
example, typography-7 samples (evaluation_mode = inpaint_reconstruction)
have:
{
"input_image": "/Users/.../lica-bench/data/gdb-dataset/.../component_018_input.png",
"mask": "/Users/.../lica-bench/data/gdb-dataset/.../component_018_mask.png",
"ground_truth_image":"/Users/.../lica-bench/data/gdb-dataset/.../component_018_gt.png"
}
layout-8 similarly stores image, mask, reference_asset as absolute host
paths into the identity-evaluation assets.
This makes the benchmark non-portable for any evaluator that runs evaluate()
on a machine or container that doesn't have the original dataset mounted at the
exact same absolute path. The verifier silently degrades (e.g. evaluated_samples=0
and nan scores) instead of failing loudly.
Impact
- Containerized evaluation: Docker / Daytona / Modal runs need to ship GT
assets into the container, but there's no canonical way to discover which
assets are referenced (every benchmark declares its own GT shape, and
references can be nested arbitrarily under ground_truth).
- Third-party harnesses (e.g. Harbor adapter, external evaluators) must walk
the serialized ground-truth dict, detect strings that look like file paths,
copy them somewhere, and rewrite the paths — a fragile convention.
- Parity runs on a different host silently produce wrong numbers
(evaluated_samples = 0, ocr_accuracy = nan) if the paths don't resolve.
Proposed fix (any of these, roughly in order of preference)
- Relative paths + explicit base dir: store references as paths relative to
a dataset_root that the caller provides, and resolve them in
BaseBenchmark.evaluate() / sample loaders.
- Asset manifest: expose a typed
sample.assets: dict[str, Path] (or
AssetRef) so harnesses can enumerate GT-referenced files programmatically
without heuristic path-sniffing.
- Bundle GT files with sample: during
load_samples, attach asset bytes
(or content hashes) directly to the sample so evaluation doesn't need
filesystem access to the original dataset layout.
Workaround used downstream
The Harbor GDB adapter now walks the serialized ground_truth, copies any
string value that resolves to an existing absolute file into
workspace/gt/<deterministic-name>, and rewrites the path to
/workspace/gt/<name> (the container-relative path inside the Harbor task).
This is a pragmatic shim but is fragile: it relies on "any absolute path string
that happens to point at an existing file must be an asset reference."
Repro
bench = Typography7(...) # evaluation_mode = inpaint_reconstruction
sample = bench.samples[0]
gt = sample.ground_truth
# all three are host-absolute paths:
assert Path(gt.input_image).is_absolute()
assert Path(gt.mask).is_absolute()
assert Path(gt.ground_truth_image).is_absolute()
# -> fails to evaluate on any machine that doesn't mirror the host's data/ layout
Problem
GDBSample.ground_truthstores references to input/mask/reference image assetsas host-absolute filesystem paths pointing into
data/gdb-dataset/…. Forexample,
typography-7samples (evaluation_mode =inpaint_reconstruction)have:
{ "input_image": "/Users/.../lica-bench/data/gdb-dataset/.../component_018_input.png", "mask": "/Users/.../lica-bench/data/gdb-dataset/.../component_018_mask.png", "ground_truth_image":"/Users/.../lica-bench/data/gdb-dataset/.../component_018_gt.png" }layout-8similarly storesimage,mask,reference_assetas absolute hostpaths into the identity-evaluation assets.
This makes the benchmark non-portable for any evaluator that runs
evaluate()on a machine or container that doesn't have the original dataset mounted at the
exact same absolute path. The verifier silently degrades (e.g.
evaluated_samples=0and
nanscores) instead of failing loudly.Impact
assets into the container, but there's no canonical way to discover which
assets are referenced (every benchmark declares its own GT shape, and
references can be nested arbitrarily under
ground_truth).the serialized ground-truth dict, detect strings that look like file paths,
copy them somewhere, and rewrite the paths — a fragile convention.
(
evaluated_samples = 0,ocr_accuracy = nan) if the paths don't resolve.Proposed fix (any of these, roughly in order of preference)
a
dataset_rootthat the caller provides, and resolve them inBaseBenchmark.evaluate()/ sample loaders.sample.assets: dict[str, Path](orAssetRef) so harnesses can enumerate GT-referenced files programmaticallywithout heuristic path-sniffing.
load_samples, attach asset bytes(or content hashes) directly to the sample so evaluation doesn't need
filesystem access to the original dataset layout.
Workaround used downstream
The Harbor GDB adapter now walks the serialized
ground_truth, copies anystring value that resolves to an existing absolute file into
workspace/gt/<deterministic-name>, and rewrites the path to/workspace/gt/<name>(the container-relative path inside the Harbor task).This is a pragmatic shim but is fragile: it relies on "any absolute path string
that happens to point at an existing file must be an asset reference."
Repro