Skip to content

GroundTruth references host-absolute asset paths (breaks containerized / off-host evaluation) #4

Description

@mohitgargai

Problem

GDBSample.ground_truth stores references to input/mask/reference image assets
as host-absolute filesystem paths pointing into data/gdb-dataset/…. For
example, typography-7 samples (evaluation_mode = inpaint_reconstruction)
have:

{
  "input_image":       "/Users/.../lica-bench/data/gdb-dataset/.../component_018_input.png",
  "mask":              "/Users/.../lica-bench/data/gdb-dataset/.../component_018_mask.png",
  "ground_truth_image":"/Users/.../lica-bench/data/gdb-dataset/.../component_018_gt.png"
}

layout-8 similarly stores image, mask, reference_asset as absolute host
paths into the identity-evaluation assets.

This makes the benchmark non-portable for any evaluator that runs evaluate()
on a machine or container that doesn't have the original dataset mounted at the
exact same absolute path. The verifier silently degrades (e.g. evaluated_samples=0
and nan scores) instead of failing loudly.

Impact

  • Containerized evaluation: Docker / Daytona / Modal runs need to ship GT
    assets into the container, but there's no canonical way to discover which
    assets are referenced (every benchmark declares its own GT shape, and
    references can be nested arbitrarily under ground_truth).
  • Third-party harnesses (e.g. Harbor adapter, external evaluators) must walk
    the serialized ground-truth dict, detect strings that look like file paths,
    copy them somewhere, and rewrite the paths — a fragile convention.
  • Parity runs on a different host silently produce wrong numbers
    (evaluated_samples = 0, ocr_accuracy = nan) if the paths don't resolve.

Proposed fix (any of these, roughly in order of preference)

  1. Relative paths + explicit base dir: store references as paths relative to
    a dataset_root that the caller provides, and resolve them in
    BaseBenchmark.evaluate() / sample loaders.
  2. Asset manifest: expose a typed sample.assets: dict[str, Path] (or
    AssetRef) so harnesses can enumerate GT-referenced files programmatically
    without heuristic path-sniffing.
  3. Bundle GT files with sample: during load_samples, attach asset bytes
    (or content hashes) directly to the sample so evaluation doesn't need
    filesystem access to the original dataset layout.

Workaround used downstream

The Harbor GDB adapter now walks the serialized ground_truth, copies any
string value that resolves to an existing absolute file into
workspace/gt/<deterministic-name>, and rewrites the path to
/workspace/gt/<name> (the container-relative path inside the Harbor task).
This is a pragmatic shim but is fragile: it relies on "any absolute path string
that happens to point at an existing file must be an asset reference."

Repro

bench = Typography7(...)  # evaluation_mode = inpaint_reconstruction
sample = bench.samples[0]
gt = sample.ground_truth
# all three are host-absolute paths:
assert Path(gt.input_image).is_absolute()
assert Path(gt.mask).is_absolute()
assert Path(gt.ground_truth_image).is_absolute()
# -> fails to evaluate on any machine that doesn't mirror the host's data/ layout

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions