Skip to content

Data Science Template: implement expect_model_artifact test function #343

Description

@ArthurCRodrigues

Parent Issue

Part of #338 — Feature: Data Science Template (data_science)


Overview

Implement the expect_model_artifact test function that executes the student's program and validates that a model file was produced in the sandbox. This test confirms that the training pipeline ran to completion and serialized a model artifact. It optionally validates minimum file size to ensure the model is non-trivial (not an empty file or a placeholder).


Interface

class ExpectModelArtifactTest(BaseExecutionTest):
    @property
    def name(self):
        return "expect_model_artifact"

Parameters

Parameter Type Required Description
program_command string/dict Command to execute. Supports multi-language dict and "CMD" auto-resolution.
artifact_path string Relative path to the expected model file (e.g., model.pkl, model.joblib, model.h5).
min_size_bytes int Minimum file size in bytes (default: 0 — only check existence). Ensures the model is non-trivial.
inputs list[string] Stdin inputs if the program requires them.

Scoring Logic

Binary scoring:

  • 100 if the file exists AND its size is >= min_size_bytes.
  • 0 if the file doesn't exist, or its size is below the minimum.

Execution Flow

  1. Run program_command in the sandbox via run_sandbox_execution.
  2. Check for base errors (timeout, compilation, runtime) → score 0.
  3. Attempt to extract file at /app/{artifact_path} using sandbox.extract_file().
  4. If file not found → score 0 with "model file not generated" message.
  5. Check file size: len(extracted.content).
  6. If min_size_bytes > 0 and file size < min_size_bytes → score 0 with "model file too small" message.
  7. Otherwise → score 100.

Error Handling

Scenario Result
Program times out Score 0, report timeout
Program crashes Score 0, report runtime error
File not found after execution Score 0, report "model file 'X' was not generated"
File exists but size < min_size_bytes Score 0, report "model file is {actual} bytes, minimum required is {min_size_bytes} bytes"
Path traversal in artifact_path Score 0, report invalid path

Example Usage

Basic existence check

{
  "name": "expect_model_artifact",
  "parameters": {
    "program_command": "python3 train.py",
    "artifact_path": "model.pkl"
  },
  "weight": 10
}

With minimum size validation

{
  "name": "expect_model_artifact",
  "parameters": {
    "program_command": "python3 train.py",
    "artifact_path": "model.pkl",
    "min_size_bytes": 2048
  },
  "weight": 15
}

Joblib model format

{
  "name": "expect_model_artifact",
  "parameters": {
    "program_command": "python3 pipeline.py",
    "artifact_path": "models/classifier.joblib",
    "min_size_bytes": 4096
  },
  "weight": 20
}

Design Rationale

This test intentionally does NOT:

  • Validate model correctness — That's the job of expect_metric (check accuracy/loss).
  • Load or deserialize the model — That would require pickle/joblib in the autograder, introducing security risks (arbitrary code execution via pickle).
  • Check model format/type — The test is format-agnostic (.pkl, .joblib, .h5, .onnx, etc.).

The test's sole purpose is confirming that the training pipeline completed successfully and produced a serialized artifact of reasonable size. Combined with expect_metric, it provides complete coverage: "did training succeed?" (model_artifact) + "is the model good?" (metric).


Implementation Notes

  • Inherit from BaseExecutionTest to reuse sandbox execution and error checking.
  • Use sandbox.extract_file(f"/app/{artifact_path}") to get file content as bytes.
  • File size = len(extracted.content) (the content is already in memory as bytes from extraction).
  • Validate artifact_path for path traversal (reject .. and absolute paths).
  • min_size_bytes defaults to 0 if not provided (any non-empty file passes).
  • All user-facing messages must go through the t() translation function.
  • This is the simplest test function in the template — good candidate for first implementation during development.

Translation Keys Needed

  • data_science.expect_model_artifact.description
  • data_science.expect_model_artifact.report.success
  • data_science.expect_model_artifact.report.file_not_found
  • data_science.expect_model_artifact.report.file_too_small
  • data_science.expect_model_artifact.report.invalid_path

Acceptance Criteria

  • Test function registered in DataScienceTemplate.tests dict with key "expect_model_artifact"
  • Passes when file exists and size >= min_size_bytes
  • Passes when file exists and min_size_bytes is not specified (existence-only check)
  • Fails with informative message when file is not generated
  • Fails with informative message when file is smaller than min_size_bytes
  • Path traversal in artifact_path is rejected
  • Works with any file extension (.pkl, .joblib, .h5, .onnx, etc.)
  • Unit tests cover existence pass, existence fail, size pass, size fail
  • Translation keys added for pt-BR and en

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions