Parent Issue
Part of #338 — Feature: Data Science Template (data_science)
Overview
Implement the expect_model_artifact test function that executes the student's program and validates that a model file was produced in the sandbox. This test confirms that the training pipeline ran to completion and serialized a model artifact. It optionally validates minimum file size to ensure the model is non-trivial (not an empty file or a placeholder).
Interface
class ExpectModelArtifactTest(BaseExecutionTest):
@property
def name(self):
return "expect_model_artifact"
Parameters
| Parameter |
Type |
Required |
Description |
program_command |
string/dict |
✓ |
Command to execute. Supports multi-language dict and "CMD" auto-resolution. |
artifact_path |
string |
✓ |
Relative path to the expected model file (e.g., model.pkl, model.joblib, model.h5). |
min_size_bytes |
int |
✗ |
Minimum file size in bytes (default: 0 — only check existence). Ensures the model is non-trivial. |
inputs |
list[string] |
✗ |
Stdin inputs if the program requires them. |
Scoring Logic
Binary scoring:
- 100 if the file exists AND its size is >=
min_size_bytes.
- 0 if the file doesn't exist, or its size is below the minimum.
Execution Flow
- Run
program_command in the sandbox via run_sandbox_execution.
- Check for base errors (timeout, compilation, runtime) → score 0.
- Attempt to extract file at
/app/{artifact_path} using sandbox.extract_file().
- If file not found → score 0 with "model file not generated" message.
- Check file size:
len(extracted.content).
- If
min_size_bytes > 0 and file size < min_size_bytes → score 0 with "model file too small" message.
- Otherwise → score 100.
Error Handling
| Scenario |
Result |
| Program times out |
Score 0, report timeout |
| Program crashes |
Score 0, report runtime error |
| File not found after execution |
Score 0, report "model file 'X' was not generated" |
| File exists but size < min_size_bytes |
Score 0, report "model file is {actual} bytes, minimum required is {min_size_bytes} bytes" |
| Path traversal in artifact_path |
Score 0, report invalid path |
Example Usage
Basic existence check
{
"name": "expect_model_artifact",
"parameters": {
"program_command": "python3 train.py",
"artifact_path": "model.pkl"
},
"weight": 10
}
With minimum size validation
{
"name": "expect_model_artifact",
"parameters": {
"program_command": "python3 train.py",
"artifact_path": "model.pkl",
"min_size_bytes": 2048
},
"weight": 15
}
Joblib model format
{
"name": "expect_model_artifact",
"parameters": {
"program_command": "python3 pipeline.py",
"artifact_path": "models/classifier.joblib",
"min_size_bytes": 4096
},
"weight": 20
}
Design Rationale
This test intentionally does NOT:
- Validate model correctness — That's the job of
expect_metric (check accuracy/loss).
- Load or deserialize the model — That would require pickle/joblib in the autograder, introducing security risks (arbitrary code execution via pickle).
- Check model format/type — The test is format-agnostic (
.pkl, .joblib, .h5, .onnx, etc.).
The test's sole purpose is confirming that the training pipeline completed successfully and produced a serialized artifact of reasonable size. Combined with expect_metric, it provides complete coverage: "did training succeed?" (model_artifact) + "is the model good?" (metric).
Implementation Notes
- Inherit from
BaseExecutionTest to reuse sandbox execution and error checking.
- Use
sandbox.extract_file(f"/app/{artifact_path}") to get file content as bytes.
- File size =
len(extracted.content) (the content is already in memory as bytes from extraction).
- Validate
artifact_path for path traversal (reject .. and absolute paths).
min_size_bytes defaults to 0 if not provided (any non-empty file passes).
- All user-facing messages must go through the
t() translation function.
- This is the simplest test function in the template — good candidate for first implementation during development.
Translation Keys Needed
data_science.expect_model_artifact.description
data_science.expect_model_artifact.report.success
data_science.expect_model_artifact.report.file_not_found
data_science.expect_model_artifact.report.file_too_small
data_science.expect_model_artifact.report.invalid_path
Acceptance Criteria
Parent Issue
Part of #338 — Feature: Data Science Template (
data_science)Overview
Implement the
expect_model_artifacttest function that executes the student's program and validates that a model file was produced in the sandbox. This test confirms that the training pipeline ran to completion and serialized a model artifact. It optionally validates minimum file size to ensure the model is non-trivial (not an empty file or a placeholder).Interface
Parameters
program_command"CMD"auto-resolution.artifact_pathmodel.pkl,model.joblib,model.h5).min_size_bytes0— only check existence). Ensures the model is non-trivial.inputsScoring Logic
Binary scoring:
min_size_bytes.Execution Flow
program_commandin the sandbox viarun_sandbox_execution./app/{artifact_path}usingsandbox.extract_file().len(extracted.content).min_size_bytes > 0and file size <min_size_bytes→ score 0 with "model file too small" message.Error Handling
Example Usage
Basic existence check
{ "name": "expect_model_artifact", "parameters": { "program_command": "python3 train.py", "artifact_path": "model.pkl" }, "weight": 10 }With minimum size validation
{ "name": "expect_model_artifact", "parameters": { "program_command": "python3 train.py", "artifact_path": "model.pkl", "min_size_bytes": 2048 }, "weight": 15 }Joblib model format
{ "name": "expect_model_artifact", "parameters": { "program_command": "python3 pipeline.py", "artifact_path": "models/classifier.joblib", "min_size_bytes": 4096 }, "weight": 20 }Design Rationale
This test intentionally does NOT:
expect_metric(check accuracy/loss)..pkl,.joblib,.h5,.onnx, etc.).The test's sole purpose is confirming that the training pipeline completed successfully and produced a serialized artifact of reasonable size. Combined with
expect_metric, it provides complete coverage: "did training succeed?" (model_artifact) + "is the model good?" (metric).Implementation Notes
BaseExecutionTestto reuse sandbox execution and error checking.sandbox.extract_file(f"/app/{artifact_path}")to get file content as bytes.len(extracted.content)(the content is already in memory as bytes from extraction).artifact_pathfor path traversal (reject..and absolute paths).min_size_bytesdefaults to 0 if not provided (any non-empty file passes).t()translation function.Translation Keys Needed
data_science.expect_model_artifact.descriptiondata_science.expect_model_artifact.report.successdata_science.expect_model_artifact.report.file_not_founddata_science.expect_model_artifact.report.file_too_smalldata_science.expect_model_artifact.report.invalid_pathAcceptance Criteria
DataScienceTemplate.testsdict with key"expect_model_artifact"artifact_pathis rejected