Data Science Template: implement expect_json_output test function

[ArthurCRodrigues]

Parent Issue

Part of #338 — Feature: Data Science Template (data_science)

---

Overview

Implement the expect_json_output test function that executes the student's program and validates a generated JSON file against expected structure (required keys) and values (with tolerance for numerics). This test is designed for assignments where students export evaluation results, configuration, or predictions as JSON.

---

Interface

class ExpectJsonOutputTest(BaseExecutionTest):
    @property
    def name(self):
        return "expect_json_output"

---

Parameters

Parameter	Type	Required	Description
program_command	string/dict	✓	Command to execute. Supports multi-language dict and "CMD" auto-resolution.
artifact_path	string	✓	Relative path to the generated JSON file (e.g., metrics.json).
expected_keys	list[string]	✗	Required top-level keys that must exist in the JSON object.
expected_values	dict	✗	Key → expected value pairs. Supports nested dot notation (e.g., "model.accuracy": 0.92).
tolerance	float	✗	Numeric tolerance for value comparisons (default: 0.001).
inputs	list[string]	✗	Stdin inputs if the program requires them.

---

Scoring Logic

Proportional scoring based on the number of satisfied checks:

1. File existence — If artifact not found, score 0.
2. JSON parsing — If file is not valid JSON, score 0.
3. Key check (if expected_keys specified):
   • Score = (found_keys / total_expected_keys) × weight of this check.
4. Value check (if expected_values specified):
   • For each key-value pair:
     • Resolve nested keys via dot notation (e.g., "model.accuracy" → json_obj["model"]["accuracy"]).
     • If key path doesn't exist: that check fails.
     • Numeric values: abs(actual - expected) <= tolerance → pass.
     • String/bool/null values: exact equality → pass.
     • Lists/dicts: deep equality (with numeric tolerance applied recursively for numbers).
   • Score = (passed_checks / total_checks) × weight.

Final score: weighted average of active checks. If all pass → 100.

---

Execution Flow

1. Run program_command in the sandbox via run_sandbox_execution.
2. Check for base errors (timeout, compilation, runtime) → score 0.
3. Extract file at /app/{artifact_path} using sandbox.extract_file().
4. Parse content with json.loads().
5. If expected_keys specified, verify each key exists at top level.
6. If expected_values specified, resolve each key path and compare values.
7. Compute proportional score and generate report.

---

Dot Notation for Nested Keys

The expected_values parameter supports dot notation to access nested JSON structures:

// Student output (metrics.json):
{
  "model": {
    "name": "RandomForest",
    "accuracy": 0.92,
    "hyperparams": {
      "n_estimators": 100
    }
  },
  "dataset_size": 1500
}

// Test configuration:
{
  "expected_values": {
    "model.accuracy": 0.92,
    "model.hyperparams.n_estimators": 100,
    "dataset_size": 1500
  }
}

Resolution: split key by ., traverse the JSON object. If any intermediate key is missing, that check fails.

---

Error Handling

Scenario	Result
Program times out	Score 0, report timeout
Program crashes	Score 0, report runtime error
File not found after execution	Score 0, report "file not generated"
File is not valid JSON	Score 0, report "invalid JSON" with parse error
Missing expected key	That key's check fails, partial score
Nested key path doesn't resolve	That check fails, partial score
Value mismatch	That check fails, report expected vs actual

---

Example Usage

Validate model evaluation output

{
  "name": "expect_json_output",
  "parameters": {
    "program_command": "python3 evaluate.py",
    "artifact_path": "metrics.json",
    "expected_keys": ["accuracy", "precision", "recall", "f1_score"],
    "expected_values": {
      "accuracy": 0.92,
      "precision": 0.89
    },
    "tolerance": 0.05
  },
  "weight": 30
}

Validate configuration export

{
  "name": "expect_json_output",
  "parameters": {
    "program_command": "python3 pipeline.py export-config",
    "artifact_path": "config.json",
    "expected_keys": ["model_type", "features", "target"],
    "expected_values": {
      "model_type": "logistic_regression",
      "features": ["age", "income", "score"]
    }
  },
  "weight": 20
}

---

Implementation Notes

• Inherit from BaseExecutionTest to reuse sandbox execution and error checking.
• Use sandbox.extract_file(f"/app/{artifact_path}") to get file content.
• Parse with json.loads() from stdlib — no external dependencies.
• Dot notation resolver: split key by ., iterate through dict.get() calls. Return None (or a sentinel) if path is broken.
• For numeric tolerance in deep comparison: check if both actual and expected are int or float, then apply tolerance; otherwise use ==.
• Validate artifact_path for path traversal (same pattern as expect_csv_output).
• All user-facing messages must go through the t() translation function.

---

Translation Keys Needed

• data_science.expect_json_output.description
• data_science.expect_json_output.report.success
• data_science.expect_json_output.report.file_not_found
• data_science.expect_json_output.report.invalid_json
• data_science.expect_json_output.report.missing_keys
• data_science.expect_json_output.report.value_mismatch
• data_science.expect_json_output.report.partial_success
• data_science.expect_json_output.report.key_path_not_found

---

Acceptance Criteria

• Test function registered in DataScienceTemplate.tests dict with key "expect_json_output"
• Passes when JSON artifact contains all expected keys and values match
• Fails with informative message when file is not generated
• Fails with informative message when JSON is malformed
• Dot notation resolves nested keys correctly
• Partial scoring works when some keys/values match and others don't
• Tolerance-based numeric comparison works for float values
• Deep equality works for lists, dicts, strings, booleans, null
• Path traversal in artifact_path is rejected
• Unit tests cover all success/failure scenarios
• Translation keys added for pt-BR and en