`create_sagemaker_execution_role` builds a role that the SDK's own validator rejects (missing `s3:PutObject`)

[Juan-Dream]

Summary

create_sagemaker_execution_role() creates an execution role without s3:PutObject, but Validator.validate() requires it. A role created entirely by this SDK therefore fails validation on the very next call, before any training job is submitted.

This makes the documented happy path (create role with the SDK, then train) fail out of the box.

Version

• amzn-nova-forge 1.4.9 (also present on main as of today)
• sagemaker pulled in as a transitive dependency
• Python 3.12.3, Linux, region us-east-1
• Model: NOVA_LITE, method: SFT_LORA, platform: SMTJ (SMTJRuntimeManager)

Steps to reproduce

import boto3
from amzn_nova_forge.iam.iam_role_creator import create_sagemaker_execution_role
from amzn_nova_forge import (
    ForgeConfig, ForgeTrainer, Model, SMTJRuntimeManager, TrainingMethod,
)

iam = boto3.client("iam")
ROLE = "NovaSageMakerRole"
BUCKET = "my-bucket"

# 1. Create the execution role using this SDK
create_sagemaker_execution_role(
    iam_client=iam,
    role_name=ROLE,
    s3_resource=BUCKET,
    region="us-east-1",
)

# 2. Use that same role to validate a training job
runtime = SMTJRuntimeManager(
    instance_type="ml.g5.12xlarge",
    instance_count=1,
    execution_role=f"arn:aws:iam::<account>:role/{ROLE}",
)
trainer = ForgeTrainer(
    model=Model.NOVA_LITE,
    method=TrainingMethod.SFT_LORA,
    infra=runtime,
    training_data_s3_path=f"s3://{BUCKET}/train/train.jsonl",
    holdout_data_s3_path=f"s3://{BUCKET}/validation/validation.jsonl",
    config=ForgeConfig(output_s3_path=f"s3://{BUCKET}/output/"),
    region="us-east-1",
)
trainer.train(job_name="repro", dry_run=True)

Expected

Validation passes. The role was created by this SDK, for this purpose, one step earlier.

Actual

ValueError: Execution role missing required permissions: s3:PutObject

Traceback ends at validation/validator.py:1297, reached from
recipe/recipe_builder.py:935 → trainer/forge_trainer.py:375.

Root cause

The two sides disagree on the required S3 actions.

src/amzn_nova_forge/validation/validator.py:577-581 requires three:

required_execution_role_permissions = [
    "s3:GetObject",
    "s3:PutObject",
    "s3:ListBucket",
]

src/amzn_nova_forge/iam/sagemaker_policies.json, key s3_read_policy, grants two:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": "S3_BUCKET_PLACEHOLDER"
}

create_sagemaker_execution_role (iam/iam_role_creator.py:287-297) attaches nine policies, and s3_read_policy is the only one touching S3. No other policy in the file grants s3:PutObject, so the created role can never satisfy the validator.

Worth noting the comment at iam_role_creator.py:274 says "S3 resources needed are the escrow bucket and the training output bucket". The intent to write to an output bucket is already there, the write action is just missing from the policy.

Suggested fix

Either add the action to the existing statement:

"s3_read_policy": {
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket", "s3:PutObject"],
    "Resource": "S3_BUCKET_PLACEHOLDER"
  }]
}

or, if read and write should stay scoped separately, add an s3_write_policy block granting s3:PutObject on the output prefix and append "s3_write_policy" to the list in create_sagemaker_execution_role. The second option keeps the _read_ name honest and lets the write scope be narrower than the read scope.

tests/unit/iam/test_iam_role_creator.py would be a good place for a regression test asserting that the policies produced by create_sagemaker_execution_role satisfy required_execution_role_permissions from the validator, so the two lists cannot drift again.

Workaround

Attach the missing action as an inline policy after calling the SDK's role creator:

import json

iam.put_role_policy(
    RoleName=ROLE,
    PolicyName="NovaSageMakerS3Write",
    PolicyDocument=json.dumps({
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
            "Resource": [
                f"arn:aws:s3:::{BUCKET}",
                f"arn:aws:s3:::{BUCKET}/*",
                f"arn:aws:s3:::customer-escrow-{ACCOUNT_ID}*",
                f"arn:aws:s3:::customer-escrow-{ACCOUNT_ID}*/*",
            ],
        }],
    }),
)

put_role_policy overwrites, so this is safe to re-run.