chore: datadog updates - infra:upgrade, remove datadog provider, rename remote state refs

NOTEBOOK_MIGRATION.md

@@ -0,0 +1,141 @@
+# SageMaker Notebook Platform Migration Guide
+
+## Problem
+You're getting this error when trying to start your SageMaker notebook instances:
+
+```
+ValidationException: Platform identifier (notebook-al2-v2) is not supported for this service. 
+Please self migrate by updating the notebook settings
+```
+
+This happens because `notebook-al2-v2` (with JupyterLab 3) was deprecated as of June 30, 2025.
+
+## Solutions
+
+### Option 1: Using the ML2P CLI (New Command)
+
+I've added an `update` command to ML2P. After pulling these changes:
+
+```bash
+# Stop, update, and start a notebook instance
+ml2p notebook stop <notebook-name>
+
+# Wait for it to stop (check with describe)
+ml2p notebook describe <notebook-name>
+
+# Update to the new platform
+ml2p notebook update <notebook-name> --platform-identifier notebook-al2023-v1
+
+# Start it again
+ml2p notebook start <notebook-name>
+```
+
+**Example:**
+```bash
+ml2p notebook stop my-analysis-notebook
+ml2p notebook update my-analysis-notebook
+ml2p notebook start my-analysis-notebook
+```
+
+### Option 2: Using the Migration Script
+
+I've created a standalone Python script that handles the entire process:
+
+```bash
+python migrate_notebooks.py <notebook-1> <notebook-2>
+```
+
+The script will:
+1. Check the current status of each notebook
+2. Stop the notebook if it's running
+3. Update the platform identifier to `notebook-al2023-v1`
+4. Ask if you want to start it immediately
+
+**Example:**
+```bash
+python migrate_notebooks.py my-notebook-1 my-notebook-2
+```
+
+### Option 3: AWS CLI Commands
+
+If you prefer using AWS CLI directly:
+
+```bash
+# For each notebook instance:
+NOTEBOOK_NAME="your-notebook-name"
+
+# Stop the notebook
+aws sagemaker stop-notebook-instance --notebook-instance-name $NOTEBOOK_NAME
+
+# Wait for it to stop
+aws sagemaker wait notebook-instance-stopped --notebook-instance-name $NOTEBOOK_NAME
+
+# Update the platform identifier
+aws sagemaker update-notebook-instance \
+  --notebook-instance-name $NOTEBOOK_NAME \
+  --platform-identifier notebook-al2023-v1
+
+# Start it again
+aws sagemaker start-notebook-instance --notebook-instance-name $NOTEBOOK_NAME
+```
+
+### Option 4: Python/Boto3 Script
+
+```python
+import boto3
+
+client = boto3.client('sagemaker')
+notebook_name = 'your-notebook-name'
+
+# Stop
+client.stop_notebook_instance(NotebookInstanceName=notebook_name)
+waiter = client.get_waiter('notebook_instance_stopped')
+waiter.wait(NotebookInstanceName=notebook_name)
+
+# Update
+client.update_notebook_instance(
+    NotebookInstanceName=notebook_name,
+    PlatformIdentifier='notebook-al2023-v1'
+)
+
+# Start
+client.start_notebook_instance(NotebookInstanceName=notebook_name)
+```
+
+## Important Notes
+
+1. **Data Preservation**: Your notebook files and data are stored on an EBS volume that persists through the platform update. Your data will NOT be lost.
+
+2. **Must Stop First**: You cannot update a running notebook instance - it must be in the "Stopped" state.
+
+3. **Recommended Platform**: Use `notebook-al2023-v1` (Amazon Linux 2023 with JupyterLab 4) as it's the latest recommended version.
+
+4. **For ML2P Projects**: If you manage notebooks via ml2p.yml, you can now add:
+   ```yaml
+   notebook:
+     instance_type: "ml.t2.medium"
+     volume_size: 8
+     platform_identifier: "notebook-al2023-v1"  # Add this line
+   ```
+
+## Available Platform Identifiers
+
+- `notebook-al2023-v1` - Amazon Linux 2023 + JupyterLab 4 (recommended)
+- `notebook-al2-v3` - Amazon Linux 2 + JupyterLab 3 (if available in your region)
+
+## What Changed in ML2P
+
+1. **New CLI Command**: `ml2p notebook update` to update existing notebook instances
+2. **Updated Notebook Creation**: `mk_notebook()` now defaults to `notebook-al2023-v1` for new notebooks
+3. **Config Support**: You can specify `platform_identifier` in your ml2p.yml file
+
+## Troubleshooting
+
+**Q: Can't access my notebook to get my data?**  
+A: Your data is safe on the EBS volume. Just update the platform identifier and you'll be able to access everything.
+
+**Q: Update fails with "ResourceNotFound"?**  
+A: Make sure you're using the correct notebook instance name. For ML2P projects, the full name is `{project-name}-{notebook-name}`.
+
+**Q: Getting "ResourceInUse" error?**  
+A: The notebook must be stopped first. Wait for it to fully stop before updating.

migrate_notebooks.py

@@ -0,0 +1,181 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Script to migrate SageMaker notebook instances to new platform identifier.
+
+This script helps migrate notebook instances from deprecated platform identifiers
+(like notebook-al2-v2) to the latest supported version (notebook-al2023-v1).
+
+Usage:
+    python migrate_notebooks.py [--profile PROFILE] <notebook-instance-name-1> [<notebook-instance-name-2> ...]
+"""
+
+import argparse
+import sys
+import boto3
+from botocore.exceptions import ClientError
+
+
+# Default SageMaker notebook platform identifier
+# Update this when AWS releases newer platform versions
+# See: https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-al2.html
+DEFAULT_NOTEBOOK_PLATFORM = "notebook-al2023-v1"
+
+
+def migrate_notebook(client, notebook_name, new_platform=None):
+    """
+    Migrate a SageMaker notebook instance to a new platform identifier.
+
+    Args:
+        client: boto3 SageMaker client
+        notebook_name: Name of the notebook instance
+        new_platform: Target platform identifier (default: from DEFAULT_NOTEBOOK_PLATFORM)
+    """
+    # Use default if not specified
+    if new_platform is None:
+        new_platform = DEFAULT_NOTEBOOK_PLATFORM
+
+    print(f"\n{'='*60}")
+    print(f"Processing notebook: {notebook_name}")
+    print('='*60)
+
+    try:
+        # Get current notebook status
+        response = client.describe_notebook_instance(
+            NotebookInstanceName=notebook_name
+        )
+        current_status = response['NotebookInstanceStatus']
+        current_platform = response.get('PlatformIdentifier', 'unknown')
+
+        print(f"Current status: {current_status}")
+        print(f"Current platform: {current_platform}")
+
+        # Handle notebook state
+        if current_status == 'InService':
+            print(f"Stopping notebook instance...")
+            client.stop_notebook_instance(NotebookInstanceName=notebook_name)
+
+            # Wait for it to stop
+            print("Waiting for notebook to stop...", end='', flush=True)
+            waiter = client.get_waiter('notebook_instance_stopped')
+            waiter.wait(
+                NotebookInstanceName=notebook_name,
+                WaiterConfig={'Delay': 10, 'MaxAttempts': 60}
+            )
+            print(" STOPPED")
+        elif current_status == 'Stopped':
+            print("Notebook is already stopped")
+        elif current_status == 'Stopping':
+            print("Notebook is currently stopping, waiting...")
+            waiter = client.get_waiter('notebook_instance_stopped')
+            waiter.wait(
+                NotebookInstanceName=notebook_name,
+                WaiterConfig={'Delay': 10, 'MaxAttempts': 60}
+            )
+            print(" STOPPED")
+        else:
+            print(f"ERROR: Notebook is in '{current_status}' state. Cannot proceed.")
+            print("Please wait for the notebook to finish its current operation.")
+            return False
+
+        # Update the platform identifier
+        print(f"Updating platform identifier to {new_platform}...")
+        client.update_notebook_instance(
+            NotebookInstanceName=notebook_name,
+            PlatformIdentifier=new_platform
+        )
+        print(f"✓ Successfully updated to {new_platform}")
+
+        # Optionally start the notebook
+        response = input(f"Start notebook instance now? [y/N]: ")
+        if response.lower() == 'y':
+            print("Starting notebook instance...")
+            client.start_notebook_instance(NotebookInstanceName=notebook_name)
+            print(f"✓ Notebook {notebook_name} is starting")
+        else:
+            print(f"Notebook {notebook_name} remains stopped. Start it manually when ready.")
+
+        return True
+
+    except ClientError as e:
+        error_code = e.response['Error']['Code']
+        error_msg = e.response['Error']['Message']
+        print(f"ERROR: {error_code} - {error_msg}")
+        return False
+    except Exception as e:
+        print(f"ERROR: {str(e)}")
+        return False
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Migrate SageMaker notebook instances to new platform identifier",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Migrate using default AWS profile
+  python migrate_notebooks.py my-notebook-1 my-notebook-2
+
+  # Migrate using specific AWS profile
+  python migrate_notebooks.py --profile production my-notebook-1
+
+  # Migrate to specific platform version
+  python migrate_notebooks.py --platform notebook-al2-v3 my-notebook-1
+"""
+    )
+    parser.add_argument(
+        'notebook_names',
+        nargs='+',
+        metavar='NOTEBOOK_NAME',
+        help='Name(s) of notebook instance(s) to migrate'
+    )
+    parser.add_argument(
+        '--profile',
+        default=None,
+        help='AWS profile to use (default: uses AWS_PROFILE env var or default profile)'
+    )
+    parser.add_argument(
+        '--platform',
+        default=None,
+        help=f'Target platform identifier (default: {DEFAULT_NOTEBOOK_PLATFORM})'
+    )
+
+    args = parser.parse_args()
+
+    print("SageMaker Notebook Migration Tool")
+    print(f"Migrating {len(args.notebook_names)} notebook instance(s) to {args.platform or DEFAULT_NOTEBOOK_PLATFORM}")
+
+    # Initialize boto3 client with optional profile
+    try:
+        if args.profile:
+            print(f"Using AWS profile: {args.profile}")
+            session = boto3.Session(profile_name=args.profile)
+            client = session.client('sagemaker')
+        else:
+            client = boto3.client('sagemaker')
+        print("✓ Connected to AWS SageMaker")
+    except Exception as e:
+        print(f"ERROR: Failed to connect to AWS: {e}")
+        sys.exit(1)
+
+    # Process each notebook
+    results = {}
+    for notebook_name in args.notebook_names:
+        success = migrate_notebook(client, notebook_name, args.platform)
+        results[notebook_name] = success
+
+    # Summary
+    print(f"\n{'='*60}")
+    print("MIGRATION SUMMARY")
+    print('='*60)
+    for notebook_name, success in results.items():
+        status = "✓ SUCCESS" if success else "✗ FAILED"
+        print(f"{notebook_name}: {status}")
+
+    successful = sum(1 for s in results.values() if s)
+    print(f"\nTotal: {successful}/{len(args.notebook_names)} successful")
+
+
+if __name__ == "__main__":
+    main()