JMES Path results in "Failed to fetch secrets"

[Jufik]

Describe the bug

When using AWS Secrets Manager with JMESPath queries in a SecretProviderClass, if a JMESPath path does not point to a valid object, the error message is misleading and unhelpful.

The error JMES Path - $KEY for object alias - $MAPPED_KEY does not point to a valid object. appears in the CSI driver logs, but the pod status shows a generic error:

Failed to fetch secret from all regions. Verify secret exists and required permissions are granted for: [your secret name]

This makes debugging difficult because:

1. The actual JMESPath error is only visible in CSI driver logs, not in pod events
2. The pod error message suggests a permissions or secret existence issue, which may not be the root cause
3. Users waste time checking IAM permissions and secret names instead of fixing the JMESPath configuration

To Reproduce

Ensure you have all the correct IRSA/Policy bindings set up correctly.

Steps to reproduce the behavior:

1. Create a secret in AWS Secrets Manager with the following structure:

   my-secret-key: {"HELLO":"WORLD"}
   

2. Create a SecretProviderClass with an invalid JMESPath query:

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: hello-secrets
     namespace: default
   spec:
     provider: aws
     parameters:
       region: eu-central-1
       objects: |
         - objectName: "my-secret-key"
           objectType: "secretsmanager"
           jmesPath:
             - path: HELLO
               objectAlias: hello-value
             - path: WORLD  # This path doesn't exist in the secret
               objectAlias: world-value
     secretObjects:
     - secretName: hello-secret
       type: Opaque
       data:
       - objectName: hello-value
         key: HELLO
       - objectName: world-value
         key: WORLD

3. Create a deployment that uses this SecretProviderClass:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hello
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: hello-develop
     template:
       metadata:
         labels:
           app: hello-develop
       spec:
         serviceAccountName: su-test
         containers:
         - name: app
           image: busybox:latest
           command: ["sleep", "infinity"]
           env:
           - name: HELLO
             valueFrom:
               secretKeyRef:
                 name: hello-secret
                 key: HELLO
           - name: WORLD
             valueFrom:
               secretKeyRef:
                 name: hello-secret
                 key: WORLD
           volumeMounts:
           - name: secrets-store
             mountPath: "/mnt/secrets-store"
             readOnly: true
         volumes:
         - name: secrets-store
           csi:
             driver: secrets-store.csi.k8s.io
             readOnly: true
             volumeAttributes:
               secretProviderClass: "hello-secrets"

4. Observe the pod status and events:

   • Pod status shows: Failed to fetch secret from all regions. Verify secret exists and required permissions are granted for: [your secret name]
   • Pod events do not show the JMESPath error
   • The actual JMESPath error is only visible in CSI driver logs

Expected behavior

The JMESPath error should be:

1. Displayed in the pod's events for easier debugging
2. More descriptive in the error message (e.g., "JMESPath query 'WORLD' does not exist in secret 'my-secret-key'")
3. Clearly distinguish JMESPath errors from permission/secret existence errors

Environment:

• CSI Secrets Store Driver version: v1.5.3

• AWS Secrets Store CSI Driver Provider version: v2.1.0

• I am able to reproduce this issue on the latest version of the CSI driver and AWS providers.

Additional context

This issue is specific to the AWS Secrets Manager provider. The error message format suggests the issue is with secret fetching/permissions, when in reality it's a JMESPath configuration problem. This leads to significant debugging time being spent on the wrong area (IAM policies, secret names) instead of the actual issue (JMESPath paths).

Workaround

Currently, users must check the CSI driver logs to see the actual JMESPath error:

kubectl logs -n kube-system -l app=csi-secrets-store-provider-aws