Enhance rejection audit events with Harvest API enrichment

packages/greenhouse/_dev/build/docs/README.md

@@ -56,6 +56,28 @@
    - **Batch Size**: Number of events per API request (100-500, default: 500)
    - **Performer IDs Filter**: Filter by specific user IDs
    - **Event Types Filter**: Filter by event type (data_change_update, data_change_create, data_change_destroy, harvest_access, action)
+   - **Enrich rejected application events**: When enabled, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event (default: disabled)
+
+### Enabling rejection enrichment (optional)
+
+When a candidate or prospect is rejected, Greenhouse audits it as an `action` event with `event.target_type` set to the literal string `Candidate or Prospect rejected` — this event does **not** include an application or candidate ID, and does not include the rejection reason or notes/comments entered at rejection time. Enabling **Enrich rejected application events** looks these up from the Harvest API and correlates them back to the audit event.
+
+1. On the same Harvest V3 (OAuth) API credential used for audit log access, grant the following read permissions (under **Configure > Dev Center > API Credentials** on the credential):
+   - **Rejection details** → *List rejection details*
+   - **Rejection reasons** → *List rejection reasons* and *Show rejection reasons*
+   - **Applications** → read access, needed for the `GET /v3/applications/{id}` lookup this integration performs
+   - **Activity Feed** → read access, needed for the `GET /v3/applications/{id}/activity_feed` lookup this integration performs to retrieve the rejection notes
+2. Enable the **Enrich rejected application events** setting on the integration.
+
+Because the "Candidate or Prospect rejected" audit event has no application ID, the integration correlates it to a second audit event that Greenhouse records for the same rejection — one with `event.target_type` of `RejectionDetails` — by matching the `request.id` shared by both events. That event's `event.meta.application_id` is then used to look up the rejection reason and notes:
+
+- **Exactly one matching `RejectionDetails` event**: the rejection reason and notes are fetched and added to the event under `greenhouse.audit.event.rejection` (`application_id`, `candidate_id`, `reason.id`/`reason.name`/`reason.type`, `notes`, `rejected_at`). The notes are also copied to `event.reason`.
+- **No matching event found**: the audit event is still indexed, with `greenhouse.audit.event.rejection.error` explaining that no matching `RejectionDetails` event was found for that request. This can happen if the two events land on different pages of a poll; increasing **Batch Size** reduces how often this occurs.
+- **More than one matching event** (for example, a bulk rejection of several candidates in one action, which shares a single `request.id` across all of them): the integration does not guess which pairs with which. The audit event is indexed with `greenhouse.audit.event.rejection.error` and `greenhouse.audit.event.rejection.ambiguous_application_ids` listing the candidates, for manual follow-up.
+
+Any of the error cases above also adds the tag `greenhouse-rejection-enrichment-failed`, and the underlying rejection audit event is never dropped.
+
+Because this issues up to two extra API requests per rejection on top of the audit log polling, be mindful of Greenhouse's rate limits (50 general requests per 10 seconds) if your organization rejects applications in bulk.
 
 ## Logs
 
@@ -93,6 +115,14 @@
 2. Check that there have been events in the last 30 days
 3. Review any filter settings that might be excluding events
 
+### Rejection Enrichment Errors
+
+If rejection events are tagged with `greenhouse-rejection-enrichment-failed` and `greenhouse.audit.event.rejection.error` is populated:
+1. Verify the OAuth credential has been granted **Rejection details** (*List rejection details*), **Rejection reasons** (*List rejection reasons*, *Show rejection reasons*), **Applications**, and **Activity Feed** read permissions
+2. Check that the authorizing user has permission to view the affected application
+3. If the error mentions no matching `RejectionDetails` event was found, try increasing **Batch Size** so the two related audit events are less likely to land on different pages
+4. If the error mentions an ambiguous match, the rejection was likely part of a bulk-reject action; `greenhouse.audit.event.rejection.ambiguous_application_ids` lists the candidates involved for manual follow-up
+
 ## Logs reference
 
 ### audit

packages/greenhouse/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.2.0"
+  changes:
+    - description: Add an optional "Enrich rejected application events" setting that, for a "Candidate or Prospect rejected" audit event, correlates it by timestamp to the matching Application via the Harvest API and adds the rejection reason and rejection notes/comments to the event as `greenhouse.audit.event.rejection`.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/17347
 - version: "0.1.0"
   changes:
     - description: Initial release.

packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json

@@ -14,6 +14,15 @@
         },
         {
             "message": "{\"request\":{\"id\":\"3456cID\",\"type\":\"jobs#destroy\"},\"performer\":{\"meta\":{\"name\":\"HR Manager\",\"username\":\"hr.manager@omniva-corp.com\"},\"id\":55555,\"ip_address\":\"172.16.0.100\",\"type\":\"user\"},\"organization_id\":456,\"event\":{\"meta\":{\"name\":[\"Software Engineer\",null]},\"target_type\":\"Job\",\"target_id\":789,\"type\":\"data_change_destroy\"},\"event_time\":\"2023-06-04T08:15:30.500Z\"}"
+        },
+        {
+            "message": "{\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"application_id\":\"987654\",\"candidate_id\":\"123123\",\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}"
+        },
+        {
+            "message": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous: 2 RejectionDetails events share request.id 7891eID (likely a bulk rejection)\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}"
+        },
+        {
+            "message": "{\"event\":{\"meta\":{\"application_id\":293344484002,\"id\":221864428002},\"target_id\":221864428002,\"target_type\":\"RejectionDetails\",\"type\":\"data_change_create\"},\"event_time\":\"2023-06-05T09:00:00.048Z\",\"organization_id\":123,\"performer\":{\"id\":22222,\"ip_address\":\"198.51.100.5\",\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"type\":\"user\"},\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"}}"
         }
     ]
 }

packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json

@@ -337,6 +337,209 @@
                 "full_name": "HR Manager",
                 "id": "55555"
             }
+        },
+        {
+            "@timestamp": "2023-06-05T09:00:00.000Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "action",
+                "category": [
+                    "iam"
+                ],
+                "id": "7890dID",
+                "kind": "event",
+                "original": "{\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"application_id\":\"987654\",\"candidate_id\":\"123123\",\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}",
+                "reason": "Candidate lacked required experience with distributed systems.",
+                "type": [
+                    "info"
+                ]
+            },
+            "greenhouse": {
+                "audit": {
+                    "event": {
+                        "rejection": {
+                            "application_id": "987654",
+                            "candidate_id": "123123",
+                            "notes": "Candidate lacked required experience with distributed systems.",
+                            "reason": {
+                                "id": "555",
+                                "name": "Not a fit",
+                                "type": "We rejected them"
+                            },
+                            "rejected_at": "2023-06-05T09:00:00.000Z"
+                        },
+                        "target_type": "Candidate or Prospect rejected",
+                        "type": "action"
+                    },
+                    "performer": {
+                        "meta": {},
+                        "type": "user"
+                    },
+                    "request": {
+                        "id": "7890dID",
+                        "type": "people#change_application_state"
+                    }
+                }
+            },
+            "organization": {
+                "id": "123"
+            },
+            "related": {
+                "ip": [
+                    "198.51.100.5"
+                ],
+                "user": [
+                    "22222",
+                    "recruiter.one@omniva-corp.com"
+                ]
+            },
+            "source": {
+                "ip": "198.51.100.5"
+            },
+            "tags": [
+                "preserve_original_event",
+                "forwarded",
+                "greenhouse-audit"
+            ],
+            "user": {
+                "email": "recruiter.one@omniva-corp.com",
+                "full_name": "Recruiter One",
+                "id": "22222"
+            }
+        },
+        {
+            "@timestamp": "2023-06-05T09:05:00.000Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "action",
+                "category": [
+                    "iam"
+                ],
+                "id": "7891eID",
+                "kind": "event",
+                "original": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous: 2 RejectionDetails events share request.id 7891eID (likely a bulk rejection)\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}",
+                "type": [
+                    "info"
+                ]
+            },
+            "greenhouse": {
+                "audit": {
+                    "event": {
+                        "rejection": {
+                            "ambiguous_application_ids": [
+                                "111111",
+                                "222222"
+                            ],
+                            "error": "ambiguous: 2 RejectionDetails events share request.id 7891eID (likely a bulk rejection)"
+                        },
+                        "target_type": "Candidate or Prospect rejected",
+                        "type": "action"
+                    },
+                    "performer": {
+                        "meta": {},
+                        "type": "user"
+                    },
+                    "request": {
+                        "id": "7891eID",
+                        "type": "people#change_application_state"
+                    }
+                }
+            },
+            "organization": {
+                "id": "123"
+            },
+            "related": {
+                "ip": [
+                    "198.51.100.6"
+                ],
+                "user": [
+                    "33333",
+                    "recruiter.two@omniva-corp.com"
+                ]
+            },
+            "source": {
+                "ip": "198.51.100.6"
+            },
+            "tags": [
+                "preserve_original_event",
+                "forwarded",
+                "greenhouse-audit",
+                "greenhouse-rejection-enrichment-failed"
+            ],
+            "user": {
+                "email": "recruiter.two@omniva-corp.com",
+                "full_name": "Recruiter Two",
+                "id": "33333"
+            }
+        },
+        {
+            "@timestamp": "2023-06-05T09:00:00.048Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "data_change_create",
+                "category": [
+                    "configuration",
+                    "iam"
+                ],
+                "id": "7890dID",
+                "kind": "event",
+                "original": "{\"event\":{\"meta\":{\"application_id\":293344484002,\"id\":221864428002},\"target_id\":221864428002,\"target_type\":\"RejectionDetails\",\"type\":\"data_change_create\"},\"event_time\":\"2023-06-05T09:00:00.048Z\",\"organization_id\":123,\"performer\":{\"id\":22222,\"ip_address\":\"198.51.100.5\",\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"type\":\"user\"},\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"}}",
+                "type": [
+                    "creation"
+                ]
+            },
+            "greenhouse": {
+                "audit": {
+                    "event": {
+                        "meta": {
+                            "application_id": 293344484002,
+                            "id": 221864428002
+                        },
+                        "target_id": "221864428002",
+                        "target_type": "RejectionDetails",
+                        "type": "data_change_create"
+                    },
+                    "performer": {
+                        "meta": {},
+                        "type": "user"
+                    },
+                    "request": {
+                        "id": "7890dID",
+                        "type": "people#change_application_state"
+                    }
+                }
+            },
+            "organization": {
+                "id": "123"
+            },
+            "related": {
+                "ip": [
+                    "198.51.100.5"
+                ],
+                "user": [
+                    "22222",
+                    "recruiter.one@omniva-corp.com"
+                ]
+            },
+            "source": {
+                "ip": "198.51.100.5"
+            },
+            "tags": [
+                "preserve_original_event",
+                "forwarded",
+                "greenhouse-audit"
+            ],
+            "user": {
+                "email": "recruiter.one@omniva-corp.com",
+                "full_name": "Recruiter One",
+                "id": "22222"
+            }
         }
     ]
 }