From 51d9493231ace0de3f74e945f69a0053fb991186 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 12 Jul 2026 14:08:41 +0000 Subject: [PATCH 1/3] greenhouse: enrich rejected application audit events via Harvest API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit log events only record that an Application's status changed to "rejected" — the rejection reason and notes/comments are not part of the audit log payload. Add an opt-in "Enrich rejected application events" setting that, when a rejection is detected, calls the Harvest V3 API (reusing the existing OAuth bearer token) to fetch the rejection reason and pull the rejection notes from the activity feed, merging them into the event under greenhouse.audit.event.rejection. --- packages/greenhouse/_dev/build/docs/README.md | 18 ++++ packages/greenhouse/changelog.yml | 5 + .../_dev/test/pipeline/test-audit-events.json | 3 + .../test-audit-events.json-expected.json | 77 +++++++++++++++ .../audit/agent/stream/cel.yml.hbs | 97 ++++++++++++++++++- .../elasticsearch/ingest_pipeline/default.yml | 11 +++ .../data_stream/audit/fields/fields.yml | 34 +++++++ .../greenhouse/data_stream/audit/manifest.yml | 9 ++ packages/greenhouse/docs/README.md | 58 +++++++++++ packages/greenhouse/manifest.yml | 10 +- 10 files changed, 320 insertions(+), 2 deletions(-) diff --git a/packages/greenhouse/_dev/build/docs/README.md b/packages/greenhouse/_dev/build/docs/README.md index 4bac41f7d00..4a40af1c261 100644 --- a/packages/greenhouse/_dev/build/docs/README.md +++ b/packages/greenhouse/_dev/build/docs/README.md @@ -56,6 +56,18 @@ To find a user's ID: - **Batch Size**: Number of events per API request (100-500, default: 500) - **Performer IDs Filter**: Filter by specific user IDs - **Event Types Filter**: Filter by event type (data_change_update, data_change_create, data_change_destroy, harvest_access, action) + - **Enrich rejected application events**: When enabled, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event (default: disabled) + +### Enabling rejection enrichment (optional) + +Audit log events only record that an Application's status changed to `rejected` — they do not include the rejection reason or the notes/comments entered at rejection time. Enabling **Enrich rejected application events** adds a lookup against the Harvest API for these details. + +1. On the same Harvest V3 (OAuth) API credential used for audit log access, add read scopes for the **Applications** and **Activity Feed** endpoints. +2. Enable the **Enrich rejected application events** setting on the integration. + +When an audit event records an Application moving to `rejected`, the integration performs up to two additional Harvest API calls (`GET /v3/applications/{id}` and `GET /v3/applications/{id}/activity_feed`) to fetch the rejection reason and the rejection notes, and adds them to the event under `greenhouse.audit.event.rejection`. If either lookup fails, the underlying audit event is still indexed, with `greenhouse.audit.event.rejection.error` describing the failure and the tag `greenhouse-rejection-enrichment-failed` added. + +Because this issues extra API requests per rejection on top of the audit log polling, be mindful of Greenhouse's rate limits (50 general requests per 10 seconds) if your organization rejects applications in bulk. ## Logs @@ -93,6 +105,12 @@ If no events are being collected: 2. Check that there have been events in the last 30 days 3. Review any filter settings that might be excluding events +### Rejection Enrichment Errors + +If rejection events are tagged with `greenhouse-rejection-enrichment-failed` and `greenhouse.audit.event.rejection.error` is populated: +1. Verify the OAuth credential has read scopes for the Harvest **Applications** and **Activity Feed** endpoints +2. Check that the authorizing user has permission to view the affected application + ## Logs reference ### audit diff --git a/packages/greenhouse/changelog.yml b/packages/greenhouse/changelog.yml index 0a1cb430a6f..006708e1ecb 100644 --- a/packages/greenhouse/changelog.yml +++ b/packages/greenhouse/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "0.2.0" + changes: + - description: Add an optional "Enrich rejected application events" setting that looks up the rejection reason and rejection notes/comments from the Harvest API and adds them to the audit event as `greenhouse.audit.event.rejection` when an Application's status changes to rejected. + type: enhancement + link: https://github.com/elastic/integrations/pull/17347 - version: "0.1.0" changes: - description: Initial release. diff --git a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json index a879a96a8b9..849baeee91a 100644 --- a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json +++ b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json @@ -14,6 +14,9 @@ }, { "message": "{\"request\":{\"id\":\"3456cID\",\"type\":\"jobs#destroy\"},\"performer\":{\"meta\":{\"name\":\"HR Manager\",\"username\":\"hr.manager@omniva-corp.com\"},\"id\":55555,\"ip_address\":\"172.16.0.100\",\"type\":\"user\"},\"organization_id\":456,\"event\":{\"meta\":{\"name\":[\"Software Engineer\",null]},\"target_type\":\"Job\",\"target_id\":789,\"type\":\"data_change_destroy\"},\"event_time\":\"2023-06-04T08:15:30.500Z\"}" + }, + { + "message": "{\"request\":{\"id\":\"7890dID\",\"type\":\"applications#update\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":{\"status\":[\"active\",\"rejected\"]},\"target_type\":\"Application\",\"target_id\":654321,\"type\":\"data_change_update\",\"rejection\":{\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}" } ] } diff --git a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json index 4463639ee4e..cf702486113 100644 --- a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json +++ b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json @@ -337,6 +337,83 @@ "full_name": "HR Manager", "id": "55555" } + }, + { + "@timestamp": "2023-06-05T09:00:00.000Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "action": "data_change_update", + "category": [ + "configuration", + "iam" + ], + "id": "7890dID", + "kind": "event", + "original": "{\"request\":{\"id\":\"7890dID\",\"type\":\"applications#update\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":{\"status\":[\"active\",\"rejected\"]},\"target_type\":\"Application\",\"target_id\":654321,\"type\":\"data_change_update\",\"rejection\":{\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}", + "reason": "Candidate lacked required experience with distributed systems.", + "type": [ + "change" + ] + }, + "greenhouse": { + "audit": { + "event": { + "meta": { + "status": [ + "active", + "rejected" + ] + }, + "rejection": { + "notes": "Candidate lacked required experience with distributed systems.", + "reason": { + "id": "555", + "name": "Not a fit", + "type": "We rejected them" + }, + "rejected_at": "2023-06-05T09:00:00.000Z" + }, + "target_id": "654321", + "target_type": "Application", + "type": "data_change_update" + }, + "performer": { + "meta": {}, + "type": "user" + }, + "request": { + "id": "7890dID", + "type": "applications#update" + } + } + }, + "organization": { + "id": "123" + }, + "related": { + "ip": [ + "198.51.100.5" + ], + "user": [ + "22222", + "recruiter.one@omniva-corp.com" + ] + }, + "source": { + "ip": "198.51.100.5" + }, + "tags": [ + "preserve_original_event", + "forwarded", + "greenhouse-audit" + ], + "user": { + "email": "recruiter.one@omniva-corp.com", + "full_name": "Recruiter One", + "id": "22222" + } } ] } diff --git a/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs b/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs index 27ec1143313..7a14729412e 100644 --- a/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs +++ b/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs @@ -16,8 +16,10 @@ resource.url: {{url}} state: initial_interval: {{initial_interval}} token_url: {{token_url}} + harvest_url: {{harvest_url}} client_id: {{client_id}} client_secret: {{client_secret}} + enrich_rejections: {{enrich_rejections}} {{#if user_id}} user_id: "{{user_id}}" {{/if}} @@ -51,6 +53,26 @@ Endpoints: - Response: {paging: {pit_id, next_search_after}, results: [...]} - next_search_after is null on the last page. +Rejection enrichment (optional, enrich_rejections): + + Audit events only record that an Application's status changed to + "rejected" (event.meta.status == [, "rejected"]); the rejection + reason and notes/comments are not part of the audit log payload, so + they are looked up from the Harvest API and merged into the event as + event.rejection before it is emitted. This reuses the same bearer + token as the audit log request, so the OAuth client must also be + granted read access to these Harvest V3 endpoints. + + GET https://harvest.greenhouse.io/v3/applications/{id} + - Provides rejection_reason {id, name, type} and rejected_at. + + GET https://harvest.greenhouse.io/v3/applications/{id}/activity_feed + - Provides the "Candidate Rejected" activity, whose body holds the + rejection notes/comments entered at rejection time. + + A failed lookup does not drop the underlying audit event: it is + emitted with event.rejection.error describing the failure instead. + State: auth_token — shallow-replaced object holding token and expires. Cleared to {} on 401 so the fields are absent, not sentinel-valued. @@ -168,7 +190,80 @@ program: |- ).as(next_max_event_time, { "auth_token": auth.auth_token, - "events": results.map(e, {"message": e.encode_json()}), + "events": results.map(e, + ( + state.enrich_rejections && + e.?event.type.orValue("") == "data_change_update" && + e.?event.target_type.orValue("") == "Application" && + e.?event.meta.status.orValue([]).size() == 2 && + e.event.meta.status[1] == "rejected" + ).as(is_rejection, + {"message": ( + is_rejection ? + request( + "GET", + state.harvest_url.trim_right("/") + "/v3/applications/" + string(e.event.target_id) + ).with( + {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} + ).do_request().as(app_resp, + (app_resp.StatusCode == 200) ? + app_resp.Body.decode_json().as(app, + request( + "GET", + state.harvest_url.trim_right("/") + "/v3/applications/" + string(e.event.target_id) + "/activity_feed" + ).with( + {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} + ).do_request().as(feed_resp, + (feed_resp.StatusCode == 200) ? + feed_resp.Body.decode_json().as(feed, + feed.?activities.orValue([]).filter(a, + a.?subject.orValue("") == "Candidate Rejected" + ).as(rejection_activities, + e.with({ + "event": e.event.with({ + "rejection": { + ?"reason": (app.?rejection_reason.orValue(null) == null) ? + optional.none() + : + optional.of({ + "id": string(app.rejection_reason.id), + "name": app.rejection_reason.name, + "type": app.rejection_reason.?type.name.orValue(""), + }), + ?"notes": (size(rejection_activities) > 0) ? + rejection_activities[0].?body + : + optional.none(), + ?"rejected_at": app.?rejected_at, + }, + }), + }) + ) + ) + : + e.with({ + "event": e.event.with({ + "rejection": { + "error": "activity_feed fetch failed: " + string(feed_resp.StatusCode), + }, + }), + }) + ) + ) + : + e.with({ + "event": e.event.with({ + "rejection": { + "error": "application fetch failed: " + string(app_resp.StatusCode), + }, + }), + }) + ) + : + e + ).encode_json()} + ) + ), "cursor": { // Keep window_start_time fixed until the final // page succeeds so the whole window can be diff --git a/packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml b/packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml index 64660b5324b..0cc54bddc13 100644 --- a/packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml +++ b/packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml @@ -187,6 +187,17 @@ processors: field: error.message value: 'Failed to convert target_id to string: {{{ _ingest.on_failure_message }}}' + # Rejection enrichment (Harvest API, requires enrich_rejections) + - set: + field: event.reason + copy_from: greenhouse.audit.event.rejection.notes + ignore_empty_value: true + - append: + field: tags + value: greenhouse-rejection-enrichment-failed + allow_duplicates: false + if: ctx.greenhouse?.audit?.event?.rejection?.error != null + # Related fields - append: field: related.user diff --git a/packages/greenhouse/data_stream/audit/fields/fields.yml b/packages/greenhouse/data_stream/audit/fields/fields.yml index 3d3d6461629..958733b492d 100644 --- a/packages/greenhouse/data_stream/audit/fields/fields.yml +++ b/packages/greenhouse/data_stream/audit/fields/fields.yml @@ -71,3 +71,37 @@ type: flattened description: > The before and after values from data change events, or other relevant data for the event. + - name: rejection + type: group + description: > + Rejection details looked up from the Harvest API when an Application's status changes to "rejected". Only populated when the `enrich_rejections` setting is enabled. + fields: + - name: reason + type: group + description: > + The rejection reason selected in Greenhouse. + fields: + - name: id + type: keyword + description: > + The ID of the rejection reason. + - name: name + type: keyword + description: > + The name of the rejection reason. + - name: type + type: keyword + description: > + The category of the rejection reason, for example "We rejected them" or "They rejected us". + - name: notes + type: match_only_text + description: > + The rejection notes/comments entered when the application was rejected, sourced from the Harvest API activity feed. + - name: rejected_at + type: date + description: > + The timestamp when the application was rejected, as reported by the Harvest API. + - name: error + type: keyword + description: > + Error message if the rejection enrichment lookup against the Harvest API failed. diff --git a/packages/greenhouse/data_stream/audit/manifest.yml b/packages/greenhouse/data_stream/audit/manifest.yml index ab78713cb94..895fab93b9d 100644 --- a/packages/greenhouse/data_stream/audit/manifest.yml +++ b/packages/greenhouse/data_stream/audit/manifest.yml @@ -45,6 +45,15 @@ streams: multi: false show_user: true required: false + - name: enrich_rejections + type: bool + title: Enrich rejected application events + multi: false + required: true + show_user: true + default: false + description: >- + When an audit event records an Application being moved to the "rejected" status, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event. This requires the OAuth client to also have read access to the Harvest "Applications" and "Activity Feed" endpoints, and results in up to two additional API calls per rejection event. - name: enable_request_tracer type: bool title: Enable request tracing diff --git a/packages/greenhouse/docs/README.md b/packages/greenhouse/docs/README.md index 87af52dbfe3..dc420b69469 100644 --- a/packages/greenhouse/docs/README.md +++ b/packages/greenhouse/docs/README.md @@ -56,6 +56,18 @@ To find a user's ID: - **Batch Size**: Number of events per API request (100-500, default: 500) - **Performer IDs Filter**: Filter by specific user IDs - **Event Types Filter**: Filter by event type (data_change_update, data_change_create, data_change_destroy, harvest_access, action) + - **Enrich rejected application events**: When enabled, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event (default: disabled) + +### Enabling rejection enrichment (optional) + +Audit log events only record that an Application's status changed to `rejected` — they do not include the rejection reason or the notes/comments entered at rejection time. Enabling **Enrich rejected application events** adds a lookup against the Harvest API for these details. + +1. On the same Harvest V3 (OAuth) API credential used for audit log access, add read scopes for the **Applications** and **Activity Feed** endpoints. +2. Enable the **Enrich rejected application events** setting on the integration. + +When an audit event records an Application moving to `rejected`, the integration performs up to two additional Harvest API calls (`GET /v3/applications/{id}` and `GET /v3/applications/{id}/activity_feed`) to fetch the rejection reason and the rejection notes, and adds them to the event under `greenhouse.audit.event.rejection`. If either lookup fails, the underlying audit event is still indexed, with `greenhouse.audit.event.rejection.error` describing the failure and the tag `greenhouse-rejection-enrichment-failed` added. + +Because this issues extra API requests per rejection on top of the audit log polling, be mindful of Greenhouse's rate limits (50 general requests per 10 seconds) if your organization rejects applications in bulk. ## Logs @@ -93,6 +105,12 @@ If no events are being collected: 2. Check that there have been events in the last 30 days 3. Review any filter settings that might be excluding events +### Rejection Enrichment Errors + +If rejection events are tagged with `greenhouse-rejection-enrichment-failed` and `greenhouse.audit.event.rejection.error` is populated: +1. Verify the OAuth credential has read scopes for the Harvest **Applications** and **Activity Feed** endpoints +2. Check that the authorizing user has permission to view the affected application + ## Logs reference ### audit @@ -108,6 +126,12 @@ If no events are being collected: | event.dataset | Event dataset | constant_keyword | | event.module | Event module | constant_keyword | | greenhouse.audit.event.meta | The before and after values from data change events, or other relevant data for the event. | flattened | +| greenhouse.audit.event.rejection.error | Error message if the rejection enrichment lookup against the Harvest API failed. | keyword | +| greenhouse.audit.event.rejection.notes | The rejection notes/comments entered when the application was rejected, sourced from the Harvest API activity feed. | match_only_text | +| greenhouse.audit.event.rejection.reason.id | The ID of the rejection reason. | keyword | +| greenhouse.audit.event.rejection.reason.name | The name of the rejection reason. | keyword | +| greenhouse.audit.event.rejection.reason.type | The category of the rejection reason, for example "We rejected them" or "They rejected us". | keyword | +| greenhouse.audit.event.rejection.rejected_at | The timestamp when the application was rejected, as reported by the Harvest API. | date | | greenhouse.audit.event.target_id | The ID of the element that was edited or accessed. | keyword | | greenhouse.audit.event.target_type | The resource name for data changes, Harvest access, or the event action type for other actions. | keyword | | greenhouse.audit.event.type | The type of event: data_change_update, data_change_create, data_change_destroy, harvest_access, or action. | keyword | @@ -204,3 +228,37 @@ An example event for `audit` looks as following: } } ``` + +When **Enrich rejected application events** is enabled, an audit event recording an Application's rejection is enriched with a `greenhouse.audit.event.rejection` object: + +```json +{ + "greenhouse": { + "audit": { + "event": { + "meta": { + "status": [ + "active", + "rejected" + ] + }, + "rejection": { + "notes": "Candidate lacked required experience with distributed systems.", + "reason": { + "id": "555", + "name": "Not a fit", + "type": "We rejected them" + }, + "rejected_at": "2023-06-05T09:00:00.000Z" + }, + "target_id": "654321", + "target_type": "Application", + "type": "data_change_update" + } + } + }, + "event": { + "reason": "Candidate lacked required experience with distributed systems." + } +} +``` diff --git a/packages/greenhouse/manifest.yml b/packages/greenhouse/manifest.yml index fa5b99bca6f..90b5230035e 100644 --- a/packages/greenhouse/manifest.yml +++ b/packages/greenhouse/manifest.yml @@ -1,7 +1,7 @@ format_version: "3.3.2" name: greenhouse title: "Greenhouse" -version: "0.1.0" +version: "0.2.0" description: Collect audit logs from Greenhouse ATS with Elastic Agent. type: integration categories: @@ -48,6 +48,14 @@ policy_templates: required: true default: https://auth.greenhouse.io/token description: The OAuth 2.0 token endpoint URL for obtaining access tokens. + - name: harvest_url + type: text + title: Greenhouse Harvest API URL + multi: false + show_user: false + required: true + default: https://harvest.greenhouse.io + description: The base URL of the Greenhouse Harvest API, used to look up rejection details when "Enrich rejected application events" is enabled. - name: client_id type: text title: OAuth Client ID From 47db5cc9069b1f228494a6fd27134c402bc04312 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 12 Jul 2026 23:29:14 +0000 Subject: [PATCH 2/3] greenhouse: correct rejection enrichment trigger and correlation using real audit event A real "Candidate or Prospect rejected" audit event showed the original implementation's assumptions were wrong: the event type is "action" (not "data_change_update"), target_type is the literal string "Candidate or Prospect rejected" (not "Application"), and the event carries no application/candidate ID at all. Since there's no ID to look up directly, correlate by time instead: query the Harvest API for rejected Applications whose rejected_at falls within a configurable window (default 5m) of the audit event's timestamp. An unambiguous single match is enriched with the rejection reason and notes; zero or multiple matches are recorded as an error (with candidate IDs listed in the ambiguous case) rather than guessed. --- packages/greenhouse/_dev/build/docs/README.md | 13 +- packages/greenhouse/changelog.yml | 2 +- .../_dev/test/pipeline/test-audit-events.json | 5 +- .../test-audit-events.json-expected.json | 89 ++++++-- .../audit/agent/stream/cel.yml.hbs | 196 ++++++++++++------ .../data_stream/audit/fields/fields.yml | 14 +- .../greenhouse/data_stream/audit/manifest.yml | 11 +- packages/greenhouse/docs/README.md | 59 ++++-- 8 files changed, 291 insertions(+), 98 deletions(-) diff --git a/packages/greenhouse/_dev/build/docs/README.md b/packages/greenhouse/_dev/build/docs/README.md index 4a40af1c261..d70a77bb9fa 100644 --- a/packages/greenhouse/_dev/build/docs/README.md +++ b/packages/greenhouse/_dev/build/docs/README.md @@ -57,17 +57,24 @@ To find a user's ID: - **Performer IDs Filter**: Filter by specific user IDs - **Event Types Filter**: Filter by event type (data_change_update, data_change_create, data_change_destroy, harvest_access, action) - **Enrich rejected application events**: When enabled, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event (default: disabled) + - **Rejection Match Window**: Time window used to correlate a rejection event to its Application (default: 5m, only used when enrichment is enabled) ### Enabling rejection enrichment (optional) -Audit log events only record that an Application's status changed to `rejected` — they do not include the rejection reason or the notes/comments entered at rejection time. Enabling **Enrich rejected application events** adds a lookup against the Harvest API for these details. +When a candidate or prospect is rejected, Greenhouse audits it as an `action` event with `event.target_type` set to the literal string `Candidate or Prospect rejected` — this event does **not** include an application or candidate ID, and does not include the rejection reason or notes/comments entered at rejection time. Enabling **Enrich rejected application events** looks these up from the Harvest API and correlates them back to the audit event. 1. On the same Harvest V3 (OAuth) API credential used for audit log access, add read scopes for the **Applications** and **Activity Feed** endpoints. 2. Enable the **Enrich rejected application events** setting on the integration. -When an audit event records an Application moving to `rejected`, the integration performs up to two additional Harvest API calls (`GET /v3/applications/{id}` and `GET /v3/applications/{id}/activity_feed`) to fetch the rejection reason and the rejection notes, and adds them to the event under `greenhouse.audit.event.rejection`. If either lookup fails, the underlying audit event is still indexed, with `greenhouse.audit.event.rejection.error` describing the failure and the tag `greenhouse-rejection-enrichment-failed` added. +Because the audit event has no application ID, the integration looks up Applications whose rejection time falls within **Rejection Match Window** of the audit event's timestamp: -Because this issues extra API requests per rejection on top of the audit log polling, be mindful of Greenhouse's rate limits (50 general requests per 10 seconds) if your organization rejects applications in bulk. +- **Exactly one match**: the rejection reason and notes are fetched and added to the event under `greenhouse.audit.event.rejection` (`application_id`, `candidate_id`, `reason.id`/`reason.name`/`reason.type`, `notes`, `rejected_at`). The notes are also copied to `event.reason`. +- **No match**: the audit event is still indexed, with `greenhouse.audit.event.rejection.error` explaining that no matching Application was found. Consider widening **Rejection Match Window**. +- **More than one match** (for example, several rejections processed in the same window): the integration does not guess. The audit event is indexed with `greenhouse.audit.event.rejection.error` and `greenhouse.audit.event.rejection.ambiguous_application_ids` listing the candidates, for manual follow-up. Consider narrowing **Rejection Match Window** if this happens often. + +Any of the error cases above also adds the tag `greenhouse-rejection-enrichment-failed`, and the underlying rejection audit event is never dropped. + +Because this issues up to two extra API requests per rejection on top of the audit log polling, be mindful of Greenhouse's rate limits (50 general requests per 10 seconds) if your organization rejects applications in bulk. ## Logs diff --git a/packages/greenhouse/changelog.yml b/packages/greenhouse/changelog.yml index 006708e1ecb..5ba0a86c61e 100644 --- a/packages/greenhouse/changelog.yml +++ b/packages/greenhouse/changelog.yml @@ -1,7 +1,7 @@ # newer versions go on top - version: "0.2.0" changes: - - description: Add an optional "Enrich rejected application events" setting that looks up the rejection reason and rejection notes/comments from the Harvest API and adds them to the audit event as `greenhouse.audit.event.rejection` when an Application's status changes to rejected. + - description: Add an optional "Enrich rejected application events" setting that, for a "Candidate or Prospect rejected" audit event, correlates it by timestamp to the matching Application via the Harvest API and adds the rejection reason and rejection notes/comments to the event as `greenhouse.audit.event.rejection`. type: enhancement link: https://github.com/elastic/integrations/pull/17347 - version: "0.1.0" diff --git a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json index 849baeee91a..f53d40206e3 100644 --- a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json +++ b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json @@ -16,7 +16,10 @@ "message": "{\"request\":{\"id\":\"3456cID\",\"type\":\"jobs#destroy\"},\"performer\":{\"meta\":{\"name\":\"HR Manager\",\"username\":\"hr.manager@omniva-corp.com\"},\"id\":55555,\"ip_address\":\"172.16.0.100\",\"type\":\"user\"},\"organization_id\":456,\"event\":{\"meta\":{\"name\":[\"Software Engineer\",null]},\"target_type\":\"Job\",\"target_id\":789,\"type\":\"data_change_destroy\"},\"event_time\":\"2023-06-04T08:15:30.500Z\"}" }, { - "message": "{\"request\":{\"id\":\"7890dID\",\"type\":\"applications#update\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":{\"status\":[\"active\",\"rejected\"]},\"target_type\":\"Application\",\"target_id\":654321,\"type\":\"data_change_update\",\"rejection\":{\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}" + "message": "{\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"application_id\":\"987654\",\"candidate_id\":\"123123\",\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}" + }, + { + "message": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous match: 2 rejected applications found within 5m of the audit event timestamp\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}" } ] } diff --git a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json index cf702486113..5ae93244e9f 100644 --- a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json +++ b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json @@ -344,29 +344,24 @@ "version": "8.11.0" }, "event": { - "action": "data_change_update", + "action": "action", "category": [ - "configuration", "iam" ], "id": "7890dID", "kind": "event", - "original": "{\"request\":{\"id\":\"7890dID\",\"type\":\"applications#update\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":{\"status\":[\"active\",\"rejected\"]},\"target_type\":\"Application\",\"target_id\":654321,\"type\":\"data_change_update\",\"rejection\":{\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}", + "original": "{\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"application_id\":\"987654\",\"candidate_id\":\"123123\",\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}", "reason": "Candidate lacked required experience with distributed systems.", "type": [ - "change" + "info" ] }, "greenhouse": { "audit": { "event": { - "meta": { - "status": [ - "active", - "rejected" - ] - }, "rejection": { + "application_id": "987654", + "candidate_id": "123123", "notes": "Candidate lacked required experience with distributed systems.", "reason": { "id": "555", @@ -375,9 +370,8 @@ }, "rejected_at": "2023-06-05T09:00:00.000Z" }, - "target_id": "654321", - "target_type": "Application", - "type": "data_change_update" + "target_type": "Candidate or Prospect rejected", + "type": "action" }, "performer": { "meta": {}, @@ -385,7 +379,7 @@ }, "request": { "id": "7890dID", - "type": "applications#update" + "type": "people#change_application_state" } } }, @@ -414,6 +408,73 @@ "full_name": "Recruiter One", "id": "22222" } + }, + { + "@timestamp": "2023-06-05T09:05:00.000Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "action": "action", + "category": [ + "iam" + ], + "id": "7891eID", + "kind": "event", + "original": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous match: 2 rejected applications found within 5m of the audit event timestamp\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}", + "type": [ + "info" + ] + }, + "greenhouse": { + "audit": { + "event": { + "rejection": { + "ambiguous_application_ids": [ + "111111", + "222222" + ], + "error": "ambiguous match: 2 rejected applications found within 5m of the audit event timestamp" + }, + "target_type": "Candidate or Prospect rejected", + "type": "action" + }, + "performer": { + "meta": {}, + "type": "user" + }, + "request": { + "id": "7891eID", + "type": "people#change_application_state" + } + } + }, + "organization": { + "id": "123" + }, + "related": { + "ip": [ + "198.51.100.6" + ], + "user": [ + "33333", + "recruiter.two@omniva-corp.com" + ] + }, + "source": { + "ip": "198.51.100.6" + }, + "tags": [ + "preserve_original_event", + "forwarded", + "greenhouse-audit", + "greenhouse-rejection-enrichment-failed" + ], + "user": { + "email": "recruiter.two@omniva-corp.com", + "full_name": "Recruiter Two", + "id": "33333" + } } ] } diff --git a/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs b/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs index 7a14729412e..85ef3d0d0c8 100644 --- a/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs +++ b/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs @@ -20,6 +20,7 @@ state: client_id: {{client_id}} client_secret: {{client_secret}} enrich_rejections: {{enrich_rejections}} + rejection_match_window: {{rejection_match_window}} {{#if user_id}} user_id: "{{user_id}}" {{/if}} @@ -55,23 +56,43 @@ Endpoints: Rejection enrichment (optional, enrich_rejections): - Audit events only record that an Application's status changed to - "rejected" (event.meta.status == [, "rejected"]); the rejection - reason and notes/comments are not part of the audit log payload, so - they are looked up from the Harvest API and merged into the event as + A candidate/prospect rejection is audited as an "action" event with + event.target_type == "Candidate or Prospect rejected" and + event.meta == null — Greenhouse does NOT include an application or + candidate ID on this event, so the rejected Application cannot be + looked up directly by ID. Instead, the rejection reason and notes are + correlated by time: the Harvest API is queried for Applications whose + last_activity/rejected_at falls within rejection_match_window of the + audit event's event_time, and the result is merged into the event as event.rejection before it is emitted. This reuses the same bearer token as the audit log request, so the OAuth client must also be granted read access to these Harvest V3 endpoints. - GET https://harvest.greenhouse.io/v3/applications/{id} - - Provides rejection_reason {id, name, type} and rejected_at. + GET https://harvest.greenhouse.io/v3/applications + ?last_activity_after= + &last_activity_before= + - Assumed to return a bare JSON array of Application objects (as + other Harvest list endpoints do), each optionally carrying + rejection_reason {id, name, type}, rejected_at, and candidate_id. + Results are filtered client-side to status == "rejected" and + rejected_at inside the window. + - Exactly one match: proceed to fetch its activity feed below. + - Zero matches: event.rejection.error notes no match was found; + widen rejection_match_window if this happens often. + - More than one match (e.g. several rejections in the same + window): no attempt is made to guess which one is correct. + event.rejection.error records the ambiguity and + event.rejection.ambiguous_application_ids lists the candidates + for manual follow-up. GET https://harvest.greenhouse.io/v3/applications/{id}/activity_feed - - Provides the "Candidate Rejected" activity, whose body holds the - rejection notes/comments entered at rejection time. + - Only called for a single, unambiguous match. Provides the + "Candidate Rejected" activity, whose body holds the rejection + notes/comments entered at rejection time. - A failed lookup does not drop the underlying audit event: it is - emitted with event.rejection.error describing the failure instead. + A failed or inconclusive lookup does not drop the underlying audit + event: it is emitted with event.rejection.error describing why no + reason/notes could be attached. State: auth_token — shallow-replaced object holding token and expires. @@ -193,71 +214,118 @@ program: |- "events": results.map(e, ( state.enrich_rejections && - e.?event.type.orValue("") == "data_change_update" && - e.?event.target_type.orValue("") == "Application" && - e.?event.meta.status.orValue([]).size() == 2 && - e.event.meta.status[1] == "rejected" + e.?event.type.orValue("") == "action" && + e.?event.target_type.orValue("") == "Candidate or Prospect rejected" ).as(is_rejection, {"message": ( is_rejection ? - request( - "GET", - state.harvest_url.trim_right("/") + "/v3/applications/" + string(e.event.target_id) - ).with( - {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} - ).do_request().as(app_resp, - (app_resp.StatusCode == 200) ? - app_resp.Body.decode_json().as(app, - request( - "GET", - state.harvest_url.trim_right("/") + "/v3/applications/" + string(e.event.target_id) + "/activity_feed" - ).with( - {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} - ).do_request().as(feed_resp, - (feed_resp.StatusCode == 200) ? - feed_resp.Body.decode_json().as(feed, - feed.?activities.orValue([]).filter(a, - a.?subject.orValue("") == "Candidate Rejected" - ).as(rejection_activities, + timestamp(e.event_time).as(event_ts, + duration(state.rejection_match_window).as(window, + request( + "GET", + state.harvest_url.trim_right("/") + "/v3/applications?" + { + "last_activity_after": [(event_ts - window).format(time_layout.RFC3339)], + "last_activity_before": [(event_ts + window).format(time_layout.RFC3339)], + }.format_query() + ).with( + {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} + ).do_request().as(list_resp, + (list_resp.StatusCode == 200) ? + list_resp.Body.decode_json().as(apps, + apps.filter(a, + a.?status.orValue("") == "rejected" && + a.?rejected_at.hasValue() && + timestamp(a.rejected_at) >= event_ts - window && + timestamp(a.rejected_at) <= event_ts + window + ).as(candidates, + (candidates.size() == 1) ? + candidates[0].as(app, + request( + "GET", + state.harvest_url.trim_right("/") + "/v3/applications/" + string(app.id) + "/activity_feed" + ).with( + {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} + ).do_request().as(feed_resp, + (feed_resp.StatusCode == 200) ? + feed_resp.Body.decode_json().as(feed, + feed.?activities.orValue([]).filter(a2, + a2.?subject.orValue("") == "Candidate Rejected" + ).as(rejection_activities, + e.with({ + "event": e.event.with({ + "rejection": { + "application_id": string(app.id), + ?"candidate_id": (app.?candidate_id.orValue(null) == null) ? + optional.none() + : + optional.of(string(app.candidate_id)), + ?"reason": (app.?rejection_reason.orValue(null) == null) ? + optional.none() + : + optional.of({ + "id": string(app.rejection_reason.id), + "name": app.rejection_reason.name, + "type": app.rejection_reason.?type.name.orValue(""), + }), + ?"notes": (size(rejection_activities) > 0) ? + rejection_activities[0].?body + : + optional.none(), + ?"rejected_at": app.?rejected_at, + }, + }), + }) + ) + ) + : + e.with({ + "event": e.event.with({ + "rejection": { + "application_id": string(app.id), + ?"reason": (app.?rejection_reason.orValue(null) == null) ? + optional.none() + : + optional.of({ + "id": string(app.rejection_reason.id), + "name": app.rejection_reason.name, + "type": app.rejection_reason.?type.name.orValue(""), + }), + ?"rejected_at": app.?rejected_at, + "error": "activity_feed fetch failed: " + string(feed_resp.StatusCode), + }, + }), + }) + ) + ) + : (candidates.size() == 0) ? e.with({ "event": e.event.with({ "rejection": { - ?"reason": (app.?rejection_reason.orValue(null) == null) ? - optional.none() - : - optional.of({ - "id": string(app.rejection_reason.id), - "name": app.rejection_reason.name, - "type": app.rejection_reason.?type.name.orValue(""), - }), - ?"notes": (size(rejection_activities) > 0) ? - rejection_activities[0].?body - : - optional.none(), - ?"rejected_at": app.?rejected_at, + "error": "no rejected application found within " + state.rejection_match_window + " of the audit event timestamp", + }, + }), + }) + : + e.with({ + "event": e.event.with({ + "rejection": { + "error": "ambiguous match: " + string(candidates.size()) + " rejected applications found within " + state.rejection_match_window + " of the audit event timestamp", + "ambiguous_application_ids": candidates.map(a3, string(a3.id)), }, }), }) - ) ) - : - e.with({ - "event": e.event.with({ - "rejection": { - "error": "activity_feed fetch failed: " + string(feed_resp.StatusCode), - }, - }), - }) - ) + ) + : + e.with({ + "event": e.event.with({ + "rejection": { + "error": "applications lookup failed: " + string(list_resp.StatusCode), + }, + }), + }) ) - : - e.with({ - "event": e.event.with({ - "rejection": { - "error": "application fetch failed: " + string(app_resp.StatusCode), - }, - }), - }) + ) ) : e diff --git a/packages/greenhouse/data_stream/audit/fields/fields.yml b/packages/greenhouse/data_stream/audit/fields/fields.yml index 958733b492d..3d8650b5e16 100644 --- a/packages/greenhouse/data_stream/audit/fields/fields.yml +++ b/packages/greenhouse/data_stream/audit/fields/fields.yml @@ -74,8 +74,20 @@ - name: rejection type: group description: > - Rejection details looked up from the Harvest API when an Application's status changes to "rejected". Only populated when the `enrich_rejections` setting is enabled. + Rejection details looked up from the Harvest API for a "Candidate or Prospect rejected" audit event, correlated by timestamp since the audit event does not include an application ID. Only populated when the `enrich_rejections` setting is enabled. fields: + - name: application_id + type: keyword + description: > + The ID of the Application matched to this rejection event by timestamp correlation. + - name: candidate_id + type: keyword + description: > + The ID of the candidate whose application was rejected. + - name: ambiguous_application_ids + type: keyword + description: > + When timestamp correlation matches more than one rejected Application, the IDs of all matching candidates. No reason or notes are attached in this case. - name: reason type: group description: > diff --git a/packages/greenhouse/data_stream/audit/manifest.yml b/packages/greenhouse/data_stream/audit/manifest.yml index 895fab93b9d..c03ed01d6fc 100644 --- a/packages/greenhouse/data_stream/audit/manifest.yml +++ b/packages/greenhouse/data_stream/audit/manifest.yml @@ -53,7 +53,16 @@ streams: show_user: true default: false description: >- - When an audit event records an Application being moved to the "rejected" status, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event. This requires the OAuth client to also have read access to the Harvest "Applications" and "Activity Feed" endpoints, and results in up to two additional API calls per rejection event. + When an audit event records a candidate or prospect being rejected, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event. Since this audit event does not include an application ID, the matching Application is found by correlating its rejected_at time to the audit event's timestamp (see "Rejection Match Window"); if that correlation is not exact, the event is emitted with an error instead of a guess. This requires the OAuth client to also have read access to the Harvest "Applications" and "Activity Feed" endpoints, and results in up to two additional API calls per rejection event. + - name: rejection_match_window + type: text + title: Rejection Match Window + description: >- + Only used when "Enrich rejected application events" is enabled. Time window, centered on the audit event's timestamp, used to find the matching rejected Application via the Harvest API. Valid time units are s, m, h. Widen this if enrichment often reports no match found; narrow it if it often reports an ambiguous match. + multi: false + required: false + show_user: true + default: 5m - name: enable_request_tracer type: bool title: Enable request tracing diff --git a/packages/greenhouse/docs/README.md b/packages/greenhouse/docs/README.md index dc420b69469..4b52cda4c20 100644 --- a/packages/greenhouse/docs/README.md +++ b/packages/greenhouse/docs/README.md @@ -57,17 +57,24 @@ To find a user's ID: - **Performer IDs Filter**: Filter by specific user IDs - **Event Types Filter**: Filter by event type (data_change_update, data_change_create, data_change_destroy, harvest_access, action) - **Enrich rejected application events**: When enabled, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event (default: disabled) + - **Rejection Match Window**: Time window used to correlate a rejection event to its Application (default: 5m, only used when enrichment is enabled) ### Enabling rejection enrichment (optional) -Audit log events only record that an Application's status changed to `rejected` — they do not include the rejection reason or the notes/comments entered at rejection time. Enabling **Enrich rejected application events** adds a lookup against the Harvest API for these details. +When a candidate or prospect is rejected, Greenhouse audits it as an `action` event with `event.target_type` set to the literal string `Candidate or Prospect rejected` — this event does **not** include an application or candidate ID, and does not include the rejection reason or notes/comments entered at rejection time. Enabling **Enrich rejected application events** looks these up from the Harvest API and correlates them back to the audit event. 1. On the same Harvest V3 (OAuth) API credential used for audit log access, add read scopes for the **Applications** and **Activity Feed** endpoints. 2. Enable the **Enrich rejected application events** setting on the integration. -When an audit event records an Application moving to `rejected`, the integration performs up to two additional Harvest API calls (`GET /v3/applications/{id}` and `GET /v3/applications/{id}/activity_feed`) to fetch the rejection reason and the rejection notes, and adds them to the event under `greenhouse.audit.event.rejection`. If either lookup fails, the underlying audit event is still indexed, with `greenhouse.audit.event.rejection.error` describing the failure and the tag `greenhouse-rejection-enrichment-failed` added. +Because the audit event has no application ID, the integration looks up Applications whose rejection time falls within **Rejection Match Window** of the audit event's timestamp: -Because this issues extra API requests per rejection on top of the audit log polling, be mindful of Greenhouse's rate limits (50 general requests per 10 seconds) if your organization rejects applications in bulk. +- **Exactly one match**: the rejection reason and notes are fetched and added to the event under `greenhouse.audit.event.rejection` (`application_id`, `candidate_id`, `reason.id`/`reason.name`/`reason.type`, `notes`, `rejected_at`). The notes are also copied to `event.reason`. +- **No match**: the audit event is still indexed, with `greenhouse.audit.event.rejection.error` explaining that no matching Application was found. Consider widening **Rejection Match Window**. +- **More than one match** (for example, several rejections processed in the same window): the integration does not guess. The audit event is indexed with `greenhouse.audit.event.rejection.error` and `greenhouse.audit.event.rejection.ambiguous_application_ids` listing the candidates, for manual follow-up. Consider narrowing **Rejection Match Window** if this happens often. + +Any of the error cases above also adds the tag `greenhouse-rejection-enrichment-failed`, and the underlying rejection audit event is never dropped. + +Because this issues up to two extra API requests per rejection on top of the audit log polling, be mindful of Greenhouse's rate limits (50 general requests per 10 seconds) if your organization rejects applications in bulk. ## Logs @@ -126,6 +133,9 @@ If rejection events are tagged with `greenhouse-rejection-enrichment-failed` and | event.dataset | Event dataset | constant_keyword | | event.module | Event module | constant_keyword | | greenhouse.audit.event.meta | The before and after values from data change events, or other relevant data for the event. | flattened | +| greenhouse.audit.event.rejection.ambiguous_application_ids | When timestamp correlation matches more than one rejected Application, the IDs of all matching candidates. No reason or notes are attached in this case. | keyword | +| greenhouse.audit.event.rejection.application_id | The ID of the Application matched to this rejection event by timestamp correlation. | keyword | +| greenhouse.audit.event.rejection.candidate_id | The ID of the candidate whose application was rejected. | keyword | | greenhouse.audit.event.rejection.error | Error message if the rejection enrichment lookup against the Harvest API failed. | keyword | | greenhouse.audit.event.rejection.notes | The rejection notes/comments entered when the application was rejected, sourced from the Harvest API activity feed. | match_only_text | | greenhouse.audit.event.rejection.reason.id | The ID of the rejection reason. | keyword | @@ -229,20 +239,16 @@ An example event for `audit` looks as following: } ``` -When **Enrich rejected application events** is enabled, an audit event recording an Application's rejection is enriched with a `greenhouse.audit.event.rejection` object: +When **Enrich rejected application events** is enabled, a `Candidate or Prospect rejected` audit event is enriched with a `greenhouse.audit.event.rejection` object, correlated to its Application by timestamp: ```json { "greenhouse": { "audit": { "event": { - "meta": { - "status": [ - "active", - "rejected" - ] - }, "rejection": { + "application_id": "987654", + "candidate_id": "123123", "notes": "Candidate lacked required experience with distributed systems.", "reason": { "id": "555", @@ -251,9 +257,8 @@ When **Enrich rejected application events** is enabled, an audit event recording }, "rejected_at": "2023-06-05T09:00:00.000Z" }, - "target_id": "654321", - "target_type": "Application", - "type": "data_change_update" + "target_type": "Candidate or Prospect rejected", + "type": "action" } } }, @@ -262,3 +267,31 @@ When **Enrich rejected application events** is enabled, an audit event recording } } ``` + +If the correlation is inconclusive, `greenhouse.audit.event.rejection` instead contains an `error` (and, for an ambiguous match, `ambiguous_application_ids`) with no `reason`/`notes` attached: + +```json +{ + "greenhouse": { + "audit": { + "event": { + "rejection": { + "ambiguous_application_ids": [ + "111111", + "222222" + ], + "error": "ambiguous match: 2 rejected applications found within 5m of the audit event timestamp" + }, + "target_type": "Candidate or Prospect rejected", + "type": "action" + } + } + }, + "tags": [ + "preserve_original_event", + "forwarded", + "greenhouse-audit", + "greenhouse-rejection-enrichment-failed" + ] +} +``` From c1555459efbe95d820b48ca09aab35dd4721f9b5 Mon Sep 17 00:00:00 2001 From: Christopher Cutajar Date: Mon, 13 Jul 2026 02:05:06 +0200 Subject: [PATCH 3/3] greenhouse: correlate rejection enrichment via request.id, document required Harvest permissions A second real audit sample showed Greenhouse also audits a separate "RejectionDetails" data_change_create event for the same rejection, sharing the same request.id and carrying event.meta.application_id. Replace the earlier timestamp-window guess with a deterministic correlation: match the "Candidate or Prospect rejected" event to its RejectionDetails sibling by request.id within the same page of audit results, then look up the resulting application_id directly via the Harvest API. Ambiguous request.id matches (e.g. bulk rejections) and no-match cases are still recorded as an error rather than guessed. Also document the specific Harvest OAuth scopes required (Rejection details / Rejection reasons, in addition to Applications / Activity Feed) based on the actual permission UI. --- packages/greenhouse/_dev/build/docs/README.md | 19 +- .../_dev/test/pipeline/test-audit-events.json | 5 +- .../test-audit-events.json-expected.json | 69 ++++- .../audit/agent/stream/cel.yml.hbs | 289 +++++++++--------- .../data_stream/audit/fields/fields.yml | 6 +- .../greenhouse/data_stream/audit/manifest.yml | 11 +- packages/greenhouse/docs/README.md | 27 +- 7 files changed, 247 insertions(+), 179 deletions(-) diff --git a/packages/greenhouse/_dev/build/docs/README.md b/packages/greenhouse/_dev/build/docs/README.md index d70a77bb9fa..b6f67bd2510 100644 --- a/packages/greenhouse/_dev/build/docs/README.md +++ b/packages/greenhouse/_dev/build/docs/README.md @@ -57,20 +57,23 @@ To find a user's ID: - **Performer IDs Filter**: Filter by specific user IDs - **Event Types Filter**: Filter by event type (data_change_update, data_change_create, data_change_destroy, harvest_access, action) - **Enrich rejected application events**: When enabled, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event (default: disabled) - - **Rejection Match Window**: Time window used to correlate a rejection event to its Application (default: 5m, only used when enrichment is enabled) ### Enabling rejection enrichment (optional) When a candidate or prospect is rejected, Greenhouse audits it as an `action` event with `event.target_type` set to the literal string `Candidate or Prospect rejected` — this event does **not** include an application or candidate ID, and does not include the rejection reason or notes/comments entered at rejection time. Enabling **Enrich rejected application events** looks these up from the Harvest API and correlates them back to the audit event. -1. On the same Harvest V3 (OAuth) API credential used for audit log access, add read scopes for the **Applications** and **Activity Feed** endpoints. +1. On the same Harvest V3 (OAuth) API credential used for audit log access, grant the following read permissions (under **Configure > Dev Center > API Credentials** on the credential): + - **Rejection details** → *List rejection details* + - **Rejection reasons** → *List rejection reasons* and *Show rejection reasons* + - **Applications** → read access, needed for the `GET /v3/applications/{id}` lookup this integration performs + - **Activity Feed** → read access, needed for the `GET /v3/applications/{id}/activity_feed` lookup this integration performs to retrieve the rejection notes 2. Enable the **Enrich rejected application events** setting on the integration. -Because the audit event has no application ID, the integration looks up Applications whose rejection time falls within **Rejection Match Window** of the audit event's timestamp: +Because the "Candidate or Prospect rejected" audit event has no application ID, the integration correlates it to a second audit event that Greenhouse records for the same rejection — one with `event.target_type` of `RejectionDetails` — by matching the `request.id` shared by both events. That event's `event.meta.application_id` is then used to look up the rejection reason and notes: -- **Exactly one match**: the rejection reason and notes are fetched and added to the event under `greenhouse.audit.event.rejection` (`application_id`, `candidate_id`, `reason.id`/`reason.name`/`reason.type`, `notes`, `rejected_at`). The notes are also copied to `event.reason`. -- **No match**: the audit event is still indexed, with `greenhouse.audit.event.rejection.error` explaining that no matching Application was found. Consider widening **Rejection Match Window**. -- **More than one match** (for example, several rejections processed in the same window): the integration does not guess. The audit event is indexed with `greenhouse.audit.event.rejection.error` and `greenhouse.audit.event.rejection.ambiguous_application_ids` listing the candidates, for manual follow-up. Consider narrowing **Rejection Match Window** if this happens often. +- **Exactly one matching `RejectionDetails` event**: the rejection reason and notes are fetched and added to the event under `greenhouse.audit.event.rejection` (`application_id`, `candidate_id`, `reason.id`/`reason.name`/`reason.type`, `notes`, `rejected_at`). The notes are also copied to `event.reason`. +- **No matching event found**: the audit event is still indexed, with `greenhouse.audit.event.rejection.error` explaining that no matching `RejectionDetails` event was found for that request. This can happen if the two events land on different pages of a poll; increasing **Batch Size** reduces how often this occurs. +- **More than one matching event** (for example, a bulk rejection of several candidates in one action, which shares a single `request.id` across all of them): the integration does not guess which pairs with which. The audit event is indexed with `greenhouse.audit.event.rejection.error` and `greenhouse.audit.event.rejection.ambiguous_application_ids` listing the candidates, for manual follow-up. Any of the error cases above also adds the tag `greenhouse-rejection-enrichment-failed`, and the underlying rejection audit event is never dropped. @@ -115,8 +118,10 @@ If no events are being collected: ### Rejection Enrichment Errors If rejection events are tagged with `greenhouse-rejection-enrichment-failed` and `greenhouse.audit.event.rejection.error` is populated: -1. Verify the OAuth credential has read scopes for the Harvest **Applications** and **Activity Feed** endpoints +1. Verify the OAuth credential has been granted **Rejection details** (*List rejection details*), **Rejection reasons** (*List rejection reasons*, *Show rejection reasons*), **Applications**, and **Activity Feed** read permissions 2. Check that the authorizing user has permission to view the affected application +3. If the error mentions no matching `RejectionDetails` event was found, try increasing **Batch Size** so the two related audit events are less likely to land on different pages +4. If the error mentions an ambiguous match, the rejection was likely part of a bulk-reject action; `greenhouse.audit.event.rejection.ambiguous_application_ids` lists the candidates involved for manual follow-up ## Logs reference diff --git a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json index f53d40206e3..f397ecd9f1d 100644 --- a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json +++ b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json @@ -19,7 +19,10 @@ "message": "{\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"id\":22222,\"ip_address\":\"198.51.100.5\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"application_id\":\"987654\",\"candidate_id\":\"123123\",\"reason\":{\"id\":\"555\",\"name\":\"Not a fit\",\"type\":\"We rejected them\"},\"notes\":\"Candidate lacked required experience with distributed systems.\",\"rejected_at\":\"2023-06-05T09:00:00.000Z\"}},\"event_time\":\"2023-06-05T09:00:00.000Z\"}" }, { - "message": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous match: 2 rejected applications found within 5m of the audit event timestamp\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}" + "message": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous: 2 RejectionDetails events share request.id 7891eID (likely a bulk rejection)\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}" + }, + { + "message": "{\"event\":{\"meta\":{\"application_id\":293344484002,\"id\":221864428002},\"target_id\":221864428002,\"target_type\":\"RejectionDetails\",\"type\":\"data_change_create\"},\"event_time\":\"2023-06-05T09:00:00.048Z\",\"organization_id\":123,\"performer\":{\"id\":22222,\"ip_address\":\"198.51.100.5\",\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"type\":\"user\"},\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"}}" } ] } diff --git a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json index 5ae93244e9f..2347eb0253a 100644 --- a/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json +++ b/packages/greenhouse/data_stream/audit/_dev/test/pipeline/test-audit-events.json-expected.json @@ -421,7 +421,7 @@ ], "id": "7891eID", "kind": "event", - "original": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous match: 2 rejected applications found within 5m of the audit event timestamp\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}", + "original": "{\"request\":{\"id\":\"7891eID\",\"type\":\"people#change_application_state\"},\"performer\":{\"meta\":{\"name\":\"Recruiter Two\",\"username\":\"recruiter.two@omniva-corp.com\"},\"id\":33333,\"ip_address\":\"198.51.100.6\",\"type\":\"user\"},\"organization_id\":123,\"event\":{\"meta\":null,\"target_type\":\"Candidate or Prospect rejected\",\"type\":\"action\",\"rejection\":{\"error\":\"ambiguous: 2 RejectionDetails events share request.id 7891eID (likely a bulk rejection)\",\"ambiguous_application_ids\":[\"111111\",\"222222\"]}},\"event_time\":\"2023-06-05T09:05:00.000Z\"}", "type": [ "info" ] @@ -434,7 +434,7 @@ "111111", "222222" ], - "error": "ambiguous match: 2 rejected applications found within 5m of the audit event timestamp" + "error": "ambiguous: 2 RejectionDetails events share request.id 7891eID (likely a bulk rejection)" }, "target_type": "Candidate or Prospect rejected", "type": "action" @@ -475,6 +475,71 @@ "full_name": "Recruiter Two", "id": "33333" } + }, + { + "@timestamp": "2023-06-05T09:00:00.048Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "action": "data_change_create", + "category": [ + "configuration", + "iam" + ], + "id": "7890dID", + "kind": "event", + "original": "{\"event\":{\"meta\":{\"application_id\":293344484002,\"id\":221864428002},\"target_id\":221864428002,\"target_type\":\"RejectionDetails\",\"type\":\"data_change_create\"},\"event_time\":\"2023-06-05T09:00:00.048Z\",\"organization_id\":123,\"performer\":{\"id\":22222,\"ip_address\":\"198.51.100.5\",\"meta\":{\"name\":\"Recruiter One\",\"username\":\"recruiter.one@omniva-corp.com\"},\"type\":\"user\"},\"request\":{\"id\":\"7890dID\",\"type\":\"people#change_application_state\"}}", + "type": [ + "creation" + ] + }, + "greenhouse": { + "audit": { + "event": { + "meta": { + "application_id": 293344484002, + "id": 221864428002 + }, + "target_id": "221864428002", + "target_type": "RejectionDetails", + "type": "data_change_create" + }, + "performer": { + "meta": {}, + "type": "user" + }, + "request": { + "id": "7890dID", + "type": "people#change_application_state" + } + } + }, + "organization": { + "id": "123" + }, + "related": { + "ip": [ + "198.51.100.5" + ], + "user": [ + "22222", + "recruiter.one@omniva-corp.com" + ] + }, + "source": { + "ip": "198.51.100.5" + }, + "tags": [ + "preserve_original_event", + "forwarded", + "greenhouse-audit" + ], + "user": { + "email": "recruiter.one@omniva-corp.com", + "full_name": "Recruiter One", + "id": "22222" + } } ] } diff --git a/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs b/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs index 85ef3d0d0c8..f2a2b3f4faf 100644 --- a/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs +++ b/packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs @@ -20,7 +20,6 @@ state: client_id: {{client_id}} client_secret: {{client_secret}} enrich_rejections: {{enrich_rejections}} - rejection_match_window: {{rejection_match_window}} {{#if user_id}} user_id: "{{user_id}}" {{/if}} @@ -58,37 +57,40 @@ Rejection enrichment (optional, enrich_rejections): A candidate/prospect rejection is audited as an "action" event with event.target_type == "Candidate or Prospect rejected" and - event.meta == null — Greenhouse does NOT include an application or - candidate ID on this event, so the rejected Application cannot be - looked up directly by ID. Instead, the rejection reason and notes are - correlated by time: the Harvest API is queried for Applications whose - last_activity/rejected_at falls within rejection_match_window of the - audit event's event_time, and the result is merged into the event as - event.rejection before it is emitted. This reuses the same bearer - token as the audit log request, so the OAuth client must also be - granted read access to these Harvest V3 endpoints. + event.meta == null — this event carries no application or candidate + ID. However, the same API request also audits a second, separate + event with event.target_type == "RejectionDetails" and + event.type == "data_change_create", whose event.meta.application_id + gives the Application ID. Both events share the same request.id, so + within a single page of audit results, a rejection event is + correlated to its RejectionDetails sibling by matching request.id, + and the resulting application_id is used to look up the rejection + reason and notes from the Harvest API. The result is merged into the + rejection event as event.rejection before it is emitted. This reuses + the same bearer token as the audit log request, so the OAuth client + must also be granted read access to these Harvest V3 endpoints. - GET https://harvest.greenhouse.io/v3/applications - ?last_activity_after= - &last_activity_before= - - Assumed to return a bare JSON array of Application objects (as - other Harvest list endpoints do), each optionally carrying - rejection_reason {id, name, type}, rejected_at, and candidate_id. - Results are filtered client-side to status == "rejected" and - rejected_at inside the window. - - Exactly one match: proceed to fetch its activity feed below. - - Zero matches: event.rejection.error notes no match was found; - widen rejection_match_window if this happens often. - - More than one match (e.g. several rejections in the same - window): no attempt is made to guess which one is correct. - event.rejection.error records the ambiguity and - event.rejection.ambiguous_application_ids lists the candidates - for manual follow-up. + Correlation by request.id: + - Exactly one RejectionDetails event shares the rejection event's + request.id: proceed to the Harvest API lookups below. + - None found: event.rejection.error records that no matching + RejectionDetails event was found for that request.id. This can + happen if the two events land on different audit-log pages; + increasing batch_size reduces how often pages split a request. + - More than one found (a bulk-reject action can audit multiple + "Candidate or Prospect rejected" and "RejectionDetails" events + under the same shared request.id, with no way to tell which + pairs with which): no attempt is made to guess. event.rejection + .error records the ambiguity and + event.rejection.ambiguous_application_ids lists the candidates. - GET https://harvest.greenhouse.io/v3/applications/{id}/activity_feed - - Only called for a single, unambiguous match. Provides the - "Candidate Rejected" activity, whose body holds the rejection - notes/comments entered at rejection time. + GET https://harvest.greenhouse.io/v3/applications/{application_id} + - Provides rejection_reason {id, name, type}, candidate_id, and + rejected_at. + + GET https://harvest.greenhouse.io/v3/applications/{application_id}/activity_feed + - Provides the "Candidate Rejected" activity, whose body holds the + rejection notes/comments entered at rejection time. A failed or inconclusive lookup does not drop the underlying audit event: it is emitted with event.rejection.error describing why no @@ -211,125 +213,122 @@ program: |- ).as(next_max_event_time, { "auth_token": auth.auth_token, - "events": results.map(e, - ( - state.enrich_rejections && - e.?event.type.orValue("") == "action" && - e.?event.target_type.orValue("") == "Candidate or Prospect rejected" - ).as(is_rejection, - {"message": ( - is_rejection ? - timestamp(e.event_time).as(event_ts, - duration(state.rejection_match_window).as(window, - request( - "GET", - state.harvest_url.trim_right("/") + "/v3/applications?" + { - "last_activity_after": [(event_ts - window).format(time_layout.RFC3339)], - "last_activity_before": [(event_ts + window).format(time_layout.RFC3339)], - }.format_query() - ).with( - {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} - ).do_request().as(list_resp, - (list_resp.StatusCode == 200) ? - list_resp.Body.decode_json().as(apps, - apps.filter(a, - a.?status.orValue("") == "rejected" && - a.?rejected_at.hasValue() && - timestamp(a.rejected_at) >= event_ts - window && - timestamp(a.rejected_at) <= event_ts + window - ).as(candidates, - (candidates.size() == 1) ? - candidates[0].as(app, - request( - "GET", - state.harvest_url.trim_right("/") + "/v3/applications/" + string(app.id) + "/activity_feed" - ).with( - {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} - ).do_request().as(feed_resp, - (feed_resp.StatusCode == 200) ? - feed_resp.Body.decode_json().as(feed, - feed.?activities.orValue([]).filter(a2, - a2.?subject.orValue("") == "Candidate Rejected" - ).as(rejection_activities, - e.with({ - "event": e.event.with({ - "rejection": { - "application_id": string(app.id), - ?"candidate_id": (app.?candidate_id.orValue(null) == null) ? - optional.none() - : - optional.of(string(app.candidate_id)), - ?"reason": (app.?rejection_reason.orValue(null) == null) ? - optional.none() - : - optional.of({ - "id": string(app.rejection_reason.id), - "name": app.rejection_reason.name, - "type": app.rejection_reason.?type.name.orValue(""), - }), - ?"notes": (size(rejection_activities) > 0) ? - rejection_activities[0].?body - : - optional.none(), - ?"rejected_at": app.?rejected_at, - }, - }), - }) - ) + "events": results.filter(rd, + rd.?event.target_type.orValue("") == "RejectionDetails" && + rd.?event.type.orValue("") == "data_change_create" && + rd.?event.meta.application_id.hasValue() + ).as(rejection_details_events, + results.map(e, + ( + state.enrich_rejections && + e.?event.type.orValue("") == "action" && + e.?event.target_type.orValue("") == "Candidate or Prospect rejected" + ).as(is_rejection, + {"message": ( + is_rejection ? + rejection_details_events.filter(rde, + rde.?request.id.orValue("") == e.?request.id.orValue("") + ).as(matches, + (matches.size() == 1) ? + string(matches[0].event.meta.application_id).as(application_id, + request( + "GET", + state.harvest_url.trim_right("/") + "/v3/applications/" + application_id + ).with( + {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} + ).do_request().as(app_resp, + (app_resp.StatusCode == 200) ? + app_resp.Body.decode_json().as(app, + request( + "GET", + state.harvest_url.trim_right("/") + "/v3/applications/" + application_id + "/activity_feed" + ).with( + {"Header": {"Authorization": ["Bearer " + auth.auth_token.token]}} + ).do_request().as(feed_resp, + (feed_resp.StatusCode == 200) ? + feed_resp.Body.decode_json().as(feed, + feed.?activities.orValue([]).filter(a, + a.?subject.orValue("") == "Candidate Rejected" + ).as(rejection_activities, + e.with({ + "event": e.event.with({ + "rejection": { + "application_id": application_id, + ?"candidate_id": (app.?candidate_id.orValue(null) == null) ? + optional.none() + : + optional.of(string(app.candidate_id)), + ?"reason": (app.?rejection_reason.orValue(null) == null) ? + optional.none() + : + optional.of({ + "id": string(app.rejection_reason.id), + "name": app.rejection_reason.name, + "type": app.rejection_reason.?type.name.orValue(""), + }), + ?"notes": (size(rejection_activities) > 0) ? + rejection_activities[0].?body + : + optional.none(), + ?"rejected_at": app.?rejected_at, + }, + }), + }) ) - : - e.with({ - "event": e.event.with({ - "rejection": { - "application_id": string(app.id), - ?"reason": (app.?rejection_reason.orValue(null) == null) ? - optional.none() - : - optional.of({ - "id": string(app.rejection_reason.id), - "name": app.rejection_reason.name, - "type": app.rejection_reason.?type.name.orValue(""), - }), - ?"rejected_at": app.?rejected_at, - "error": "activity_feed fetch failed: " + string(feed_resp.StatusCode), - }, - }), - }) - ) + ) + : + e.with({ + "event": e.event.with({ + "rejection": { + "application_id": application_id, + ?"reason": (app.?rejection_reason.orValue(null) == null) ? + optional.none() + : + optional.of({ + "id": string(app.rejection_reason.id), + "name": app.rejection_reason.name, + "type": app.rejection_reason.?type.name.orValue(""), + }), + ?"rejected_at": app.?rejected_at, + "error": "activity_feed fetch failed: " + string(feed_resp.StatusCode), + }, + }), + }) ) - : (candidates.size() == 0) ? - e.with({ - "event": e.event.with({ - "rejection": { - "error": "no rejected application found within " + state.rejection_match_window + " of the audit event timestamp", - }, - }), - }) - : - e.with({ - "event": e.event.with({ - "rejection": { - "error": "ambiguous match: " + string(candidates.size()) + " rejected applications found within " + state.rejection_match_window + " of the audit event timestamp", - "ambiguous_application_ids": candidates.map(a3, string(a3.id)), - }, - }), - }) - ) + ) + : + e.with({ + "event": e.event.with({ + "rejection": { + "application_id": application_id, + "error": "application fetch failed: " + string(app_resp.StatusCode), + }, + }), + }) ) - : - e.with({ - "event": e.event.with({ - "rejection": { - "error": "applications lookup failed: " + string(list_resp.StatusCode), - }, - }), - }) - ) + ) + : (matches.size() == 0) ? + e.with({ + "event": e.event.with({ + "rejection": { + "error": "no RejectionDetails audit event found sharing request.id " + e.?request.id.orValue(""), + }, + }), + }) + : + e.with({ + "event": e.event.with({ + "rejection": { + "error": "ambiguous: " + string(matches.size()) + " RejectionDetails events share request.id " + e.?request.id.orValue("") + " (likely a bulk rejection)", + "ambiguous_application_ids": matches.map(m, string(m.event.meta.application_id)), + }, + }), + }) ) - ) - : - e - ).encode_json()} + : + e + ).encode_json()} + ) ) ), "cursor": { diff --git a/packages/greenhouse/data_stream/audit/fields/fields.yml b/packages/greenhouse/data_stream/audit/fields/fields.yml index 3d8650b5e16..01267c94beb 100644 --- a/packages/greenhouse/data_stream/audit/fields/fields.yml +++ b/packages/greenhouse/data_stream/audit/fields/fields.yml @@ -74,12 +74,12 @@ - name: rejection type: group description: > - Rejection details looked up from the Harvest API for a "Candidate or Prospect rejected" audit event, correlated by timestamp since the audit event does not include an application ID. Only populated when the `enrich_rejections` setting is enabled. + Rejection details looked up from the Harvest API for a "Candidate or Prospect rejected" audit event. This audit event does not include an application ID; it is obtained by correlating a sibling "RejectionDetails" audit event that shares the same request.id. Only populated when the `enrich_rejections` setting is enabled. fields: - name: application_id type: keyword description: > - The ID of the Application matched to this rejection event by timestamp correlation. + The ID of the Application matched to this rejection event via its sibling "RejectionDetails" audit event. - name: candidate_id type: keyword description: > @@ -87,7 +87,7 @@ - name: ambiguous_application_ids type: keyword description: > - When timestamp correlation matches more than one rejected Application, the IDs of all matching candidates. No reason or notes are attached in this case. + When more than one "RejectionDetails" audit event shares the rejection event's request.id (for example, a bulk rejection), the Application IDs of all of them. No reason or notes are attached in this case, since it is not possible to tell which one corresponds to this rejection event. - name: reason type: group description: > diff --git a/packages/greenhouse/data_stream/audit/manifest.yml b/packages/greenhouse/data_stream/audit/manifest.yml index c03ed01d6fc..70c9f5ee360 100644 --- a/packages/greenhouse/data_stream/audit/manifest.yml +++ b/packages/greenhouse/data_stream/audit/manifest.yml @@ -53,16 +53,7 @@ streams: show_user: true default: false description: >- - When an audit event records a candidate or prospect being rejected, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event. Since this audit event does not include an application ID, the matching Application is found by correlating its rejected_at time to the audit event's timestamp (see "Rejection Match Window"); if that correlation is not exact, the event is emitted with an error instead of a guess. This requires the OAuth client to also have read access to the Harvest "Applications" and "Activity Feed" endpoints, and results in up to two additional API calls per rejection event. - - name: rejection_match_window - type: text - title: Rejection Match Window - description: >- - Only used when "Enrich rejected application events" is enabled. Time window, centered on the audit event's timestamp, used to find the matching rejected Application via the Harvest API. Valid time units are s, m, h. Widen this if enrichment often reports no match found; narrow it if it often reports an ambiguous match. - multi: false - required: false - show_user: true - default: 5m + When an audit event records a candidate or prospect being rejected, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event. This audit event does not include an application ID, but Greenhouse also audits a separate "RejectionDetails" event under the same request.id that does; the Application is identified by correlating the two via request.id, and if that correlation is not exact, the event is emitted with an error instead of a guess. This requires the OAuth client to also have read access to the Harvest "Applications" and "Activity Feed" endpoints, and results in up to two additional API calls per rejection event. - name: enable_request_tracer type: bool title: Enable request tracing diff --git a/packages/greenhouse/docs/README.md b/packages/greenhouse/docs/README.md index 4b52cda4c20..47f885139d8 100644 --- a/packages/greenhouse/docs/README.md +++ b/packages/greenhouse/docs/README.md @@ -57,20 +57,23 @@ To find a user's ID: - **Performer IDs Filter**: Filter by specific user IDs - **Event Types Filter**: Filter by event type (data_change_update, data_change_create, data_change_destroy, harvest_access, action) - **Enrich rejected application events**: When enabled, look up the rejection reason and rejection notes/comments from the Harvest API and add them to the event (default: disabled) - - **Rejection Match Window**: Time window used to correlate a rejection event to its Application (default: 5m, only used when enrichment is enabled) ### Enabling rejection enrichment (optional) When a candidate or prospect is rejected, Greenhouse audits it as an `action` event with `event.target_type` set to the literal string `Candidate or Prospect rejected` — this event does **not** include an application or candidate ID, and does not include the rejection reason or notes/comments entered at rejection time. Enabling **Enrich rejected application events** looks these up from the Harvest API and correlates them back to the audit event. -1. On the same Harvest V3 (OAuth) API credential used for audit log access, add read scopes for the **Applications** and **Activity Feed** endpoints. +1. On the same Harvest V3 (OAuth) API credential used for audit log access, grant the following read permissions (under **Configure > Dev Center > API Credentials** on the credential): + - **Rejection details** → *List rejection details* + - **Rejection reasons** → *List rejection reasons* and *Show rejection reasons* + - **Applications** → read access, needed for the `GET /v3/applications/{id}` lookup this integration performs + - **Activity Feed** → read access, needed for the `GET /v3/applications/{id}/activity_feed` lookup this integration performs to retrieve the rejection notes 2. Enable the **Enrich rejected application events** setting on the integration. -Because the audit event has no application ID, the integration looks up Applications whose rejection time falls within **Rejection Match Window** of the audit event's timestamp: +Because the "Candidate or Prospect rejected" audit event has no application ID, the integration correlates it to a second audit event that Greenhouse records for the same rejection — one with `event.target_type` of `RejectionDetails` — by matching the `request.id` shared by both events. That event's `event.meta.application_id` is then used to look up the rejection reason and notes: -- **Exactly one match**: the rejection reason and notes are fetched and added to the event under `greenhouse.audit.event.rejection` (`application_id`, `candidate_id`, `reason.id`/`reason.name`/`reason.type`, `notes`, `rejected_at`). The notes are also copied to `event.reason`. -- **No match**: the audit event is still indexed, with `greenhouse.audit.event.rejection.error` explaining that no matching Application was found. Consider widening **Rejection Match Window**. -- **More than one match** (for example, several rejections processed in the same window): the integration does not guess. The audit event is indexed with `greenhouse.audit.event.rejection.error` and `greenhouse.audit.event.rejection.ambiguous_application_ids` listing the candidates, for manual follow-up. Consider narrowing **Rejection Match Window** if this happens often. +- **Exactly one matching `RejectionDetails` event**: the rejection reason and notes are fetched and added to the event under `greenhouse.audit.event.rejection` (`application_id`, `candidate_id`, `reason.id`/`reason.name`/`reason.type`, `notes`, `rejected_at`). The notes are also copied to `event.reason`. +- **No matching event found**: the audit event is still indexed, with `greenhouse.audit.event.rejection.error` explaining that no matching `RejectionDetails` event was found for that request. This can happen if the two events land on different pages of a poll; increasing **Batch Size** reduces how often this occurs. +- **More than one matching event** (for example, a bulk rejection of several candidates in one action, which shares a single `request.id` across all of them): the integration does not guess which pairs with which. The audit event is indexed with `greenhouse.audit.event.rejection.error` and `greenhouse.audit.event.rejection.ambiguous_application_ids` listing the candidates, for manual follow-up. Any of the error cases above also adds the tag `greenhouse-rejection-enrichment-failed`, and the underlying rejection audit event is never dropped. @@ -115,8 +118,10 @@ If no events are being collected: ### Rejection Enrichment Errors If rejection events are tagged with `greenhouse-rejection-enrichment-failed` and `greenhouse.audit.event.rejection.error` is populated: -1. Verify the OAuth credential has read scopes for the Harvest **Applications** and **Activity Feed** endpoints +1. Verify the OAuth credential has been granted **Rejection details** (*List rejection details*), **Rejection reasons** (*List rejection reasons*, *Show rejection reasons*), **Applications**, and **Activity Feed** read permissions 2. Check that the authorizing user has permission to view the affected application +3. If the error mentions no matching `RejectionDetails` event was found, try increasing **Batch Size** so the two related audit events are less likely to land on different pages +4. If the error mentions an ambiguous match, the rejection was likely part of a bulk-reject action; `greenhouse.audit.event.rejection.ambiguous_application_ids` lists the candidates involved for manual follow-up ## Logs reference @@ -133,8 +138,8 @@ If rejection events are tagged with `greenhouse-rejection-enrichment-failed` and | event.dataset | Event dataset | constant_keyword | | event.module | Event module | constant_keyword | | greenhouse.audit.event.meta | The before and after values from data change events, or other relevant data for the event. | flattened | -| greenhouse.audit.event.rejection.ambiguous_application_ids | When timestamp correlation matches more than one rejected Application, the IDs of all matching candidates. No reason or notes are attached in this case. | keyword | -| greenhouse.audit.event.rejection.application_id | The ID of the Application matched to this rejection event by timestamp correlation. | keyword | +| greenhouse.audit.event.rejection.ambiguous_application_ids | When more than one "RejectionDetails" audit event shares the rejection event's request.id (for example, a bulk rejection), the Application IDs of all of them. No reason or notes are attached in this case, since it is not possible to tell which one corresponds to this rejection event. | keyword | +| greenhouse.audit.event.rejection.application_id | The ID of the Application matched to this rejection event via its sibling "RejectionDetails" audit event. | keyword | | greenhouse.audit.event.rejection.candidate_id | The ID of the candidate whose application was rejected. | keyword | | greenhouse.audit.event.rejection.error | Error message if the rejection enrichment lookup against the Harvest API failed. | keyword | | greenhouse.audit.event.rejection.notes | The rejection notes/comments entered when the application was rejected, sourced from the Harvest API activity feed. | match_only_text | @@ -239,7 +244,7 @@ An example event for `audit` looks as following: } ``` -When **Enrich rejected application events** is enabled, a `Candidate or Prospect rejected` audit event is enriched with a `greenhouse.audit.event.rejection` object, correlated to its Application by timestamp: +When **Enrich rejected application events** is enabled, a `Candidate or Prospect rejected` audit event is enriched with a `greenhouse.audit.event.rejection` object, correlated to its Application via a sibling `RejectionDetails` audit event sharing the same `request.id`: ```json { @@ -280,7 +285,7 @@ If the correlation is inconclusive, `greenhouse.audit.event.rejection` instead c "111111", "222222" ], - "error": "ambiguous match: 2 rejected applications found within 5m of the audit event timestamp" + "error": "ambiguous: 2 RejectionDetails events share request.id 7891eID (likely a bulk rejection)" }, "target_type": "Candidate or Prospect rejected", "type": "action"