[release-6.5]LOG-8697:Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes by vparfonov · Pull Request #3244 · openshift/cluster-logging-operator

vparfonov · 2026-04-03T08:40:35Z

Description

backport #3226

/cc @cahartma @Clee2691
/assign @jcantrill

Links

Depending on PR(s):
GitHub issue:
JIRA: https://redhat.atlassian.net/browse/LOG-8697
Enhancement proposal:

@timestamp

…ernetes event logs EventRouter updates existing Event objects instead of creating new ones. The original metadata.creationTimestamp may therefore be days/weeks old while lastTimestamp reflects the most recent occurrence. Using creationTimestamp caused Loki (others sinks possibly) to reject records with "greater_than_max_sample_age". Use lastTimestamp for @timestamp and timestamp fields so event updates are ingested with the correct time. The fallback chain is: lastTimestamp -> firstTimestamp -> eventTime -> creationTimestamp

vparfonov · 2026-04-03T08:40:44Z

/hold

openshift-ci · 2026-04-03T08:40:55Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vparfonov
Once this PR has been reviewed and has the lgtm label, please assign alanconway for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

qodo-code-review · 2026-04-03T08:40:55Z

Review Summary by Qodo

Use event.lastTimestamp for Kubernetes event log timestamps

🐞 Bug fix

Walkthroughs

Description

• Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes event logs
• Implement timestamp fallback chain: lastTimestamp → firstTimestamp → eventTime → creationTimestamp
• Set both @timestamp and timestamp fields with correct event time
• Add Loki output support and timestamp validation tests

Diagram

flowchart LR
  A["EventRouter Events"] -->|"Extract timestamp"| B["Timestamp Fallback Chain"]
  B -->|"lastTimestamp available"| C["Use lastTimestamp"]
  B -->|"lastTimestamp null/empty"| D["Try firstTimestamp"]
  D -->|"firstTimestamp null/empty"| E["Try eventTime"]
  E -->|"eventTime null/empty"| F["Use creationTimestamp"]
  C --> G["Set @timestamp and timestamp fields"]
  D --> G
  E --> G
  F --> G
  G -->|"Prevent Loki rejection"| H["Correct log ingestion"]

File Changes

1. internal/generator/vector/filter/openshift/viaq/v1/normalize.go 🐞 Bug fix +28/-17

Implement timestamp fallback chain for EventRouter logs

• Replaced single creationTimestamp assignment with timestamp fallback chain logic
• Added checks for null, empty string, and zero-value timestamps ("0001-01-01T00:00:00Z")
• Set both ._internal."@timestamp" and ._internal.timestamp fields with determined timestamp
• Fixed indentation and whitespace formatting throughout the file

internal/generator/vector/filter/openshift/viaq/v1/normalize.go

2. test/functional/normalization/eventrouter_test.go 🧪 Tests +95/-20

Add Loki output tests and timestamp fallback validation

• Replaced hardcoded timestamp with dynamic time.Now().UTC() for test execution time
• Added Loki receiver support and output type handling in test framework
• Enhanced ExpectedLogTemplateBuilder to handle different output types with conditional timestamp
 expectations
• Added new test table for timestamp fallback chain validation with four scenarios
• Updated event data builder to set FirstTimestamp and LastTimestamp on Kubernetes events
• Added Loki query logic to retrieve logs from Loki receiver instead of framework logs

test/functional/normalization/eventrouter_test.go

3. test/helpers/loki/receiver.go ✨ Enhancement +3/-0

Add time range to Loki query parameters

• Added time range parameters to Loki query: 24-hour window around current time
• Set start and end query parameters in Unix nanosecond format

test/helpers/loki/receiver.go

View more (3)

4. internal/generator/vector/conf/complex.toml 🐞 Bug fix +15/-1

Implement timestamp fallback chain in complex config

• Replaced single creationTimestamp assignment with timestamp fallback chain logic
• Added checks for null, empty string, and zero-value timestamps
• Set both @timestamp and timestamp fields with determined timestamp value

internal/generator/vector/conf/complex.toml

5. internal/generator/vector/conf/complex_http_receiver.toml 🐞 Bug fix +15/-1

Implement timestamp fallback chain in HTTP receiver config
• Replaced single creationTimestamp assignment with timestamp fallback chain logic
• Added checks for null, empty string, and zero-value timestamps
• Set both @timestamp and timestamp fields with determined timestamp value
internal/generator/vector/conf/complex_http_receiver.toml

6. internal/generator/vector/conf/container.toml 🐞 Bug fix +15/-1

Implement timestamp fallback chain in container config

• Replaced single creationTimestamp assignment with timestamp fallback chain logic
• Added checks for null, empty string, and zero-value timestamps
• Set both @timestamp and timestamp fields with determined timestamp value

internal/generator/vector/conf/container.toml

qodo-code-review · 2026-04-03T08:40:56Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX Issues (0)

1. Null timestamp clobbers time 🐞 Bug ≡ Correctness

Description

The EventRouter timestamp selection unconditionally assigns ._internal.timestamp = ts even when
all candidate event timestamp fields are null/empty, which can erase the original CRI/source
timestamp and emit records with null/zero @timestamp/timestamp. This can break downstream
time-based indexing/sorting and violates the prior behavior where the CRI timestamp would have
remained intact if the event lacked timestamps.

Code

internal/generator/vector/filter/openshift/viaq/v1/normalize.go[R22-33]

+      ts = ._internal.kubernetes.event.lastTimestamp
+      if ts == null || ts == "" || ts == "0001-01-01T00:00:00Z" {
+        ts = ._internal.kubernetes.event.firstTimestamp
+      }
+      if ts == null || ts == "" || ts == "0001-01-01T00:00:00Z" {
+        ts = ._internal.kubernetes.event.eventTime
+      }
+      if ts == null || ts == "" {
+        ts = ._internal.kubernetes.event.metadata.creationTimestamp
+      }
+      ._internal."@timestamp" = ts
+      ._internal.timestamp = ts

Evidence
HandleEventRouterLog computes ts from event fields and then always sets `._internal.timestamp =
ts without a final guard, so ts == null results in ._internal.timestamp` becoming null. The
container pipeline later derives root .timestamp and .@timestamp directly from
._internal.timestamp, meaning a null overwrite propagates to the emitted record. Repository
fixtures demonstrate EventRouter payloads where all relevant event timestamp fields can be null,
making this path realistic.
internal/generator/vector/filter/openshift/viaq/v1/normalize.go[19-33]
internal/generator/vector/conf/container.toml[35-39]
internal/generator/vector/conf/container.toml[370-371]
test/helpers/types/types_test.go[295-321]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
EventRouter timestamp fallback assigns `._internal.timestamp = ts` even when `ts` remains `null`/empty after checking `lastTimestamp`, `firstTimestamp`, `eventTime`, and `metadata.creationTimestamp`. This can overwrite the original container log timestamp (from the kubernetes_logs source) and lead to output records with missing/zero `@timestamp` and `timestamp`.

### Issue Context
The Vector pipeline ultimately sets root `.timestamp` and `.@timestamp` from `._internal.timestamp`, so clobbering it with null propagates to all sinks.

### Fix Focus Areas
- internal/generator/vector/filter/openshift/viaq/v1/normalize.go[19-33]
- internal/generator/vector/conf/complex.toml[722-736]
- internal/generator/vector/conf/complex_http_receiver.toml[760-774]
- internal/generator/vector/conf/container.toml[313-327]

### Suggested change
After the fallback chain, add a final safeguard:
- If `ts` is still `null`/""/"0001-01-01T00:00:00Z", fall back to the pre-existing `._internal.timestamp` (the CRI/source timestamp) or `now()`.
- Only override `._internal.timestamp` (and any related fields) when `ts` is valid.

Example VRL pattern:
```vrl
# ...existing fallback chain...
if ts == null || ts == "" || ts == "0001-01-01T00:00:00Z" {
 ts = ._internal.timestamp
}
if ts != null && ts != "" && ts != "0001-01-01T00:00:00Z" {
 ._internal.timestamp = ts
 ._internal."@timestamp" = ts
}
```

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

ⓘ The new review experience is currently in Beta. Learn more

qodo-code-review · 2026-04-03T08:47:33Z

+      ts = ._internal.kubernetes.event.lastTimestamp
+      if ts == null || ts == "" || ts == "0001-01-01T00:00:00Z" {
+        ts = ._internal.kubernetes.event.firstTimestamp
+      }
+      if ts == null || ts == "" || ts == "0001-01-01T00:00:00Z" {
+        ts = ._internal.kubernetes.event.eventTime
+      }
+      if ts == null || ts == "" {
+        ts = ._internal.kubernetes.event.metadata.creationTimestamp
+      }
+      ._internal."@timestamp" = ts
+      ._internal.timestamp = ts


1. Null timestamp clobbers time 🐞 Bug ≡ Correctness

The EventRouter timestamp selection unconditionally assigns ._internal.timestamp = ts even when all candidate event timestamp fields are null/empty, which can erase the original CRI/source timestamp and emit records with null/zero @timestamp/timestamp. This can break downstream time-based indexing/sorting and violates the prior behavior where the CRI timestamp would have remained intact if the event lacked timestamps.

Agent Prompt

### Issue description EventRouter timestamp fallback assigns `._internal.timestamp = ts` even when `ts` remains `null`/empty after checking `lastTimestamp`, `firstTimestamp`, `eventTime`, and `metadata.creationTimestamp`. This can overwrite the original container log timestamp (from the kubernetes_logs source) and lead to output records with missing/zero `@timestamp` and `timestamp`. ### Issue Context The Vector pipeline ultimately sets root `.timestamp` and `.@timestamp` from `._internal.timestamp`, so clobbering it with null propagates to all sinks. ### Fix Focus Areas - internal/generator/vector/filter/openshift/viaq/v1/normalize.go[19-33] - internal/generator/vector/conf/complex.toml[722-736] - internal/generator/vector/conf/complex_http_receiver.toml[760-774] - internal/generator/vector/conf/container.toml[313-327] ### Suggested change After the fallback chain, add a final safeguard: - If `ts` is still `null`/""/"0001-01-01T00:00:00Z", fall back to the pre-existing `._internal.timestamp` (the CRI/source timestamp) or `now()`. - Only override `._internal.timestamp` (and any related fields) when `ts` is valid. Example VRL pattern: ```vrl # ...existing fallback chain... if ts == null || ts == "" || ts == "0001-01-01T00:00:00Z" { ts = ._internal.timestamp } if ts != null && ts != "" && ts != "0001-01-01T00:00:00Z" { ._internal.timestamp = ts ._internal."@timestamp" = ts } ```

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

vparfonov · 2026-04-07T08:08:16Z

/retest

openshift-ci-robot · 2026-04-07T08:09:39Z

@vparfonov: This pull request references LOG-8697 which is a valid jira issue.

Details

In response to this:

Description

backport #3226

/cc @cahartma @Clee2691
/assign @jcantrill

Links

Depending on PR(s):

GitHub issue:

JIRA: https://redhat.atlassian.net/browse/LOG-8697

Enhancement proposal:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

vparfonov · 2026-04-07T08:10:10Z

/hold cancel

vparfonov · 2026-04-07T09:42:01Z

/test e2e-target

qodo-code-review · 2026-04-07T09:42:06Z

PR-Agent: could not fine a component named e2e-target in a supported language in this PR.

openshift-ci · 2026-04-07T12:03:34Z

@vparfonov: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 3, 2026

openshift-ci bot requested review from cahartma and jcantrill April 3, 2026 08:40

qodo-code-review bot reviewed Apr 3, 2026

View reviewed changes

vparfonov changed the title ~~Use event.lastTimestamp instead of metadata.creationTimestamp for Kub…~~ [release-6.5]LOG-8697:Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes Apr 7, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 7, 2026

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release-6.5]LOG-8697:Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes#3244

[release-6.5]LOG-8697:Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes#3244
vparfonov wants to merge 1 commit intoopenshift:release-6.5from
vparfonov:log8697-6.5

vparfonov commented Apr 3, 2026 •

edited

Loading

Uh oh!

vparfonov commented Apr 3, 2026

Uh oh!

openshift-ci bot commented Apr 3, 2026

Uh oh!

qodo-code-review bot commented Apr 3, 2026

Uh oh!

qodo-code-review bot commented Apr 3, 2026 •

edited

Loading

Uh oh!

qodo-code-review bot Apr 3, 2026

Uh oh!

vparfonov commented Apr 7, 2026

Uh oh!

openshift-ci-robot commented Apr 7, 2026 •

edited by openshift-ci bot

Loading

Description

Links

Uh oh!

vparfonov commented Apr 7, 2026

Uh oh!

vparfonov commented Apr 7, 2026

Uh oh!

qodo-code-review bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

openshift-ci bot commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vparfonov commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

vparfonov commented Apr 3, 2026

Uh oh!

openshift-ci bot commented Apr 3, 2026

Uh oh!

qodo-code-review bot commented Apr 3, 2026

Review Summary by Qodo

Walkthroughs

File Changes

Uh oh!

qodo-code-review bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

qodo-code-review bot Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

vparfonov commented Apr 7, 2026

Uh oh!

openshift-ci-robot commented Apr 7, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

vparfonov commented Apr 7, 2026

Uh oh!

vparfonov commented Apr 7, 2026

Uh oh!

qodo-code-review bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vparfonov commented Apr 3, 2026 •

edited

Loading

qodo-code-review bot commented Apr 3, 2026 •

edited

Loading

openshift-ci-robot commented Apr 7, 2026 •

edited by openshift-ci bot

Loading

qodo-code-review bot commented Apr 7, 2026 •

edited

Loading