Enable PHP FFE evaluation metric system tests by leoromanovsky · Pull Request #7033 · DataDog/system-tests

leoromanovsky · 2026-05-28T13:39:46Z

Motivation

PHP FFE evaluation metrics need system-test coverage before DataDog/dd-trace-php#3911 can be treated as review-ready. The PHP runtime now records feature_flag.evaluations through native/libdatadog-sidecar code, so we need the shared FFE metric tests enabled against a locally built PHP tracer artifact rather than relying only on PHPT coverage.

Design doc: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0

Changes

This PR enables tests/ffe/test_flag_eval_metrics.py for PHP at v1.21.0-dev in the manifest only. It intentionally does not enable exposure tests; exposure validation is tracked separately in #7031 so each PHP milestone can be reviewed independently.

The PR is stacked on #7003, which adds the PHP FFE scaffold and evaluation-test enablement. The PHP behavior being validated here is implemented in DataDog/dd-trace-php#3911 and the sidecar metric delivery support is implemented in DataDog/libdatadog#2052.

Decisions

The scope is deliberately one manifest activation. The shared metric tests cover the expected feature_flag.evaluations behavior, including successful evaluations, missing flags, malformed/empty RC handling, disabled flags, type mismatches, targeting keys, and allocation metadata.

The local proof uses a matched PHP 8.2 NTS artifact and the php-fpm-8.2 weblog, which matches the PHP 8.2 target Bob recommended for FFE system-test iteration.

Related PRs

PHP runtime evaluation base: feat(ffe): add runtime-backed PHP feature flag evaluation dd-trace-php#3906
PHP metric implementation: Add FFE evaluation metrics dd-trace-php#3911
System-tests PHP FFE scaffold/evaluations base: Enable PHP FFE evaluation system tests #7003
Sidecar metric delivery: feat(sidecar): forward FFE evaluation metrics to OTLP intake libdatadog#2052

Validation

Local behavior validation before this conflict-resolution rebase (previous system-tests PR head a412f4f6), dd-trace-php metric branch now pushed as dc19ce479, and libdatadog submodule 96d9a7bae:

cd /Users/leo.romanovsky/go/src/github.com/DataDog/dd-trace-php-ffe-metrics-restack
./tooling/bin/build-debug-artifact gnu-aarch64-8.2-nts \
  /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests-pr7033/binaries

cd /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests-pr7033
DOCKER_CONFIG=/tmp/system-tests-docker-config-nocredsstore \
SYSTEM_TEST_BUILD_TIMEOUT=1800 \
./build.sh --library php --weblog-variant php-fpm-8.2

DOCKER_CONFIG=/tmp/system-tests-docker-config-nocredsstore \
TEST_LIBRARY=php \
WEBLOG_VARIANT=php-fpm-8.2 \
./run.sh +l php FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_metrics.py

Result: 17 passed in 81.28s.

Post-rebase verification: branch head is now fbc98222a, stacked on the updated parent, enabling tests/ffe/test_flag_eval_metrics.py at v1.21.0-dev to match #7003. PATH=/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH ./format.sh passed. The rebase only changed the manifest activation floor; the test behavior and local PHP artifact path above did not change.

MacOS arm64 local-run note: the published PHP 8.2 base image inspected locally was amd64-only, so the local validation reused the locally built arm64 datadog/system-tests:php-fpm-8.2.base-v1. This PR remains only the manifest activation.

github-actions · 2026-05-28T13:40:25Z

CODEOWNERS have been resolved as:

manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php

datadog-prod-us1-5 · 2026-05-28T13:48:45Z

Tests

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 26 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.0 1

See error
17 failed tests due to assertion failures on missing feature_flag.evaluations metrics for multiple flags in tests/ffe/test_flag_eval_metrics.py.
🧪 1 Test failed
tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.0] from system_tests_suite (Fix with Cursor)
AssertionError: Expected metric for non-existent flag &#39;non-existent-eval-metric-flag&#39;, found none. All: []
assert 0 &gt; 0
 &#43;  where 0 = len([])

self = &lt;tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7f9448900d40&gt;

    def test_ffe_eval_config_exists_flag_missing(self):
        &#34;&#34;&#34;Test that missing flag (with config loaded) produces error.type=flag_not_found.&#34;&#34;&#34;
        assert self.r.status_code == 200, f&#34;Flag evaluation request failed: {self.r.text}&#34;
    
...
Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.0-zts 1

See error
17 failed tests due to AssertionError: Expected at least one feature_flag.evaluations metric for several flags, but found none.
🧪 1 Test failed
tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.0-zts] from system_tests_suite (Fix with Cursor)
AssertionError: Expected metric for non-existent flag &#39;non-existent-eval-metric-flag&#39;, found none. All: []
assert 0 &gt; 0
 &#43;  where 0 = len([])

self = &lt;tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7ff8e45b6780&gt;

    def test_ffe_eval_config_exists_flag_missing(self):
        &#34;&#34;&#34;Test that missing flag (with config loaded) produces error.type=flag_not_found.&#34;&#34;&#34;
        assert self.r.status_code == 200, f&#34;Flag evaluation request failed: {self.r.text}&#34;
    
...
Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.1 1

See error
17 failed tests due to missing metrics for various flags.
🧪 1 Test failed
tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.1] from system_tests_suite (Fix with Cursor)
AssertionError: Expected metric for non-existent flag &#39;non-existent-eval-metric-flag&#39;, found none. All: []
assert 0 &gt; 0
 &#43;  where 0 = len([])

self = &lt;tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7f73287437d0&gt;

    def test_ffe_eval_config_exists_flag_missing(self):
        &#34;&#34;&#34;Test that missing flag (with config loaded) produces error.type=flag_not_found.&#34;&#34;&#34;
        assert self.r.status_code == 200, f&#34;Flag evaluation request failed: {self.r.text}&#34;
    
...
View all 26 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 581f541 | Docs | Datadog PR Page | Give us feedback!}

## Motivation PHP FFE evaluation metrics need a native path for aggregation, OTLP encoding, and delivery without building PHP OTLP writer/transport machinery. The shared design doc is the cross-PR reference: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0 This PR is metric-only. Exposures remain in #2026 so reviewers can evaluate OTLP metric delivery independently from exposure cache semantics. ## Changes This adds caller-driven FFE evaluation metric sidecar actions and OTLP export for `feature_flag.evaluations`. The reusable FFE-domain pieces now live in `datadog-ffe` behind the `evaluation-metrics` feature: evaluation metric input types, metric attribute normalization, aggregation by matching attribute sets, and OTLP/protobuf payload encoding. `datadog-sidecar` keeps only sidecar-specific work: parsing the configured endpoint URL, building the HTTP request, applying the timeout, logging delivery failures, and integrating with sidecar lifecycle/actions. The PHP companion PR uses this from native/C code for raw `DDTrace\ffe_evaluate` calls and from a thin PHP OpenFeature adapter for final OpenFeature-aware results. PHP does not aggregate, encode, or transport OTLP payloads. Current PHP MVP path: ```mermaid flowchart LR Eval["PHP evaluation raw API or OpenFeature adapter"] Record["PHP tracer native call record typed evaluation metric"] Action["sidecar action record FFE evaluation metrics"] Domain["datadog-ffe feature: evaluation-metrics attributes + aggregation + OTLP encoder"] Sidecar["shared sidecar metric flush lifecycle"] Collector["OTLP endpoint Agent or local collector"] Intake["feature_flag.evaluations"] Eval --> Record Record --> Action Action --> Domain Domain --> Sidecar Sidecar --> Collector Collector --> Intake ``` Future Python/Ruby connection: ```mermaid flowchart LR PyToday["dd-trace-py today OpenFeature hook + host metric writer"] RbToday["dd-trace-rb today OpenFeature hook + host metric writer"] PyFuture["dd-trace-py future explicit native opt-in"] RbFuture["dd-trace-rb future explicit native opt-in"] Native["libdatadog caller-driven FFE metric action"] Shared["shared sidecar aggregation + OTLP delivery"] Otlp["OTLP endpoint"] PyToday -. "current host metric path" .-> Otlp RbToday -. "current host metric path" .-> Otlp PyFuture -. "after ownership switch" .-> Native RbFuture -. "after ownership switch" .-> Native Native --> Shared Shared --> Otlp ``` The future Python/Ruby arrows are intentionally not active behavior in this PR. They show the reusable target for a later migration while preserving today's host-language metric writers. Why Python/Ruby do not double count today: - Python and Ruby use libdatadog for evaluation only; the evaluator returns assignment metadata and does not record `feature_flag.evaluations` as a side effect. - This PR adds a separate caller-driven sidecar action. Metric emission happens only when an SDK explicitly records a typed evaluation metric into that action. PHP wires this in its companion PR; Python and Ruby do not. - Python and Ruby therefore keep exactly their current host-language OpenFeature metric writers. They are not also sending evaluation metrics through this native sidecar path. - Evaluation metrics intentionally count every evaluation and do not have exposure-cache deduplication semantics. Future Python/Ruby migration must switch ownership to native logging and disable/bypass the host metric writer for the same evaluations. Reference implementation check: dd-trace-java's canonical metric path is OpenFeature hook based. Java's `Provider` creates `FlagEvalMetrics` and returns a `FlagEvalHook`; the hook runs in `finallyAfter`, reads the final OpenFeature `FlagEvaluationDetails` including flag key, variant, reason, error code, and allocation metadata, and records one `feature_flag.evaluations` counter. Application code only calls OpenFeature; it does not call a metric API. PHP mirrors that canonical OpenFeature shape. The PHP OpenFeature provider disables raw native metric recording while it asks the native evaluator for an assignment, then records exactly one final OpenFeature-aware metric through the Datadog-owned recorder. The raw Datadog PHP client has no direct Java equivalent, but it keeps the same SDK-owned ergonomics: normal evaluation APIs record one native metric per evaluation internally. For future Python/Ruby migration, the same rule applies: either keep the existing host-language OpenFeature metric hook, or switch ownership to the native recorder and disable/bypass the host metric writer for those evaluations. ## Decisions No telemetry is emitted automatically from shared libdatadog evaluator calls. SDKs must explicitly enqueue FFE telemetry actions. This avoids double counting for Python/Ruby, which currently log feature-flag telemetry in host-language code. Evaluation metrics intentionally count evaluations and do not use exposure-cache deduplication semantics. Future Python/Ruby migration must be an ownership switch, not an additional writer. If those SDKs opt into this native metric path, their host-language OpenFeature metric writers must stop recording the same evaluations. ## Validation Current head (`96d9a7bae`) local validation: ```sh cd /Users/leo.romanovsky/go/src/github.com/DataDog/libdatadog-ffe-sidecar-metrics cargo fmt --check cargo test -p datadog-ffe --features evaluation-metrics telemetry::evaluation_metrics cargo test -p datadog-sidecar ffe_metric cargo check -p datadog-ffe cargo check -p datadog-sidecar-ffi ``` Results: datadog-ffe metric tests passed (2 passed), sidecar metric tests passed (6 passed), default datadog-ffe check passed, sidecar FFI check passed, fmt check passed with only the repo stable-rustfmt warnings. Prior downstream PHP behavior validation before the reusable-crate refactor, from DataDog/dd-trace-php#3911 using this PR at `1f1fca439`: ```text ffe-dogfooding subject=php-3911-split-1779981881 php7_metrics=3 php8_metrics=3 php7_exposures=0 php8_exposures=0 ``` System-tests downstream validation: ```sh TEST_LIBRARY=php ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_metrics.py -vv ``` Result: 17 passed in 81.26 seconds. Related PRs: DataDog/dd-trace-php#3906, DataDog/dd-trace-php#3911, #2026, DataDog/system-tests#7033. Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>

leoromanovsky mentioned this pull request May 28, 2026

Add FFE evaluation metrics DataDog/dd-trace-php#3911

Open

This was referenced May 28, 2026

feat(sidecar): forward FFE evaluation metrics to OTLP intake DataDog/libdatadog#2052

Merged

Enable PHP FFE exposure system tests #7031

Open

leoromanovsky force-pushed the leo.romanovsky/pr-i-php-ffe-metrics branch from a412f4f to fbc9822 Compare May 29, 2026 00:59

Base automatically changed from leo.romanovsky/pr-g-php-ffe-scaffold to main May 29, 2026 15:48

Enable PHP FFE evaluation metric system tests

581f541

leoromanovsky force-pushed the leo.romanovsky/pr-i-php-ffe-metrics branch from fbc9822 to 581f541 Compare June 3, 2026 20:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable PHP FFE evaluation metric system tests#7033

Enable PHP FFE evaluation metric system tests#7033
leoromanovsky wants to merge 1 commit into
mainfrom
leo.romanovsky/pr-i-php-ffe-metrics

leoromanovsky commented May 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

datadog-prod-us1-5 Bot commented May 28, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

leoromanovsky commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Related PRs

Validation

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

datadog-prod-us1-5 Bot commented May 28, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

leoromanovsky commented May 28, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented May 28, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading