chore(remoteconfig): refresh client identity on runtime refresh by litianningdatadog · Pull Request #19818 · DataDog/dd-trace-py

litianningdatadog · 2026-08-23T21:51:59Z

Description

Split from #19780.

Remote Config clients include process/runtime identity in their generated client id, and the native Remote Config client keeps that id in its own state. When runtime identity is refreshed for a restored MicroVM instance, rebuild the Remote Config client identity and drop the stale native client so future requests use the refreshed identity.

Testing

scripts/lint format_check ddtrace/internal/remoteconfig/client.py tests/internal/remoteconfig/test_remoteconfig_native.py
git diff --check

Stack

Draft split branch. Stacked on #19816.

cit-pr-commenter-54b7da · 2026-08-23T21:52:02Z

Codeowners resolved as

Resolved from the full PR diff against tianning.li/2-flask-web-request-starting-event using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/remoteconfig/client.py                                 @DataDog/remote-config @DataDog/apm-core-python
tests/internal/remoteconfig/test_remoteconfig_native.py                 @DataDog/remote-config @DataDog/apm-core-python

cit-pr-commenter-54b7da · 2026-08-23T21:52:10Z

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

cit-pr-commenter-54b7da · 2026-08-23T21:52:14Z

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 250 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 250 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=135)
ddtrace.llmobs._integrations.crewai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.llmobs._integrations.mcp -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.profiling.collector.pytorch -×-> ddtrace.trace  (product:profiling -> product:tracing, score=133)
ddtrace.internal.ci_visibility.filters -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=133)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

datadog-prod-us1-5 · 2026-08-23T21:58:58Z

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 40 Pipeline jobs failed

System Tests | tracer-release / End-to-end #1 / flask-poc 1 — ❌ 10 tests failed · 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitHub Actions

10 failed tests. Error: No appsec event validates this condition in multiple tests while asserting WAF attack.

❌ tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction.test_lfi_get[flask-poc] from system_tests_suite

AssertionError: assert 200 == 403
 &#43;  where 200 = HttpResponse(status_code:200, headers:{&#39;Server&#39;: &#39;gunicorn&#39;, &#39;Date&#39;: &#39;Sun, 23 Aug 2026 22:10:05 GMT&#39;, &#39;Connection&#39;: &#39;keep-alive&#39;, &#39;Content-Type&#39;: &#39;text/html; charset=utf-8&#39;, &#39;Content-Length&#39;: &#39;33&#39;}, text:../etc/passwd open with 839 bytes).status_code
 &#43;    where HttpResponse(status_code:200, headers:{&#39;Server&#39;: &#39;gunicorn&#39;, &#39;Date&#39;: &#39;Sun, 23 Aug 2026 22:10:05 GMT&#39;, &#39;Connection&#39;: &#39;keep-alive&#39;, &#39;Content-Type&#39;: &#39;text/html; charset=utf-8&#39;, &#39;Content-Length&#39;: &#39;33&#39;}, text:../etc/passwd open with 839 bytes) = &lt;tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction object at 0x7f3bf4afdaf0&gt;.r1

self = &lt;tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction object at 0x7f3bf4afdaf0&gt;

    def test_lfi_get(self):
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
        assert self.config_state_1b.state == rc.ApplyState.ACKNOWLEDGED
&gt;       assert self.r1.status_code == 403
...

❌ tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig.test_block_405[flask-poc] from system_tests_suite

ValueError: No appsec event validate this condition

self = &lt;tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig object at 0x7f3bf56cd460&gt;

    def test_block_405(self):
        # normal block
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
&gt;       interfaces.library.assert_waf_attack(self.response_1, rule=&#34;ua0-600-56x&#34;)

tests/appsec/test_remote_config_rule_changes.py:73: 
...

❌ tests.appsec.test_remote_config_rule_changes.Test_Empty_Config.test_empty_config[flask-poc] from system_tests_suite

ValueError: No appsec event validate this condition

self = &lt;tests.appsec.test_remote_config_rule_changes.Test_Empty_Config object at 0x7f3bf56cd6a0&gt;

    def test_empty_config(self):
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
&gt;       interfaces.library.assert_waf_attack(self.response_1, rule=&#34;ua0-600-56x&#34;)

tests/appsec/test_remote_config_rule_changes.py:537: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...

↳ and 7 more — View all

System Tests | tracer-release / End-to-end #1 / uds-flask 1 — ❌ 10 tests failed · 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitHub Actions

10 failed tests. Error: No appsec event validate this condition in multiple instances of assert_waf_attack.

❌ tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction.test_lfi_get[uds-flask] from system_tests_suite

AssertionError: assert 200 == 403
 &#43;  where 200 = HttpResponse(status_code:200, headers:{&#39;Server&#39;: &#39;gunicorn&#39;, &#39;Date&#39;: &#39;Sun, 23 Aug 2026 22:09:51 GMT&#39;, &#39;Connection&#39;: &#39;keep-alive&#39;, &#39;Content-Type&#39;: &#39;text/html; charset=utf-8&#39;, &#39;Content-Length&#39;: &#39;33&#39;}, text:../etc/passwd open with 839 bytes).status_code
 &#43;    where HttpResponse(status_code:200, headers:{&#39;Server&#39;: &#39;gunicorn&#39;, &#39;Date&#39;: &#39;Sun, 23 Aug 2026 22:09:51 GMT&#39;, &#39;Connection&#39;: &#39;keep-alive&#39;, &#39;Content-Type&#39;: &#39;text/html; charset=utf-8&#39;, &#39;Content-Length&#39;: &#39;33&#39;}, text:../etc/passwd open with 839 bytes) = &lt;tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction object at 0x7f65e79cc860&gt;.r1

self = &lt;tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction object at 0x7f65e79cc860&gt;

    def test_lfi_get(self):
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
        assert self.config_state_1b.state == rc.ApplyState.ACKNOWLEDGED
&gt;       assert self.r1.status_code == 403
...

❌ tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig.test_block_405[uds-flask] from system_tests_suite

ValueError: No appsec event validate this condition

self = &lt;tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig object at 0x7f65e7c7bdd0&gt;

    def test_block_405(self):
        # normal block
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
&gt;       interfaces.library.assert_waf_attack(self.response_1, rule=&#34;ua0-600-56x&#34;)

tests/appsec/test_remote_config_rule_changes.py:73: 
...

❌ tests.appsec.test_remote_config_rule_changes.Test_Empty_Config.test_empty_config[uds-flask] from system_tests_suite

ValueError: No appsec event validate this condition

self = &lt;tests.appsec.test_remote_config_rule_changes.Test_Empty_Config object at 0x7f65e7cb8290&gt;

    def test_empty_config(self):
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
&gt;       interfaces.library.assert_waf_attack(self.response_1, rule=&#34;ua0-600-56x&#34;)

tests/appsec/test_remote_config_rule_changes.py:537: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...

↳ and 7 more — View all

System Tests | tracer-release / End-to-end #10 / flask-poc 10 — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitHub Actions

2 failed tests. Assertion Error: Expected probes to be emitting after enabling dynamic instrumentation and Expected snapshots to be emitting after enabling exception replay.

❌

tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation.test_inproduct_enablement_di[flask-poc]

from system_tests_suite

AssertionError: Expected probes to be emitting after enabling dynamic instrumentation
assert False
 &#43;  where False = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation object at 0x7f820516ee70&gt;.di_explicit_enabled

self = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation object at 0x7f820516ee70&gt;

    def test_inproduct_enablement_di(self):
        self.assert_rc_state_not_error()
        self.assert_all_weblog_responses_ok()
    
...

❌

tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay.test_inproduct_enablement_exception_replay[flask-poc]

from system_tests_suite

AssertionError: Expected snapshots to be emitting after enabling exception replay
assert False
 &#43;  where False = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay object at 0x7f820516ef30&gt;.er_explicit_enabled

self = &lt;tests.debugger.test_debugger_inproduct_enablement.Test_Debugger_InProduct_Enablement_Exception_Replay object at 0x7f820516ef30&gt;

    def test_inproduct_enablement_exception_replay(self):
        self.assert_rc_state_not_error()
        self.assert_all_weblog_responses_ok(expected_code=500)
    
...

View all 40 failed jobs.

📋 Copy prompt for your agent

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: tianning.li/3-1-remoteconfig-identity-client

System Tests | tracer-release / End-to-end #1 / flask-poc 1
Commit: 8fd5f56614dfbd81a3179ef58b2ee01a7c15f384
Error (code / test):
10 failed tests. Error: No appsec event validates this condition in multiple tests while asserting WAF attack.
CI job: https://github.com/DataDog/dd-trace-py/actions/runs/32668821950/job/97267220336

System Tests | tracer-release / End-to-end #1 / uds-flask 1
Commit: 8fd5f56614dfbd81a3179ef58b2ee01a7c15f384
Error (code / test):
10 failed tests. Error: No appsec event validate this condition in multiple instances of assert_waf_attack.
CI job: https://github.com/DataDog/dd-trace-py/actions/runs/32668821950/job/97267220969

System Tests | tracer-release / End-to-end #10 / flask-poc 10
Commit: 8fd5f56614dfbd81a3179ef58b2ee01a7c15f384
Error (code / test):
2 failed tests. Assertion Error: Expected probes to be emitting after enabling dynamic instrumentation and Expected snapshots to be emitting after enabling exception replay.
CI job: https://github.com/DataDog/dd-trace-py/actions/runs/32668821950/job/97267220528

Plus 24 more failing jobs not shown here.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 3e763c0 | Docs | View more details | Give us feedback!}

pr-commenter · 2026-08-23T22:23:10Z

Benchmarks

Benchmark execution time: 2026-08-23 22:23:08

Comparing candidate commit 3e763c0 in PR branch tianning.li/3-1-remoteconfig-identity-client with baseline commit d59e112 in branch tianning.li/2-flask-web-request-starting-event.

📊 Benchmarking dashboard

Found 0 performance improvements and 10 performance regressions! Performance is the same for 612 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.009µs; +2.210µs] or [+10.501%; +11.550%]

scenario:iastaspects-add_aspect

🟥 execution_time [+14.503µs; +18.243µs] or [+14.308%; +17.998%]

scenario:iastaspects-join_aspect

🟥 execution_time [+48.155µs; +52.033µs] or [+22.761%; +24.594%]

scenario:iastaspects-ljust_noaspect

🟥 execution_time [+61.771µs; +66.015µs] or [+21.705%; +23.197%]

scenario:iastaspects-title_noaspect

🟥 execution_time [+29.311µs; +33.463µs] or [+14.999%; +17.123%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+144.547µs; +149.975µs] or [+35.431%; +36.762%]

scenario:iastaspectssplit-rsplit_aspect

🟥 execution_time [+16.594µs; +21.953µs] or [+11.579%; +15.318%]

scenario:span-start

🟥 execution_time [+1.458ms; +1.600ms] or [+9.896%; +10.864%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+444.116ns; +496.305ns] or [+16.478%; +18.414%]

scenario:tracer-small

🟥 execution_time [+26.370µs; +28.613µs] or [+7.810%; +8.474%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

unstable execution_time [-648.212ns; +841.395ns] or [-5.882%; +7.635%]

scenario:coreapiscenario-core_dispatch_1_listener

unstable execution_time [-32.301ns; +34.033ns] or [-5.300%; +5.584%]

scenario:coreapiscenario-core_dispatch_50_listeners

unstable execution_time [-1651.484ns; +1698.057ns] or [-9.607%; +9.878%]

scenario:coreapiscenario-core_dispatch_exception_listeners

unstable execution_time [-1097.152ns; +1380.020ns] or [-8.524%; +10.721%]

scenario:coreapiscenario-core_dispatch_listeners

unstable execution_time [-300.988ns; +356.964ns] or [-8.180%; +9.701%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

unstable execution_time [-278.579ns; +229.758ns] or [-9.510%; +7.843%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

unstable execution_time [-97.921ns; +48.341ns] or [-8.463%; +4.178%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

unstable execution_time [-3382.122ns; +4546.634ns] or [-8.478%; +11.398%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

unstable execution_time [-705.994ns; +835.274ns] or [-8.839%; +10.457%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

unstable execution_time [-61592.105ns; +59899.871ns] or [-9.571%; +9.308%]

fix(remoteconfig): refresh client identity on runtime refresh

3e763c0

litianningdatadog added changelog/no-changelog A changelog entry is not required for this PR. aws-microvm Work related to AWS MicroVM onboarding labels Aug 23, 2026

litianningdatadog changed the title ~~fix(remoteconfig): refresh client identity on runtime refresh~~ chore(remoteconfig): refresh client identity on runtime refresh Aug 23, 2026

Navigation Menu

Uh oh!

chore(remoteconfig): refresh client identity on runtime refresh - #19818

chore(remoteconfig): refresh client identity on runtime refresh#19818
litianningdatadog wants to merge 1 commit into
tianning.li/2-flask-web-request-starting-eventfrom
tianning.li/3-1-remoteconfig-identity-client

litianningdatadog commented Aug 23, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 23, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 23, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 23, 2026

Uh oh!

datadog-prod-us1-5 Bot commented Aug 23, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

Uh oh!

pr-commenter Bot commented Aug 23, 2026

Explanation

More details about the CI and significant changes

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

litianningdatadog commented Aug 23, 2026

Description

Testing

Stack

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 23, 2026

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 23, 2026

Circular import analysis

⚠️ Existing circular imports

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 23, 2026

Dependency direction analysis

⚠️ Existing dependency direction violations

Uh oh!

datadog-prod-us1-5 Bot commented Aug 23, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

pr-commenter Bot commented Aug 23, 2026

Benchmarks

Explanation

More details about the CI and significant changes

scenario:httppropagationinject-ids_only

scenario:iastaspects-add_aspect

scenario:iastaspects-join_aspect

scenario:iastaspects-ljust_noaspect

scenario:iastaspects-title_noaspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:iastaspectssplit-rsplit_aspect

scenario:span-start

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:tracer-small

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

datadog-prod-us1-5 Bot commented Aug 23, 2026 •

edited by datadog-prod-us1-3 Bot

Loading