Follow-up: a6 test / skills / docs review findings

[guoqqqi]

Background

Before putting a6 into real use, we completed a full review and local rerun of the current test system, skills, and documentation.

Detailed Feishu report:

• https://www.feishu.cn/docx/AR0Qd3d1xo4I0Bx2PGQccwCBnXe

Current Conclusions

• The baseline CI test suite passes.
• With the current CI E2E workflow deployment model, the a6 E2E suite can also be reproduced successfully in a local environment.
• skills metadata validation passes.
• After running the skills E2E suite locally against a real environment, all cases pass except skywalking, which is skipped because the required conditional plugin is missing.
• The current overall code coverage is approximately 52.4%.

This indicates that a6 already has a relatively complete automation baseline, but there are still gaps before it can be considered ready for “100% coverage / full pre-release validation”.

Follow-up Items

P0

• Clarify the conditional capability boundary for the skywalking skill
  • TestSkillPluginSkywalking is currently skipped with the current image
  • We need to make it explicit that this is a conditional skip caused by a missing plugin, not a fully covered and passing case

P1

• Add compatibility tests for debug logs container naming

  • The current setup passes with the CI container name apisix
  • However, the test currently assumes a hard-coded container name, which can easily lead to mismatches in local reproduction

• Add capability-boundary coverage for Global Rule labels

  • The related case is currently skipped because APISIX does not support it

• Add tests for shared modules and low-coverage modules

  • Such as selector, tableprinter, root, configutil, and similar modules

• Add more combined scenario coverage

  • service + route + plugin-config
  • Real traffic verification for multi-node upstreams
  • More non-happy-path and invalid-parameter branches

P2

• Continue raising low-coverage packages one by one based on the coverage report by adding more unit tests

Suggested Outputs

• Track conditional-capability cases separately from real passing cases
• Add another round of test supplementation using semantic scenarios as the unit of coverage
• Keep tests and documentation aligned in follow-up PRs

[guoqqqi]

The notes below break down the current a6 status so the issue can continue to be analyzed and advanced directly.

Current Status

• Baseline CI tests: successfully rerun locally
• CI E2E: successfully rerun locally using the current workflow assumptions, including host-network mode and container names apisix / httpbin
• Skills validation: passed
• Skills E2E: executed against a real environment; all cases passed except skywalking, which was skipped because the required plugin condition was not present
• Overall coverage: 52.4%

Semantic Cases That Have Already Passed in a Real Environment

• Basic CLI execution, version, and completion
• Context create, switch, delete, and error handling
• Config dump / validate / diff / sync
• Route CRUD, list, filtering, YAML/JSON output, and real traffic forwarding
• Service CRUD, Upstream binding, and Route binding via Service ID
• Upstream CRUD, multi-node behavior, and health checks
• Consumer / Credential / Consumer Group CRUD
• Real key-auth traffic flow for Consumers
• Plugin / Plugin Metadata / Plugin Config CRUD
• Global Rule CRUD
• Secret CRUD
• SSL CRUD / create-from-file / status querying
• Stream Route CRUD
• Bulk delete / export / label operations
• Basic debug trace capability

Current Skills Status

• ./scripts/validate-skills.sh: passed
• test/e2e/skills: executed against a real environment
• Skills currently verified in a real environment include:
  • auth: key-auth, basic-auth, jwt-auth, hmac-auth
  • traffic: proxy-rewrite, traffic-split, limit-count, limit-req, redirect, response-rewrite
  • observability: http-logger, datadog, zipkin, prometheus
  • recipes: canary, blue-green, api-versioning, health-check, circuit-breaker, graphql-proxy, mtls, multi-tenant
  • persona/shared: developer, operator, shared
• The only skill case currently not passing is:
  • TestSkillPluginSkywalking
  • Reason: the current APISIX image does not include the skywalking plugin, so this is a conditional skip rather than a functional failure

Current Skips / Conditional Capabilities

• TestCoverageFlags_DebugLogsTypeFlag
• TestCoverageFlags_DebugLogsOutputJSONFlag
• TestExport_GlobalRule
• TestSkillPluginSkywalking

These should not be counted directly as “fully covered”.

Follow-up TODOs

P0

• Clarify the conditional capability boundary for skywalking
  • There should be a passing validation path when the plugin exists
  • There should be a stable skip semantic when the plugin does not exist

P1

• Add compatibility tests for debug logs container naming
  • The current setup passes with the CI container name
  • But the test still assumes a hard-coded container name

- Add capability-boundary coverage for Global Rule labels

• Add more combined scenario coverage

  • service + route + plugin-config
  • Real traffic verification for multi-node upstreams
  • More non-happy-path and invalid-parameter branches

• Add tests for low-coverage shared modules

  • selector
  • tableprinter
  • root
  • configutil

P2

• Continue adding unit tests package by package based on the coverage report

Documentation Follow-ups

• Add clarification for conditional plugins such as skywalking
• Add more direct entry points in the user guide for context, config, update, and version
• The documentation should clearly distinguish between “verified in a real environment” and “conditional capability / skip”

[guoqqqi]

The next-stage testing architecture work has now been split into dedicated tracking issues so that the follow-up can be executed in phases:

• test: refactor automation strategy around real APISIX E2E and Ginkgo #19 umbrella: refactor automation strategy around real APISIX E2E and Ginkgo
• docs(test): redefine unit-vs-e2e policy for a6 #20 policy/docs update
• test(e2e): bootstrap Ginkgo/Gomega framework for real APISIX scenarios #23 Ginkgo/Gomega E2E bootstrap
• test: migrate core command coverage from mocked Admin API tests to real E2E #22 first migration batch for core resources
• test: migrate remaining resource commands and remove legacy mock-command coverage #21 remaining migration and legacy cleanup

This keeps #14 as the original findings issue while the actual strategy migration and execution are tracked separately. The current execution order is: #20 -> #23 -> #22 -> #21.

[guoqqqi]

Current status after merging #26, #27, #28, #29, #30, and #31:

Completed:

• Ginkgo/Gomega E2E framework is in place and running in CI.
• Route/service/upstream/plugin-config/global-rule coverage has been migrated to Ginkgo E2E.
• Consumer/credential/consumer-group/proto/secret/ssl/stream-route/plugin-metadata/config/group/utility/debug command coverage has been supplemented with Ginkgo E2E.
• E2E assertions now follow the updated boundary: prefer CLI create/update/delete/list/export plus CLI get/read-back assertions against real APISIX Admin API.
• Existing mock Admin API unit tests are intentionally retained for now.
• The strategy execution issues test: refactor automation strategy around real APISIX E2E and Ginkgo #19, docs(test): redefine unit-vs-e2e policy for a6 #20, test: migrate remaining resource commands and remove legacy mock-command coverage #21, test: migrate core command coverage from mocked Admin API tests to real E2E #22, and test(e2e): bootstrap Ginkgo/Gomega framework for real APISIX scenarios #23 have been updated/closed according to the current boundary.

Boundary corrections:

• We should not require new E2E tests to validate the real proxy/data-plane chain by default. That should only be added when a feature specifically requires proxy behavior.
• We should not remove legacy mock Admin API unit tests in the current phase. If we decide to remove them later, that should be tracked as a separate cleanup issue.

Still worth tracking from this original findings issue:

• skywalking remains a conditional plugin/capability case unless the test image includes that plugin.
• Any future low-coverage pure helper work should stay limited to self-contained unit tests, not mocked Admin API command tests.

Given this, I recommend keeping #14 open as the original findings ledger unless we split the remaining skywalking/conditional-capability follow-up into a smaller dedicated issue.

[guoqqqi]

SkyWalking follow-up update:

The issue is not that a6 needs to validate a full SkyWalking trace pipeline. Under the current E2E boundary, we only need to verify that a6 can configure the plugin through the real APISIX Admin API and read the resource back through the CLI.

The root cause was that APISIX does not enable the skywalking plugin in the default plugin list used by the test configuration. PR #32 enables skywalking in the CI and docker-compose APISIX E2E configs while preserving the default plugin list, so TestSkillPluginSkywalking can pass as a config/Admin API validation instead of skipping due to the plugin being unavailable.

PR: #32