Skip to content

test: fix failing integration tests#1040

Merged
karenc-bq merged 3 commits into
mainfrom
test/hanging-efm2-integration-test
Dec 1, 2025
Merged

test: fix failing integration tests#1040
karenc-bq merged 3 commits into
mainfrom
test/hanging-efm2-integration-test

Conversation

@karenc-bq

@karenc-bq karenc-bq commented Nov 18, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Hanging Tests

  • When there's a network issue, calling conn.close() may hang indefinitely. This PR wraps these direct calls in a timeout to ensure application doesn't block the main workflow when we only want to do a best attempt at closing a connection.

Failing Tests

  • PG occasionally throws a different type of network error that we weren't handling properly
  • We were testing EFM2 against MySQL. Since EFM2 isn't supported for MySQL, some integration test cases would fail. This PR reverted some of those test changes. We will need to do handle this more carefully in a separate PR.

Chores

  • Added a few more log messages
  • Refactored the integration tests code to allow running tests locally with a specific number of instances.
  • Ensure all thread pool executors have a name so they are easier to identify when debugging
  • Updated GH workflow to stagger the LTS and Latest integration tests so we are not creating as many instances at once

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@karenc-bq karenc-bq changed the title test: add timeouts to prevent integration tests from hanging [WIP] test: add timeouts to prevent integration tests from hanging Nov 18, 2025
@karenc-bq
karenc-bq force-pushed the test/hanging-efm2-integration-test branch 8 times, most recently from 4ea7fd3 to a73f3af Compare November 20, 2025 19:36
@karenc-bq
karenc-bq force-pushed the test/hanging-efm2-integration-test branch 10 times, most recently from b3ddddd to bad6e9f Compare November 28, 2025 07:44
@karenc-bq karenc-bq changed the title [WIP] test: add timeouts to prevent integration tests from hanging test: fix failing integration tests Nov 28, 2025
@karenc-bq
karenc-bq force-pushed the test/hanging-efm2-integration-test branch from bad6e9f to 7b405cb Compare November 28, 2025 23:05
karenc-bq
karenc-bq commented Nov 28, 2025
View reviewed changes
Comment thread .github/workflows/integration_tests.yml Outdated
push:
branches:
- main
- test/hanging-efm2-integration-test

@karenc-bq karenc-bq Nov 28, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will remove this once integration tests pass

@karenc-bq
karenc-bq force-pushed the test/hanging-efm2-integration-test branch 3 times, most recently from 10647db to ff4f6a1 Compare November 28, 2025 23:18
sophia-bq
sophia-bq reviewed Nov 29, 2025
View reviewed changes
Comment thread aws_advanced_python_wrapper/writer_failover_handler.py Outdated
Comment thread tests/integration/container/conftest.py
Comment thread tests/integration/container/test_aurora_failover.py Outdated
@karenc-bq
karenc-bq force-pushed the test/hanging-efm2-integration-test branch from 1b61e4f to dc5e9f2 Compare November 29, 2025 00:32
sophia-bq
sophia-bq approved these changes Dec 1, 2025
View reviewed changes
Comment thread tests/integration/container/test_aurora_failover.py Outdated
karenc-bq
karenc-bq commented Dec 1, 2025
View reviewed changes
Comment thread tests/integration/container/test_aurora_failover.py
assert idle_connection.is_closed is True

@enable_on_features([TestEnvironmentFeatures.NETWORK_OUTAGES_ENABLED])
def test_failover__socket_timeout(

@karenc-bq karenc-bq Dec 1, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleting this test case because it is already being covered by other tests

@karenc-bq
karenc-bq force-pushed the test/hanging-efm2-integration-test branch from 4816e8d to 865b506 Compare December 1, 2025 22:39
@karenc-bq
karenc-bq merged commit 3b4d524 into main Dec 1, 2025
8 checks passed
@karenc-bq
karenc-bq deleted the test/hanging-efm2-integration-test branch December 1, 2025 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@faviansamatha faviansamatha faviansamatha approved these changes
@sophia-bq sophia-bq sophia-bq approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@karenc-bq @faviansamatha @sophia-bq