Bump apache-beam[gcp] from 2.28.0 to 2.41.0 in /megalist_dataflow

[dependabot]

Bumps apache-beam[gcp] from 2.28.0 to 2.41.0.

Release notes

Sourced from apache-beam[gcp]'s releases.

Beam 2.41.0 release

We are happy to present the new 2.41.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.41.0, check out the detailed release notes.

I/Os

• Projection Pushdown optimizer is now on by default for streaming, matching the behavior of batch pipelines since 2.38.0. If you encounter a bug with the optimizer, please file an issue and disable the optimizer using pipeline option --experiments=disable_projection_pushdown.

New Features / Improvements

• Previously available in Java sdk, Python sdk now also supports logging level overrides per module. (#18222).

Breaking Changes

• Projection Pushdown optimizer may break Dataflow upgrade compatibility for optimized pipelines when it removes unused fields. If you need to upgrade and encounter a compatibility issue, disable the optimizer using pipeline option --experiments=disable_projection_pushdown.

Deprecations

• Support for Spark 2.4.x is deprecated and will be dropped with the release of Beam 2.44.0 or soon after (Spark runner) (#22094).
• The modules amazon-web-services and kinesis for AWS Java SDK v1 are deprecated in favor of amazon-web-services2 and will be eventually removed after a few Beam releases (Java) (#21249).

Bugfixes

• Fixed a condition where retrying queries would yield an incorrect cursor in the Java SDK Firestore Connector (#22089).
• Fixed plumbing allowed lateness in Go SDK. It was ignoring the user set value earlier and always used to set to 0. (#22474).

Known Issues

• See a full list of open issues that affect this version.

List of Contributors

According to git shortlog, the following people contributed to the 2.41.0 release. Thank you to all contributors!

Ahmed Abualsaud Ahmet Altay akashorabek Alexey Inkin Alexey Romanenko Anand Inguva andoni-guzman Andrew Pilloud

... (truncated)

Changelog

Sourced from apache-beam[gcp]'s changelog.

[2.41.0] - 2022-08-23

I/Os

• Projection Pushdown optimizer is now on by default for streaming, matching the behavior of batch pipelines since 2.38.0. If you encounter a bug with the optimizer, please file an issue and disable the optimizer using pipeline option --experiments=disable_projection_pushdown.

New Features / Improvements

• Previously available in Java sdk, Python sdk now also supports logging level overrides per module. (#18222).

Breaking Changes

• Projection Pushdown optimizer may break Dataflow upgrade compatibility for optimized pipelines when it removes unused fields. If you need to upgrade and encounter a compatibility issue, disable the optimizer using pipeline option --experiments=disable_projection_pushdown.

Deprecations

• Support for Spark 2.4.x is deprecated and will be dropped with the release of Beam 2.44.0 or soon after (Spark runner) (#22094).
• The modules amazon-web-services and kinesis for AWS Java SDK v1 are deprecated in favor of amazon-web-services2 and will be eventually removed after a few Beam releases (Java) (#21249).

Bugfixes

• Fixed a condition where retrying queries would yield an incorrect cursor in the Java SDK Firestore Connector (#22089).
• Fixed plumbing allowed lateness in Go SDK. It was ignoring the user set value earlier and always used to set to 0. (#22474).

[2.40.0] - 2022-06-25

Highlights

• Added RunInference API, a framework agnostic transform for inference. With this release, PyTorch and Scikit-learn are supported by the transform. See also example at apache_beam/examples/inference/pytorch_image_classification.py

I/Os

• Upgraded to Hive 3.1.3 for HCatalogIO. Users can still provide their own version of Hive. (Java) (Issue-19554).

New Features / Improvements

• Go SDK users can now use generic registration functions to optimize their DoFn execution. (BEAM-14347)
• Go SDK users may now write self-checkpointing Splittable DoFns to read from streaming sources. (BEAM-11104)
• Go SDK textio Reads have been moved to Splittable DoFns exclusively. (BEAM-14489)
• Pipeline drain support added for Go SDK has now been tested. (BEAM-11106)
• Go SDK users can now see heap usage, sideinput cache stats, and active process bundle stats in Worker Status. (BEAM-13829)

Breaking Changes

• The Go Sdk now requires a minimum version of 1.18 in order to support generics (BEAM-14347).

... (truncated)

Commits

• f37795e Set version for 2.41.0 RC2
• 7ef51bc Merge pull request #22759 from kileys/releasebranch
• c855da6 Revert "[BEAM-14117] Unvendor bytebuddy dependency (#17317)"
• 53c2398 Merge pull request #22636 from AnandInguva/distutils-update-release
• 7a4b3db Add stdlib distutils while building the wheels
• 5d5bac7 Merge pull request #22589 from reuvenlax/release-2.41.0
• c5b630a Merge pull request #22347: [22188]Set allowed timestamp skew
• 8bd1b10 Merge pull request #22519 from AnandInguva/bug_fix_julia_seup
• fb7e7ba Add comment on setuptools and distutils
• 4afc22a Fix lint
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Superseded by #92.