Hotfix remove unreachedexception#1
Open
lamberken wants to merge 2294 commits into
Open
Conversation
…tions This closes apache#7211.
…itting records We need to make sure that concurrent access to the RecordWriter is protected by a lock. It seems that everything but the StreamIterationHead was synchronizing on the checkpoint lock and hence we sync here as well.
…k start document. This closes apache#7269
…ANCELED This closes apache#7004.
… tests should fail if exception isn't thrown This commit strengthens tests in StateBackendMigrationTestBase that depend on a certain state operation (restoring state, accessing state, etc.) to be failing to assert correct behaviour. However, we previously do not really fail the test if no exception was thrown when there should be. This also caught some bugs in the test itself which had the tests verifying incorrect behaviour.
…ackend should always be compatible Previously, we were only checking if the new namespace serializer is incompatible, while properly we should be checking that it is strictly compatible. This doesn't affect any user expected behaviour, since the namespace serializer is never exposed to users.
…ore understandable The StateBackendMigrationTestBase previously mocked user behaviour of upgrading serializers by using a single serializer class, that can be configured with different target compatibility results when they are checked for compatibility. This is a bit hard to understand, also doesn't really reflect how a user would actually approach the feature. Instead, instead of configuring a single serializer class with different compatibility "personalities", this commit uses actual different classes, V1TestTypeSerializer, V2TestTypeSerializer, and IncompatibleTestTypeSerializer, to simulate the compatibility cases in tests. This commit also refactors the serializer migration related test serializers / types / snapshots to the testutil package, so that it can be shared by other state migration related tests in the future.
…in RocksDBKeyedStateBackend This ensures that all restored state, even non-accessed ones after the restore, have a meta info available on future snapshots.
…store serializer This commit introduces StateSerializerProvider that wraps logic on how to obtain serializers for registered state, either with the previous schema of state in checkpoints or the current schema of state. All state meta info subclasses use StateSerializerProviders to replace direct serializer instances. This allows meta infos that were instantiated with restored serializer snapshots to not eagerly access the restore serializer when restoring state. This needs to be avoided since when restoring from 1.6, the restore serializer might not be available; for RocksDB, this should be tolerable.
… map for restored StateMetaInfoSnapshots Since now all restored state meta info snapshots are handled so that we always eagerly create the corresponding RegisteredStateMetaInfoBase for it, the information is already part of the registered state infos map. As can be seen in the changes, those maps are no longer queried and can therefore be safely removed. This closes apache#7264.
…MetaInfoSnapshot This commit removes the `restoreTypeSerializer(...)` method. That method is no longer used after the series of changes in FLINK-11094. This also corresponds to the new principle that the restore serializer is only accessed, when the state backends attempt request it from their state meta infos. We do not create restore serializers eagerly when creating meta infos from a StateMetaInfoSnapshot. It also removes "config" from names of methods and fields related to serializer snapshotting. This corresponds to the abstraction rework of retiring TypeSerializerConfigSnapshot to be replaced by TypeSerializerSnapshot. The related fields / methods should not mention "config" anymore.
…ot association when restoring from 1.5.x When restoring a broadcast state's meta information from a 1.5.x savepoint, the LegacyStateMetaInfoReaders.OperatorBackendStateMetaInfoReaderV2V3 incorrectly associates the first restored serializer as the value serializer, and the second restored serializer as the key serializer. The actual order of this should be the other way around. This bug prevents successful broadcast state restores from 1.5, both for Flink 1.6.x and 1.7.0. The commit also modifies the StatefulJobWBroadcastStateMigrationITCase to have different key / value types for its tested broadcast tests, which otherwise would not have caught this bug.
…estoring 1.5.x braodcast state in later versions. This closes apache#7256.
This closes apache#7254.
…hould hold checkpointing lock
- use UnregisteredMetricGroups instead - simplify metrics - encapsulate TestActor fields
…ends.md In order to use the RocksDBStateBackend, the user has to add the flink-statebackend-rocksdb dependency to his project. This commit adds this information to the state_backends.md. This closes apache#7480.
This closes apache#7491.
…ceStreamTask This closes apache#7485.
The clock parameter is missing in the docstring. This closes apache#7484.
…toreMockitoTest with JDK 9
…specific to MacOS X users
This changes the S3AccessHelper API to take a file instead of an input stream. This allows s3 client to properly reset a file instead of a file over stream for writes. This fixes an issue where the underlying s3 implementation has an intermittent failure, tries to reset the stream, fails to do so, and results in hung requests with delayed errors.
…cases. This closes apache#7384.
…locks for finite stream jobs when resources are limited This commit adds a job config InputDependencyConstraint, which helps to avoid resource deadlocks in LAZY_FROM_SOURCES scheduling when resources are limited. The InputDependencyConstraint controls across multiple inputs when consumers are scheduled. Currently it supports ANY and ALL. ANY means that any input intermediate result partition must be consumable and ALL means that all input intermediate result partitions (from all inputs) need to be consumable in order to schedule the consumer task. This closes apache#7255.
…rom query.x to queryable-state.x Change-Id: Idb561da3982de07e77a05ffef9ad227094f527b0
…stem. This closes apache#7458.
…yncCallRunnable method
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)
Brief change log
(for example:)
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
@Public(Evolving): (yes / no)Documentation