-
Notifications
You must be signed in to change notification settings - Fork 9
fix: event source tracking #470
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
6172c98
f38e5fb
fa3950e
9c7d8f0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -275,6 +275,7 @@ class Requests { | |
| trackingSet: state.returnedIds, | ||
| outController: state.controller, | ||
| eventOutFilters: _eventOutFilters, | ||
| cacheManager: _cacheWrite.cacheManager, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Honor
💡 Suggested fix- cacheManager: _cacheWrite.cacheManager,
+ cacheManager: request.cacheWrite ? _cacheWrite.cacheManager : null,🤖 Prompt for AI Agents |
||
| )(); | ||
|
|
||
| // Record fetched ranges when network requests complete (EOSE received) | ||
|
|
@@ -345,7 +346,7 @@ class Requests { | |
| }) { | ||
| final requestId = '$name-paginated-${Helpers.getRandomString(10)}'; | ||
| final aggregatedController = ReplaySubject<Nip01Event>(); | ||
| final seenEventIds = <String>{}; | ||
| final seenEvents = <String, Set<String>>{}; // event_id -> sources | ||
|
|
||
| Future<void> paginate() async { | ||
| final since = filter.since; | ||
|
|
@@ -373,9 +374,20 @@ class Requests { | |
| final relayState = <String, _RelayPaginationState>{}; | ||
|
|
||
| for (final event in initialEvents) { | ||
| if (!seenEventIds.contains(event.id)) { | ||
| seenEventIds.add(event.id); | ||
| final existingSources = seenEvents[event.id]; | ||
| if (existingSources == null) { | ||
| // First time seeing this event | ||
| seenEvents[event.id] = event.sources.toSet(); | ||
| aggregatedController.add(event); | ||
| } else { | ||
| // Merge sources if this event has new sources | ||
| if (event.sources.isNotEmpty) { | ||
| final newSources = existingSources..addAll(event.sources); | ||
| if (newSources.length > (seenEvents[event.id]?.length ?? 0)) { | ||
| seenEvents[event.id] = newSources; | ||
| aggregatedController.add(event.copyWith(sources: newSources.toList())); | ||
| } | ||
| } | ||
|
Comment on lines
+377
to
+390
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: fd -t f -i 'requests.dart' | head -5Repository: relaystr/ndk Length of output: 191 🏁 Script executed: wc -l packages/ndk/lib/domain_layer/usecases/requests/requests.dartRepository: relaystr/ndk Length of output: 121 🏁 Script executed: sed -n '370,400p' packages/ndk/lib/domain_layer/usecases/requests/requests.dart | cat -nRepository: relaystr/ndk Length of output: 1470 🏁 Script executed: sed -n '460,485p' packages/ndk/lib/domain_layer/usecases/requests/requests.dart | cat -nRepository: relaystr/ndk Length of output: 1413 🏁 Script executed: # Let me verify the exact context around these lines and understand the data flow
sed -n '375,405p' packages/ndk/lib/domain_layer/usecases/requests/requests.dart | cat -nRepository: relaystr/ndk Length of output: 1385 🏁 Script executed: cat > /tmp/test_dart.txt << 'EOF'
In Dart:
- The cascade operator (..) mutates the object and returns self
- existingSources is a reference to the Set in seenEvents
- existingSources..addAll(event.sources) mutates that Set and returns it
- So seenEvents[event.id] points to the same mutated Set
- Comparison happens after mutation, so both sides have same length
- This means the condition is always false when sources are added
EOF
cat /tmp/test_dart.txtRepository: relaystr/ndk Length of output: 450 🏁 Script executed: # Let me check the actual line numbers in the original review to map them correctly
sed -n '377,390p' packages/ndk/lib/domain_layer/usecases/requests/requests.dart | cat -nRepository: relaystr/ndk Length of output: 810 🏁 Script executed: sed -n '468,480p' packages/ndk/lib/domain_layer/usecases/requests/requests.dart | cat -nRepository: relaystr/ndk Length of output: 843 🏁 Script executed: cat << 'EOF'
Dart Set behavior with cascade operator:
Given:
final existingSources = seenEvents[event.id]; // reference to Set in map
When executing:
final newSources = existingSources..addAll(event.sources);
The cascade operator (..) on existingSources:
1. Calls addAll() which MUTATES existingSources in-place
2. Returns the mutated existingSources (same object reference)
3. Assigns it to newSources
Result: newSources and existingSources reference the SAME Set object
When checking:
if (newSources.length > (seenEvents[event.id]?.length ?? 0))
Both sides reference the SAME mutated Set, so:
- newSources.length == seenEvents[event.id].length (always)
- Condition is always FALSE
- Event is never emitted with merged sources
The suggested fix {...existingSources, ...event.sources} creates a NEW Set
via spread syntax, avoiding mutation and allowing proper comparison.
EOFRepository: relaystr/ndk Length of output: 928 Clone the tracked set before comparing sizes.
Replace with 🤖 Prompt for AI Agents |
||
| } | ||
|
|
||
| // Track oldest timestamp per relay | ||
|
|
@@ -453,9 +465,20 @@ class Requests { | |
|
|
||
| int? oldestTimestamp; | ||
| for (final event in pageEvents) { | ||
| if (!seenEventIds.contains(event.id)) { | ||
| seenEventIds.add(event.id); | ||
| final existingSources = seenEvents[event.id]; | ||
| if (existingSources == null) { | ||
| // First time seeing this event | ||
| seenEvents[event.id] = event.sources.toSet(); | ||
| aggregatedController.add(event); | ||
| } else { | ||
| // Merge sources if this event has new sources | ||
| if (event.sources.isNotEmpty) { | ||
| final newSources = existingSources..addAll(event.sources); | ||
| if (newSources.length > (seenEvents[event.id]?.length ?? 0)) { | ||
| seenEvents[event.id] = newSources; | ||
| aggregatedController.add(event.copyWith(sources: newSources.toList())); | ||
| } | ||
| } | ||
| } | ||
| // Track oldest timestamp for this relay | ||
| if (oldestTimestamp == null || event.createdAt < oldestTimestamp) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,14 +2,16 @@ import 'dart:async'; | |
| import '../../../shared/logger/logger.dart'; | ||
| import '../../entities/event_filter.dart'; | ||
| import '../../entities/nip_01_event.dart'; | ||
| import '../../repositories/cache_manager.dart'; | ||
|
|
||
| /// given a stream with Nip01 events it tracks the id and adds the one to the provided stream controller \ | ||
| /// tracking of the happens in the tracking list | ||
| class StreamResponseCleaner { | ||
| final Set<String> _trackingSet; | ||
| final Map<String, Set<String>> _trackingMap; // event_id -> set of sources | ||
| final List<Stream<Nip01Event>> _inputStreams; | ||
| final StreamController<Nip01Event> _outController; | ||
| final List<EventFilter> _eventOutFilters; | ||
| final CacheManager? _cacheManager; | ||
|
|
||
|
Comment on lines
+14
to
15
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: fd -t f "stream_response_cleaner.dart"Repository: relaystr/ndk Length of output: 147 🏁 Script executed: fd -t f "requests.dart" packages/ndk/lib/domain_layer/usecases/Repository: relaystr/ndk Length of output: 117 🏁 Script executed: cat -n packages/ndk/lib/domain_layer/usecases/stream_response_cleaner/stream_response_cleaner.dartRepository: relaystr/ndk Length of output: 4389 🏁 Script executed: sed -n '260,285p' packages/ndk/lib/domain_layer/usecases/requests/requests.dartRepository: relaystr/ndk Length of output: 756 🏁 Script executed: # Also check the CacheManager interface to understand the contract
fd -t f "cache_manager.dart" packages/ndk/libRepository: relaystr/ndk Length of output: 195 🏁 Script executed: cat -n packages/ndk/lib/domain_layer/repositories/cache_manager.dartRepository: relaystr/ndk Length of output: 8290 Queue cache writes per event ID to ensure sources never shrink. Line 61 fires an unordered async cache update that can silently drop on cache miss (line 91) or cause earlier writes to complete last and shrink stored sources. In Suggested fix final List<EventFilter> _eventOutFilters;
final CacheManager? _cacheManager;
+ final Map<String, Future<void>> _pendingCacheUpdates = {};
@@
- final mergedEvent = event.copyWith(sources: newSources.toList());
- _outController.add(mergedEvent);
- // Update cache with merged sources
- _updateCacheSources(event.id, newSources);
+ final mergedEvent = event.copyWith(sources: newSources.toList());
+ _outController.add(mergedEvent);
+ _updateCacheSources(mergedEvent);
@@
- void _updateCacheSources(String eventId, Set<String> sources) {
- if (_cacheManager == null) return;
-
- // Load existing event from cache and update sources
- _cacheManager.loadEvent(eventId).then((cachedEvent) {
- if (cachedEvent != null) {
- final updatedEvent = cachedEvent.copyWith(sources: sources.toList());
- _cacheManager.saveEvent(updatedEvent);
- }
- });
+ Future<void> _updateCacheSources(Nip01Event event) {
+ if (_cacheManager == null) return Future.value();
+
+ final previous = _pendingCacheUpdates[event.id] ?? Future.value();
+ final queued = previous.then((_) async {
+ final cachedEvent = await _cacheManager!.loadEvent(event.id);
+ final mergedSources = {
+ ...?cachedEvent?.sources,
+ ...event.sources,
+ }.toList();
+
+ await _cacheManager!.saveEvent(
+ (cachedEvent ?? event).copyWith(sources: mergedSources),
+ );
+ });
+
+ _pendingCacheUpdates[event.id] = queued;
+ return queued;
+ }Also applies to: 58-61, 85-95 🤖 Prompt for AI Agents |
||
| int get _numStreams => _inputStreams.length; | ||
|
|
||
|
|
@@ -18,16 +20,19 @@ class StreamResponseCleaner { | |
| /// - [trackingSet] a set of ids that are already returned \ | ||
| /// - [inputStreams] a list of streams that are be listened to \ | ||
| /// - [outController] the controller that is used to add the events to \ | ||
| /// - [cacheManager] optional cache manager to persist event sources \ | ||
|
|
||
| StreamResponseCleaner({ | ||
| required Set<String> trackingSet, | ||
| required List<Stream<Nip01Event>> inputStreams, | ||
| required StreamController<Nip01Event> outController, | ||
| required List<EventFilter> eventOutFilters, | ||
| }) : _trackingSet = trackingSet, | ||
| CacheManager? cacheManager, | ||
| }) : _trackingMap = {for (final id in trackingSet) id: {}}, | ||
| _outController = outController, | ||
| _inputStreams = inputStreams, | ||
| _eventOutFilters = eventOutFilters; | ||
| _eventOutFilters = eventOutFilters, | ||
| _cacheManager = cacheManager; | ||
|
|
||
| void call() { | ||
| for (final stream in _inputStreams) { | ||
|
|
@@ -37,16 +42,30 @@ class StreamResponseCleaner { | |
|
|
||
| void _addStreamListener(Stream<Nip01Event> stream) { | ||
| stream.listen((event) { | ||
| // check if event id is in the set | ||
| if (_trackingSet.contains(event.id)) { | ||
| if (_outController.isClosed) { | ||
| return; | ||
| } | ||
|
|
||
| if (_outController.isClosed) { | ||
| // check if event id is already seen | ||
| final existingSources = _trackingMap[event.id]; | ||
|
nogringo marked this conversation as resolved.
|
||
| if (existingSources != null) { | ||
| // Event already seen - merge sources if this event has new sources | ||
| if (event.sources.isNotEmpty) { | ||
| final newSources = Set<String>.from(existingSources)..addAll(event.sources); | ||
| // Only emit if we have new sources to add | ||
| if (newSources.length > existingSources.length) { | ||
| _trackingMap[event.id] = newSources; | ||
| final mergedEvent = event.copyWith(sources: newSources.toList()); | ||
| _outController.add(mergedEvent); | ||
| // Update cache with merged sources | ||
| _updateCacheSources(event.id, newSources); | ||
| } | ||
| } | ||
| return; | ||
| } | ||
|
|
||
| _trackingSet.add(event.id); | ||
| // First time seeing this event | ||
| _trackingMap[event.id] = event.sources.toSet(); | ||
|
|
||
| // check against filters | ||
| for (final filter in _eventOutFilters) { | ||
|
|
@@ -63,6 +82,19 @@ class StreamResponseCleaner { | |
| }); | ||
| } | ||
|
|
||
| /// Updates the cache with merged sources for an event | ||
| void _updateCacheSources(String eventId, Set<String> sources) { | ||
| if (_cacheManager == null) return; | ||
|
|
||
| // Load existing event from cache and update sources | ||
| _cacheManager.loadEvent(eventId).then((cachedEvent) { | ||
| if (cachedEvent != null) { | ||
| final updatedEvent = cachedEvent.copyWith(sources: sources.toList()); | ||
| _cacheManager.saveEvent(updatedEvent); | ||
| } | ||
| }); | ||
| } | ||
|
|
||
| /// used to wait on all streams | ||
| Future<void> _canClose() async { | ||
| _closedStreams++; | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| import 'package:ndk/ndk.dart'; | ||
| import 'package:ndk/shared/nips/nip01/bip340.dart'; | ||
| import 'package:test/test.dart'; | ||
|
|
||
| import '../mocks/mock_event_verifier.dart'; | ||
| import '../mocks/mock_relay.dart'; | ||
|
|
||
| void main() async { | ||
| test("broadcast should update source", () async { | ||
| final relay = MockRelay(name: "relay"); | ||
|
|
||
| await relay.startServer(); | ||
| addTearDown(() => relay.stopServer()); | ||
|
|
||
| final ndk = Ndk(NdkConfig( | ||
| eventVerifier: MockEventVerifier(), | ||
| cache: MemCacheManager(), | ||
| bootstrapRelays: [relay.url], | ||
| )); | ||
| addTearDown(() => ndk.destroy()); | ||
|
|
||
| final keypair = Bip340.generatePrivateKey(); | ||
| final signer = Bip340EventSigner( | ||
| privateKey: keypair.privateKey, | ||
| publicKey: keypair.publicKey, | ||
| ); | ||
| ndk.accounts.loginExternalSigner(signer: signer); | ||
|
|
||
| final event = Nip01Event( | ||
| pubKey: keypair.publicKey, | ||
| kind: 1, | ||
| tags: [], | ||
| content: "content", | ||
| ); | ||
|
|
||
| await ndk.broadcast.broadcast(nostrEvent: event).broadcastDoneFuture; | ||
|
|
||
| final localEvent = await ndk.config.cache.loadEvent(event.id); | ||
|
|
||
| expect(localEvent, isNotNull); | ||
| expect(localEvent!.sources, isNotEmpty); | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| }); | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| import 'package:ndk/shared/nips/nip01/bip340.dart'; | ||
| import 'package:test/test.dart'; | ||
| import 'package:ndk/ndk.dart'; | ||
|
|
||
| import '../../mocks/mock_event_verifier.dart'; | ||
| import '../../mocks/mock_relay.dart'; | ||
|
|
||
| void main() async { | ||
| test("requests should update sources", () async { | ||
| final bannedWord = "cow"; | ||
|
|
||
| final relay1 = MockRelay(name: "relay 1"); | ||
| final relay2 = MockRelay(name: "relay 2"); | ||
| final relay3 = MockRelay(name: "relay 3", bannedWord: bannedWord); | ||
|
|
||
| await relay1.startServer(); | ||
| await relay2.startServer(); | ||
| await relay3.startServer(); | ||
|
|
||
| addTearDown(() async { | ||
| await relay1.stopServer(); | ||
| await relay2.stopServer(); | ||
| await relay3.stopServer(); | ||
| }); | ||
|
|
||
| final ndk = Ndk(NdkConfig( | ||
| eventVerifier: MockEventVerifier(), | ||
| cache: MemCacheManager(), | ||
| bootstrapRelays: [relay1.url, relay2.url, relay3.url], | ||
| )); | ||
|
|
||
| addTearDown(() => ndk.destroy()); | ||
|
|
||
| final keypair = Bip340.generatePrivateKey(); | ||
| final signer = Bip340EventSigner( | ||
| privateKey: keypair.privateKey, | ||
| publicKey: keypair.publicKey, | ||
| ); | ||
| ndk.accounts.loginExternalSigner(signer: signer); | ||
|
|
||
| final event = Nip01Event( | ||
| pubKey: keypair.publicKey, | ||
| kind: 1, | ||
| tags: [], | ||
| content: bannedWord, | ||
| ); | ||
| await ndk.broadcast.broadcast(nostrEvent: event).broadcastDoneFuture; | ||
|
|
||
| await ndk.config.cache.clearAll(); | ||
|
|
||
| final query = ndk.requests.query(filter: Filter(ids: [event.id])); | ||
| final events = await query.future; | ||
|
|
||
| // The last event should have all merged sources | ||
| expect(events.last.sources.length, equals(2)); | ||
|
|
||
| final localEvent = await ndk.config.cache.loadEvent(event.id); | ||
|
|
||
| expect(localEvent, isNotNull); | ||
| expect(localEvent!.sources.length, equals(2)); | ||
| }); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 121
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 115
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 2890
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 2053
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 844
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 123
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 333
🏁 Script executed:
Repository: relaystr/ndk
Length of output: 1503
🏁 Script executed:
cat -n packages/ndk/lib/domain_layer/entities/broadcast_state.dart | head -30Repository: relaystr/ndk
Length of output: 1066
🏁 Script executed:
rg -A 3 "class BroadcastState" packages/ndk/lib/domain_layer/entities/broadcast_state.dartRepository: relaystr/ndk
Length of output: 219
Use the in-memory broadcast state to avoid losing relay sources to cache read-modify-write races.
Lines 75–91 listen to broadcast responses and load the cached event once per relay, merge only that relay's URL, and save. If two relays respond before the first cache write completes, both listeners read the same stale cached snapshot, and the second save loses the first relay's source. Since
relay_manager.dart(lines 504–517) already maintainsbroadcastState.eventwith the monotonic union of all successful relays before emitting the response, merge the cached sources withbroadcastState.event.sourcesinstead of patching the cache one relay at a time.Suggested fix
🤖 Prompt for AI Agents