waggle: initial version - #732
Conversation
|
|
||
| Objects are disseminated through incremental publication of their constituent pieces. A publisher may divide an object into multiple pieces and publish those pieces independently, including through repeated publication attempts. Pieces do not need to originate from a single publisher; any peer possessing a piece may participate in its propagation. Each published piece belongs to a topic and MUST be disseminated through the corresponding topic stream. | ||
|
|
||
| To publish a piece, a node selects a random subset of its connected peers and transmits it to them. Upon receiving a previously unknown piece, a peer updates its local object view and begins propagating the piece through its own connected peers. |
There was a problem hiding this comment.
I think this should be "subset of peers subscribed to the piece's topic". I'd also not explicitly require that this subset is random, the application should choose the subset. Of course, the application may choose that subset randomly.
There was a problem hiding this comment.
that's a great insight, agree and updated, thanks Daniel
|
|
||
| To discover additional dissemination opportunities, a peer also selects a random subset of connected peers for which it has no current metadata view and advertises its current object metadata to them. These exchanges allow peers to progressively discover missing pieces and establish additional synchronization relationships throughout the network. |
There was a problem hiding this comment.
again, subset of subscribed peers.
Also, I am unsure if this should be codified here. At the very least, there should be a MAY somewhere in here IMO.
There was a problem hiding this comment.
updated to remove the random reference. Why MAY? this is part of the dissemination, the protocol should guarantee that applications are able to in the limit flood the network so that a full dissemination is achieved.
If applications set the gossip factor to 0 this step is skipped
| ## Object Cache | ||
|
|
||
| The lifetime of cached object and piece state SHOULD be configurable by implementations and tunable by applications according to their resource constraints and dissemination requirements. |
There was a problem hiding this comment.
There should not be a waggle maintained object cache. Waggle can not by itself utilize such a cache, as all fields are opaque, so the caching should just be responsibility of the application,
There was a problem hiding this comment.
the fields being opaque doesn't mean you can't store them as bytes as we do with partial messages cache, which compares and validate against new received pieces
There was a problem hiding this comment.
such a general purpose cache might be very inefficient, depending on the object being disseminated. I'd just let the application manage it.
In the end, maybe this boils down to being an API question: Either a highly configurable and modifiable cache that lives in the waggle implementation, or letting this be the responsibility of the application. I lean towards the latter.
| message Message { | ||
| oneof payload { | ||
| TopicSubscription subscription = 1; | ||
| ObjectPieces publication = 2; | ||
| } | ||
| } |
There was a problem hiding this comment.
As TopicSubscription is only sent exactly once, I'd omit this message definition
There was a problem hiding this comment.
thanks Daniel, meanwhile updated the protobuf schema according to what we spoke
2 participants
first draft to start the conversation, info is in the spec.