PolderCast Layers Not Communicating

[michaeljfazio]

As per the PolderCast paper. Each layer (Cyclon, Rings, Vicinity) are meant to communicate with one another. Each layer should be acting as a filter on the next. The current implementation has each layer acting independently, sourcing nodes from the available set, rather than for the set filtered by the layer below it. Furthermore, where applicable, layers should producing and maintaining node profile sets per-topic.

[NicolasDP]

They are working independently as per the paper. They are not meant to filter nodes for each others.

[michaeljfazio]

From section 5

The Vicinity module is responsible for feeding the Rings module with a
few neighbors for each topic t ∈ Tp, of arbitrary ids

I believe the paper specifies that the layers are to operate in parallel when sourcing gossips, but it looks to me that they are still meant to be communicating.

[NicolasDP]

And following the same quote, for Cylon, they also say for the whole network:

Finally, the Cyclon module [24], is a lightweight peer sampling service [12], providing each node with a continuous stream of neighbors chosen uniformly at random from the whole network.

If we were to follow the paper by the letter we would have a headache 😄 and nothing would be done. In the rings, the neighbours for the rings are selected per topic too (the TopicView contains the neighbour nodes):

poldercast/src/poldercast/rings.rs

Line 32 in f882186

neighbors: BTreeMap<Topic, TopicView>,

While this might not be perfect this is as good as we can do whilst still allowing us to have a nice separation of concerns and allowing adding more (or less) modules for the topology.

[michaeljfazio]

I agree there are some ambiguities in that paper. But, taking the whole thing in context I have to stand by my original assessment with there definitely being communication between those layers.

I looked to the Haskell code for guidance... But they seem to be doing something that isn't PolderCast at all...

[NicolasDP]

The fact that the layers are not communicating is a design decision that we made in order to facilitate adding more layers and working the layers in isolation.

That being said, this was not done without consideration of pros and cons. I believe this is not only more efficient (at least since 0.12 and the introduction of a max capacity of the peer collection driven by a LRU cache), it is also more convenient and offers the different module a broader choice. Also, rings is still using a similar algorithm to Vicinity in selecting the candidates for its view (filtering the subscriptions in common, giving more or less the same results as vicinity in few topic settings).