A workable way to retire old gossips throughout the network

[mzabaluev]

Currently, it's not feasible to implement a forgetful policy that would reliably retire obsolete gossips circulating on the network. The gossip profile items lack timestamps, so the only way for a node to check if gossip is outdated is to try to connect to a node. The connection attempts are only performed on a subset of nodes, that may be smaller than the average number of unique gossips that the node receives during the time window needed for the quarantine policy to decide to forget about the bad node. The network-wide effect is that old gossips live indefinitely, bloating the topology sets to impractical sizes.

A simple solution is to add timestamps to gossip items. The timestamp for a node profile is only updated by the node whenever it sends gossip about itself. Other (faithful) nodes pass the timestamps unchanged on the gossip items they forward.

[NicolasDP]

This is not really true. When applying the Policy we get the whole Node.

And from the Logs you have a lot of data you can already leverage. to remove or not that node. The Policy you have in jormungandr could be updated to make use of that:

• you have the creation time (i.e. when your node added this node in the db);
• you have the last_update (the last time the gossip was updated by another node through gossiping);
• you have the last_use_of a topic and last_gossip. This will tell you when/if the node has been used by your node recently or not for block/transactions or if it has been selected as a gossip of interest for another node.

[NicolasDP]

There is also the Topology::force_reset_layer that will force applying the Policy on every node. That can be used to GC some of the nodes. In jormungandr this is only use optionally. Maybe it would make sense to perform GC operation from time to time?

[mzabaluev]

you have the last_update (the last time the gossip was updated by another node through gossiping);

The problem with this is, at least with current parameters, gossip continually circulates through other nodes so the last_update field may never get old enough to retire; intuitively, a gossip entry needs to be forgotten by almost all nodes in the network to be eventually removed.

you have the last_use_of a topic and last_gossip.

These are local again. The policy can, in fact, only retain topology information on nodes that have been recently contacted by this node. But would this diverge from the Poldercast algorithm into some unverified ad-hoc territory?

Currently, with topology_force_reset_interval set to 5 minutes, I never see any entries forgotten.

[michaeljfazio]

See Jormungandr 1690.

Also, I don't know if this explains your observation @mzabaluev but until v0.11.1 and the inclusion of #8, setting topology_force_reset_interval would not correctly cause re-evaluation of the policy over all nodes.

[NicolasDP]

Old unused gossips will be retired faster with 0.12 without needing to add new tooling. There is a LRU Cache driving the overall peer collection, peers that are not used will be replaced by new peers.

Adding the timestamp to the gossip won't really help as this is: permission less protocol and there is no way to know if a timestamp is valid or not. In this context, only one node with bad intent could continuously propagate gossips with a recent timestamp (especially that we could not verify the information, unlike the case of the address field of the gossip).