Modification of NGSIElasticsearchSink

[santcorc]

Changes in the NGSIElasticsearchSink to not append the date at the end of the index created. This was done due to the limit of 1000 shards per node of elasticsearch. Since each index by default needs 2 shards, in long term scenarios all the shards would be full leading to a stop in the storage. Thus, the addition of the date in the index does not provide useful information due to each attribute contains the hour and the date when it was created.

The documentation has also been modified to change the specification of this Sink.

[fgalan]

Thanks for your contribution! :)

Could you provide a little of context, please? In which project are you using Cygnus ElasticSearch sink? Thanks!

[santcorc]

I was using it in the european project Smart2B, to connect a orion context broker to elasticsearch. The objective of this is to store information from several sensors across different locations during a long term period (around 2 years). This is the reason why having a new index created everyday for each sensor did not make sense due to the great amount of shards needed in that case, taking account that could be around 200 sensors sending information every 10 minutes everyday.

[fgalan]

Thanks for your explanation!

It seems that your proposed modification have broken the tests (see https://github.com/telefonicaid/fiware-cygnus/actions/runs/2970976609/jobs/4759074174). Could you have a look to them an fix it, please? We cannot merge a PR with broken tests...

[santcorc]

I have modified the NGSIElasticsearchSinkTest according to the new behaviour, not requering to add the date at the end of the entities.

[fgalan]

Note that after your last modifications in commit bda85fa GitHub is still reporting test fails. Could you have a look please?

Apart from that, please add a line in CHANGES_NEXT_RELEASE file regarding the change in this PR. Suggestion:

- [cygnus-ngsi][Elasticsearch] Simplify index generation removing date prefix

O something like that.

Thanks!