Some data are not reported to Orion when updating a device

[GuillaumeOd]

IoT Agent Node Lib version the issue has been seen with

4.6.0

Bound or port used (API interaction)

Southbound (Devices data API)

NGSI version

NGSIv2

Are you running a container?

Yes, I am using a contaner (Docker, Kubernetes...)

Image type

normal

Expected behaviour you didn't see

When upgrading iot-agent-node-lib from 3.3.4 to 4.6.0

It should correctly updating a device attribute when using iotAgentLib.update(device.name, device.type, "", [ { name: 'value', type: 'number', value: '10' }], device)

iot-agent-node-lib 3.3.4 is still working perfectly

Unexpected behaviour you saw

The payload request made to Orion is perfectly sent but does not contain de requested change on the device attribute

Here's what is sent to Orion in 3.3.4 :

{ op: 'IoTAgentNGSI.Request' } Options: %s {
    "url": "http://orion:1026/v2/entities?options=upsert",
    "method": "POST",
    "headers": {
        "fiware-service": "ngeconnect",
        "fiware-servicepath": "/"
    },
    "json": {
        "id": "urn:ngsild:TemperatureSensor:CypressDeviceA1A",
        "type": "TemperatureSensor",
        "value": {
            "value": "10",
            "type": "number",
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2024-09-30T08:41:12.495Z"
                }
            }
        }
        (.........)
    }
}

In 4.6.0 the payload is exactly the same but the "value" attribute in the request has disappeared, so the request is sent to Orion but without the updated value on the device. End to end, this leads to having the exact same device in orion as the upsert request did not sent the attribute to update.

I believe this comes from sendUpdateValueNgsi2() in entities-NGSI-v2.js as this function changed a lot between 3.3.4 and 4.6.0.

After some debugging and analysis, it's coming from the line measures = measures.filter((item) => item.name !== currentAttr.object_id && item.name !== currentAttr.name); that removes the measures value. The second part of the filter item.name !== currentAttr.name is going to filter all attributes names that are present in preprocessedAttr but it contains typeInformation.active from originTypeInformation which contains the value attribute...

Steps to reproduce the problem

No response

Configs

environment:
    - "IOTA_CB_HOST=orion"
    - "IOTA_CB_PORT=1026"
    - "IOTA_NORTH_PORT=5005"
    - "IOTA_REGISTRY_TYPE=mongodb"
    - "IOTA_MONGO_HOST=mongodb"
    - "IOTA_MONGO_PORT=27017"
    - "IOTA_MONGO_DB=iotagent-json"
    - "IOTA_HTTP_PORT=5004"
    - "IOTA_PROVIDER_URL=http://localhost:5005"

iot-agent-node-lib config

iotAgentLib.activate({
    logLevel:    process.env.LOG_LEVEL, // 'info'
    ngsiVersion: 'v2',
    providerUrl: process.env.EXTERNAL_PROVIDER_URL, // 'http://localhost:5005'
    service:     process.env.SERVICE_NAME, // 'iotagent'
    subservice:  process.env.SERVICE_PATH, // '/'
    deviceRegistry: {
        type: 'mongodb'
    },
    server:{
        port: process.env.NORTH_PORT // '5005'
    },
    defaultType: "Thing",
    types:       {},
    apiKey: process.env.IOT_AGENT_API_KEY
})

Log output

No response

[tzzed]

@fgalan I ask you if @GuillaumeOd can provide the PR that fixes this issue. We need it to fix our use case.

[AlvaroVega]

Could you please provide equivalent json sent to Orion in 4.6.0 for the same example?
And maybe provide device/group provisioned and measure received?

A test case about his POC could added to functional tests: https://github.com/telefonicaid/iotagent-node-lib/blob/master/test/functional/testCases.js

[fgalan]

@fgalan I ask you if @GuillaumeOd can provide the PR that fixes this issue. We need it to fix our use case.

PRs with fixes are always welcome! :)

Anyway, take into account the feedback provided by @AlvaroVega in previous comment.

[GuillaumeOd]

The primary difference between the two payloads is the absence of the "value" attribute in version 4.6.0.

4.6.0 JSON Structure

{
    "json": {
        "id": "urn:ngsild:TemperatureSensor:CypressDeviceA1A",
        "type": "TemperatureSensor",
        "project": {
            "type": "Text",
            "value": "97e3b48e-3cf3-477a-a784-b958ce35cf9c",
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2024-11-05T13:27:08.039Z"
                }
            }
        },
        "model": {
            "type": "Text",
            "value": "TemperatureSensor",
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2024-11-05T13:27:08.039Z"
                }
            }
        },
        "provider": {
            "type": "Text",
            "value": "...",
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2024-11-05T13:27:08.039Z"
                }
            }
        },
        "isPhysical": {
            "type": "Boolean",
            "value": true,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2024-11-05T13:27:08.039Z"
                }
            }
        },
        "TimeInstant": {
            "type": "DateTime",
            "value": "2024-11-05T13:27:08.039Z"
        }
    }
}

Device Update Logs

The following logs detail the data structure when updating the device, captured via function sendUpdateValue(entityName, attributes, typeInformation, token, callback).

Device Update Log

entityName: urn:ngsild:TemperatureSensor:CypressDeviceA1A
attributes: [ { name: 'value', type: 'number', value: '10' } ]
typeInformation: {
  timestamp: true,
  defaultResource: undefined,
  explicitAttrs: undefined,
  pollingExpiration: undefined,
  pollingDaemonFrequency: undefined,
  multiCore: false,
  relaxTemplateValidation: false,
  defaultEntityNameConjunction: ':',
  defaultType: 'Thing',
  lazy: [],
  active: [
    {
      name: 'temperatureTooHigh',
      type: 'boolean',
      object_id: 'tap_th'
    },
    { name: 'temperatureTooLow', type: 'boolean', object_id: 'tap_tl' },
    {
      name: 'temperatureHighThreshold',
      type: 'number',
      object_id: 't_ht'
    },
    {
      name: 'temperatureLowThreshold',
      type: 'number',
      object_id: 't_lt'
    },
    {
      name: 'fireDetectionThreshold',
      type: 'number',
      object_id: 'f_dt'
    },
    { name: 'value', type: 'number', object_id: 't' }
  ],
  commands: [],
  staticAttributes: [
    {
      name: 'project',
      type: 'Text',
      value: '97e3b48e-3cf3-477a-a784-b958ce35cf9c'
    },
    { name: 'model', type: 'Text', value: 'TemperatureSensor' },
    {
      name: 'provider',
      type: 'Text',
      value: '...'
    },
    { name: 'isPhysical', type: 'Boolean', value: true },
    { name: 'network', type: 'Text' },
    { name: 'physical', type: 'Text' }
  ],
  subscriptions: [],
  _id: 65e1ec5e3884a211e783f62f,
  creationDate: 2024-03-01T14:55:26.641Z,
  id: 'TemperatureSensor:CypressDeviceA1A',
  type: 'TemperatureSensor',
  name: 'urn:ngsild:TemperatureSensor:CypressDeviceA1A',
  service: 'ngeconnect',
  subservice: '/',
  internalId: null,
  internalAttributes: [ { name: 'decoder', type: 'string' } ],
  apikey: '907055d45e0f04ae1f0e69c2b11e9ef2',
  protocol: 'PDI-IoTA-JSON',
  transport: 'HTTP',
  trust: undefined,
  cbHost: undefined,
  ngsiVersion: undefined
}
token: null
newCallback: function accounting(...args) {
    const counter = args.length > 0 && args[0] ? errCounter : okCounter;
    add(counter, 1, function () {
        callback(...args);
    });
}

[AlvaroVega]

Then this happens when an attribute has name 'value' like: ?

{ name: 'value', type: 'number', object_id: 't' }

Maybe type, value are reserved words...

This is the requested test case #1668 which demostrate this case and which make not neccesarry this PR #1667

[tzzed]

I got the problem with object_id that I encountered #1621

[AlvaroVega]

I got the problem with object_id that I encountered #1621
#1621 (comment)

[fgalan]

@GuillaumeOd @tzzed @edwinvautier should this issue be closed now that PR #1721 has been merged (and the fix included in release 4.13.0 of the library) or is anything else pending?