OBIS reference filter help

[Proxy5]

hello,

it is possible to filter this out of the data sent by the meter
0-0: 96.13.0.255,
it is not relevant data and its processing is problematic, because the service provider, 0xff, fills in the location of the message.

Thanks, Tomas

[lowdef]

Hi Tomas, can you send an example message?

[lowdef]

In principle you can delete the line, before you offer it to the parser. Make sure you switch of CRC check in this case.

[Proxy5]

Hi, yes of course dsmr_console --host 192.168.1.71 --port 23 --version 5 --verbose ***@***.***:/# dsmr_console --host 192.168.1.71 --port 23 --version 5 --verbose DEBUG:asyncio:Using selector: EpollSelector DEBUG:dsmr_parser.clients.protocol:connected DEBUG:dsmr_parser.clients.protocol:received data: /AUX59940192257 0-0:1.0.0(220212153250W) 0-0:42.0.0(AUX1020340192257) 0-0:96.1.0(9940192257) 0-0:96.14.0(0002) 0-0:96.50.68(ON) 0-0:17.0.0(24.000*kW) 1-0:1.8.0(003970.901*kWh) 1-0:1.8.1(001827.309*kWh) 1-0:1.8.2(002143.592*kWh) 1-0:1.8.3(000000.000*kWh) 1-0:1.8.4(000000.000*kWh) 1-0:2.8.0(000000.000*kWh) 1-0:2.8.1(000000.000*kWh) 1-0:2.8.2(000000.000*kWh) 1-0:2.8.3(000000.000*kWh) 1-0:2.8.4(000000.000*kWh) 1-0:3.8.0(000081.870*kvarh) 1-0:4.8.0(000353.778*kvarh) 1-0:5.8.0(000081.870*kvarh) 1-0:6.8.0(000000.000*kvarh) 1-0:7.8.0(000000.000*kvarh) 1-0:8.8.0(000353.778*kvarh) 1-0:15.8.0(003970.901*kWh) 1-0:32.7.0(232.0*V) 1-0:31.7.0(002*A) 1-0:33.7.0(0.848) 1-0:14.7.0(49.99*Hz) 1-0:1.7.0(00.572*kW) 1-0:2.7.0(00.000*kW) 1-0:5.7.0(00.000*kvar) 1-0:6.7.0(00.000*kvar) 1-0:7.7.0(00.000*kvar) 1-0:8.7.0(00.239*kvar) 0-0:98.1.0(220201000000W)(003574.321*kWh)(001624.224*kWh)(001950.097*kWh)(000000.000*kWh)(000000.000*kWh)(000000.000*kWh)(000072.292*kva rh)(000317.886*kvarh)(000072.292*kvarh)(000000.000*kvarh)(000000.000*kvarh)(000317.886*kvarh)(003574.321*kWh)(05.248*kW)(05.248*kW)(04.2 28*kW)(00.000*kW)(00.000*kW)(00.000*kW) 0-0:96.13.0(\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff DEBUG:dsmr_parser.clients.protocol:received data: \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff!3CE6 ERROR:asyncio:Exception in callback _SelectorSocketTransport._read_ready() handle: <Handle _SelectorSocketTransport._read_ready()> Traceback (most recent call last):         File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run                 self._callback(*self._args)         File "/usr/lib/python3.6/asyncio/selector_events.py", line 721, in _read_ready                 self._protocol.data_received(data)         File "/usr/local/lib/python3.6/dist-packages/dsmr_parser/clients/protocol.py", line 124, in data_received                 telegram = telegram.encode("latin1").decode("ascii") UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1177: ordinal not in range(128) ^CINFO:dsmr_parser.clients.protocol:disconnected because of close/abort. Tamás

…

On 2022. Feb 12. 12:34 +0100, lowdef ***@***.***>, wrote: Hi Tomas, can you send an example message? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[lowdef]

In principle recent versions of the parser can handle this, the offending line is just ignored:

P1_MESSAGE_TIMESTAMP: 	 2022-02-12T15:32:50+01:00	[None]
ELECTRICITY_IMPORTED_TOTAL: 	 3970.901	[kWh]
ELECTRICITY_USED_TARIFF_1: 	 1827.309	[kWh]
ELECTRICITY_USED_TARIFF_2: 	 2143.592	[kWh]
ELECTRICITY_DELIVERED_TARIFF_1: 	 0.000	[kWh]
ELECTRICITY_DELIVERED_TARIFF_2: 	 0.000	[kWh]
ELECTRICITY_ACTIVE_TARIFF: 	 0002	[None]
CURRENT_ELECTRICITY_USAGE: 	 0.572	[kW]
CURRENT_ELECTRICITY_DELIVERY: 	 0.000	[kW]
INSTANTANEOUS_VOLTAGE_L1: 	 232.0	[V]
INSTANTANEOUS_CURRENT_L1: 	 2	[A]
EQUIPMENT_IDENTIFIER_GAS: 	 9940192257	[None]

ignore line with signature \d-\d:96\.13\.0.+?\r\n, because parsing failed.
Traceback (most recent call last):
  File "/home/hanserik/projects/dsmr_parser/dsmr_parser/parsers.py", line 59, in parse
    telegram[signature] = parser.parse(match.group(0))
  File "/home/hanserik/projects/dsmr_parser/dsmr_parser/parsers.py", line 201, in parse
    return CosemObject(self._parse(line))
  File "/home/hanserik/projects/dsmr_parser/dsmr_parser/parsers.py", line 151, in _parse
    raise ParseError("Invalid '%s' line for '%s'", line, self)
dsmr_parser.exceptions.ParseError: ("Invalid '%s' line for '%s'", '0-0:96.13.0(\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\r\n', <dsmr_parser.parsers.CosemParser object at 0x7ff488d029a0>)

[apulai]

Hello,

I also have the same problem. 0-0: 96.13.0.255 is filled with 0xff.
Provider's manual says it is reserved for further use and is filled with '????'s. In reality it is 0xff.

I understand that it is ignored when parsing as lowdef wrote.

However I would like to use it from Home Assisstant.

This is the log I see in Home Assisstant:

File "/usr/local/lib/python3.12/site-packages/dsmr_parser/clients/protocol.py", line 132, in data_received
telegram = telegram.encode("latin1").decode("ascii")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1371: ordinal not in range(128)

The dsmr_parser/clients/protocol.py
Tries to convert it to an ASCII string and rightfully fails, since 0xff cannot be converted to ASCII.

So I am proposing to change this code below.
I think a try block would help:

        for telegram in self.telegram_buffer.get_all():
            # ensure actual telegram is ascii (7-bit) only (ISO 646:1991 IRV required in section 5.5 of IEC 62056-21)

             try:
                telegram = telegram.encode("latin1").decode("ascii")
            except UnicodeDecodeError as e:
                pass
                or maybe ???:
                telegram = telegram.encode("latin1").decode("latin1")

            self.handle_telegram(telegram)

Please comment on it.
Is there an elegant way to get around this issue?
Thanks,
Andras

[apulai]

Hello,

Actually after modifiying /usr/local/lib/python3.12/site-packages/dsmr_parser/clients/protocol.py
around line 132 in Homassistant (inside the container) from the original to

        for telegram in self.telegram_buffer.get_all():
            # ensure actual telegram is ascii (7-bit) only (ISO 646:1991 IRV required in section 5.5 of IEC 62056-21)

             try:
                telegram = telegram.encode("latin1").decode("ascii")
            except:
                telegram = telegram.encode("latin1").decode("latin1")

            self.handle_telegram(telegram)

It works even with the non-comformant telegram (0xff bytes under last part of the message).

Dear @ndokter, please advise or comment.

Thanks,
Andras

[ndokter]

I don't feel like i'm knowledged enough to comment on it. I'd think that maybe just processing it as latin1 might be better then? I think it will be fine, but am unable to predict side effects.