Field error in first line of data file.

[SteveWilcox303]

Two related issues:

1. If the input data file contains an extra field in the first data record (line 2 after the header), an error appears (but with no line number):

ValueError:
Input data (0:00) is incompatible with data format (0: MM/DD/YYYY)

This is inconsistent with the input line:
5/20/2018,0:00,25.9683,-0.347231,30.427999,999.9
(999.9 is the added field)

2. If the additional field is on line 3 or greater, it is identified as a parsing error. The software specs actually allow for any number of CSV fields after the DHI field, which are to be ignored, e.g. an extra field should not trigger an error. Only too few fields should trigger an error. IF that spec unduly complicates the parsing algorithm, we can drop it and strictly count the fields, but the user's manual will have to be modified. Let me know your thoughts.

Attached is a short input file that should illustrate #1 above. You can run it as an input file with a valid configuration for any other process.

brs.b1_formatted_extrafield_test.csv

[ppinchuk]

Fixed this in #101. We'll have to ask @sjanzou to use the updated main branch in his next build so that the new behavior goes through.

I'll leave this issue open until you have a chance to test it on your end @SteveWilcox303

[SteveWilcox303]

Thanks. I’ll test with next release. Steve W. From: Paul Pinchuk ***@***.***> Sent: Wednesday, September 11, 2024 2:47 PM To: NREL/SUNI ***@***.***> Cc: SteveWilcox303 ***@***.***>; Mention ***@***.***> Subject: Re: [NREL/SUNI] Field error in first line of data file. (Issue #99) Fixed this in #101 <#101> . We'll have to ask @sjanzou <https://github.com/sjanzou> to use the updated main branch in his next build so that the new behavior goes through. I'll leave this issue open until you have a chance to test it on your end @SteveWilcox303 <https://github.com/SteveWilcox303> — Reply to this email directly, view it on GitHub <#99 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4EPBA6OW7YSTM2L47VPG53ZWCT4LAVCNFSM6AAAAABN3MDSLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBUGY3TAOJZHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[SteveWilcox303]

Testing with 2024.9.26:

The extra field described in #1 is now correctly passed without error.

If there are too few solar fields (e.g. GHI, DNI, or DHI), the record is incorrectly passed without error. If a field is missing, it should generate an error message with line # and halt processing.

If the time field is corrupt (e.g. 1 rather than 1:00) a cryptic message is generated without line number: "not enough values to unpack (expected 2, got 1)"

It correctly identifies and reports a corrupt date field, but without line number. All these messages should include the number of the offending line in the file.

[ppinchuk]

Addressed all of these in #104.

leaving issue open until @sjanzou can make a new build and Steve W. has a chance to test the updates.

[SteveWilcox303]

Confirming that these changes are not included in V2024.10.25 (testing indicate the errors persist). Steve From: Paul Pinchuk ***@***.***> Sent: Friday, October 25, 2024 3:11 PM To: NREL/SUNI ***@***.***> Cc: SteveWilcox303 ***@***.***>; Mention ***@***.***> Subject: Re: [NREL/SUNI] Field error in first line of data file. (Issue #99) Addressed all of these in #104 <#104> . leaving issue open until @sjanzou <https://github.com/sjanzou> can make a new build and Steve W. has a chance to test the updates. — Reply to this email directly, view it on GitHub <#99 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4EPBA55EVY4LLW76OGLOL3Z5KXXXAVCNFSM6AAAAABN3MDSLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZYHA3DQNRYGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ppinchuk]

Right, we need to wait until the next build from @sjanzou

[SteveWilcox303]

Testing 2024.11.08
Most input formatting error detection now works except for case of a completely missing date field (second example record below) or time field (third record). This results in an error attributable to insufficient irradiance fields ("One or more solar irradiance values are missing.")

If another field is added to the end of the corrupt record (record 4 below), it correctly identifies the missing date or time field.

5/20/2018,0:04,19.952499,0.675435,23.588499
0:05,19.6052,5.53121,21.8599
5/20/2018,19.952499,0.675435,23.588499
0:05,19.6052,5.53121,21.8599,999

This is now getting into the weeds to accommodate an unlikely circumstance. Because the system's response to the error results in blocking further processing, let's leave this problem documented, but GIVE IT A LOW PRIORITY for additional work.