In the latest version the preprocessing routines, I was getting:
...
reference_water has invalid values.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[43], line 1
----> 1 dswx_s1_workflow(cfg)
File ~/bekaert-team/DSWX-SAR/src/dswx_sar/dswx_s1.py:51, in dswx_s1_workflow(cfg)
49 processing_cfg.polarizations = pol_set
50 # preprocessing (relocating ancillary data and filtering)
---> 51 pre_processing.run(cfg)
53 # Estimate threshold for given polarizations
54 initial_threshold.run(cfg)
File ~/bekaert-team/DSWX-SAR/src/dswx_sar/pre_processing.py:706, in run(cfg)
703 logger.warning(f'{anc_type} has invalid values.')
704 # update reference water if any values of reference water
705 # are invalid.
--> 706 replace_reference_water_nodata_from_ancillary(
707 os.path.join(scratch_dir,
708 relocated_ancillary_filename_set['reference_water']),
709 os.path.join(scratch_dir,
710 relocated_ancillary_filename_set['landcover']),
711 os.path.join(scratch_dir,
712 relocated_ancillary_filename_set['hand']),
713 reference_water_max=ref_water_max,
714 reference_water_no_data=ref_water_no_data,
715 line_per_block=line_per_block
716 )
718 # apply SAR filtering
719 pad_shape = (filter_size, 0)
File ~/bekaert-team/DSWX-SAR/src/dswx_sar/pre_processing.py:537, in replace_reference_water_nodata_from_ancillary(reference_water_path, landcover_path, hand_path, reference_water_max, reference_water_no_data, line_per_block)
534 landcover_label = get_label_landcover_esa_10()
536 for block_param in block_params:
--> 537 ref_water_block, landcover_block, hand_block = [
538 dswx_sar_util.get_raster_block(path, block_param)
539 for path in [temp_water_path,
540 landcover_path,
541 hand_path]]
543 # Both invalid and no-water areas have zero value
544 # in reference water. The area with zero values are
545 # replaced with maximum values where the permanent water
546 # area in landcover.
548 replaced_area = np.logical_and(
549 ref_water_block == reference_water_no_data,
550 landcover_block == landcover_label['Permanent water bodies'])
File ~/bekaert-team/DSWX-SAR/src/dswx_sar/pre_processing.py:538, in <listcomp>(.0)
534 landcover_label = get_label_landcover_esa_10()
536 for block_param in block_params:
537 ref_water_block, landcover_block, hand_block = [
--> 538 dswx_sar_util.get_raster_block(path, block_param)
539 for path in [temp_water_path,
540 landcover_path,
541 hand_path]]
543 # Both invalid and no-water areas have zero value
544 # in reference water. The area with zero values are
545 # replaced with maximum values where the permanent water
546 # area in landcover.
548 replaced_area = np.logical_and(
549 ref_water_block == reference_water_no_data,
550 landcover_block == landcover_label['Permanent water bodies'])
File ~/bekaert-team/DSWX-SAR/src/dswx_sar/dswx_sar_util.py:608, in get_raster_block(raster_path, block_param)
605 ds_data = gdal.Open(raster_path, gdal.GA_Update)
607 # Number of bands in the raster
--> 608 num_bands = ds_data.RasterCount
609 # List to store blocks from each band
610 data_blocks = []
AttributeError: 'NoneType' object has no attribute 'RasterCount'
At first, I thought it had to do with auxiliary data not having enough buffer since it said that the nodata was being carried over into the reprojected reference water map. However, buffering by 1 degree yielded a similar issue.
Specifically, the function validate_gtiff here makes sure there are no nodata values as indicated here:
|
# check if relocated ancillaries are filled with invalid values |
|
validation_result = validate_gtiff( |
|
os.path.join(scratch_dir, anc_filename), |
|
[no_data, np.inf]) |
. The particular reprojected Pekel occurence map (this reprojection was done running the
dswx-s1 workflow) looked like this:
.
First, I removed the no_data in the validation here:
|
# check if relocated ancillaries are filled with invalid values |
|
validation_result = validate_gtiff( |
|
os.path.join(scratch_dir, anc_filename), |
|
[no_data, np.inf]) |
Still same error.
So I went into dswx_utils and modifying this line:
ds_data = gdal.Open(raster_path, gdal.GA_Update)
to
ds_data = gdal.Open(raster_path)
Allowed the code to continue. Not sure if it's correct.
This seems like it could be gdal issue. I am using version 3.8.3 for the above as indicated with conda list | grep gdal:
gdal 3.8.3 py311h4a095a9_0 conda-forge
libgdal 3.8.3 h7e86f1f_0 conda-forge
My issues could be related to the minor release in November and making the runtime environment inconsistent with that which is being developed here. I have not checked this as I had a hard time pinpointing the bug. I am saying this because I had similar issues with DSWx-HLS recently as documented here. It turned out the expected runtime environment required gdal<3.8 and I was similarly using the latest gdal.
Please let me know:
- what the development gdal version is
- if removing
gdal.GA_Update in this particular place is intended/acceptable/etc.
In the latest version the preprocessing routines, I was getting:
At first, I thought it had to do with auxiliary data not having enough buffer since it said that the nodata was being carried over into the reprojected reference water map. However, buffering by 1 degree yielded a similar issue.
Specifically, the function
validate_gtiffhere makes sure there are nonodatavalues as indicated here:DSWX-SAR/src/dswx_sar/pre_processing.py
Lines 688 to 691 in 5f4c320
dswx-s1workflow) looked like this:First, I removed the
no_datain the validation here:DSWX-SAR/src/dswx_sar/pre_processing.py
Lines 688 to 691 in 5f4c320
Still same error.
So I went into
dswx_utilsand modifying this line:to
Allowed the code to continue. Not sure if it's correct.
This seems like it could be gdal issue. I am using version
3.8.3for the above as indicated withconda list | grep gdal:My issues could be related to the minor release in November and making the runtime environment inconsistent with that which is being developed here. I have not checked this as I had a hard time pinpointing the bug. I am saying this because I had similar issues with DSWx-HLS recently as documented here. It turned out the expected runtime environment required
gdal<3.8and I was similarly using the latestgdal.Please let me know:
gdal.GA_Updatein this particular place is intended/acceptable/etc.