Problem while running Polymer

[dmantsis]

Hi
I am using Polymer to perform atmospheric correction on a Sentinel-2 image and I am running the following code

run_atm_corr(Level1_MSI('/home/dmantsis/Data_Temp/Polyfytos/'\
'S2B_MSIL1C_20171230T092359_N0206_R093_T34TEK_20171230T115949.SAFE/'\
'GRANULE/L1C_T34TEK_A004266_20171230T092450/',
ancillary=Ancillary_ERA5(),
resolution='10'),
Level2())

The code runs for many hours and after it shows on the screen the following it crashes with an error. Can anyone help me figure out what the problem is? Thank you for helping.

Processing block: size (198, 10980), offset (0, 0)
Processing block: size (198, 10980), offset (198, 0)
Processing block: size (198, 10980), offset (396, 0)
Processing block: size (198, 10980), offset (594, 0)
.........
Processing block: size (198, 10980), offset (10692, 0)
Processing block: size (90, 10980), offset (10890, 0)

Write compressed dataset Rgli
Write compressed dataset Rnir
Write compressed dataset Rw1610
Write compressed dataset Rw443
Write compressed dataset Rw490
Write compressed dataset Rw560
Write compressed dataset Rw665
Write compressed dataset Rw705

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_1599480/2043740210.py in <module>
----> 1 run_atm_corr(Level1_MSI('/home/dmantsis/Data_Temp/Polyfytos/'\
2 'S2B_MSIL1C_20171230T092359_N0206_R093_T34TEK_20171230T115949.SAFE/'\
3 'GRANULE/L1C_T34TEK_A004266_20171230T092450/',
4 ancillary=Ancillary_ERA5(),
5 resolution='10'),

~/AtmCorr/Polymer/polymer-v4.14/polymer/main.py in run_atm_corr(level1, level2, **kwargs)
522 params.update(**l1.attributes('%Y-%m-%d %H:%M:%S'))
523 params.update(**l2.attributes())
--> 524 l2.finish(params)
525
526 if params.multiprocessing != 0:

~/AtmCorr/Polymer/polymer-v4.14/polymer/level2_hdf.py in finish(self, params)
133 sds2 = hdf.create(name, dtype, self.shape)
134 sds2.setcompress(SDC.COMP_DEFLATE, 9)
--> 135 sds2[:] = sds[:]
136
137 sds2.endaccess()

~/anaconda3/envs/polymer/lib/python3.8/site-packages/pyhdf/SD.py in __setitem__(self, elem, data)
1813 data = [data]
1814 # Assign.
-> 1815 self.set(data, start, count, stride)
1816
1817 def endaccess(self):

~/anaconda3/envs/polymer/lib/python3.8/site-packages/pyhdf/SD.py in set(self, data, start, count, stride)
1999 'with the SDS data type')
2000
-> 2001 _C._SDwritedata_0(self._id, data_type, start, count, data, stride)
2002
2003 def __buildStartCountStride(self, elem):

ValueError: SDwritedata failure
-------------------------------------------------------------------------------------------------------------
Can anyone help me figure out what the problem is? Thank you for helping.

[fsteinmetz]

Hello,
This may be due to a limitation of hdf4 file format, which does not support files greater than 2Gb.
Please try to use Level2_NETCDF, or Level2(fmt='netcdf4'), instead of Level2().
Cheers,
François

[dmantsis]

Thank you
I am using NetCDF now and it is working
Damianos