The hdf output can not processed completely and the output netcdf image can not match with the original input in SNAP

Post Reply
yifanaaa
Posts: 3
Joined: Fri Aug 14, 2020 8:59 am
company / institution: Institute of oceanology, Chinese academy of sciences
Location: qingdao, China

The hdf output can not processed completely and the output netcdf image can not match with the original input in SNAP

Post by yifanaaa »

I have used a python script, attached in the following, to use polymer in msi images. When i use nc as output format, the process is processed very well. However, when i change it to hdf as output format, the wrong message comes out, which I have attached as follows. Can anyone help me with this question?

Also, I am really confused with the output netcdf result. When I open it in SNAP software, with synchronizes views across multiple image windows tool, the atmosphere corrected image is not positioned quite well with the original msi level1 image. The scale of the output is quite smaller than the original level1 image. But I think the geographical range is similar, because the image looks similar in single grey band. Is there anything I should do to process the nc file or do i need to set the spatial resolution image in run_atm_corr script.


Python Script:

Code: Select all

from polymer.main import run_atm_corr
from polymer.level2_hdf import Level2_HDF
from polymer.level1_msi import Level1_MSI
def example_msi():
   
    run_atm_corr(
            Level1_MSI('/home/data/S2A_MSIL1C_20170526T022551_N0205_R046_T51SVT_20170526T023526/S2A_MSIL1C_20170526T022551_N0205_R046_T51SVT_20200812T040015.SAFE/GRANULE/L1C_T51SVT_A010053_20170526T023526/'                     
                       ),
            Level2_HDF(outdir='/home/data/test/')  
            )
if __name__ == "__main__":
    example_msi()
    

Result:
Initialize MSI projection EPSG:32651
/home/anaconda3/lib/python3.6/site-packages/pyproj/crs/crs.py:280: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable ... -in-proj-6
projstring = _prepare_from_string(projparams)
Starting processing at 2020-08-25 14:57:53.979668
Initializing output file "/home/data/test/L1C_T51SVT_A010053_20170526T023526.hdf"
Processing block: size (198, 1830), offset (0, 0)
Processing block: size (198, 1830), offset (198, 0)
Processing block: size (198, 1830), offset (396, 0)
Processing block: size (198, 1830), offset (594, 0)
Processing block: size (198, 1830), offset (792, 0)
Processing block: size (198, 1830), offset (990, 0)
Processing block: size (198, 1830), offset (1188, 0)
Processing block: size (198, 1830), offset (1386, 0)
Processing block: size (198, 1830), offset (1584, 0)
Processing block: size (48, 1830), offset (1782, 0)
Write compressed dataset Rgli
Write compressed dataset Rnir
Write compressed dataset Rw1610
Write compressed dataset Rw443
Write compressed dataset Rw490
Write compressed dataset Rw560
Write compressed dataset Rw665
Write compressed dataset Rw705
Write compressed dataset Rw740
Write compressed dataset Rw783
Write compressed dataset Rw842
Write compressed dataset Rw865
Write compressed dataset bbs
Write compressed dataset bitmask
Write compressed dataset latitude
Write compressed dataset logchl
Write compressed dataset longitude
Traceback (most recent call last):
File "/home/polymer-v4.13/polymer/main.py", line 524, in run_atm_corr
l2.finish(params)
File "/home/polymer-v4.13/polymer/level2_hdf.py", line 147, in finish
setattr(hdf, str(k), pstr(v))
File "/home/anaconda3/lib/python3.6/site-packages/pyhdf/SD.py", line 1455, in __setattr__
_setattr(self, name, value, ['_id'])
File "/home/anaconda3/lib/python3.6/site-packages/pyhdf/SD.py", line 3268, in _setattr
a.set(xtype, value)
File "/home/anaconda3/lib/python3.6/site-packages/pyhdf/SD.py", line 1330, in set
buf[n] = values[n]
File "/home/anaconda3/lib/python3.6/site-packages/pyhdf/hdfext.py", line 168, in __setitem__
return _hdfext.array_byte___setitem__(self, index, value)
OverflowError: in method 'array_byte___setitem__', argument 3 of type 'unsigned char'
Image
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./example2.py", line 29, in <module>
example_msi()
File "./example2.py", line 24, in example_msi
Level2_HDF(outdir='/home/data/test/') # by default, output in the same folder as input
File "/home/polymer-v4.13/polymer/main.py", line 532, in run_atm_corr
return l2
File "/home/polymer-v4.13/polymer/level2_hdf.py", line 177, in __exit__
self.cleanup()
File "/home/polymer-v4.13/polymer/level2_hdf.py", line 171, in cleanup
remove(f)
OSError: [Errno 26] Text file busy: '/home/data/test/L1C_T51SVT_A010053_20170526T023526.hdf.tmp_latitude.tmp'
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: The hdf output can not processed completely and the output netcdf image can not match with the original input in SNA

Post by fsteinmetz »

Hello,

I tried to reproduce your error with HDF output but couldn't, in my case it didn't fail. Maybe if you could export your anaconda environment, if you are using anaconda, that could be helpful (conda env export > environment.yml).

Concerning the error of geolocation in SNAP, this could be due to a bug discussed in this thread : viewtopic.php?f=7&t=73

Cheers,
François
yifanaaa
Posts: 3
Joined: Fri Aug 14, 2020 8:59 am
company / institution: Institute of oceanology, Chinese academy of sciences
Location: qingdao, China

Re: The hdf output can not processed completely and the output netcdf image can not match with the original input in SNA

Post by yifanaaa »

fsteinmetz wrote:
Wed Aug 26, 2020 1:20 pm
Hello,

I tried to reproduce your error with HDF output but couldn't, in my case it didn't fail. Maybe if you could export your anaconda environment, if you are using anaconda, that could be helpful (conda env export > environment.yml).

Concerning the error of geolocation in SNAP, this could be due to a bug discussed in this thread : viewtopic.php?f=7&t=73

Cheers,
François

Thanks so much for your reply. I have exported the enviroment.yml out.
Wish it is easier for you to help me figure out this trouble. Thanks again for your kind help.
download/file.php?mode=view&id=38
You do not have the required permissions to view the files attached to this post.
Post Reply