run lots of images

[lanzhiqishi]

Dear fsteinmetz,

by the way, any way to limit the lat or lon? i want release the shoratge , thanks.

i try to calibration with msi, i find the file "para" only for gain msi,how could i put new calibration coefficent for both S2A and S2B, thanks.

Best

Binbin

[fsteinmetz]

Dear Binbin,
- To avoid writing band 1610, you can pass the argument bands_rw = [443,490,560,665,705,740,783,842,865] to run_atm_corr. You can take example on params.py, in method defaults_msi.
- To avoid writing some parameters, you can pass the argument datasets = ['latitude', 'longitude','Rw', 'Rnir', 'bitmask', 'logchl', 'bbs', 'Rgli'] to your level2 object. You can remove the variables you don't want from that list. The default values for this list are in level2.py
- The lat and lon can not be limited directly. However, if you first identify the rows and columns you want to process in the image, you can pass the arguments sline, eline, scol and ecol to Level1_MSI. This will allow processing only a subset of an image.
- Other calibration gains can be used by passing argument calib to run_atm_corr, following the dictionnary defined in params.py:defaults_msi. Currently this dictionnay contains only "1." because vicarious calibration gains have not been calculated for MSI.

Cheers,
François

[lanzhiqishi]

Dear festeinmetz,


I saw the calibration coefficient (Nima Pahlevan 2019 RSE) and (sentinel2 L1C 2020 April ESA), so use the latter one. but when I change it, run well at the beginning. but now it not work, the error detail below. it look likes my storage is full. But I check it have more than 1Tb. Thanks



Best

Binbin






/home/bjiang/.conda/envs/binbin1/lib/python3.7/site-packages/pyproj/crs/crs.py:280: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable ... -in-proj-6
projstring = _prepare_from_string(projparams)
Traceback (most recent call last):
File "sentinelbatchrun.py", line 44, in <module>
Level1_MSI(result,resolution='20'),
File "/nfs3-home/bjiang/software/pythondepen/polymer-v4.13/polymer/level1_msi.py", line 151, in __init__
self.init_ancillary()
File "/nfs3-home/bjiang/software/pythondepen/polymer-v4.13/polymer/level1_msi.py", line 181, in init_ancillary
self.wind_speed = self.ancillary.get('wind_speed', self.date)
File "/nfs3-home/bjiang/software/pythondepen/polymer-v4.13/polymer/ancillary.py", line 251, in get
D1 = self.read(param, res[0])
File "/nfs3-home/bjiang/software/pythondepen/polymer-v4.13/polymer/ancillary.py", line 179, in read
decomp_file.write(decompdata)
File "/home/bjiang/.conda/envs/binbin1/lib/python3.7/tempfile.py", line 481, in func_wrapper
return func(*args, **kwargs)
OSError: [Errno 28] No space left on device

[fsteinmetz]

Hi Binbin,
Here the error is raised when writing a temporary file for ancillary data. This happens by default in /tmp (you can check this by running python -c 'import tempfile ; print(tempfile.gettempdir())'). It may be that your /tmp directory is full.
The location for temporary files can also be changed with environment variables, see https://docs.python.org/3/library/tempf ... gettempdir
Cheers,
François

[lanzhiqishi]

Dear fsteinmetz,


it looks download file not complied from NASA. the first run very well. But the second that it is not work any more. so the issue is network or NASA website, or software 'polymer'? How could i fix it? Do you have some suggestions? thanks




Best

Binbin




error:

Initialize MSI projection EPSG:32619
Starting processing at 2021-05-19 10:10:41.063704
Removing file /home/bjiang/bjiang_tmp_20210303/Machias2016-2020/polymer/L1C_T19TFK_A015967_20200327T152624_20mpolymer_S2B.nc
Initializing output file "/home/bjiang/bjiang_tmp_20210303/Machias2016-2020/polymer/L1C_T19TFK_A015967_20200327T152624_20mpolymer_S2B.nc"
/home/bjiang/.conda/envs/binbin1/lib/python3.7/site-packages/pyproj/crs/crs.py:280: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable ... -in-proj-6
projstring = _prepare_from_string(projparams)
Processing block: size (198, 5490), offset (0, 0)
Processing block: size (198, 5490), offset (198, 0)
Processing block: size (198, 5490), offset (396, 0)
Processing block: size (198, 5490), offset (594, 0)
Processing block: size (198, 5490), offset (792, 0)
Processing block: size (198, 5490), offset (990, 0)
Processing block: size (198, 5490), offset (1188, 0)
Processing block: size (198, 5490), offset (1386, 0)
Processing block: size (198, 5490), offset (1584, 0)
Processing block: size (198, 5490), offset (1782, 0)
Processing block: size (198, 5490), offset (1980, 0)
Processing block: size (198, 5490), offset (2178, 0)
Processing block: size (198, 5490), offset (2376, 0)
Processing block: size (198, 5490), offset (2574, 0)
Processing block: size (198, 5490), offset (2772, 0)
Processing block: size (198, 5490), offset (2970, 0)
Processing block: size (198, 5490), offset (3168, 0)
Processing block: size (198, 5490), offset (3366, 0)
Processing block: size (198, 5490), offset (3564, 0)
Processing block: size (198, 5490), offset (3762, 0)
Processing block: size (198, 5490), offset (3960, 0)
Processing block: size (198, 5490), offset (4158, 0)
Processing block: size (198, 5490), offset (4356, 0)
Processing block: size (198, 5490), offset (4554, 0)
Processing block: size (198, 5490), offset (4752, 0)
Processing block: size (198, 5490), offset (4950, 0)
Processing block: size (198, 5490), offset (5148, 0)
Processing block: size (144, 5490), offset (5346, 0)
Done in 0:08:29.492616
Initialize MSI projection EPSG:32619
Trying to download https://oceandata.sci.gsfc.nasa.gov/cgi ... MI_24h.hdf ...
2021-05-19 10:19:27 URL:https://oceandata.sci.gsfc.nasa.gov/ob/ ... MI_24h.hdf [201285/201285] -> "ANCILLARY/METEO/2019/357/N201935700_O3_AURAOMI_24h.hdf.tmp" [1]
Trying to download https://oceandata.sci.gsfc.nasa.gov/cgi ... 6h.hdf.bz2 ...
2021-05-19 10:19:28 URL:https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2 [848287/848287] -> "ANCILLARY/METEO/2019/357/N201935712_MET_NCEPR2_6h.hdf.bz2.tmp" [1]
sh: bzip2: command not found
‘ANCILLARY/METEO/2019/357/N201935712_MET_NCEPR2_6h.hdf.bz2’ -> ‘ANCILLARY/METEO/2019/357/N201935712_MET_NCEPR2_6h.hdf’
Trying to download https://oceandata.sci.gsfc.nasa.gov/cgi ... 6h.hdf.bz2 ...
2021-05-19 10:19:30 URL:https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2 [848804/848804] -> "ANCILLARY/METEO/2019/357/N201935718_MET_NCEPR2_6h.hdf.bz2.tmp" [1]
sh: bzip2: command not found
‘ANCILLARY/METEO/2019/357/N201935718_MET_NCEPR2_6h.hdf.bz2’ -> ‘ANCILLARY/METEO/2019/357/N201935718_MET_NCEPR2_6h.hdf’
/home/bjiang/.conda/envs/binbin1/lib/python3.7/site-packages/pyproj/crs/crs.py:280: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable ... -in-proj-6
projstring = _prepare_from_string(projparams)
Traceback (most recent call last):
File "sentinelbatchrun_2018Machias.py", line 44, in <module>
Level1_MSI(result,resolution='20'),
File "/home/bjiang/software/pythondepen/polymer-v4.13/polymer/level1_msi.py", line 151, in __init__
self.init_ancillary()
File "/home/bjiang/software/pythondepen/polymer-v4.13/polymer/level1_msi.py", line 181, in init_ancillary
self.wind_speed = self.ancillary.get('wind_speed', self.date)
File "/home/bjiang/software/pythondepen/polymer-v4.13/polymer/ancillary.py", line 251, in get
D1 = self.read(param, res[0])
File "/home/bjiang/software/pythondepen/polymer-v4.13/polymer/ancillary.py", line 190, in read
hdf = SD(filename)
File "/home/bjiang/.conda/envs/binbin1/lib/python3.7/site-packages/pyhdf/SD.py", line 1429, in __init__
_checkErr('SD', id, "cannot open %s" % path)
File "/home/bjiang/.conda/envs/binbin1/lib/python3.7/site-packages/pyhdf/error.py", line 23, in _checkErr
raise HDF4Error(err)
pyhdf.error.HDF4Error: SD (15): File is supported, must be either hdf, cdf, netcdf

[fsteinmetz]

Dear Binbin,
You can try and remove the auxiliary files (at least the problematic one), run the code again, maybe there was a download error.
From your output, it looks like the executable bzip2 is missing on your system. I suggest intalling it and see if it solves your problem.
Cheers,
François

[southocean]

i try to run the code as follow, the error"NameError: name 'files' is not defined" is displayed, someone can help me ?
other,i want to know what is the "xml" in the "filename="xml"" ? a filesname list ?

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from polymer.main import run_atm_corr,Level1,Level2
#from polymer.level2_hdf import Level2_HDF
from polymer.level2_nc import Level2_NETCDF
#from polymer.ancillary_era5 import Ancillary_ERA5
#from polymer.level1_nasa import Level1_NASA
#from polymer.level1_olci import Level1_OLCI
from polymer.level1_msi import Level1_MSI
import os

#find the dir
path="/home/huangwei/data/YX/"
#fin the file name
filename="xml"
#definte the result of find
result=[]

def findfiles(files_path,files_list):
#find the file code
files=os.listdir(files_path)
for s in files:
s_path=os.path.join(files_path,s)
if os.path.isdir(s_path):
findfiles(s_path,files_list)
elif os.path.isfile(s_path)and 'MTD_TL' in s:
result.append(s_path[0:-10])


if __name__=="__main__":
findfiles(path.result)
for i in range(len(result)):
print(result)

run_atm_corr(
level1_MSI(result,resolution='10'),# ancillary=ancillary_era5() # if not provided, use the 60m resolution
Level2_NETCDF(outdir='/home/huangwei/data/YX/out'),multiprocessing=-1)
# by default, output in the same folder as input