batch processing erorr
Posted: Tue Dec 21, 2021 9:37 pm
Hi,
I am having an issue processing multi year data sets of OLCI. It was working for some time and then I get an error "OSError: [Errno 24] too many files open". Now everytime i run it after the error it gets through 36 files and errors out.
I see this was an issue in the past for someone else (see this thread- viewtopic.php?f=7&t=145&p=565&hilit=%5BErrno+24%5D#p565)
However in the thread while the issue was resolved the answer wasn't posted only mentioned in broad terms.
I understand that there is a pileup of .tmp files somewhere but I can't find them. As polymer runs I see the .tmp file be created in the outdir I designate and then the .nc file is created and the .tmp file is removed (recycle bin is empty the whole time). So I am not sure where the buildup of files occurs or where they should be closed. I have had a look in the root tmp folders and don't see anything there. any suggestions or recommendation?
Below is the script I use to run each years worth of data.
Thank you,
Jonathan Sherman
def polymer_olci_batch(data_path):
"""
Batch processing of Sentinel-3 OLCI
"""
from polymer.main import run_atm_corr, Level1, Level2
from polymer.level2_nc import Level2_NETCDF
from polymer.level1_olci import Level1_OLCI
import glob
l1_path = f'{data_path}/l1'
l2_path = f'{data_path}/l2/'
flist = sorted(glob.glob(f'{l1_path}/*'))
for fname in flist:
run_atm_corr(Level1_OLCI(fname), Level2_NETCDF(outdir=l2_path, ext='_polymer_L2.nc'), multiprocessing=-1)
# run_atm_corr(Level1_OLCI(fname), Level2_NETCDF(outdir=l2_path, ext='_polymer_L2.nc'))
I am having an issue processing multi year data sets of OLCI. It was working for some time and then I get an error "OSError: [Errno 24] too many files open". Now everytime i run it after the error it gets through 36 files and errors out.
I see this was an issue in the past for someone else (see this thread- viewtopic.php?f=7&t=145&p=565&hilit=%5BErrno+24%5D#p565)
However in the thread while the issue was resolved the answer wasn't posted only mentioned in broad terms.
I understand that there is a pileup of .tmp files somewhere but I can't find them. As polymer runs I see the .tmp file be created in the outdir I designate and then the .nc file is created and the .tmp file is removed (recycle bin is empty the whole time). So I am not sure where the buildup of files occurs or where they should be closed. I have had a look in the root tmp folders and don't see anything there. any suggestions or recommendation?
Below is the script I use to run each years worth of data.
Thank you,
Jonathan Sherman
def polymer_olci_batch(data_path):
"""
Batch processing of Sentinel-3 OLCI
"""
from polymer.main import run_atm_corr, Level1, Level2
from polymer.level2_nc import Level2_NETCDF
from polymer.level1_olci import Level1_OLCI
import glob
l1_path = f'{data_path}/l1'
l2_path = f'{data_path}/l2/'
flist = sorted(glob.glob(f'{l1_path}/*'))
for fname in flist:
run_atm_corr(Level1_OLCI(fname), Level2_NETCDF(outdir=l2_path, ext='_polymer_L2.nc'), multiprocessing=-1)
# run_atm_corr(Level1_OLCI(fname), Level2_NETCDF(outdir=l2_path, ext='_polymer_L2.nc'))