run lots of images

[lanzhiqishi]

Dear fsteinmetz,

i will run polymer more than thousand Sentinel2 images on linux, how could I do best? do you have any good exprience to share with me? Thank you so much.


Best

Binbin

[fsteinmetz]

Dear Binbin,
Good to know that you are planning for a massive processing. You can activate parallel processing with the argument multiprocessing=-1 which will make the processing faster on a single computer. Then you can use a simple loop over the files, or set up a processing on a cluster, depending on what is available at your institute.
Cheers,
François

[lanzhiqishi]

Dear Post by fsteinmetz ,

thanks for your suggestion. My school provide the supercomputer cluster for run it.


Best

Binbin

[lanzhiqishi]

Dear fsteinmetz,

when I prepare run lots of image, it fails. when I run each script, it works. but put them together , it get error. Could you help me? thanks.


best

Binbin

This is the error:
(binbin1) [bjiang@viz3]~/software/pythondepen/polymer-v4.13% python ./sentinelbatchrun.py
/home/bjiang/Sentinel2/20200618/GRANULE/L1C_T19TEK_A017154_20200618T153657/
Traceback (most recent call last):
File "./sentinelbatchrun.py", line 44, in <module>
Level1_MSI('result' ,resolution='20'),
File "/nfs3-home/bjiang/software/pythondepen/polymer-v4.13/polymer/level1_msi.py", line 122, in __init__
assert len(xmlfiles) == 1
AssertionError
(binbin1) [bjiang@viz3]~/software/pythondepen/polymer-v4.13%


this is my script 'sentinelbatchrun':

from polymer.main import run_atm_corr
from polymer.level1 import Level1
from polymer.level2 import Level2
from polymer.level1_msi import Level1_MSI
from polymer.level2_nc import Level2_NETCDF
import os



#find the dir
path = "/home/bjiang/Sentinel2/20200618/"
#find the file name
filename = "xml"
#definte the result of find
result = []

def findfiles(files_path, files_list):
#find the file code
files = os.listdir(files_path)
for s in files:
s_path = os.path.join(files_path, s)
if os.path.isdir(s_path):
findfiles(s_path, files_list)
elif os.path.isfile(s_path) and 'MTD_TL' in s:
result.append(s_path[0:-10])


if __name__ == "__main__":
findfiles(path,result)
for i in range(len(result)):


print(result)

run_atm_corr(
Level1_MSI('result' ,resolution='20'),
Level2_NETCDF(outdir='/home/bjiang/Sentinel2', ext='20mpolymer.nc',
overwrite=True), multiprocessing = -1
)

[fsteinmetz]

You are passing 'result' to Level1_MSI, which is not a variable but a string.

[lanzhiqishi]

Dear fsteinmetz，


Thanks a lot. it works.


Best

Binbin

[giuliasent@gmail.com]

Hi lanzhiqishi and fsteinmetz,

I also need to run Polymer for a big number of Sentinel-2 granules. Since I'm a begginer with bash and python, I was trying to adapt the script that lanzhiqishi posted on this forum (thanks a lot) but I am getting this error:

Traceback (most recent call last):
File "msi_multiprocessing.py", line 44, in <module>
Level1_MSI(result ,resolution='10'),
File "/home/gsent/Documents/Installers/polymer-v4.13/polymer/level1_msi.py", line 93, in __init__
if dirname.endswith('/'):
AttributeError: 'list' object has no attribute 'endswith'

I've been trying to solve it but with bad results.
Could you help me?

Thank you a lot in advance.

Best,
Giulia

[fsteinmetz]

Dear Julia,

It seems you are passing a list to Level1_MSI instead of a filename (str).
A simple loop on many granules may look like this in python:

Code: Select all

from pathlib import Path
for filename in Path('/path/to/directory/').glob('**/GRANULE/*'):
    run_atm_corr(Level1_MSI(str(filename)), ...)

[giuliasent@gmail.com]

Dear fsteinmetz,

Thanks a lot for your fast reply. That worked!

Cheers,

Giulia

[lanzhiqishi]

Dear fsteinmetz,

i want to save with Rw_443,.Rw560,Rw660... not Rw_1610 ,chl,,Rgli, Could you give me some suggestion? Thanks


Best

Binbin