L8 ancillary error

[sydney.baratta]

Hi François,

I am receiving this error in regards to ancillary.py when trying to run my L8 imagery:

(polymer) [sbaratta@katahdin polymer-v4.16]$ python runPolymer.py
Traceback (most recent call last):
File "runPolymer.py", line 19, in <module>
Level1_OLI('/home/sbaratta/Downloads/polymer-v4.16/dur'),
File "/home/sbaratta/Downloads/polymer-v4.16/polymer/level1_landsat8.py", line 69, in __init__
self.ancillary = Ancillary_NASA()
File "/home/sbaratta/Downloads/polymer-v4.16/polymer/ancillary.py", line 181, in __init__
assert isdir(directory), 'Directory {} does not exist. Please create it, by default it will be automatically populated with ancillary data. Please see help for class Ancillary_NASA for more details.'.format(directory)
AssertionError: Directory ANCILLARY/METEO/ does not exist. Please create it, by default it will be automatically populated with ancillary data. Please see help for class Ancillary_NASA for more details.

And the links within the comments on ancillary.py (https://oceancolor.gsfc.nasa.gov/...) aren't working on my end. Any help would be greatly appreciated!

Best,
Sydney

[sydney.baratta]

Just an update and new error that has arisen:

I created the directory ANCILLARY/METEO/ in my polymer v4.16 directory, along with a .netrc file that includes my Earthdata login. However, I receive a value error ("ValueError: A value (1.1496814974406155) in x_new is above the interpolation range's maximum value (0.4000000059604645)") after connecting to Earthdata and trying to run the imagery.

Here is the full error:
Reading coordinates from /home/sbaratta/Downloads/polymer-v4.16/LC08_L1TP_001017_20230618_20230623_02_T1/LC08_L1TP_001017_20230618_20230623_02_T1_B1.TIF
/opt/ohpc/pub/anaconda3/users/sbaratta/2023.07/envs/polymer/lib/python3.8/site-packages/osgeo/gdal.py:287: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
warnings.warn(
Image size is 8201x8271
Starting processing at 2023-08-26 16:33:06.610714
Initializing output file "/home/sbaratta/Downloads/polymer-v4.16/LC08_L1TP_001017_20230618_20230623_02_T1test.nc"
Processing block: size (500, 400), offset (0, 0)
Processing block: size (500, 400), offset (0, 400)
Processing block: size (500, 400), offset (0, 800)
Processing block: size (500, 400), offset (0, 1200)
Processing block: size (500, 400), offset (0, 1600)
Processing block: size (500, 400), offset (0, 2000)
Processing block: size (500, 400), offset (0, 2400)
Processing block: size (500, 400), offset (0, 2800)
Processing block: size (500, 400), offset (0, 3200)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/ohpc/pub/anaconda3/users/sbaratta/2023.07/envs/polymer/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/sbaratta/Downloads/polymer-v4.16/polymer/main.py", line 413, in process_block
c.cloudmask(block)
File "/home/sbaratta/Downloads/polymer-v4.16/polymer/main.py", line 289, in cloudmask
block.Rnir[ok] = block.Rtoa_gc[:,:,inir_block][ok] - self.mlut['Rmol'][
File "/home/sbaratta/Downloads/polymer-v4.16/polymer/luts.py", line 399, in __getitem__
keys = k.index(self.axes)
File "/home/sbaratta/Downloads/polymer-v4.16/polymer/luts.py", line 1110, in index
res = interp1d(axis, np.arange(len(axis)),
File "/opt/ohpc/pub/anaconda3/users/sbaratta/2023.07/envs/polymer/lib/python3.8/site-packages/scipy/interpolate/_polyint.py", line 80, in __call__
y = self._evaluate(x)
File "/opt/ohpc/pub/anaconda3/users/sbaratta/2023.07/envs/polymer/lib/python3.8/site-packages/scipy/interpolate/_interpolate.py", line 752, in _evaluate
below_bounds, above_bounds = self._check_bounds(x_new)
File "/opt/ohpc/pub/anaconda3/users/sbaratta/2023.07/envs/polymer/lib/python3.8/site-packages/scipy/interpolate/_interpolate.py", line 786, in _check_bounds
raise ValueError("A value ({}) in x_new is above "
ValueError: A value (1.1496814974406155) in x_new is above the interpolation range's maximum value (0.4000000059604645).
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "runPolymer.py", line 18, in <module>
run_atm_corr(
File "/home/sbaratta/Downloads/polymer-v4.16/polymer/main.py", line 523, in run_atm_corr
for block in block_iter:
File "/opt/ohpc/pub/anaconda3/users/sbaratta/2023.07/envs/polymer/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
ValueError: A value (1.1496814974406155) in x_new is above the interpolation range's maximum value (0.4000000059604645).
Processing block: size (500, 400), offset (0, 3600)

Thanks,
Sydney

[sakvaka_env]

Are you using version 4.16.1 instead of 4.16? It has fixed an issue related to ancillary data, which manifested itself as an index overflow in LUT table lookup.

viewtopic.php?f=5&t=210

[sydney.baratta]

I am using 4.16, should I instead try 4.16.1? Also, what version of python should I be using with this?

Best,
Sydney

[sydney.baratta]

Another update and question about 4.16.1 installation -

I tried v4.16.1 with python v3.8, and am receiving this error when trying to run my imagery:

(polymer) [sbaratta@katahdin polymer-v4.16.1]$ python runPolymer.py

Traceback (most recent call last):
File "runPolymer.py", line 1, in <module>
from polymer.main import run_atm_corr
File "/home/sbaratta/Downloads/polymer-v4.16.1/polymer/main.py", line 16, in <module>
from polymer.polymer_main import PolymerMinimizer
ModuleNotFoundError: No module named 'polymer.polymer_main'


Do you know what I need to do that is different from 4.16?

[sakvaka_env]

Another update and question about 4.16.1 installation -

I tried v4.16.1 with python v3.8, and am receiving this error when trying to run my imagery:

(polymer) [sbaratta@katahdin polymer-v4.16.1]$ python runPolymer.py

Traceback (most recent call last):
File "runPolymer.py", line 1, in <module>
from polymer.main import run_atm_corr
File "/home/sbaratta/Downloads/polymer-v4.16.1/polymer/main.py", line 16, in <module>
from polymer.polymer_main import PolymerMinimizer
ModuleNotFoundError: No module named 'polymer.polymer_main'


Do you know what I need to do that is different from 4.16?

Did you compile Polymer? Quoting from the README.md:

### 1.3 Compilation

The pyx files are cython files which need to be converted to C, then compiled.
A makefile is provided, so just type:
```
$ make
```

NOTE: the command `make all` will download the auxiliary files and proceed to the compilation.