polymer 4.12 throws ValueError "masked arrays not supported"

Post Reply
martin.boettcher.bc
Posts: 2
Joined: Thu Oct 10, 2019 3:04 pm
company / institution: Brockmann Consult GmbH
Location: Hamburg, Germany

polymer 4.12 throws ValueError "masked arrays not supported"

Post by martin.boettcher.bc » Fri Oct 11, 2019 7:11 am

Dear Francois, dear colleagues,

after Marco has left us it is on me to compile the newest version of Polymer for Calvalus. When running it with ancillary data from either ERA5 or NASA I run into an exception. Maybe, I made a mistake when setting up the miniconda environment. But I do not see it myself. Do you have an idea?

Best regards,
Martin

The output is as follows:
-----
height=4091, width=4865
Download 2019-06-03 11:00:00 -> ANCILLARY/ERA5/2019/06/03/era5_20190603_110000.nc
2019-10-10 23:11:23,198 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2019-10-10 23:11:24,073 INFO Request is queued
2019-10-10 23:11:25,104 INFO Request is running
2019-10-10 23:11:28,917 INFO Request is completed
2019-10-10 23:11:28,917 INFO Downloading http://136.156.133.39/cache-compute-0012/cache/data1/adaptor.mars.internal-1570741884.5034971-18471-3-b3af2ac7-76d7-45e0-a866-63a5621c4616.nc to ANCILLARY/ERA5/2019/06/03/era5_20190603_110000.nc.tmp (9.9M)
2019-10-10 23:11:33,290 INFO Download rate 2.3M/s
Download 2019-06-03 12:00:00 -> ANCILLARY/ERA5/2019/06/03/era5_20190603_120000.nc
2019-10-10 23:11:33,363 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2019-10-10 23:11:33,449 INFO Request is queued
2019-10-10 23:11:34,478 INFO Request is running
2019-10-10 23:11:36,009 INFO Request is completed
2019-10-10 23:11:36,009 INFO Downloading http://136.156.132.110/cache-compute-0001/cache/data7/adaptor.mars.internal-1570741893.5790102-19835-5-9c654a13-3d5c-4371-a878-51f2b733faab.nc to ANCILLARY/ERA5/2019/06/03/era5_20190603_120000.nc.tmp (9.9M)
2019-10-10 23:11:42,552 INFO Download rate 1.5M/s
Starting processing at 2019-10-10 23:11:42.752118
Initializing output file "/home/yarn/S3B_OL_1_EFR____20190603T110400_20190603T110700_20190604T174819_0179_026_094_1980_MAR_O_NT_002.SEN3.nc"
Traceback (most recent call last):
File "example_boe.py", line 19, in <module>
example_olci2()
File "example_boe.py", line 14, in example_olci2
Level2_NETCDF(outdir='/home/yarn')
File "/home/yarn/polymer-v4.12/polymer/main.py", line 517, in run_atm_corr
for block in block_iter:
File "/home/yarn/polymer-v4.12/polymer/main.py", line 434, in blockiterator
for block in level1.blocks(params.bands_read()):
File "/home/yarn/polymer-v4.12/polymer/level1_olci.py", line 350, in blocks
yield self.read_block(size, offset, bands_read)
File "/home/yarn/polymer-v4.12/polymer/level1_olci.py", line 285, in read_block
block.ozone = self.ozone[block.latitude, block.longitude]
File "/home/yarn/polymer-v4.12/polymer/ancillary.py", line 58, in __getitem__
return self.data[Idx(lat), Idx(lon)]
File "/home/yarn/polymer-v4.12/polymer/luts.py", line 389, in __getitem__
keys[i] = k.index(self.axes[i])
File "/home/yarn/polymer-v4.12/polymer/luts.py", line 1056, in index
fill_value=fv)(self.value)
File "/home/yarn/opt/miniconda3-polymer/lib/python3.7/site-packages/scipy/interpolate/polyint.py", line 78, in __call__
x, x_shape = self._prepare_x(x)
File "/home/yarn/opt/miniconda3-polymer/lib/python3.7/site-packages/scipy/interpolate/polyint.py", line 90, in _prepare_x
x = _asarray_validated(x, check_finite=False, as_inexact=True)
File "/home/yarn/opt/miniconda3-polymer/lib/python3.7/site-packages/scipy/_lib/_util.py", line 237, in _asarray_validated
raise ValueError('masked arrays are not supported')
ValueError: masked arrays are not supported
-----

My installation procedure and the actual call are:

sudo apt install gcc
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod 755 Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh
# install to /home/yarn/opt/miniconda3-polymer
export PATH=/home/yarn/opt/miniconda3-polymer/bin:$PATH

tar xf polymer-v4.12.tar.gz
cd polymer-v4.12
./install-anaconda-deps.sh
make auxdata_all
make
make ancillary

cat example_boe.py
-----
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from polymer.ancillary_era5 import Ancillary_ERA5
from polymer.ancillary import Ancillary_NASA
from polymer.main import run_atm_corr, Level1, Level2
from polymer.level2_nc import Level2_NETCDF
from polymer.level1_olci import Level1_OLCI
def example_olci2():
run_atm_corr(
Level1_OLCI('/home/yarn/S3B_OL_1_EFR____20190603T110400_20190603T110700_20190604T174819_0179_026_094_1980_MAR_O_NT_002.SEN3',
sline=3000, eline=3299, ancillary=Ancillary_ERA5()),
Level2_NETCDF(outdir='/home/yarn')
)
if __name__ == "__main__":
example_olci2()
-----

# a call without using ancillary data that succeeds:
./polymer_cli.py ../S3B_OL_1_EFR____20190603T110400_20190603T110700_20190604T174819_0179_026_094_1980_MAR_O_NT_002.SEN3 test.nc
# runs for ~30 minutes and generates an .nc file

mkdir ANCILLARY/ERA5/
# the actual call that returns the error:
python example_boe.py
User avatar
fsteinmetz
Site Admin
Posts: 50
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: polymer 4.12 throws ValueError "masked arrays not supported"

Post by fsteinmetz » Mon Oct 14, 2019 10:03 am

Hi Martin,

Thanks for the detailed report !
Indeed there is a bug, external ancillary data can not be applied to OLCI due to the NetCDF library returning masked arrays.

This can be fixed with the following patch :

Code: Select all

diff --git a/polymer/level1_olci.py b/polymer/level1_olci.py
index 10a4a63..55787d6 100644
--- a/polymer/level1_olci.py
+++ b/polymer/level1_olci.py
@@ -170,6 +170,7 @@ class Level1_OLCI(Level1_base):
             return self.nc_datasets[filename]
 
         self.nc_datasets[filename] = Dataset(os.path.join(self.dirname, filename))
+        self.nc_datasets[filename].set_auto_mask(False)
 
         return self.nc_datasets[filename]
I will include this in next Polymer release.
Cheers,
François
martin.boettcher.bc
Posts: 2
Joined: Thu Oct 10, 2019 3:04 pm
company / institution: Brockmann Consult GmbH
Location: Hamburg, Germany

Re: polymer 4.12 throws ValueError "masked arrays not supported"

Post by martin.boettcher.bc » Tue Oct 15, 2019 8:12 am

Thank you, Francois, the patch fixes the problem.
Post Reply