Sanity check for ancillary data

Post Reply
blake
Posts: 4
Joined: Thu Oct 04, 2018 12:19 pm
company / institution: University of Stirling

Sanity check for ancillary data

Post by blake »

Hi,

Is this repeatable for anyone else (i am using 4.11) or have I done something else wrong?

When polymer goes to download the ancillary data required for S2MSI processing from nasa ocean colour I am only receiving the header file containing no data. (size 12.7kb) the normal is around 200kb for the ozone data. So polymer thinks it has the data then moves on until it attempts to read the file then errors out with a HDF read error as the file is not complete so cannot read it. Equally because it has the file (or thinks it does) it will not move onto any of the secondary options for download.

Thanks.
User avatar
fsteinmetz
Site Admin
Posts: 74
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: Sanity check for ancillary data

Post by fsteinmetz »

Hi Blake, and thanks for your feedback.

Indeed, you have done nothing wrong.
The problem is that downloading on oceancolor.gsfc.nasa.gov now (since January) requires an EarthData login. There will need to be a modification in polymer/ancillary.py to reflect this change.
In the meantime, I suggest you use polymer/ancillary_era5.py as an alternate source of ancillary data.
This can be done like so:

Code: Select all

from polymer.ancillary_era5 import Ancillary_ERA5
run_atm_corr(
    Level1_MSI(<S2_product>, ancillary=Ancillary_ERA5()),
    Level2...
    )
You will have to use a ECMWF login as described here : https://cds.climate.copernicus.eu/api-how-to

Cheers,
François
blake
Posts: 4
Joined: Thu Oct 04, 2018 12:19 pm
company / institution: University of Stirling

Re: Sanity check for ancillary data

Post by blake »

Hi François,

Many thanks for your response but I am trying to keep everything the same so would prefer to use the nasa data.

by editing line 295 in anciallry.py to :

cmd = 'wget --user=insert_username --password=insert_password --auth-no-challenge=on -nv {} -O {} '.format(url, target+'.tmp')

Will make it run correctly however my understanding of using "--auth-no-challenge=on" is that it will send your username and password in plain text unencrypted for authorisation because that is they only way it will work with nasa ocean colour. So long as no one is re-using password then its maybe not so bad. Hopefully there is a better way for long term fix (and more elegant).

Cheers
User avatar
fsteinmetz
Site Admin
Posts: 74
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: Sanity check for ancillary data

Post by fsteinmetz »

Great, thanks for sending your solution !
It's not so bad as you say, but it will need a fix for the long term.
Cheers
User avatar
fsteinmetz
Site Admin
Posts: 74
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: Sanity check for ancillary data

Post by fsteinmetz »

Fixed in v4.13.
Post Reply