Download ancillary data on-the-fly

Post Reply
scheunro
Posts: 4
Joined: Mon Mar 23, 2020 10:16 am
company / institution: Eawag
Location: Zurich

Download ancillary data on-the-fly

Post by scheunro »

Hi,

How do I configure polymer to make the download of ancillary data work?

I tried executing the code suggested here before calling polymer but it did not help.

Any help appreciated.

Best,
Rolf

polymer version: 4.13
OS: CentOS 7
wget: 1.14-18.el7_6.1.x86_64
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: Download ancillary data on-the-fly

Post by fsteinmetz »

Hello Rolf,

The different ancillary data modules (NASA, ERA-Interim, ERA5) should already download the necessary ancillary files by themselves.
Please have a try; the error messages should guide you if any further action is required on your side. If anything is unclear, let me know of the error message.
Cheers,
François
scheunro
Posts: 4
Joined: Mon Mar 23, 2020 10:16 am
company / institution: Eawag
Location: Zurich

Re: Download ancillary data on-the-fly

Post by scheunro »

I am using the NASA module.
From error message it says me and from the code I see it requires a cookie file ".urs_cookie", how do I create this cookie file?
scheunro
Posts: 4
Joined: Mon Mar 23, 2020 10:16 am
company / institution: Eawag
Location: Zurich

Re: Download ancillary data on-the-fly

Post by scheunro »

I found this here: https://wiki.earthdata.nasa.gov/display ... L+And+Wget
I followed the instructions, but Step 4 does not work. What am I doing wrong?
What is the name of the application to be preauthorized in my https://urs.earthdata.nasa.gov/ login?
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: Download ancillary data on-the-fly

Post by fsteinmetz »

What error message do you get ?
With the .netrc set, could you try with the following command ?

Code: Select all

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge https://oceandata.sci.gsfc.nasa.gov/ob/getfile/N198101000_PROFILE_MERRA2_3h.nc
scheunro
Posts: 4
Joined: Mon Mar 23, 2020 10:16 am
company / institution: Eawag
Location: Zurich

Re: Download ancillary data on-the-fly

Post by scheunro »

Thank you for your reply.

Sadly, there is no error message; it just hangs. After about a minute, I cancel it. To me it looks like the forwarding to https://urs.earthdata.nasa.gov does not work with my wget.

Following is my output:

Code: Select all

$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge https://oceandata.sci.gsfc.nasa.gov/ob/getfile/N198101000_PROFILE_MERRA2_3h.nc
--2020-04-15 06:11:52--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/N198101000_PROFILE_MERRA2_3h.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
^C
And with verbose and debug enabled:

Code: Select all

$ wget -d -v --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge https://oceandata.sci.gsfc.nasa.gov/ob/getfile/N198101000_PROFILE_MERRA2_3h.nc
Setting --verbose (verbose) to 1
Setting --load-cookies (loadcookies) to /home/eouser/.urs_cookies
Setting --save-cookies (savecookies) to /home/eouser/.urs_cookies
Setting --keep-session-cookies (keepsessioncookies) to 1
Setting --auth-no-challenge (authnochallenge) to 1
DEBUG output created by Wget 1.14 on linux-gnu.

URI encoding = ‘UTF-8’
Converted file name 'N198101000_PROFILE_MERRA2_3h.nc' (UTF-8) -> 'N198101000_PROFILE_MERRA2_3h.nc' (UTF-8)
Converted file name 'N198101000_PROFILE_MERRA2_3h.nc' (UTF-8) -> 'N198101000_PROFILE_MERRA2_3h.nc' (UTF-8)
--2020-04-15 06:12:23--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/N198101000_PROFILE_MERRA2_3h.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Caching oceandata.sci.gsfc.nasa.gov => 169.154.128.84 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
Created socket 3.
Releasing 0x0000000000b47740 (new refcount 1).
Initiating SSL handshake.
^C
My .netrc looks like following:

Code: Select all

machine urs.earthdata.nasa.gov login scheunro password <super-secret-password>
Post Reply