All NaN Values - Landsat 8

GHesketh4
Posts: 10
Joined: Tue Jun 09, 2020 4:42 pm
company / institution: University of Maine
Location: Orono, ME

All NaN Values - Landsat 8

Post by GHesketh4 »

Hello, I am trying to process Landsat-8 data. Polymer has been successful with some files such as this:
LC08_L1TP_011029_20160823_20180130_01_T1.jpg
But has given me large files of only NaN values for other files such as these:
LC08_L1TP_011029_20200615_20200625_01_T1.jpg
LC08_L1TP_012030_20140809_20170304_01_T1.jpg
I realize some of the files have some cloud cover and a large amount of landmass, but there are clearly a decent amount of unobscured ocean pixels in all of them. Any idea what I might be doing wrong? I have tried re-downloading and re-installing both POLYMER and l8angles, and deleting the ancillary data in ANCILLARY?METEO on each run so it would have to get that data for each file.

My basic run script looks like this:
...
run_atm_corr(
Level1_OLI(fileLoc),
Level2_NETCDF(outdir=outLoc, ext=(fileName + '.nc'),
overwrite=True), multiprocessing=-1
)
Thanks!
You do not have the required permissions to view the files attached to this post.
dingky
Posts: 37
Joined: Tue Jun 30, 2020 3:56 am
company / institution: China University of Geosciences
Location: China

Re: All NaN Values - Landsat 8

Post by dingky »

Dear GHesketh4
Hello,Can you further talk about the l8angles installation code?

Thank you very much

Best regards,
ding
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: All NaN Values - Landsat 8

Post by fsteinmetz »

Dear GHesketh4,

I tried the same file but did not get a file full of NaNs. Water reflectance is recovered over the glint area. Which version of Polymer did you use ?
LC08_L1TP_011029_20200615_20200625_01_T1_polymer.jpg
ding: please open a new thread if you have a precise question that is unrelated to the current thread, thanks.
You do not have the required permissions to view the files attached to this post.
GHesketh4
Posts: 10
Joined: Tue Jun 09, 2020 4:42 pm
company / institution: University of Maine
Location: Orono, ME

Re: All NaN Values - Landsat 8

Post by GHesketh4 »

I am using Polymer 4.13 on a MAC. Would you recommend I download an earlier version? Would you be able to post your ancilliary files and l8angles files for the 20200615 Landsat image so I could check to see if I downloaded either of those incorrectly somehow?
Thanks!
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: All NaN Values - Landsat 8

Post by fsteinmetz »

Hello,
Please find my input and output files for this scene at the following address :
https://we.tl/t-JxBXtlrjxf
Cheers,
François
GHesketh4
Posts: 10
Joined: Tue Jun 09, 2020 4:42 pm
company / institution: University of Maine
Location: Orono, ME

Re: All NaN Values - Landsat 8

Post by GHesketh4 »

Thank you! The folder you provided me did not work, but it did teach me a lot. Our Landsat 8 files and l8angle files are the same as far as I can tell ( I used diff -rq from the command line, which displayed no differences). The two sets of solar and sensor HDR files match but I don't know how to directly compare .img files if that is important to check.

I did a three-way comparison of Polymer, using:
1. Results of entirely my own files
2. My Landsat-8 imagery with your l8angle sensor and solar files
3. Results using only your posted files

Importantly, I was not generate your polymer_result.nc using my own version of Polymer even with option 3.

I also found all of the files swap the latitude and longitude in the resulting netcdf? For the 20200615 file for the Landsat Path 11 Row 29 file, the latitude range is ~ 43.5 to 45.5 and the longitude range is ~ -67 to -70. I don't know if this could be causing the root problem; however, the 20160823 file from the same location also has reverse lat-lon values but has proper non-NaN values in the expected areas.
Also importantly, your polymer_result.nc file does have the correct latitude and longitude values.
incorrectLats.png
Any advice? Re-running anaconda-deps.sh says everything has installed successfully, so I don't believe it's a problem with my installation of the osgeo/gdal package as I originally suspected.
You do not have the required permissions to view the files attached to this post.
GHesketh4
Posts: 10
Joined: Tue Jun 09, 2020 4:42 pm
company / institution: University of Maine
Location: Orono, ME

Re: All NaN Values - Landsat 8

Post by GHesketh4 »

As a secondary update, my co-worker with a Linux setup of Polymer tried the two files and got the same failing results as me for the 20200615 file, with the lat-lon arrays swapped for both files as well.
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: All NaN Values - Landsat 8

Post by fsteinmetz »

Hello,
It seems indeed that there are cases where lat and lon are swapped. I'm investigating and will keep you updated !
Cheers,
François
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: All NaN Values - Landsat 8

Post by fsteinmetz »

Ok I finally got it. The problem was that due to a swapping of the order of lat/lon in gdal v3.
A description of the issue is found here https://github.com/OSGeo/gdal/issues/1546
I am attaching an updated version of level1_landsat8.py which should fix the issue. Could you check ?

Though it is not quite clear to me why this would have caused issues for one file, and not for the other.
You do not have the required permissions to view the files attached to this post.
GHesketh4
Posts: 10
Joined: Tue Jun 09, 2020 4:42 pm
company / institution: University of Maine
Location: Orono, ME

Re: All NaN Values - Landsat 8

Post by GHesketh4 »

That did the trick! Both the 2016 and 2020 file now process correctly. Thanks for the speedy assistance.
Post Reply