No-data values in Polymer Level2

Post Reply
fecgiannini
Posts: 6
Joined: Wed Jan 09, 2019 5:51 pm
company / institution: FURG
Location: Rio Grande, RS

No-data values in Polymer Level2

Post by fecgiannini »

Hi
I am trying to use the gpt-snap tool to run a mosaic operator on polymer products. However I am having a problem to use the following expression:

if bit_set(bitmask,10) then NaN else logchl

It works perfectly in the SNAP GUI interface, but in the gpt-snap, the final netcdf turn 'NaN' into '0.0000' which is a problem.

I understand that the logchl band takes a generic default number as No-data and probably because of this the mosaic script is using 0.00 instead of NaN. Is there a way of changing this to be able to get NaN in the mosaic output in gpt, similar to what I get in the GUI interface?

Thank you!
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: No-data values in Polymer Level2

Post by fsteinmetz »

Hi Fernanda,
You would probably have better luck asking in the ESA step forum - it seems that the main issue is why it works in the GUI but not using gpt-snap, but I'm afraid I can not help you on this issue.
Cheers,
François
fecgiannini
Posts: 6
Joined: Wed Jan 09, 2019 5:51 pm
company / institution: FURG
Location: Rio Grande, RS

Re: No-data values in Polymer Level2

Post by fecgiannini »

Hi François
No problem. Thank you
I actually found a discussion in that forum and it seems the problem is the way that the original files to be mosaiced have their "no-data" set up. For example, instead of NaN, the netcdf files have No-data as a generic number 9.969E36. However, the GUI is able to convert it to NaN, while the gpt is not. So I thought there could be a way to change how the No-data is written in the Level-2.
But I will certainly contact them about it.
Thank you very much
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: No-data values in Polymer Level2

Post by fsteinmetz »

Hi Fernanda,

Thanks for the details. If you are looking for a workaround, it is possible to tweak how the "no-data" is written (the fill_value, in netCDF terms), however I don't really know how you should do it.
You can have a look at polymer/level2_nc.py, line 98 and after. Maybe removing the option "fill_value=fill_value" from createVariable, or removing line 111, "data[np.isnan(data)] = fill_value", could help.

But shouldn't the detection of NaNs be overridden by the use of bitmask and the function bit_set ?
bit_set(bitmask,10) tests only the bit number 10, CASE2, right ? So maybe you could also include bits number 1 to 9, which would have the effect of discarding NaNs?

Cheers,
François
fecgiannini
Posts: 6
Joined: Wed Jan 09, 2019 5:51 pm
company / institution: FURG
Location: Rio Grande, RS

Re: No-data values in Polymer Level2

Post by fecgiannini »

Hi François
Thank you for your reply. Yes, I thought of removing line 111 from Level2.py. I will give it a try.

If you dont mind, I will try to explain in more details what I am doing, so maybe you have a clue. Usually in the mosaic script I set a couple of "conditions" to exclude pixels that I want to mask out, for example:

<conditions>
<condition>
<name>condition_neg_BB</name>
<expression>bit_set(bitmask,3)==false</expression>
</condition>
<condition>
<name>condition_oob</name>
<expression>bit_set(bitmask,4)==false</expression>
</condition>
<condition>
<name>condition_exc</name>
<expression>bit_set(bitmask,5)==false</expression>
</condition>
<condition>
<name>condition_ham</name>
<expression>bit_set(bitmask,7)==false</expression>
</condition>
</conditions>

With this, any band (or variable) that I include in the mosaic (TSM, logchl, Rw, etc) will have those flagged pixels excluded from the mosaic process. Because I said the script to mosaic those images with the 'condition' that the pixel have bit_set(bitmask,3)==false, for example. That works great, but it applies for ALL bands.
But, I would like to include an extra condition only for logchl, the Case-2. So I had the idea of creating a new band during this process, with an expression that states:
<variable>
<name>chl_conc_c2flag</name>
<expression> if bit_set(bitmask,10) then NaN else logchl </expression>
</variable>

In this case, if the pixel has case-2 flag, than turn that pixel into NaN, otherwise keep the logchl.
And this worked for the mosaic process in the GUI interface, and I got stuck, because instead of turning into NaN in the gpt, it turned into 'zero'.

The difference is that "conditions" does not imply in the values turning into NaN, it just doesn't consider them. But as a new band, the expression would have to be written somehow like that.

Sorry for the long message, if you have a clue, please let me know. But don't worry about it.
I can run two different mosaic steps to apply different conditions to the variables, but this way would save me processing time.

Thank you very much for your reply!
User avatar
fsteinmetz
Site Admin
Posts: 306
Joined: Fri Sep 07, 2018 1:34 pm
company / institution: Hygeos
Location: Lille, France
Contact:

Re: No-data values in Polymer Level2

Post by fsteinmetz »

Thanks for the details, Fernanda.
So the NaNs that are source of the issue do not come from the NetCDF file, but from the variable chl_conc_c2flag.

Here are my advices :
- The simplest solution seems indeed to use different masking expressions for different variables. Does it take significantly more processing time to do so ?
- What is your motivation to remove the CASE2 pixels ? The CASE2 flag is not designed to detect case 2 waters, but to diagnose when Polymer has worked in case2 mode - and it should have no impact on the final results. Thus it is not recommended to reject the pixels flagged as CASE2.
- You could probably mask all recommended flags are once, using a binary and operator. In snap, that should be "bitmask & 1023 == 0" (for valid pixel), or maybe "bitmask & 1023 == 0"

I hope this helps.
Cheers,
François
fecgiannini
Posts: 6
Joined: Wed Jan 09, 2019 5:51 pm
company / institution: FURG
Location: Rio Grande, RS

Re: No-data values in Polymer Level2

Post by fecgiannini »

Hi François,

Sorry that you ended up replying the same thing twice - just saw the email.
In a match-up work for British Columbia waters, I observed that this flag perfectly indicate pixels where the Chl-a retrievals failed. Removing those pixels, which are not that many, the statistical performance improved significantly.
But I will re-process the data for Maycira's lab, and keep the two variables (chl and chl_c2_flag), so they can decide whether to use it or not.
Yes, to have another mosaic run only to apply specific conditions for different variables will be twice the processing time. We are working with the entire coast of Canada and 4 years of data. =)
I will still run a couple of tests with the level2_nc.py (line 111) modified and I suspect that will work out.

Thank you very much
Post Reply