CE02SHBP OPTAA Files Missing Wavelengths

HelpDesk
1

I pulled this file from the Thredds Catalog.

https://thredds.dataexplorer.oceanobservatories.org/thredds/catalog/ooigoldcopy/public/CE02SHBP-LJ01D-08-OPTAAD106-streamed-optaa_sample/catalog.html?dataset=ooigoldcopy/public/CE02SHBP-LJ01D-08-OPTAAD106-streamed-optaa_sample/deployment0009_CE02SHBP-LJ01D-08-OPTAAD106-streamed-optaa_sample_20230621T040013.945106-20230622T030301.821923.nc

In xarray, the wavelength coordinate and the wavelength_a/wavelength_c variables are populated with NaNs. Panoply shows the same thing.

I have observed this in other deployment files too.
I looped over all CE02SHBP files and have found that this issue occurs in files associated with deployments 1, 2, and 9. This data cannot be used without wavelength information.

2

Ian,

Are you seeing this in all files for a deployment, or just random ones in a deployment? My guess is the later?

Chris

3

Hi Chris,

Here is a complete list of files. I opened and checked each file to see if it had a NaN value in the wavelength dimension. It looks like it is full deployment.

Cheers,
Ian

4

OK, thanks. I’ll put together a ticket to get this tracked down.

Cheers,
Chris

5

Hi Ian,

The download script keeps throwing this error message, sometimes after downloading one file, sometimes after downloading several files:

~WRD0001.jpg

6

Hi Chris,

I was able to parse out a couple of the .dat files for deployment 1.
I have confirmed that the ACS output 77 wavelength bins, which is the size of the wavelength coordinate in the netCDF files. This suggests to me that the packet parser deployed for the cabled OPTAAs works okay, but the assignment of wavelengths fails somewhere. My guess is that there is a discrepancy in the device file that caused the wavelength values to be read in as NaNs.

As a brief example, I parsed out a .dat file and selected a single packet. I then compared the elapsed run time in the packet to the elapsed run time in the .netCDF. The a_signal values were the same. scratch/ce02shbp_scratch.ipynb at main · IanTBlack/scratch · GitHub

I would recommend this check for the other deployments and for maybe a couple packets in each .dat file, but I think the best case is that this issue can be resolved by re-parsing the device files for the afflicted netCDFs and then adding the wavelengths to each file.

Cheers,

  • Ian
7

There appears to be a mismatch in the length of the output wavelengths in the coeffs requested through m2m and the wavelengths in the netCDF.
I used the node and sensor attributes and the data start and stop times in a file to build the m2m request url.

If I query M2M for the coefficients for deployment 1, I get a response that has 83 wavelengths instead of 77, but as I showed earlier I verified that the sensor output was 77 wavelengths.

https://ooinet.oceanobservatories.org/api/m2m/12587/asset/cal/?refdes=CE02SHBP-LJ01D-08-OPTAAD106&beginDT=2014-09-25T18%3A29%3A00.000000Z&endDT=2014-11-18T09%3A59%3A59.000000Z

There is a dev file with a matching serial number that occurs 5 days before (ATOSU-58332-00009__20140910.csv, which does have 83 wavelengths). Maybe an older cal file was applied?
If I had to guess, the wrong device file was applied and right one is ATOSU-58332-00009__20140415.dev

If I query for deployment 2 I get 88 wavelengths instead of the expected 77 in the file. I did not check the packets for this deployment.
https://ooinet.oceanobservatories.org/api/m2m/12587/asset/cal/?refdes=CE02SHBP-LJ01D-08-OPTAAD106&beginDT=2015-11-05T02%3A00%3A15.000000Z&endDT=2015-11-05T05%3A59%3A53.000000Z

However, if I request for deployment 9, I get coeffs with 77 wavelengths and a file dimension of 77, so I don’t know what is going on there. I also did not check the packets for this deployment.
https://ooinet.oceanobservatories.org/api/m2m/12587/asset/cal/?refdes=CE02SHBP-LJ01D-08-OPTAAD106&beginDT=2023-08-07T03%3A03%3A01.233452Z&endDT=2023-08-08T03%3A03%3A12.270190Z

8

FYI, the file I linked to in Alfresco appear to only be on Alfresco, not on the OOI asset-management GitHub repo.

9

Looking at one of the raw .dat files for CE02SHBP deployment 2, it looks like the number of output wavelengths should be 88, so the netCDFs do not have the right dimensions anyway. Good news is that the corresponding device file also says 88 wavelengths, so this deployment is likely recoverable.

10

Hi Ian,

We are still actively investigating this issue. While digging into the wavelength mismatch we did identify some missing calibrations including deployment 1 for OPTAAD106. Correcting these cals seems to have resolved the NAN wavelengths for OPTAAD106 deployment 1. However, we have not yet identified what is causing NAN wavelengths in the other deployments - which appears to be a separate issue.

I’ll update this thread when we know more.

11

Thanks for fixing deployment 1. This appears to be one of the issues the Data Review Team identified back in 2019. In case you haven’t already found them, several other deployments with NaNs were flagged in their final report.