On April 15, 2026, following routine maintenance on the Apache Cassandra database cluster, the system entered an unstable state known as an asymmetric network partition. While this did not cause a full outage, it resulted in intermittent read failures and, beginning April 21, intermittent failures to persist incoming data to Cassandra. The root cause was a split-brain condition in which cluster nodes held conflicting views of cluster health – each node believed itself healthy while marking others as offline. These mismatched perspectives prevented the cluster from reliably reaching quorum, causing inconsistent slowdowns and errors for users making data requests.
The issue was resolved on May 19, 2026 by sequentially restarting affected nodes, allowing each to fully synchronize with the cluster before proceeding to the next.
Analysis of EDEX ingest logs confirmed that write failures occurred in discrete time windows across seven affected days: April 21, May 7, and May 15 through May 19. Failures were not uniformly distributed across instruments – on the worst-affected day (May 19), 75 of 110 instruments had at least one stream with data loss, while May 7 saw only 3 affected instruments. Log analysis produced per-day, per-stream breakdowns identifying the specific time ranges and record counts for successful writes, failed writes, and streams with no failures, enabling targeted deletions and re-loads of the data at the instrument/method/stream level.
OOI personnel are reloading data into the impacted streams. We will update this ticket with information on the impacted streams and their reload status as they are addressed. Users who downloaded OOI data between April 15 and May 19, 2026 are encouraged to monitor this ticket and re-download the data from any impacted streams they may have downloaded once those streams have been reloaded to ensure they have the complete data record.
@mvardaro – Regional Cabled Array
@chris.wingard – Coastal Endurance Array
@cdobson – Global Irminger and Coastal Pioneer Arrays
Coastal Endurance Array – Reloaded Streams
All impacted Coastal Endurance Array sensors (currently deployed assets only were impacted) have been deleted and reloaded in the Cassandra database. The affected sensors are listed below. Note, for uncabled assets, the deletion and reload process impacts all streams.
| Site |
Node |
Sensor |
Start |
End |
| CE02SHSM |
SBD11 |
02-HYDGN0000 |
2026-04-21 |
2026-04-21 |
| CE02SHSM |
SBD11 |
06-METBKA000 |
2026-04-21 |
2026-04-21 |
| CE02SHSM |
SBD11 |
06-METBKA001 |
2026-04-21 |
2026-04-21 |
| CE02SHSM |
SBD11 |
06-METBKA000 |
2026-05-07 |
2026-05-07 |
| CE02SHSM |
SBD11 |
01-MOPAK0000 |
2026-05-15 |
2026-05-19 |
| CE02SHSM |
SBD11 |
02-HYDGN0000 |
2026-05-15 |
2026-05-19 |
| CE02SHSM |
SBD11 |
06-METBKA000 |
2026-05-15 |
2026-05-19 |
| CE02SHSM |
SBD11 |
06-METBKA001 |
2026-05-15 |
2026-05-19 |
| CE02SHSM |
SBD12 |
03-HYDGN0000 |
2026-05-15 |
2026-05-19 |
| CE02SHSM |
SBD12 |
04-PCO2AA000 |
2026-05-15 |
2026-05-19 |
| CE02SHSM |
SBD12 |
05-WAVSSA000 |
2026-05-15 |
2026-05-19 |
| CE05MOAS |
G1153 |
00-ENG000000 |
2026-05-15 |
2026-05-19 |
| CE05MOAS |
G1153 |
01-PARADM000 |
2026-05-15 |
2026-05-19 |
| CE05MOAS |
G1153 |
02-FLORTM000 |
2026-05-15 |
2026-05-19 |
| CE05MOAS |
G1153 |
05-CTDGVM000 |
2026-05-15 |
2026-05-19 |
An internal ticket has been created to reload the data into the Gold Copy THREDDS catalog. I’ll update this ticket once that process is complete for Endurance Array assets.
Chris Wingard
All impacted Coastal Pioneer and Global Irminger sensors have been deleted and reloaded in the Cassandra database. The affected sensors are listed in the table below.
| Site |
Node |
Sensor |
Start |
End |
| CP10CNSM |
RID26 |
00-DCLENG000 |
2026-04-21 |
2026-04-21 |
| CP10CNSM |
RID26 |
03-TURBDA000 |
2026-04-21 |
2026-04-21 |
| CP10CNSM |
RID26 |
04-VELPTA000 |
2026-04-21 |
2026-04-21 |
| CP10CNSM |
SBD11 |
02-HYDGN0000 |
2026-04-21 |
2026-04-21 |
| CP10CNSM |
SBD11 |
06-METBKA000 |
2026-04-21 |
2026-04-21 |
| CP10CNSM |
SBD11 |
06-METBKA001 |
2026-04-21 |
2026-04-21 |
| CP10CNSM |
SBD12 |
06-METBKA001 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
MFD36 |
05-PCO2WB000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
MFD37 |
04-DOSTAD000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
RID26 |
04-VELPTA000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
RID27 |
02-FLORTD000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
RID27 |
03-CTDBPC000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
SBD11 |
00-DCLENG000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
SBD11 |
06-METBKA001 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
SBD12 |
00-DCLENG000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
SBD12 |
03-HYDGN0000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
SBD12 |
05-WAVSSA000 |
2026-04-21 |
2026-04-21 |
| CP11NOSM |
RID26 |
00-DCLENG000 |
2026-05-07 |
2026-05-07 |
| CP11NOSM |
MFD36 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
MFD36 |
03-TURBDA000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
MFD36 |
05-PCO2WB000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
MFD37 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
MFD37 |
02-ADCPTF000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
MFD37 |
04-DOSTAD000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID26 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID26 |
01-PRTSZA000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID26 |
03-TURBDA000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID26 |
04-VELPTA000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID26 |
07-NUTNRB000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID26 |
08-SPKIRB000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID27 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID27 |
01-OPTAAD000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID27 |
02-FLORTD000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID27 |
03-CTDBPC000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
RID27 |
04-DOSTAD000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
SBD11 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
SBD11 |
05-SPKIRB000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
SBD11 |
06-METBKA000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
SBD11 |
06-METBKA001 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
SBD12 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
SBD12 |
03-HYDGN0000 |
2026-05-15 |
2026-05-19 |
| CP11NOSM |
SBD12 |
05-WAVSSA000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
MFD37 |
04-DOSTAD000 |
2026-04-21 |
2026-04-21 |
| CP11SOSM |
RID26 |
07-NUTNRB000 |
2026-04-21 |
2026-04-21 |
| CP11SOSM |
RID27 |
01-OPTAAD000 |
2026-04-21 |
2026-04-21 |
| CP11SOSM |
RID27 |
03-CTDBPC000 |
2026-04-21 |
2026-04-21 |
| CP11SOSM |
SBD11 |
06-METBKA000 |
2026-04-21 |
2026-04-21 |
| CP11SOSM |
SBD11 |
06-METBKA001 |
2026-04-21 |
2026-04-21 |
| CP11SOSM |
SBD12 |
03-HYDGN0000 |
2026-04-21 |
2026-04-21 |
| CP11SOSM |
MFD36 |
00-DCLENG000 |
2026-05-07 |
2026-05-07 |
| CP11SOSM |
MFD36 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
MFD36 |
01-PRTSZA000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
MFD36 |
03-TURBDA000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
MFD36 |
05-PCO2WB000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
MFD37 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
MFD37 |
03-CTDBPD000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
MFD37 |
04-DOSTAD000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID26 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID26 |
03-TURBDA000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID26 |
07-NUTNRB000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID26 |
08-SPKIRB000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID27 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID27 |
01-OPTAAD000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID27 |
02-FLORTD000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
RID27 |
03-CTDBPC000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD11 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD11 |
02-HYDGN0000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD11 |
05-SPKIRB000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD11 |
06-METBKA000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD11 |
06-METBKA001 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD12 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD12 |
03-HYDGN0000 |
2026-05-15 |
2026-05-19 |
| CP11SOSM |
SBD12 |
05-WAVSSA000 |
2026-05-15 |
2026-05-19 |
| CP12CNSW |
MFI01 |
02-CTDMOS031 |
2026-04-21 |
2026-04-21 |
| CP13SOPM |
WFP01 |
00-WFPENG001 |
2026-04-21 |
2026-04-21 |
| CP13SOPM |
WFP01 |
01-VEL3DK000 |
2026-04-21 |
2026-04-21 |
| CP13SOPM |
WFP01 |
02-DOFSTK000 |
2026-04-21 |
2026-04-21 |
| CP13SOPM |
WFP01 |
03-CTDPFK000 |
2026-04-21 |
2026-04-21 |
| GI01SUMO |
RID16 |
08-SPKIRB000 |
2026-04-21 |
2026-04-21 |
| GI01SUMO |
RII11 |
02-CTDBPP033 |
2026-04-21 |
2026-04-21 |
| GI01SUMO |
RII11 |
02-CTDMOQ011 |
2026-04-21 |
2026-04-21 |
| GI01SUMO |
RII11 |
02-FLORDG033 |
2026-04-21 |
2026-04-21 |
| GI01SUMO |
SBD11 |
04-DOSTAD000 |
2026-04-21 |
2026-04-21 |
| GI01SUMO |
SBD11 |
06-METBKA001 |
2026-04-21 |
2026-04-21 |
| GI01SUMO |
RID16 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RID16 |
01-OPTAAD000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RID16 |
02-FLORTD000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RID16 |
03-CTDBPF000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RID16 |
06-DOSTAD000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RID16 |
07-NUTNRB000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RID16 |
08-SPKIRB000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-ADCPSN010 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDBPP031 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDBPP032 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDBPP033 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOQ011 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOQ012 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOQ013 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOQ014 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOQ015 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOQ016 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOQ017 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOR018 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOR019 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-CTDMOR020 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-FLORDG031 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-FLORDG033 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-PCO2WC052 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
RII11 |
02-PCO2WC053 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD11 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD11 |
03-HYDGN0000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD11 |
04-DOSTAD000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD11 |
05-SPKIRB000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD11 |
06-METBKA001 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD11 |
08-NUTNRB000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD12 |
00-DCLENG000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD12 |
01-OPTAAD000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD12 |
02-FLORTD000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD12 |
03-HYDGN0000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD12 |
04-PCO2AA000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD12 |
06-METBKA000 |
2026-05-15 |
2026-05-19 |
| GI01SUMO |
SBD12 |
06-METBKA001 |
2026-05-15 |
2026-05-19 |
Deletion and playback of impacted Regional Cabled Array instruments is underway. The incident affected 270 instruments (106 science sensors and 164 engineering sensors) across three time ranges. Due to the density of the streaming datasets, playback will take some time to complete individually without impacting system stability.
So far, the following 10 sensors have been fully purged and reloaded:
| Reference Designator |
|
Start |
End |
| CE02SHBP-LJ01D-05-ADCPTB104 |
0 |
2026-04-21 17:00:00 |
2026-04-21 23:00:00 |
| CE02SHBP-LJ01D-05-ADCPTB104 |
0 |
2026-05-07 7:00:00 |
2026-05-07 12:00:00 |
| CE02SHBP-LJ01D-05-ADCPTB104 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |
| CE02SHBP-LJ01D-07-VEL3DC108 |
0 |
2026-04-21 17:00:00 |
2026-04-21 23:00:00 |
| CE02SHBP-LJ01D-07-VEL3DC108 |
0 |
2026-05-07 7:00:00 |
2026-05-07 12:00:00 |
| CE02SHBP-LJ01D-07-VEL3DC108 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |
| CE02SHBP-LJ01D-10-PHSENH110 |
0 |
2026-04-21 17:00:00 |
2026-04-21 23:00:00 |
| CE02SHBP-LJ01D-10-PHSENH110 |
0 |
2026-05-07 7:00:00 |
2026-05-07 12:00:00 |
| CE02SHBP-LJ01D-10-PHSENH110 |
0 |
2026-05-16 0:00:00 |
2026-05-20 0:00:00 |
| CE04OSBP-LJ01C-05-ADCPSI103 |
0 |
2026-04-21 17:00:00 |
2026-04-21 23:00:00 |
| CE04OSBP-LJ01C-05-ADCPSI103 |
0 |
2026-05-07 7:00:00 |
2026-05-07 12:00:00 |
| CE04OSBP-LJ01C-05-ADCPSI103 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |
| CE04OSBP-LJ01C-06-CTDBPO108 |
0 |
2026-04-21 17:00:00 |
2026-04-21 23:00:00 |
| CE04OSBP-LJ01C-06-CTDBPO108 |
0 |
2026-05-07 7:00:00 |
2026-05-07 12:00:00 |
| CE04OSBP-LJ01C-06-CTDBPO108 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |
| CE04OSBP-LJ01C-06-DOSTAD108 |
0 |
2026-04-21 17:00:00 |
2026-04-21 23:00:00 |
| CE04OSBP-LJ01C-06-DOSTAD108 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |
| CE04OSBP-LJ01C-07-VEL3DC107 |
0 |
2026-04-21 17:00:00 |
2026-04-21 23:00:00 |
| CE04OSBP-LJ01C-07-VEL3DC107 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |
| CE04OSBP-LJ01C-08-OPTAAC104 |
0 |
2026-05-07 7:00:00 |
2026-05-07 12:00:00 |
| CE04OSBP-LJ01C-08-OPTAAC104 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |
| CE04OSBP-LJ01C-09-PCO2WB104 |
0 |
2026-05-16 0:00:00 |
2026-05-20 0:00:00 |
| CE04OSBP-LJ01C-10-PHSEND107 |
0 |
2026-05-14 15:00:00 |
2026-05-20 0:00:00 |