Hi,
Apologies if this is explained somewhere and I missed it. Is there a way to download data as whole folder from the raw data repository and not have to download each individual file? I am trying to download all the CTD files (hex, xlmcon, hdr, etc) required to process the data using the seabird software. As an example, a folder I would like all the data from is https://rawdata.oceanobservatories.org/files/cruise_data/Station_Papa/Station_Papa-02_MV1404_2014-06-10/Ship_Data/
Thank you!
Kristen
Hi Kristen,
Yes, you can use wget to download entire directories (you get the whole folder structure, but it’s fairly easy to then combine them all into a single directory once it’s local).
Let me know if you have any issues with it.
-Mike V.
Hi Mike, Thanks for the response. I am having trouble getting this to work. I am using a bash environment in OOI JupyterHub but am new to this. Trying the provided example, I get the following error.
wget -r -np -e robots=off https://rawdata.oceanobservatories.org/files/CE01ISSP/D00001/
--2026-05-05 16:09:57-- https://rawdata.oceanobservatories.org/files/CE01ISSP/D00001/
Resolving rawdata.oceanobservatories.org (rawdata.oceanobservatories.org)... 192.168.220.241
Connecting to rawdata.oceanobservatories.org (rawdata.oceanobservatories.org)|192.168.220.241|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
rawdata.oceanobservatories.org/files/CE01ISSP/D00001: No such file or directory
rawdata.oceanobservatories.org/files/CE01ISSP/D00001/index.html: No such file or directory
Cannot write to ‘rawdata.oceanobservatories.org/files/CE01ISSP/D00001/index.html’ (Success).
Thoughts?
Kristen,
If you are on the JupyterHub, you don’t need to use wget to download those files. They are directly available to you under:
/home/jovyan/ooi/uncabled/CE01ISSP/D00001
The cruise data is as well:
/home/jovyan/ooi/cruise_data
All of the raw data that you can see on the Raw Data Server is available to you (read-only) directly
Chris
Yes, what Chris said. Also, wget is a console command that I was able to use by just opening a local console on my laptop and pasting in the command text. But it’s a somewhat crude method and JupyterHub is probably the better way to go, if Chris’s approach works for you.
Stace Beaulieu also pointed out that a lot of the raw CTD data for OOI cruises (including your target cruise) are available for download at R2R: Send, Uwe (2014). CTD (SeaBird SBE-911+) data as collected during the cruise MV1404, Ocean Observatories Initiative (OOI): Station Papa Array, Leg 2. Rolling Deck to Repository (R2R). doi:https://doi.org/10.7284/114908. Accessed [YYYY-MM-DD]
Good luck!
-Mike
I wish I could work with all the data on Jupyter Hub but the files need to be on my local machine to do the processing with the proprietary SBE processing software. The only way I have been able to do this is to download each individual file from the Raw Data Server. It’s the same file by file download on R2R unfortunately and I need over 1000 files. I’ll try to get the wget function working on my machine.
My current workaround is downloading them from alfresco since I can zip and manually download the entire folder for each cruise needed.
That’s a good workaround too. But try using the console (“Terminal” on a Mac) and see if it works. My only other note is that you can get the ‘cannot write to’ error if you don’t have enough space in the directory where you’re trying to save the files, or if you don’t have write permissions on the user profile you’re logged in with, so navigate to the target directory within the console first.
If none of this makes sense, we could try setting up a call and I can walk you through what I’m doing and see if we can get it working for you.
-Mike