Skip to content

Commit 8c499ba

Browse files
authored
Merge pull request #883 from peer35/yoda-webdav
improve Yoda data acess introduction
2 parents 51c409f + 13bc40d commit 8c499ba

2 files changed

Lines changed: 14 additions & 12 deletions

File tree

tools/yoda/analysing_data.qmd

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ There are several ways in which you can download and upload the files:
6767
| Tool | Typical dataset | Platform | Explanation |
6868
| --- | --- | --- | --- |
6969
| **Yoda web portal** | up to 10GB, up to 100 files | PC, some VRE | This can be done if you have an internet browser available (e.g., your PC and some VREs). You could choose this option when you do not want to install additional tools on your system. However, this method is not very reliable when transferring large files. Also, the web portal will not give you clear feedback on whether a download was completed correctly. |
70-
| **WebDAV client**<br>[manual](/tools/yoda/introduction.qmd) | up to 100GB, up to 1000 files | PC, VRE | WebDAV can be slow when transferring a large amount of small files. It is possible to automate the transfer files using WebDAV with Python, but it would be better to use the iRODS interface, see below. |
70+
| **WebDAV client**<br>[manual](/tools/yoda/data_access_instructions.qmd#access-via-webdav) | up to 100GB, up to 1000 files | PC, VRE | WebDAV can be slow when transferring a large amount of small files. It is possible to automate the transfer files using WebDAV with Python, but it would be better to use the iRODS interface, see below. |
7171
| **iCommands or GoCommands**<br>[manual](/tools/yoda/yoda_using_icommands.qmd) | Small to very large | PC, VRE, HPC | These command line tools can handle very large datasets and also offer many features for working with file-level metadata. It is also possible to check the integrity of uploaded and downloaded files, see the [ichksum command](https://docs.irods.org/4.3.4/icommands/user/). |
7272
| **iBridges or the Python iRODS Client**<br>[manual](/tools/yoda/yoda_using_python.qmd) | Small to very large | PC, VRE, HPC | If you use Python for your analysis, you could include transfer of the source data and results in your scripts. This way you can automate data management and avoid duplicates or temporary data. For some workflows it is also possible to access a file directly by streaming, [see below](#workflow-streaming). |
7373

tools/yoda/data_access_instructions.qmd

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -5,22 +5,24 @@ categories: []
55

66
There are three main ways to transfer data to and from Yoda. Note that creating a data access password is necessary to transfer data from and to Yoda.
77

8-
You can transfer files to and from Yoda dependent on the size of the file:
8+
You can transfer files to and from Yoda dependent on the size and number of files:
99

10-
- For small amounts of data you can upload files via the Drag & Drop via de Yoda Portal.
11-
- For transferring larger amounts of data (tens of GB) it is better to use the WebDAV interface.
10+
- For small amounts of data (typically up to 10GB, up to 100 files) you can upload files via the Drag & Drop via de Yoda Portal.
11+
- For larger amounts of data (typically up to 100GB, up to 1000 files) it is better to use the WebDAV interface.
12+
- For very large amounts of data you can connect directly to iRODS.
1213

14+
::: {.callout-tip}
15+
For practical tips on working with your data, check the [Analysing Data](analysing_data.qmd) page.
16+
:::
1317

18+
## Access via WebDAV
1419
There are several methods to access Yoda data via WebDAV, they depend on the Operating System you are using:
1520

16-
- Windows (link)
17-
- MacOS (link)
18-
- Linux (Link)
19-
20-
Note that performance of WebDAV is limited, especially when transferring a large amount of files.
21+
- [Windows](data_access_windows.qmd)
22+
- [MacOS](data_access_macos.qmd)
23+
- [Linux](data_access_linux.qmd)
2124

25+
## Access via iRODS
2226
For transferring large amounts of data (>100GB) consider using direct access to iRODS. A direct iRODS connection offers better performance by enabling multi-threaded transfers and the option to compare checksums.
2327

24-
Power users who are comfortable with command line tools can use the [iRODS iCommands](yoda_using_icommands.html) to interact with the iRODS backend of Yoda directly. If iCommands are not available on your OS, [Gocommands](yoda_using_gocommands.qmd) can be used as an alternative.
25-
26-
28+
Power users who are comfortable with command line tools can use the [iRODS iCommands](yoda_using_icommands.html) to interact with the iRODS backend of Yoda directly. If iCommands are not available on your OS, [Gocommands](yoda_using_gocommands.qmd) can be used as an alternative.

0 commit comments

Comments
 (0)