Using Dropbox to archive files

About

• The Langille Lab has a Business Dropbox account with "Unlimited" storage.
• Instead of using the regular "syncing" dropbox client (which takes too many resources), we can use rClone
• This cloud storage provides a location to upload files that are not routinely required.
• This helps with archiving projects and freeing up space on our servers (e.g. kronos and vulcan).
• Before, uploading files to Dropbox you should remove unnecessary files and compress directories as much as possible.
• Upload speeds from vulcan to dropbox is about 1GB per minute.

Setup

• rClone is already installed on Vulcan and Kronos for all users.
• Each user needs their own config file which authenticates to the Langille Lab Dropbox account and is unique for each user.
• Create an empty config file at ~/.config/rclone/rclone.conf

rclone config touch

• Ask Morgan for a new rclone config account, and add that information to your rclone.conf file:

nano ~/.config/rclone/rclone.conf

For example the contents should look something like this:

[morgan_backup]
type = dropbox
client_id = nfoeui02
client_secret = gdbjkasg
token = {"access_token":"sl.A5wOhOxybg=dg;=aglgauZ_-x8EfvMfa2VD-Z9jxu0tPrVHP3YUhDI3yg41e9B7BNcvE1nEcez-CSOt5YXp3-nDGZPpbvCI0rtqmhT2aiicCdnT4zFWGn9kkLBv9IgndklsgajHZVmUwXxh6w","token_type":"bearer","refresh_token":"r4VMcqdEvmIAAAAAAAAAAXu5E0Fvo5BhkHt9MQDb8aBZCGhg-YWbKKwUpF7X6uZj","expiry":"2021-10-05T16:23:07.487353-03:00"}

• Take note of the name on the first line, it will be something like "user_backup". You will need this name for all commands.

• You can test that everything is ok, but doing a listing of your files on dropbox (should be empty)

rclone ls user_backup:

Usage

• A list of all commands are listed on the rClone website
• The main ones to check out are copy and sync
• Example 1, if I wanted up upload a single file (note, -P flag shows progress:

rclone copy -P project1.tar.gz  user_backup:projects/

• Example2, if I had the directory project2 which I wanted to upload I would run the following command (note the trailing /)

  rclone -P sync project2 user_backup:project2/

• After ensuring that all files are uploaded (see below), you can then remove your files locally.

Browsing uploaded files on Dropbox

• To see a simple list of directories on Dropbox

rclone lsd user_backup:

• To view a graphical tree structure of your files try:

rclone tree user_backup:

• You can even mount your entire dropbox space to a local directory for easier browsing and transferring of files

mkdir ~/dropbox_mount
rclone mount user_backup: ~/dropbox_mount 

Making archives.

• If you have a project that you would like to compress and store as a single file just use a tar command (Note: the -I pigz option allows parallelization of the gzipping which speeds things up).

tar cf your_project.tar.gz your_path_to_project/project -I pigz