Uploading large objects to Cloudflare R2 using rclone

OSS Support
1

We often use Cloudflare R2 to store STIX 2.1 bundles we distribute. Many of these bundles are 100’s of Mb’s.

Here’s how we do it;

Install rclone:

To create a rclone.conf file in the path of ~/.config/rclone/.

Contents of the conf file are as follows:

[r2]
type = s3
provider = Cloudflare
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_ACCESS_KEY>
region = auto
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
acl = private

Where:

  • [r2]: A custom name(an alias) for storage service. We need to use it to operate files.
  • type = s3: The type of file operation API. R2 supports the S3 standard protocol.
  • provider = Cloudflare: The storage provider ID. You could use man rclone in your terminal to get the supported providers.
  • access_key_id: You need to create a token with Admin Read & Write permissions on the R2 console (note, I am not sure if this is a bug, but I couldn’t get it to work with any other permissions levels)
  • secret_access_key: Same as above.
  • endpoint: The URL that rclone uses to operate files. To get the account id on the top-right of the R2 homepage.

To run rclone lsf r2: to see your buckets and rclone lsf r2:your-bucket to show the file list within a bucket.

e.g.

rclone lsf r2:
cti-knowledge-base-store/

rclone lsf r2:cti-knowledge-base-store/
disarm/
locations/
mitre-attack-enterprise/
mitre-attack-ics/
mitre-attack-mobile/
mitre-capec/
mitre-cwe/
nvd-cpe/
nvd_cve/
sigma-rules/
yara-rules/

To upload a file to the root of the bucket

rclone copy /path/to/file.type r2:your-bucket

To upload a file to a specific directory in the bucket

rclone copy /path/to/file.type r2:your-bucket/DIRECTORY

To upload a directory of files to a specific directory in the bucket

rclone copy /path/to/directory r2:your-bucket/DIRECTORY

other useful rclone flags…

print progress, and retry on failures during upload

rclone --exclude '.*' copy bundles/ r2:BUCKET -v --retries 10 --low-level-retries 10

delete all the bucket contents

rclone delete r2:BUCKET/

delete the bucket

rclone purge r2:BUCKET/