Uploading large objects to Cloudflare R2 using rclone

OSS Support
1

We often use Cloudflare R2 to store STIX 2.1 bundles we distribute. Many of these bundles are 100’s of Mb’s.

Here’s how we do it;

Install rclone:

To create a rclone.conf file in the path of ~/.config/rclone/.

Contents of the conf file are as follows:

[r2]
type = s3
provider = Cloudflare
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_ACCESS_KEY>
region = auto
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
acl = private

Where:

  • [r2]: A custom name(an alias) for storage service. We need to use it to operate files.
  • type = s3: The type of file operation API. R2 supports the S3 standard protocol.
  • provider = Cloudflare: The storage provider ID. You could use man rclone in your terminal to get the supported providers.
  • access_key_id: You need to create a token with Admin Read & Write permissions on the R2 console (note, I am not sure if this is a bug, but I couldn’t get it to work with any other permissions levels)
  • secret_access_key: Same as above.
  • endpoint: The URL that rclone uses to operate files. To get the account id on the top-right of the R2 homepage.

To run rclone lsf r2: to see your buckets and rclone lsf r2:your-bucket to show the file list within a bucket.

e.g.

rclone lsf r2:
cti-knowledge-base-store/

rclone lsf r2:cti-knowledge-base-store/
disarm/
locations/
mitre-attack-enterprise/
mitre-attack-ics/
mitre-attack-mobile/
mitre-capec/
mitre-cwe/
nvd-cpe/
nvd_cve/
sigma-rules/
yara-rules/

To upload a file to the root of the bucket

rclone copy /path/to/file.type r2:your-bucket

To upload a file to a specific directory in the bucket

rclone copy /path/to/file.type r2:your-bucket/DIRECTORY

To upload a directory of files to a specific directory in the bucket

rclone copy /path/to/directory r2:your-bucket/DIRECTORY

Note, if on Mac, use add the following to the end of the command --exclude '.*{/**,}' (to exclude hidden files, namely .DS_Store).

You can see an example of this implementation in a Github action we created for cxe2stix_helper here: