Uploading large objects to Cloudflare R2

dtp1900 · 7 December 2025 00:00

I see cve2stix and CTI Butler both store objects in public Cloudflare R2 buckets.

Can you explain how you’re uploading very large files (e.g. CVE bundles) to R2?

I am trying to do this locally for something not related to dogesec tools, but keep getting issues when using rclone.

dgreenwood-dogesec · 30 December 2025 10:46

We use boto3 for CVE uploads, you can see the logic here;

github.com/muchdogesec/cve2stix

daily_uploader.py

main

import base64
from datetime import datetime, timedelta, timezone
import io
import os
from pathlib import Path
import traceback
import boto3
from cve2stix.celery import start_celery
from cve2stix.main import main as download_bundle
import logging


summary_file = open(os.getenv('GITHUB_STEP_SUMMARY', '/tmp/github_step_summary.md'), 'a')
print(f"### Create CVE bundle\n", file=summary_file)

def upload_file_to_s3(filepath, s3_path):
    logging.info("uploading to %s", s3_path)
    client = boto3.client('s3', endpoint_url=os.getenv('S3_ENDPOINT_URL'))
    with open(filepath, 'rb') as f:
        client.upload_fileobj(f, os.getenv('S3_BUCKET_NAME') or "cti-public", s3_path)

This file has been truncated. show original

But honestly, rclone is enough, and more flexible for sporadic uploads.

Here’s a copy of some local notes that will solve your rclone issues (it would be helpful if you shared the actual issue if these don’t solve it)…

Install rclone:

To create a rclone.conf file in the path of ~/.config/rclone/.

Contents of the conf file are as follows:

[r2]
type = s3
provider = Cloudflare
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_ACCESS_KEY>
region = auto
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
acl = private

Where:

[r2]: A custom name(an alias) for storage service. We need to use it to operate files.
type = s3: The type of file operation API. R2 supports the S3 standard protocol.
provider = Cloudflare: The storage provider ID. You could use man rclone in your terminal to get the supported providers.
access_key_id: You need to create a token with Write permissions on the bucket
secret_access_key: Same as above.
endpoint: The URL that rclone uses to operate files. To get the account id on the top-right of the R2 homepage.

To run rclone lsf r2: to see your buckets and rclone lsf r2:your-bucket to show the file list within a bucket.

e.g.

rclone lsf r2:
cti-knowledge-base-store/

rclone lsf r2:cti-knowledge-base-store/

gives all the directories in the bucket, e.g.

disarm/
locations/
mitre-attack-enterprise/
mitre-attack-ics/
mitre-attack-mobile/
mitre-capec/
mitre-cwe/
nvd-cpe/
nvd_cve/
sigma-rules/
yara-rules/

To upload a file to the root of the bucket

rclone copy /path/to/file.type r2:your-bucket

To upload a file to a specific directory in the bucket

rclone copy /path/to/file.type r2:your-bucket/DIRECTORY

To upload a directory of files to a specific directory in the bucket

rclone copy /path/to/directory r2:your-bucket/DIRECTORY

other useful rclone flags we use…

include hidden mac files, print progress, and retry on failures during upload

rclone --exclude '.*' copy bundles/ r2:BUCKET -v --retries 10 --low-level-retries 10

delete all the bucket contents (but keep the bucket):

rclone delete r2:BUCKET/

delete the bucket

rclone purge r2:BUCKET/