BUD-12

Multi-part uploads

draft optional

This bud defines a new PATCH method for the /upload endpoint to allow clients to upload blobs in multiple parts.

Signaling support for multi-part uploads

The server SHOULD respond to an OPTIONS /upload request with a 204 "No Content" response according to RFC-9110 including the Allow header with PATCH

Upload requirements

The server SHOULD implement BUD-06 "Upload requirements" to allow clients to check if a blob can be uploaded before uploading any chunks.

Chunking strategy

The client MAY split the blob into as many chunks as needed and MAY include overlap in the chunks if needed.

The server MUST concatenate the chunks based on the Upload-Offset and Content-Length headers to reconstruct the final blob to ensure any overlap is accounted for.

The server MUST support chunks arriving out of order and MUST support concurrent chunk uploads from the same client.

Uploading chunks

Clients MUST send the following headers in each PATCH /upload request:

X-SHA-256: The sha256 hash of the final blob. this should be considered the "ID" of the multi-part upload.
Upload-Type: The mine type of the final blob. should be set like Content-Type defined in RFC-9110
Upload-Length: The total length of the blob. should be set like Content-Length defined in RFC-9110
Content-Length: The length of the chunk in bytes.
Upload-Offset: The offset of the chunk in the blob.
Content-Type: The type of the chunk. MUST be set to application/octet-stream

The server MUST respond with a 204 "No Content" response if the chunk was accepted or a 4xx status code if it was not.

Uploading the final chunk

Once the server has received enough chunks to cover the Upload-Length of the blob the server MUST respond with a 2xx status code following BUD-02 or a 4xx status code if the blob hash does not match the X-SHA-256 header

Authorization

The server MAY require authorization for PATCH /upload requests as defined in BUD-11. The authorization event MUST use the upload verb and include one x tag per chunk hash plus a final x tag for the complete blob hash.

When the number of chunks is large enough that all chunk hashes would exceed HTTP header size limits, clients MAY split the chunk hashes across multiple authorization events and send each event in a separate request using the same X-SHA-256 upload ID.

Partial upload timeout

Servers MAY impose a timeout on in-progress uploads. If no PATCH request is received within the timeout window, the server MAY discard the partial upload and free any associated resources. The RECOMMENDED timeout is 60 seconds of inactivity between chunks.

Resuming uploads

The client SHOULD keep track of a chunks it has uploaded in order to resume uploads after a failure.

Example upload flow

# Client splits the blob into 4 chunks
split -b 46073 bitcoin.pdf chunk_

# Client uploads the first chunk
curl -X PATCH http://cdn.example.com/upload \
  -H "X-SHA-256: b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553" \
  -H "Upload-Type: application/pdf" \
  -H "Upload-Length: 184292" \
  -H "Upload-Offset: 0" \
  -H "Content-Length: 46073" \
  -H "Content-Type: application/octet-stream" \
  --data-binary "@chunk_aa"

# Server accepts the chunk and responds with a 204
HTTP/1.1 204 No Content

# CLient uploads remaining chunks (2-4)
curl -X PATCH http://cdn.example.com/upload
	# ..
	--data-binary "@chunk_ab"
curl -X PATCH http://cdn.example.com/upload
	# ...
	--data-binary "@chunk_ac"
curl -X PATCH http://cdn.example.com/upload
	# ...
	--data-binary "@chunk_ad"

# Server responds with 200 OK
HTTP/1.1 200 OK
Content-Type: application/pdf
Content-Length: 184292

{
  "url": "https://cdn.example.com/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf",
	"sha256": "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553",
  "uploaded": 1725105921,
	"type": "application/pdf",
	"length": 184292
}

4.2 KiB Raw Blame History