Problem
dvc-s3 already maps multipart_chunksize and multipart_threshold into boto3's TransferConfig via _TRANSFER_CONFIG_ALIASES (see dvc_s3/__init__.py lines 59β60, 117), but dvc/config_schema.py's s3 block (lines ~165β193) doesn't list either key. Result: any attempt to set them is rejected at config-load time, before the plugin ever sees them.
$ dvc remote modify --local codap-dvc multipart_chunksize 64MiB
ERROR: configuration error - config file error: extra keys not allowed @ data['remote']['PROFILE']['multipart_chunksize']
Reproduced on DVC 3.66.1 and 3.67.1.
Why this matters
Scaleway Object Storage caps multipart uploads at 1000 parts (vs AWS S3's 10,000). With boto3's default 8 MiB part size, the maximum object size is 8 MiB Γ 1000 = 8 GiB. Any file larger than that fails with:
ERROR: failed to transfer '<hash>' - [Errno 22] Part number must be an integer between 1 and 1000, inclusive: An error occurred (InvalidArgument) when calling the UploadPart operation
The intended fix β multipart_chunksize 64MiB, lifting the ceiling to 64 GiB β is correctly handled by dvc-s3 once the value reaches the plugin, but the config schema blocks it from ever getting there.
This is not Scaleway-specific in any architectural sense; the same gap blocks any S3-compatible backend with stricter multipart limits, and blocks legitimate tuning on AWS itself for very large objects.
Proposed fix
Add the two keys to the s3 block in dvc/config_schema.py:
"s3": {
# ...existing keys...
"read_timeout": Coerce(int),
"connect_timeout": Coerce(int),
"multipart_threshold": str, # human-readable size, e.g. "64MiB"
"multipart_chunksize": str, # human-readable size, e.g. "64MiB"
Optional("verify", default=False): Bool,
**REMOTE_COMMON,
},
dvc-s3's _split_s3_config already calls human_readable_to_bytes on these two keys, so accepting str is correct. Optionally, max_queue_size and max_concurrent_requests (also in the alias map but not the schema) could be added in the same change β same gap, same rationale.
Workaround
For users blocked today, content-addressed objects can be uploaded directly via rclone (or any S3 client) with a sufficient chunk_size, written to the same <bucket>/files/md5/<2chars>/<rest> path DVC uses. dvc push becomes a HEAD-check no-op for those objects; dvc pull works transparently from any other machine since the path layout is the contract, not the upload mechanism. This isn't ideal β every operator with a >8 GiB object hits it β but it unblocks the immediate failure.
Environment
- DVC: 3.66.1 (also reproduced on 3.67.1)
- dvc-s3: 3.3.0
- aiobotocore: 3.6.0, botocore: 1.43.0
- Python 3.13.2, Linux
- Remote: Scaleway Object Storage (S3-compatible,
s3.fr-par.scw.cloud)
Problem
dvc-s3already mapsmultipart_chunksizeandmultipart_thresholdinto boto3'sTransferConfigvia_TRANSFER_CONFIG_ALIASES(seedvc_s3/__init__.pylines 59β60, 117), butdvc/config_schema.py'ss3block (lines ~165β193) doesn't list either key. Result: any attempt to set them is rejected at config-load time, before the plugin ever sees them.Reproduced on DVC 3.66.1 and 3.67.1.
Why this matters
Scaleway Object Storage caps multipart uploads at 1000 parts (vs AWS S3's 10,000). With boto3's default 8 MiB part size, the maximum object size is 8 MiB Γ 1000 = 8 GiB. Any file larger than that fails with:
The intended fix β
multipart_chunksize 64MiB, lifting the ceiling to 64 GiB β is correctly handled bydvc-s3once the value reaches the plugin, but the config schema blocks it from ever getting there.This is not Scaleway-specific in any architectural sense; the same gap blocks any S3-compatible backend with stricter multipart limits, and blocks legitimate tuning on AWS itself for very large objects.
Proposed fix
Add the two keys to the
s3block indvc/config_schema.py:dvc-s3's_split_s3_configalready callshuman_readable_to_byteson these two keys, so acceptingstris correct. Optionally,max_queue_sizeandmax_concurrent_requests(also in the alias map but not the schema) could be added in the same change β same gap, same rationale.Workaround
For users blocked today, content-addressed objects can be uploaded directly via
rclone(or any S3 client) with a sufficientchunk_size, written to the same<bucket>/files/md5/<2chars>/<rest>path DVC uses.dvc pushbecomes a HEAD-check no-op for those objects;dvc pullworks transparently from any other machine since the path layout is the contract, not the upload mechanism. This isn't ideal β every operator with a >8 GiB object hits it β but it unblocks the immediate failure.Environment
s3.fr-par.scw.cloud)