Skip to content

fix: stream remote io paths and benchmark decompression backends#67

Merged
digizeph merged 2 commits intomainfrom
dev/fix-streaming-and-benchmarks
Mar 6, 2026
Merged

fix: stream remote io paths and benchmark decompression backends#67
digizeph merged 2 commits intomainfrom
dev/fix-streaming-and-benchmarks

Conversation

@digizeph
Copy link
Member

@digizeph digizeph commented Mar 6, 2026

Summary

  • make async downloads preserve raw bytes and stream cache population instead of buffering entire payloads in memory
  • reuse default blocking HTTP clients across GET and HEAD paths
  • replace string-parsed S3 status handling with a structured status error and stream S3 reads through a bounded channel
  • add gzip backend and bz2 decompression benchmarks plus a helper script to run the comparison matrix
  • update changelog, docs, and tests for the new behavior

Benchmarks

16 MiB generated text corpus, measured on both the raw decoder path and oneio::get_reader.

  • cloudflare-zlib: raw 1.162 GiB/s, oneio 1.135 GiB/s
  • zlib-rs: raw 1.035 GiB/s, oneio 1.017 GiB/s
  • zlib-ng: raw 1.020 GiB/s, oneio 1.014 GiB/s
  • miniz_oxide: raw 686 MiB/s, oneio 680 MiB/s
  • bz2: raw 76.8 MiB/s, oneio 76.9 MiB/s

cloudflare-zlib was the fastest gzip backend in this run. zlib-rs and zlib-ng were close behind and materially faster than miniz_oxide.

Example

Run the backend comparison locally:

./scripts/bench_decompression_backends.sh

Behavior note:

oneio::download_async("https://example.com/data.csv.gz", "local_data.csv.gz").await?;

The async download now preserves the remote bytes, matching download().

digizeph added 2 commits March 6, 2026 06:50
Update async download to preserve raw bytes and stream cache and S3 reads instead of buffering whole payloads.

Reuse default HTTP clients for blocking requests, add structured S3 status errors, expand decompression benchmarks for gzip backends and bz2, and refresh tests and changelog for the new behavior.
@digizeph digizeph merged commit b720b9a into main Mar 6, 2026
5 checks passed
@digizeph digizeph deleted the dev/fix-streaming-and-benchmarks branch March 6, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant