Skip to content

Add lambda-s3-download pattern for streaming URL downloads to S3#2956

Open
roblOnTour wants to merge 1 commit intoaws-samples:mainfrom
roblOnTour:roblontour-feature-lambda-s3-download
Open

Add lambda-s3-download pattern for streaming URL downloads to S3#2956
roblOnTour wants to merge 1 commit intoaws-samples:mainfrom
roblOnTour:roblontour-feature-lambda-s3-download

Conversation

@roblOnTour
Copy link

Issue #, if available:
N/A

Description of changes:

New pattern: lambda-s3-download

A Lambda function (Python 3.12) that downloads a file from a URL and uploads it to S3 using multipart upload with SHA256 checksums. The file is streamed in configurable chunks through /tmp, allowing it to handle files larger than Lambda's memory limit.

Features:

  • Chunked streaming download with configurable chunk size (clamped between 5 MB and 5 GB)
  • S3 multipart upload with SHA256 checksum verification
  • Automatic cleanup of incomplete multipart uploads on failure
  • 15-minute timeout, 1 GB memory, 10 GB ephemeral storage

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants