OOM and request cancellation when reading large objects

[michaelpaliy]

Problem:

When attempting to read a large file (~1GB) from S3 using the v3 Encryption SDK, the request fails with a hidden OutOfMemoryError. In v2 SDK, the same operation works without issues.

Stack trace:

java.io.IOException: Request cancelled
	at software.amazon.awssdk.http.nio.netty.internal.FutureCancelHandler.exceptionCaught(FutureCancelHandler.java:63)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
	at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
	at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
	at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
	at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1221)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1324)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:856)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$createExecutionFuture$3(NettyRequestExecutor.java:135)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:840)

The exception reported is misleading (FutureCancelledException), but after debugging I found it’s masking an actual OOM:

software.amazon.awssdk.http.nio.netty.internal.FutureCancelledException: java.lang.OutOfMemoryError: Java heap space

Steps to reproduce:

S3EncryptionClient s3EncryptionClient = S3EncryptionClient.builder()
        .enableDelayedAuthenticationMode(true)
        .kmsKeyId("XXX")
        .build();
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
        .bucket("XXX")
        .key("XXX")
        .build();
try (BufferedReader stream = new BufferedReader(new InputStreamReader(
        s3EncryptionClient.getObject(getObjectRequest, ResponseTransformer.toInputStream())))) {
    String line;
    while ((line = stream.readLine()) != null) {
        System.out.println("Read line: " + line);
        // Process the line
    }
}

I am using the latest version 3.4.0. The file size being tested is approximately 1GB. The runtime environment is Java 17.

[lucasmcdonald3]

Hi @michaelpaliy --

I wasn't able to reproduce the issue on my end -- v2 and v3 used comparable amounts of memory in my testing.

Could you profile your memory usage and let us know what you're seeing? We'd be happy to dig into this with more information.

Thanks,
Lucas

[michaelpaliy]

Hi @lucasmcdonald3 , Thanks for the reply!

Changing the crypto provider to Bouncy Castle fixed the issue for me, as it's done in the tests: S3EncryptionClientStreamTest:
https://github.com/aws/amazon-s3-encryption-client-java/blob/main/src/test/java/software/amazon/encryption/s3/S3EncryptionClientStreamTest.java#L422

It's important to note that the exception thrown before applying this fix doesn’t really point to the actual cause, and the OutOfMemoryError seems to be swallowed, which makes debugging quite tricky.

[sid22]

@michaelpaliy how did you RCA it to Out of memory ? I am getting similar issue wherein the file download works correctly with v1 sdk but fails with similar stack with v2 sdk ( accp and crypto comply )

[FoKas90]

I've just faced exactly same issue. I had tried to download 1Gb encrypred file via S3EncriptionClient and got the OOM.
The fraction of heap dump:

Thread 'aws-java-sdk-NettyEventLoop-1-0' with ID = 91
	java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
	java.util.Arrays.copyOf(Arrays.java:3541)
	java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
	java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:132)
	com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doUpdate(GaloisCounterMode.java:1458)
	com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doUpdate(GaloisCounterMode.java:1441)
	com.sun.crypto.provider.GaloisCounterMode.engineUpdate(GaloisCounterMode.java:349)
	javax.crypto.Cipher.update(Cipher.java:1908)
	software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:56)
	software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:19)
	software.amazon.awssdk.core.async.listener.SubscriberListener$NotifyingSubscriber.onNext(SubscriberListener.java:85)
	software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onNext(BytesReadTrackingPublisher.java:64)
	software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onNext(BytesReadTrackingPublisher.java:47)
	software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onNext(ResponseHandler.java:501)
	software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onNext(ResponseHandler.java:487)
	software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.lambda$onNext$4(ResponseHandler.java:335)
	software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1$$Lambda+0x00007f5d2c125ad8.run(Native method)
	software.amazon.awssdk.http.nio.netty.internal.utils.ExceptionHandlingUtils.tryCatch(ExceptionHandlingUtils.java:40)
	software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onNext(ResponseHandler.java:335)
	software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onNext(ResponseHandler.java:286)
	software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:407)
	software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)
	software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)
	software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)
	software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	java.lang.Thread.runWith(Thread.java:1596)
	java.lang.Thread.run(Thread.java:1583)

Calling stacktrace

Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception java.io.IOException: Request cancelled
	at software.amazon.awssdk.http.nio.netty.internal.FutureCancelHandler.exceptionCaught(FutureCancelHandler.java:63)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
	at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
	at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
	at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
	at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1210)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1324)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:856)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$createExecutionFuture$3(NettyRequestExecutor.java:135)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1583)

S3EncryptionClient

    S3EncryptionClient.builderV4() 
                .wrappedClient(s3Client) 
                .wrappedAsyncClient(s3AsyncClient) 
                .enableDelayedAuthenticationMode(true) 
                .region(Region.US_EAST_1) 
                .keyring(aesKeyring) 
                .build();

Maven setup

        <aws.sdk.version>2.33.7</aws.sdk.version>
        <s3.encryption.version>4.0.0</s3.encryption.version>

[lucasmcdonald3]

Hi @FoKas90 -- try switching your crypto provider from Sun JCE to BouncyCastle as is done here: https://github.com/aws/amazon-s3-encryption-client-java/blob/main/src/test/java/software/amazon/encryption/s3/S3EncryptionClientStreamTest.java#L320-L321

[FoKas90]

Hi @lucasmcdonald3!
Thanks for responding. I had switched crypto provider and it worked!