I am using

• Spring Cloud Gateway version 3.0.3
• reactor-netty-core 1.0.10

and i am facing a io.netty.util.internal.OutOfDirectMemoryError when the spring cloud gateway modifyRequestBody filter is being used. When it is being commented out, load tests runs fine without any error.

In our filters, we are doing a cache of the request body which will be used in our Fallback when there is a timeout or errors. Cache is just using the default ServerWebExchangeUtils.cacheRequestBody method.Fallback is implemented using Resilience4j Circuit Breaker.

In part of our chain, we are also using the spring gateway modifyRequestBody filter to modify our request body before sending it to the resource server.

OUT 2022-01-19 18:10:54.344 ERROR 16 --- [r-http-epoll-12] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
OUT Recent access records:
OUT Created at:
OUT io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
OUT io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
OUT org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71)
OUT org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
OUT org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:91)
OUT org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:75)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
OUT reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
OUT reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
OUT reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
OUT reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
OUT reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
OUT reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
OUT reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
OUT reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
OUT reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1817)
OUT reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
OUT reactor.core.publisher.Mono.subscribe(Mono.java:4338)
OUT reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)
OUT reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
OUT reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.Mono.subscribe(Mono.java:4338)
OUT reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)
OUT reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
OUT reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
OUT reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
OUT reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
OUT reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
OUT reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
OUT reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
OUT reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
OUT reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:468)
OUT reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:260)
OUT reactor.netty.channel.FluxReceive.request(FluxReceive.java:129)
OUT reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:162)
OUT reactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)
OUT reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:162)
OUT reactor.core.publisher.MonoCollect$CollectSubscriber.onSubscribe(MonoCollect.java:103)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92)
OUT reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:171)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92)
OUT reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:167)
OUT reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:146)
OUT io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
OUT io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
OUT java.lang.Thread.run(Thread.java:748)
Netty.docs: Reference counted objects

Sample Code Snippet:

return predicateSpec
                .path(pathPattern)
                .filters(f -> f
                        .filter(cachingRequestBodyFilter)
                        .modifyRequestBody(String.class, String.class, (exchange, modified) -> {
                            //Modify request body method here
                        })
                        .circuitBreaker(c -> c.setName("sessionTimeoutCircuitBreaker").setFallbackUri("forward:/timeoutFallback/"+country+"/"+domain))
                )
                .uri(uri);