0

I am using selenium with browsermob-proxy, ultimately powered by "netty-all", to access a site (outside my control) which offers up enormous headers as part of its authentication process. Proxy fails with a netty error:

io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 16384 bytes., version: HTTP/1.1

I need to remove all such limits from netty-alljar that my browsermob-proxy depends on, scalability, performance and memory conservation are not relevant in this use case.

Having cloned the repo, I changed:

  • DEFAULT_MAX_FRAME_SIZE in WebSocket00FrameDecoder (io.netty.handler.codec.http.websocketx)

  • HttpObjectDecoder default constructor in io.netty.handler.codec.http

to Integer.MAX_VALUE where appropriate.

However, even with these new settings it keeps throwing out "HTTP header is larger than 16384 bytes" in use.

  1. Where else could this 16384 limit be coming from?
  2. How does one remove it while retaining full functionality (at the acceptable cost to efficiency/memory usage etc)
granitezero
  • 101
  • 1
  • 4

1 Answers1

0

Arrived at solution, its far from elegant but it works - my use case is inefficiency/fault tolerant so use with care.

  1. I wont pollute this answer with Maven shenanigans as they are not strictly relevant, however, please note that netty-all by default pulls all of its components from the Maven repo. To change netty-all internals you will need to produce a jar of a required component (handler.codec.http in this case), then change pom.xml to pull in your modified jar. There are several methods to do this, the only one that worked for me was using mvn install to place the jar in the local .m2 repo:

    mvn install:install-file -Dfile=netty-codec-http-4.1.25.Final-SNAPSHOT.jar -DgroupId=io.netty -DartifactId=netty-codec-http -Dversion=4.1.25.Final-SNAPSHOT -Dpackaging=jar

    Then build netty-all to get the final jar, which you then use in your own project instead of the original.

  2. Files modified to remove size limits from http operation:

    • all/pom.xml
    • codec-http/pom.xml
    • codec-http/src/main/java/io/netty/handler/codec/http/HttpObjectDecoder.java
    • codec-http/src/main/java/io/netty/handler/codec/http/websocketx/WebSocket00FrameDecoder.java
    • codec-http/src/test/java/io/netty/handler/codec/http/HttpRequestDecoderTest.java
    • codec-http/src/test/java/io/netty/handler/codec/http/HttpResponseDecoderTest.java

  3. Aside from setting various size restrictions to Integer.MAX_VALUE, I commented out relevant tests to ensure that Maven "package" command succeeds in producing the jar.

  4. The git diff of the changes is available here:

    https://gist.github.com/granite-zero/723fa55ae628494ff9b833dde1973a00

    You could apply it as a patch to netty commit 04fac00c8c98ed26c5a75887c8e7e53b1e1b68d0

granitezero
  • 101
  • 1
  • 4