In this blog post, we explore potential vulnerabilities with reverse proxies
that use hop-by-hop headers. We first explain hop-by-hop headers and their
distinction from end-to-end headers. After that, we introduce the Go standard
library's httputil.ReverseProxy. We explain its flawed handling of hop-by-hop
headers. Finally, we will look into how you can fix the vulnerability by looking
at how we fixed Ory Oathkeeper, a cloud native identity & access proxy.
Overall, we believe that the impact of this vulnerability will not be huge in
most cases, but it could potentially be exploited by an attacker.
What are Hop-by-Hop Headers
HTTP/1.1 distinguishes between so-called hop-by-hop and end-to-end headers. Per
spec, hop-by-hop headers are only meant for the next hop on the request path, in
a cloud-native stack often some kind of reverse proxy. By default all headers
are considered end-to-end, except a list of standard headers that only really
make sense on a hop-by-hop basis, like Transfer-Encoding.
However, a client can mark a header to be treated as hop-by-hop by adding the
header name to the Connection header. For example, to treat the Cookie
header as hop-by-hop, one can send this request:
curl -H 'Cookie: value' -H 'Connection: close,cookie' https://some.url
According to the spec, the server receiving this request should remove the
Cookie header before forwarding the request.
Go Stdlib httputil.ReverseProxy
Now that we looked at the client side, let’s see how a standard reverse proxy
handles the Connection header. In general, Go comes with a great standard
library and is widely used. Because reverse proxies are such a common thing in a
modern, cloud-native stack, the Go standard library ships a simple yet powerful
reverse proxy (httputil.ReverseProxy).
Without going into too many confusing details on how to write your own reverse
proxy, I want to focus on the request forwarding part of it. The
httputil.ReverseProxy allows you to set a so-called Director hook. It is a
function that takes the incoming *http.Request and may modify it to set the
upstream host, rewrite the path, add headers, rewrite the body, … the sky is the
limit.
httputil.ReverseProxy handles incoming requests in this order:
- Incoming request is cloned and passed to the
Director hook
- Hop-by-hop headers are removed
- Request is forwarded to the upstream host
Hop-by-Hop Headers and httputil.ReverseProxy
You might have already spotted that the hop-by-hop headers are only removed
after the director potentially added new headers, so it is totally possible
that these are removed again if they were specified in the Connection header.
This leaves us with an inherently flawed design for the reverse proxy in the
standard library. Fortunately we don’t have to stop here and present some
third-party library that fixes this issue, but instead it
was identified and fixed as a potential
vulnerability already by the Go team. The fix landed with Go 1.20. Because the
previous design was flawed, they needed to add a new hook supersede the
Director which is now called Rewrite (way better naming as well).
We don't know why Director was not marked as deprecated, nor why no CVE was
assigned to this. The issue apparently flew under most user's radars, and has
not received any comments on GitHub as of the writing of this article.
How to Fix the Vulnerability
Ory Oathkeeper, our reverse proxy, was also affected. We fixed the issue by
migrating to the new Rewrite hook,
and we also
issued a security advisory
on this topic. In general, you can quite easily migrate the Director to the
Rewrite hook by replacing all usages of the *http.Request with the
*httputil.ProxyRequest.Out. See
the PR in Oathkeeper
for more details and examples.
We believe that even if the vulnerability is exploitable, the impact will not be
huge in most cases, but it could potentially. It only takes one developer who
decides that a request without an X-Forwarded-For header means it is an
internal request -- a reasonable assumption --, and allow such a request to do
all kinds of privileged calls. Think for yourself how much you trust your
reverse proxies and if you make such assumptions.
