Prelude

There are many cases where the requests have to route through intermediate servers before reaching Application Server. Such intermediate servers may include Reverse Proxy, CDN, Load balancers, etc.

Most common is the case with CDN.

There are many CDN services available e.g. Cloudflare, KeyCDN, MaxCDN, Google CDN, AWS Cloudfront and many more.

Some CDNs e.g. KeyCDN, etc work simply by providing a simple domain which needs to be configured in the application configuration. Then, the application uses configured filters to find the eligible URLs and replaces the domain with that of CDN alias in the HTML content.

There are some CDNs which need all the traffic to route through them e.g Cloudflare, Google CDN, etc. Then, the static content is served by Service and rest is routed to origin (i.e. Source Application server). In this case, you need migrate your DNS Name Servers.

Such services may offer some extra benefits. There can be downsides too. A while ago Cloudflare Service was entirely down & sites were not accessible. This is higly rare.

In the later case, the traffic has to go through extra intermediate server(s) i.e. CDN servers, Load balancers, etc. This does add a bit of latency but that is often negligible.

This causes changes in HTTP headers (e.g Source IP) which otherwise should reach the Application server unchanged.

Our Case

One such case that we recently pursued was with Google Cloud Platform. There we needed to setup the application with Nginx Reverse Proxy, Google Load Balancer & Google CDN.  The application could run independently but that is not recommended.

As the request is routed to Google CDN first and then to Google Load Balancer, the client/source IP header was getting changed to the load balancer IP.

Note:- Generally, most of HTTP headers tend to remain same.

Application logs the source/client IP. But the client/source IP there would always be load balancer IP.

It doesn’t mean that the real client IP is lost. It was available in a separate header instead.

Proxy forwards X-Forwarded-For header which contains the sever IPs that the request visited before reaching proxy. Such header should be passed on by every intermediate server.

We configured the Nginx to forward the headers X-Forwarded-For & X-Real-IP(containing IP of last Server i.e. Load Balancer) to application. The application was logging the received header X-Real-IP as the source IP when used in proxy mode. This behaviour can be justified with the argument that Proxy server was supposed to receive the direct client traffic.

We needed to log the client IP (available in X-Forwarded-For) not the load balancer IP. The client IPs in logs can be really helpful if we need to track the our traffic origin.

The application was receiving the IPs of last 3 servers the request has been through. The received three IPs were Client IP, Google CDN IP and Google LB IP respectively.

Available Workarounds

1. Changes at the application code level.
2. Change Nginx configuration to forward only the client IP.

Making code changes can be time-taking and is often not suggested as it may break something else in the system. Also, the application was correct in its place.

So, Nginx config should be changed.

As Nginx had the IPs in “proxy_add_x_forwarded_for” header we just needed to grep the client IP only. This header has three comma seperated IPs. The IPs are appended to this string as the request passes through. Hence, the first IP is the client IP.

We could use a regex to extract the first IP from the string. We could just map the result to another variable which could then easily be forwarded to application as X-Forwared-For header.

Below the Nginx configuration greps and stores Client IP in a variable.

map $proxy_add_x_forwarded_for $client_ip {"~(?<IP>([0-9]{1,3}\.){3}[0-9]{1,3}),.*" $IP;}

Note:- We just grep the first IP only.

The above block can be considered a function with $proxy_add_x_forwarded_for as an argument and the returned value is stored in variable named client_ip.

Below proxy_set_header directive sets the X-Forwarded-For.

proxy_set_header X-Forwarded-For $client_ip;

Our application was Odoo. So, we integrated the above changes with that. Similar changes can be made for any other application.

The change in logs after reloading the Nginx.

App now logs the Client IP.

The CDN setup with Google CDN is bit different. Some changes should be made in Nginx.

Hope the above helps!!!!

Feel free to add a ticket if any issue and let us know your views.

You can always reach us at https://webkul.uvdesk.com/en/