How To Debug Nginx Reverse Proxy Issues

Nginx server is loved by many people for its proxy capabilities and ease of configuration. A simple proxy_pass can allow you to connect to any of the backends such as GoLang, php-fpm, NodeJS, another Nginx, Tomcat, Apache, Gunicorn, uwsgi, Flask, Django, an external CDN and many more.

When proxying a request to another server, you may or may not have access to a log of the server. So it is important to be able to debug the problem if an issue occurs. Common problems that you may face when proxying request are below

  • 502 Bad Gateway
  • 504 Gateway Timeout
  • 404 Page Not Found
  • 403 Access Denied
  • 400 Bad Request request header or cookie too large
  • Wrong Redirect
  • upstream sent too big header while reading response header from upstream
  • Primary script unknown while reading response header from upstream
  • upstream prematurely closed connection while reading response header from upstream

Few of these issues can occur because of below possible reasons

  • Error in the your server code
  • Error in Nginx config
  • Error in the information server receives
  • Fine tuning of timeouts

In this article I will explain few techniques to debug Nginx + PHP-FPM. These techniques will be applicable to other servers like Gunicorn, uwsgi, or other sites as well

Sample PHP Backend with Error

Consider the below Nginx config with a PHP config

The above config passes all php scripts to the running php-fpm server listening on 127.0.0.1:9000. And anything other than php is denied.

Now let’s create a php file error.php in /var/www/html/

/var/www/html/error.php

Now when we call the above script through curl

If we look at nginx error log, it would show the error below

Now this error indicates that there is an issue with the headers that were sent back. In our demo case there is just one file to check, but in a real project there would be multiple files and it becomes hard to say where the error might be coming from. So what we need is to be able to see the interaction between Nginx and our backend used in proxy_pass

Logging traffic with socat

Socat is a command line based utility that establishes two bidirectional byte streams and transfers data between them. Install socat based on the package manager you have on your OS. One of the below commands should do

Once installed we need to run socat to listen on a port different than 9000 and pass the data to 9000 port.

So we will run a socat in a new terminal or screen or tmux. The communication transfer between the two ports will only happen till socat is running

The command is self explanatory. It is to listen on 9001 and pass traffic bidirectionaly between the two 127.0.0.1:9000. The -v is for verbose output, it helps us print the traffic

How to set up socat if application runs on unix socket?

No problems. You can adapt your socat command to something like below

Note: You will need to make sure the user has access to the socket. Some sockets run with 600 permissions and the owners are different than root or your current user. For example /run/php/php7.0-fpm.sock by default is owned by www-data. So you would need to change the owner of these socket or chmod them to 777 for your testing

Now let us get back to our testing the curl localhost/error.php. When we run

The call fail as usual. Let’s look at the socat terminal

From this we can easily see that X-MY-Header is coming with large data and after reading 8192 bytes that is 8KB of headers, nginx just leaves it and fails the request.

Now let us execute another curl statement this time

This time the output on the socat window will be

We can see that SCRIPT_FILENAME is getting a value /var/www/html/test/not_exists.php. And now we know this doesn’t exists in our code. So it becomes easy to understand what values the backend is getting and why it is not working.

Now let use test another scenario with proxy_pass to another site. So we want to load images from http://itgala.xyz/images/ from our localhost. So we update our config to

Now if we curl a image from the site http://itgala.xyz/images/docker_swarm_security_group.png as http://localhost/images/docker_swarm_security_group.png

This is weird, we requested a image and got an html. Let us open another socat session

And change our proxy_pass in nginx from tarunlalwani.com to 127.0.0.1:8080

In the window we can see the request as below

You can see that we are requesting /docker_swarm_security_group.png instead of /images/docker_swarm_security_group.png. If we check our proxy_pass, it is

When we add a trailing / to proxy_pass address then original uri is not sent to the server. So we need to modify our proxy_pass to

Now let us re-run our curl statement. The logs in socat will be

As we can see the url is now corrected. But we still have a 404. This could be because of Host: 127.0.0.1:8080. So we should set the host name also with our request

Note: This is only needed because we are sending the proxy to socat listener on localhost. If we had use https://itgala.xyz as the proxy_pass address, this was not needed

Now our curl command will work. So the final proxy_pass that we need is