Varnish error Backend fetch failed

Here are some generic tips on finding the actual cause of “Backend Fetch Failed” error shown by Varnish Cache.

Why backend fetch fails

Common reasons of getting “Backend fetch failed” error:

• Varnish timeout was hit
• Varnish header size limit reached
• A backend health probe failed – your backend is marked as sick
• Your PHP has generated fatal error (HTTP code > 500)
• Varnish can’t communicate to the backend
• Varnish cannot find an object to evict from cache while serving large file (ExpKill LRU_Exhausted and FetchError Could not get storage in varnishlog), similiar to this issue. Solution is often to return (pipe) for large files

Use varnishlog to see why backend fetch failed for particular request

It much better to hunt down specific request that generated the error. When you get “Backend fetch failed” in browser, take note of “XID: 12345” at the bottom of the page. Use that number to get relevant requests quickly.

In versions prior to 5.1 you can use the X-Varnish header:

First, you will get client side request using the following command:

This will display the client side request to Varnish. Investigate it for ID of the backend request. Find a line that looks like this:

Now you can investigate actual errors. Review request details of Varnish connection to the backend:

Use varnishlog to see all failed requests

Will show latest in-memory Varnish log entries for “Backend fetch failed”.

It may produce too many entries and quickly fill your screen. A better approach is to save in-memory Varnish log entries into a file.

Now we can investigate the file and find out details why we have failed requests.

Enable varnishlog service for collecting longer history of failed requests

One issue with the above approaches is that they allow us to lookup only recent requests. Varnish memory log obviously can’t hold for a long period of logged data.

Let’s enable varnishlog service which logs only failed requests. Run:

Then paste in:

This overrides the default startup command of the service, to log only failed requests.

You can then enable the logging with:

Now we can read all the failed requests from the binary log that is being recorded by varnishlog daemon.

You can adjust how many days of varnish logs are being kept on server by looking into /etc/logrotate.d/varnish.