Magento 2 Bypass Varnish For Specific URL

There is no arguing that Varnish does a really good job in speeding up the page response time for Magento. However there will always cases where we don't want Varnish to cache the URL. A good example of this is product feed files

• They can be extremely large and requiring massive increases in variables that determine the maximum page size, which even if configured ends up being wasteful.
• We don't want the content to be cached, when the feed is requested we want it to have the content from the file at that point in time.

There are a number of ways to avoid this, the best we have found is bypass Varnish for feed files.

How To

In this example we are going to assume:

• You are using Nginx as your webserver
• You are using Nginx to terminate SSL before handing off to Varnish
• All your feed files are in the directory pub/feeds (though if yours aren't you can change the config to apply to your setup)
• All your feed files are either .xml or .csv (again if yours aren't you can change the config to apply to your setup)

We first need to find where the SSL termination happens, and the request is passed to Varnish. This usually looks something like

server {
    listen 443 ssl http2;
    server_name SERVER_NAME;

    ssl_certificate cert.crt;
    ssl_certificate_key cert.key;

    location / {
        proxy_pass http://varnish;
        proxy_set_header Host $host;
    }
}

We then need to add in a location block in to match our feed files location and stop it from being passed to Varnish.

server {
    listen 443 ssl http2;
    server_name SERVER_NAME;

    ssl_certificate cert.crt;
    ssl_certificate_key cert.key;

    location ~* /feeds/(.*)\.(xml|csv)$ {
        root /path/to/magento/pub;
        try_files $uri =404;
    }

    location / {
        proxy_pass http://varnish;
        proxy_set_header Host $host;
    }
}

To explain we will break down into it's parts

location ~* /feeds/(.*)\.(xml|csv)$ - Match any request that starts with /feeds/ and ends with .xml or .csv

If you wanted this to include .json files you could change to something like location ~* /feeds/(.*)\.(xml|csv|json)$

If you wanted to use a different directory you could change to location ~* /myDirectory/(.*)\.(xml|csv)$

root /path/to/magento/pub; - Tell Nginx this is where we should look for files, so if the request was /feeds/products.xml Then the full path would be /path/to/magento/pub/feeds/products.xml

try_files $uri =404; - Return the file if it exists, if not return a 404 page

Once this change is in place, all you need to do is reload Nginx for the change to take effect.

How To MDOQ

If you're on MDOQ managed Magento hosting you can implement this by:

1. Create an instance
2. Create the file mdoq/nginx/templates/default_https.conf if it doesn't exist
3. Add the following to it

   location ~* /feeds/(.*)\.(xml|csv)$ {
       root /home/magento/htdocs/pub;
       try_files $uri =404;
   }

4. Commit to source control
5. Follow the "I'm Done" process and do a zero downtime release.
6. Recreate "Environment Changes"