Ttdfjt

We have a Magento 2.3.1 store that uses CloudFlare and has Varnish enabled. If we use wget or curl to load a page, that page is cached in Varnish if we try wget or curl again on that same page.

However, if we test that URL via a browser, the page is not cached. It is cached after the first browser visit, and then remains cached on reloads from a browser.

We added the CF+Varnish rules as listed at:
https://support.cloudflare.com/hc/en-us/articles/200169376-Can-I-use-Cloudflare-and-Varnish-together-

but that did not seem to make a difference. We're using the recommended Magento default.vcl file that is generated.

Any ideas what could be stopping a wget/curl from loading a page into the Varnish cache such that any request from a browser would be served that cached page instead of causing Varnish to cache the page again?

Wget request:

/usr/bin/wget --no-cache --wait=0.3 --no-check-certificate --delete-after --header="user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" --header="Cache-control: max-age=0" --header="Accept-Language: en-US,en;q=0.5" --header="Accept-encoding: gzip, deflate" -S 'https://XXXX'
--2019-07-20 09:14:14-- https://XXXX
Resolving XXXX... 1.2.3.4
Connecting to XXXX|1.2.3.4|:443... connected.
HTTP request sent, awaiting response... 
 HTTP/1.1 200 OK
 Server: nginx
 Date: Sat, 20 Jul 2019 13:14:16 GMT
 Content-Type: text/html; charset=UTF-8
 Content-Length: 51021
 Connection: keep-alive
 X-Content-Type-Options: nosniff
 X-XSS-Protection: 1; mode=block
 X-Frame-Options: SAMEORIGIN
 Vary: Accept-Encoding
 X-Magento-Cache-Control: max-age=86400000, public, s-maxage=86400000
 Content-Encoding: gzip
 Age: 0
 X-Magento-Cache-Debug: MISS
 Pragma: no-cache
 Expires: -1
 Cache-Control: no-store, no-cache, must-revalidate, max-age=0
 Accept-Ranges: bytes
 Access-Control-Allow-Origin: *
Length: 51021 (50K) [text/html]

Browser response/header after wget runs:

accept-ranges: bytes
access-control-allow-origin: *
age: 0
cache-control: no-store, no-cache, must-revalidate, max-age=0
content-encoding: gzip
content-length: 51023
content-type: text/html; charset=UTF-8
date: Sat, 20 Jul 2019 13:14:29 GMT
expires: -1
pragma: no-cache
server: nginx
status: 200
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-magento-cache-control: max-age=86400000, public, s-maxage=86400000
x-magento-cache-debug: MISS
x-xss-protection: 1; mode=block
----
----
:authority: XXXX
:method: GET
:path: /YYYY
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cookie: mage-translation-storage=%7B%7D; mage-translation-file-version=%7B%7D; newsletter_popup=dontshowitagain; ku1-vid=e5cf0293-7c71-74d9-10df-ffd19dcd11bb; ku1-sid=UvdxKNYS6MKRhvKZFJN9a; _fbp=fb.1.1563628017878.1828386074; recordID=c7f615a9-d0c2-41d5-8ccc-7a11beee21b1; dmSessionID=c2397d68-85b5-4f04-ad24-0c5e92e69a57; __tawkuuid=e::XXXX::dr5R22jZlQVd2e0p7XTrs0xIAaO+8YriwgvRE8yBIalE8AkHbZFPjEheVHdRnQCI::2; form_key=WRPXsJkeLSDLVwoN; _gcl_au=1.1.997920531.1563628021; mage-cache-storage=%7B%7D; mage-cache-storage-section-invalidation=%7B%7D; mage-banners-cache-storage=%7B%7D; mage-messages=; _ga=GA1.3.129667006.1563628021; _gid=GA1.3.1515936369.1563628021; X-Magento-Vary=8bed0844a67603e244f2ca107da80466ff383efd; recently_viewed_product=%7B%7D; recently_viewed_product_previous=%7B%7D; recently_compared_product=%7B%7D; recently_compared_product_previous=%7B%7D; product_data_storage=%7B%7D; PHPSESSID=2197268281d0aedc4015f7be1f5974d7; mage-cache-sessid=true; section_data_ids=%7B%22cart%22%3A1563628090%7D; TawkConnectionTime=0
referer: https://XXXX/ZZZZ
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
----

If I put this in the default.vcl file (along with the M2 rules) Varnish caches each page the first time it is hit, regardless if from wget/curl or from a web browser:

sub vcl_recv {
 if (req.http.cookie) COOKIE3)=", "; 1=");
 set req.http.cookie = regsuball(req.http.cookie, ";[^ ][^;]*", "");
 set req.http.cookie = regsuball(req.http.cookie, "^[; ]+

This comes from the guide at https://www.varnish-software.com/wiki/content/tutorials/magento2/m2_step_by_step.html

I don't see any issues with doing this, but I find it odd that this is not mentioned in any docs when there are multiple cookies in play on pages...