Drupal high cache miss rate — finding the path that's bypassing your cache layer

1. Problem

Your Drupal site is slower than it should be, PHP-FPM is sweating under traffic that used to be a non-event, and your reverse-proxy cache hit ratio has drifted from 92% to 61% over the last week. New Relic blames "PHP execution time." Your hosting dashboard shows MySQL CPU climbing in lockstep with anonymous traffic. You're staring at the classic "drupal high cache miss rate which pages not caching" question and the dashboard doesn't tell you which path.

The page-cache module is enabled. system.performance has cache.page.max_age set. Varnish is in front. And yet every request to a specific subset of paths punches through to PHP, hits Drupal's full bootstrap, queries node, users_field_data, path_alias, and renders fresh — for anonymous users. You can see it: drupal_cache_hit_total flatlining while drupal_cache_miss_total and drupal_uncacheable_requests_total climb together. What you can't see from those numbers alone is which paths.

This usually surfaces as a drupal_uncacheable_requests_total increase paired with new entries in drupal_top_uncached_paths — Drupal's top-N panel naming the exact URLs skipping the cache. Without it you are guessing.

2. Impact

Every uncached anonymous request is a full Drupal bootstrap: autoloader, container build, route match, render array, twig compile, DB round-trips per entity load. On a stock Drupal 10 install that's 80–250ms of PHP-FPM time per request, versus 2ms from Varnish. The math gets ugly fast.

Concrete pattern: a publisher running Drupal 10 on a 4-vCPU app server sees baseline 60 req/s with 92% Varnish hit ratio. A contrib update silently injects Cache-Control: no-cache, must-revalidate on every node teaser block render. Hit ratio drops to 64%. PHP-FPM CPU goes from 8% to 71% at lunchtime peak. p95 latency goes from 110ms to 1.8s. The site becomes the kind of slow that loses readers and tanks Core Web Vitals. Three weeks later the SEO team asks why ranking dropped.

For Drupal Commerce the cost is worse: anonymous category pages that served from cache now hit the DB every request. Under a spike (newsletter blast, Black Friday landing), max_connections saturates, db.connection_failed fires, and the slow-path becomes a no-path. A 10-minute blast to 200k subscribers can take a healthy site offline if cacheability silently broke that morning.

You usually find this after the slowdown is customer-visible, because Drupal's admin reports don't tell you which path is uncacheable — only that the global ratio dropped.

3. Why It’s Hard to Spot

Cacheability in Drupal is a layered system that fails silently. A response is cacheable if and only if every contributing render array, block, and entity agrees it's cacheable. Any one of them can flip the response to uncacheable by setting max-age = 0, adding a cache context that varies per-user, or emitting a Cache-Control: no-cache header from anywhere in the render pipeline.

When that happens, Drupal does not log it. It does not warn. The page renders correctly, returns 200, and Varnish dutifully refuses to cache it. The only artifact is the response header. Nothing in /admin/reports tells you which path is the culprit.

Standard tooling makes this worse:

1. Page-cache module reports a single global ratio. /admin/reports/status shows "Internal Page Cache: enabled." It does not show that 8% of paths account for 73% of misses.
2. APM tools see PHP time, not cache state. New Relic will tell you /articles/foo is slow — not that its Cache-Control flipped to no-cache after a contrib update.
3. Varnish's varnishstat aggregates. You can see cache_miss climbing, but you can't get a top-N of which URLs are missing without sampling varnishlog and parsing it yourself.

The signal you actually need is "what are the top uncacheable paths on this site, ranked by request volume, in the last hour?" Drupal doesn't compute it. Varnish doesn't compute it. APM doesn't. So in practice nobody runs the query, and the regression sits there for weeks.

4. Cause

Drupal classifies every response into one of three states at the moment it leaves the kernel:

• Cacheable, hit — page-cache served it from storage. Counted in drupal_cache_hit_total.
• Cacheable, miss — response can be cached but wasn't in storage at request time. Will be cached for the next request. Counted in drupal_cache_miss_total.
• Uncacheable — response carries Cache-Control: no-cache, max-age=0, private, or Set-Cookie. Will never be cached. Counted in drupal_uncacheable_requests_total.

The Logystera Drupal module hooks KernelEvents::RESPONSE, inspects the outgoing Cache-Control and Vary headers, and emits one of three signals per request. It also samples the request URI for any response that increments drupal_uncacheable_requests_total and feeds it into a top-N reservoir, surfaced as drupal_top_uncached_paths — the metric that names which paths are uncacheable.

The distinction matters: a high drupal_cache_miss_total is often benign — a path was simply not yet in cache and the next request will hit. A high drupal_uncacheable_requests_total is never benign for an anonymous-traffic path — that path will always be uncacheable, every request, until something on the cacheability chain changes. The top-N metric tells you which paths fall into the "always uncacheable" bucket — where the real CPU is burning.

5. Solution

5.1 Diagnose (logs first)

The diagnosis path is mechanical: confirm that uncacheable requests are actually rising, get the top paths, then for each one, find the cache-control header on the response and trace it back to the code that set it.

1. Confirm the uncacheable rate is actually elevated, not just cache miss.

# Drupal's structured request log — cache_state field appended by Logystera module.
grep -E "cache_state=(uncacheable|miss|hit)" /var/log/drupal/request.log \
  | awk -F'cache_state=' '{print $2}' | awk '{print $1}' | sort | uniq -c

If uncacheable is climbing vs hit, that's drupal_uncacheable_requests_total rising. If only miss is climbing, you have a cache warming problem — different fix.

2. Get the top uncacheable paths from the access log.

# nginx/apache access log with cache-status logged. Find BYPASS paths over the last hour.
awk '$NF == "BYPASS" {print $7}' /var/log/nginx/drupal-access.log \
  | sort | uniq -c | sort -rn | head -20

This is the manual version of drupal_top_uncached_paths. The top of the list is your culprit set.

3. For each top path, dump the response headers and find the cacheability killer.

curl -sI 'https://example.com/api/v1/cart-summary' | grep -iE "cache-control|vary|set-cookie|x-drupal-cache"

What to look for, in order of frequency:

• Cache-Control: no-cache, max-age=0 → response itself is marked uncacheable.
• Set-Cookie: SESS... on an anonymous response → Drupal started a session, killing cacheability.
• X-Drupal-Cache: UNCACHEABLE → dynamic page cache explicitly opted out.
• Vary: Cookie with max-age > 0 → cacheable in theory, per-user in practice.

4. Time-correlate with the most recent change window.

drupal_uncacheable_requests_total rarely climbs on its own. It climbs immediately after a composer update, a contrib module enable, or a custom-module deploy. Find the change.

cd /var/www/drupal && git log --since="7 days ago" --oneline -- web/modules/contrib web/modules/custom
stat -c '%y' composer.lock
git log -p --since="7 days ago" -- composer.lock | grep -E "^\+" | grep "drupal/" | head

If drupal_top_uncached_paths started showing /api/v1/cart-summary last Tuesday at 14:03, and composer.lock shows drupal/commerce_cart updated at 14:01, you have your answer. The signal-to-deploy correlation is the entire fix.

5. Trace it back to code with drush.

drush -r /var/www/drupal/web ev '
  $request = \Symfony\Component\HttpFoundation\Request::create("/api/v1/cart-summary");
  $response = \Drupal::service("http_kernel")->handle($request);
  print "Cache-Control: " . $response->headers->get("Cache-Control") . "\n";
  print "max-age: " . $response->getCacheableMetadata()->getCacheMaxAge() . "\n";
'

max-age: 0 is the smoking gun. Grep contrib for the line that sets it:

grep -rn "setCacheMaxAge(0)\|'max-age' => 0\|->setPrivate()" \
  /var/www/drupal/web/modules/contrib/<suspect_module>/

5.2 Root Causes

Each cause maps to a specific signal and a specific code path. Prioritized by frequency.

• Contrib module sets max-age = 0 without thinking — most common. A module's hook_preprocess_HOOK or block plugin calls $build['#cache']['max-age'] = 0 to be "safe." The cache metadata bubbles up and kills the whole response. Produces drupal_uncacheable_requests_total for any path that renders the affected block. Path appears in drupal_top_uncached_paths with high volume.

• AJAX/REST endpoint with anonymous-bypass logic — a custom controller returns CacheableJsonResponse with setMaxAge(0) because it's "always dynamic." Correct for authenticated users, accidentally applied to anonymous. Surfaces as a single high-traffic path in drupal_top_uncached_paths (e.g., /api/v1/cart-summary, /views/ajax).

• Session cookie issued on an anonymous request — a contrib module (form-cache, anti-spam, custom analytics) calls \Drupal::service('session')->start() during render of an anonymous page. The Set-Cookie: SESS... header forces Cache-Control: must-revalidate, no-cache, private. Surfaces as drupal_uncacheable_requests_total with Set-Cookie on response and a path that should obviously be public.

• Admin pages legitimately uncacheable — /admin/, /user/, /node//edit are correctly uncacheable. They appear in drupal_top_uncached_paths as baseline — not a regression. Filter the panel to exclude /admin if they crowd out real signal.

• Vary: Cookie from a context-sensitive block — a sidebar block uses 'cookies:foo' cache context, which adds Vary: Cookie to the response. Technically cacheable, but Varnish treats Vary: Cookie as effectively uncacheable because every visitor has a different cookie set. Surfaces in drupal_cache_by_request_type as high cacheable-but-not-cached volume.

• CDN stripping Cache-Control upstream — Drupal sends Cache-Control: max-age=600, public; CloudFront's "ignore origin cache headers" replaces it. Drupal-side drupal_uncacheable_requests_total looks healthy, Varnish hit ratio is still bad — diagnosis moves to the proxy layer.

5.3 Fix

Match the fix to which root cause drupal_top_uncached_paths pointed at.

Cause A — module sets max-age=0: find the offending render-array tweak and remove it, or replace with a sensible TTL. If it's contrib, file an issue and patch with composer patches.

// Bad — what you found:
$build['#cache']['max-age'] = 0;

// Good — let metadata bubble up correctly:
$build['#cache']['max-age'] = 300;          // 5 minutes
$build['#cache']['contexts'][] = 'url.path'; // vary by path, not all cookies

Cause B — AJAX endpoint anon-bypass: scope the setMaxAge(0) to authenticated users only.

$response = new CacheableJsonResponse($data);
if (\Drupal::currentUser()->isAuthenticated()) {
  $response->setMaxAge(0);
} else {
  $response->setMaxAge(60);
  $response->getCacheableMetadata()->addCacheContexts(['url.path']);
}

Cause C — session cookie on anon request: find the module starting a session. Common offenders: eu_cookie_compliance (some configs), older flag versions, custom analytics modules. Either disable the session start path or move it to authenticated users.

# Find session starts in custom + contrib modules:
grep -rn "session()->start\|->migrate()\|setSession" /var/www/drupal/web/modules/

Cause D — admin pages baseline noise: filter drupal_top_uncached_paths to exclude expected uncacheable paths. Add path_pattern_exclude: ["/admin", "/user/", "/node/*/edit"] to your metric set's labels filter so admin traffic isn't the top of the list.

Cause E — Vary: Cookie blowup: narrow the cache context. Replace 'cookies:foo' with 'cookies:my_specific_cookie' so Varnish varies on a single named cookie instead of the whole header.

Cause F — proxy stripping headers: verify on the CDN side. For CloudFront, set the cache policy to "Origin Cache Control"; for Cloudflare, disable "Cache Everything" rules that override origin.

After every fix: drush cr and re-curl the path. The Cache-Control response header is the verification.

5.4 Verify

You're looking for two things to hold: drupal_uncacheable_requests_total returns to baseline for the affected paths, and drupal_top_uncached_paths no longer lists them in the top-N (or lists them with traffic matching admin-only volume).

# Re-pull headers for the path that was the top offender:
curl -sI 'https://example.com/api/v1/cart-summary' | grep -i cache-control
# Expected: Cache-Control: max-age=600, public

# Watch the BYPASS rate from access log — should drop within minutes:
tail -f /var/log/nginx/drupal-access.log | grep "BYPASS" | wc -l

In Logystera's entity view, healthy state looks like: drupal_uncacheable_requests_total under 8% of total requests on average (baseline is admin + user + edit forms — never zero). drupal_top_uncached_paths lists /admin/, /user/, /node//edit — not* anonymous content paths or REST endpoints. Hit ratio (hit / (hit + miss)) sits at 85%+ for a content-heavy site.

The baseline matters: this signal is never zero. Any production Drupal site has 5–10% uncacheable traffic from legitimately uncacheable paths. The regression is when the ratio jumps to 25–40%, or when the top-N starts including paths that should be cacheable. If the suspect endpoint stays in the top-N after a deploy, the max-age=0 is still being set somewhere — keep grepping. If the ratio recovers but reverts within 24h, you patched a symptom downstream of the real source — usually a config-export issue where the next deploy reverts your fix.

6. How to Catch This Early

Fixing it is straightforward once you know the cause. The hard part is knowing it happened at all.

This issue surfaces as drupal_uncacheable_requests_total.

Everything you just did manually — sample access logs, sort BYPASS by URL, curl each suspect path for Cache-Control, correlate with the deploy window — Logystera does automatically. The Drupal module hooks KernelEvents::RESPONSE, classifies every response, and emits the cache-state signal with the request URI so the top-N panel is computed in real time.

!Logystera dashboard — drupal_uncacheable_requests_total over time drupal_uncacheable_requests_total rate, last 24h — step change at 14:03 immediately after composer update drupal/commerce_cart.

The companion panel — drupal_top_uncached_paths — names exactly which paths are bypassing cache, ranked by request volume. That panel is what turns "the site is slow" into "/api/v1/cart-summary started returning Cache-Control: no-cache at 14:03 and is now 38% of your uncacheable traffic." Compared to global hit ratios, the top-N is the metric that points at code.

The rule that fires is id 511 — Drupal cacheability regression, severity warning, threshold drupal_uncacheable_requests_total / total_requests > 20% sustained over 10 minutes AND a new path enters the top-5 of drupal_top_uncached_paths that wasn't there in the previous 24h. The path-stability check suppresses the alert during legitimate admin-traffic spikes (which raise the ratio without changing top-N composition).

!Logystera alert — Drupal cacheability regression Warning fires when uncacheable ratio crosses 20% AND a new path enters the top-N — alert payload includes the new path and its Cache-Control header.

The alert payload includes the timestamp, the new path (e.g., /api/v1/cart-summary), the Cache-Control header observed, and the deploy that preceded it (correlated from drupal_state_change). That's enough to walk straight to §5.3 cause A or B without opening a terminal.

The fix is simple once you know the problem. The hard part is knowing it happened at all. Logystera turns this kind of regression from a three-week SEO casualty into a 10-minute notification with the path that proves it.

7. Related Silent Failures

• drupal_cache_miss_total spike without uncacheable — cache eviction or restart, not a cacheability bug. Different fix: warm the cache or raise eviction headroom.
• drupal_cache_by_request_type showing AJAX/REST dominance — sign your API surface is uncached even when content is. Often the biggest lever.
• drupal_state_change (config import) — config exports that flip system.performance.cache.page.max_age to 0 on deploy. Pairs with drupal_uncacheable_requests_total as the trigger event.
• drupal_php_error_total rising in lockstep — uncached anon requests hit PHP and surface deprecation noise that cache previously absorbed. Cacheability regressions unmask other latent bugs.
• db.connection_limit (1040) — the downstream failure when uncacheable + traffic spike. Cache regressions are the most common pre-cursor.