How to configure Varnish Cache for Drupal with SSL Termination Using Pound or Nginx

Secure Socket Layer (SSL) is the protocol that allows web sites to serve traffic in HTTPS. This provides end to end encryption between the two end points (the browser and the web server). The benefits of using HTTPS is that traffic between the two end points cannot be deciphered by anyone snooping on the connection. This reduces the odds of exposing sensitive information such as passwords, or getting the web site hacked by malicious parties. Google has also indicated that sites serving content exclusively in HTTPS will get a small bump in Page Rank.

Historically, SSL certificate issuers have served a secondary purpose: identity verification. This is when the issuing authority vouches that a host or a domain is indeed owned by the entity that requests the SSL certificate for it. This is traditionally done by submitting paper work including government issued documentation, incorporation certificates, ...etc.

Historically, SSL certificates were costly. However, with the introduction of the Let's Encrypt initiative, functional SSL certificates are now free, and anyone who wants to use them can do so, minus the identity verification part, at least for now.

Implementing HTTPS with Drupal can be straightforward with low traffic web sites. The SSL certificate is installed in the web server, and that is about it. With larger web sites that handle a lot of traffic, a caching layer is almost always present. This caching layer is often Varnish. Varnish does not handle SSL traffic, and just passes all HTTPS traffic straight to Drupal, which means a lot of CPU and I/O load.

This article will explain how to avoid this drawback, and how to have it all: caching in Varnish, plus serving all the site using HTTPS.

The idea is quite simple in principle: terminate SSL before Varnish, which will never know that the content is encrypted upstream. Then pass the traffic from the encryptor/decryptor to Varnish on port 81. From there, Varnish will pass it to Apache on port 8080.

We assume you are deploying all this on Ubuntu 16.04 LTS, which uses Varnish 4.0, although the same can be applied to Ubuntu 14.04 LTS with Varnish 3.0.

Note that we use either one of two possible SSL termination daemons: Pound and Nginx. Each is better in certain cases, but for the large part, they are interchangeable.

One secondary purpose for this article is documenting how to create SSL bundles for intermediate certificate authorities, and to generate a combined certificate / private key. We document this because of the sparse online information on this very topic.

Install Pound

aptitude install pound

Preparing the SSL certificates for Pound

Pound does not allow the private key to be in a separate file or directory from the certificate itself. It has to be included with the main certificate, and with intermediate certificate authorities (if there are any).

We create a directory for the certificates:

mkdir /etc/pound/certs

cd /etc/pound/certs

We then create a bundle for the intermediate certificate authority. For example, if we are using using NameCheap for domain registration, they use COMODO for certificates, and we need to do the following. The order is important.

cat COMODORSADomainValidationSecureServerCA.crt \ 
  COMODORSAAddTrustCA.crt \ 
  AddTrustExternalCARoot.crt >> bundle.crt

Then, as we said earlier, we need to create a host certificate that includes the private key.

cat example_com.key example_com.crt > host.pem

And we make sure the host certificate (which contains the private key as well) and the bundle, are readable only to root.

chmod 600 bundle.crt host.pem 

Configure Pound

We then edit /etc/pound/pound.cfg

# We have to increase this from the default 128, since it is not enough
# for medium sized sites, where lots of connections are coming in
Threads 3000 

# Listener for unencrypted HTTP traffic 
ListenHTTP
  Address 0.0.0.0
  Port    80
  
  # If you have other hosts add them here
  Service 
    HeadRequire "Host: admin.example.com"
    Backend 
      Address 127.0.0.1
      Port 81
    End
  End
  
  # Redirect http to https
  Service 
    HeadRequire "Host: example.com"
    Redirect "https://example.com"
  End
  
  # Redirect from www to domain, also https
  Service 
    HeadRequire "Host: www.example.com"
    Redirect "https://example.com"
  End
End

# Listener for encrypted HTTP traffic 
ListenHTTPS
  Address 0.0.0.0
  Port    443
  # Add headers that Varnish will pass to Drupal, and Drupal will use to switch to HTTPS
  HeadRemove      "X-Forwarded-Proto"
  AddHeader       "X-Forwarded-Proto: https"
  
  # The SSL certificate, and the bundle containing intermediate certificates
  Cert      "/etc/pound/certs/host.pem"
  CAList    "/etc/pound/certs/bundle.crt"
  
  # Send all requests to Varnish
  Service 
    HeadRequire "Host: example.com"
    Backend 
      Address 127.0.0.1
      Port 81
    End
  End
  
  # Redirect www to the domain
  Service
    HeadRequire "Host: www.example.com.*"
    Redirect "https://example.com"
  End
End

Depending on the amount of concurrent traffic that your site gets, you may need to increase the number of open files for Pound. You also want to increase the backend time out times, and the browser time out time.

To do this, edit the file /etc/default/pound, and add the following lines:

# Timeout value, for browsers
Client  45

# Timeout value, for backend
Timeout 40

# Increase the number of open files, so pound does not log errors like:
# "HTTP Acces: Too many open files"
ulimit -n 20000

You also need to create a run directory for pound and change the owner for it, since the pound package does not do that automatically.

mkdir /var/run/pound
chown www-data.www-data /var/run/pound

Do not forget to change the 'startup' line from 0 to 1, otherwise pound will not start.

Configure SSL Termination for Drupal using Nginx

You may want to use Nginx instead of the simpler Pound in certain cases.

For example, if you want to process your site's traffic using analysis tools, for example Awstats, you need to capture those logs. Although Pound can output logs in Apache combined format, it also outputs errors to the same log, at least on Ubuntu 16.04, and that makes these logs unusable by analysis tools.

Also, Pound has a basic mechanism to handle redirects from the plain HTTP URLs to the corresponding SSL HTTPS URLs. But, you cannot do more complex rewrites, or more configurable and flexible options.

First install Nginx:

aptitude install nginx

Create a new virtual host under /etc/nginx/sites-available/example.com, with this in it:

# Redirect http www to https no-www
server {
  server_name www.example.com;
  access_log off;
  return 301 https://example.com$request_uri;
}

# Redirect http no-www to https no-www
server {
  listen      80 default_server;
  listen [::]:80 default_server;
  server_name example.com;
  access_log off;
  return 301 https://$host$request_uri;
}

# Redirect http www to https no-www
server {
  listen      443 ssl;
  server_name www.example.com;
  access_log off;
  return 301 https://example.com$request_uri;
}

server {
  listen      443 ssl default_server;
  listen [::]:443 ssl default_server ipv6only=on;

  server_name example.com;

  # We capture the log, so we can feed it to analysis tools, e.g. Awstats
  # This will be more comprehensive than what Apache captures, since Varnish
  # will end up removing a lot of the traffic from Apache
  #
  # Replace this line with: 'access_log off' if logging ties up the disk
  access_log /var/log/nginx/access-example.log;

  ssl on;

  # Must contain the a bundle if it is a chained certificate. Order is important.
  # cat example.com.crt bundle.crt > example.com.chained.crt  
  ssl_certificate      /etc/ssl/certs/example.com.chained.crt;
  ssl_certificate_key  /etc/ssl/private/example.com.key;

  # Test certificate
  #ssl_certificate     /etc/ssl/certs/ssl-cert-snakeoil.pem;
  #ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

  # Restrict to secure protocols, depending on whether you have visitors
  # from older browsers
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

  # Restrict ciphers to known secure ones
  ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;

  ssl_prefer_server_ciphers on;
  ssl_ecdh_curve secp384r1;
  ssl_stapling on;
  ssl_stapling_verify on;

  add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
  add_header X-Frame-Options SAMEORIGIN;
  add_header X-Content-Type-Options nosniff;

  location / {
    proxy_pass                         http://127.0.0.1:81;
    proxy_read_timeout                 90;
    proxy_connect_timeout              90;
    proxy_redirect                     off;

    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port  443;
    
    proxy_buffers                      8 24k;
    proxy_buffer_size                  2k;
  }
}

Then link this to an entry in the sites-enabled directory

cd /etc/nginx/sites-enabled

ln -s /etc/nginx/sites-available/example.com

Then we add some performance tuning parameters, by creating a new file: /etc/nginx/nginx.conf. These will make sure that we handle higher traffic than the default configuration allows:

At the top of the file, modify these two parameters, or add them if they are not present:

  
worker_processes       auto;
worker_rlimit_nofile   20000;

Then, under the 'events' section, add or modify to look like the following:

events {
  use epoll;
  worker_connections 19000;
  multi_accept       on;
}

And under the 'http' section, make sure the following parameters are added or modified to the following values:

http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 80;
keepalive_requests 10000;
client_max_body_size 50m;
}

We now have either Pound or Nginx in place, handling port 443 with SSL certifcates, and forwarding the plain text traffic to Varnish.

Change Varnish configuration to use an alternative port

First, we need to make Varnish work on port 81.

On 16.04 LTS, we edit the file: /lib/systemd/system/varnish.service. If you are using Ubuntu 14.04 LTS, then the changes should go into /etc/default/varnish instead.

Change the 'ExecStart' line for the following:

Port that Varnish will listen on (-a :81)
Varnish VCL Configuration file name (/etc/varnish/main.vcl)
Size of the cache (-s malloc,1536m)

You can also change the type of Varnish cache storage, e.g. to be on disk if it is too big to fit in memory (-s file,/var/cache/varnish/varnish_file.bin,200GB,8K). Make sure to create the directory and assign it the correct owner and permissions.

We use a different configuration file name so as to not overwrite the default one, and make updates easier (no questions asks during update to resolve differences).

In order to inform systemd that we changed a daemon startup unit, we need to issue the following command:

systemctl daemon-reload

Add Varnish configuration for SSL

We add the following section to the Varnish VCL configuration file. This will pass a header to Drupal for SSL, so Drupal will enforce HTTPS for that request.

# Routine used to determine the cache key if storing/retrieving a cached page.
sub vcl_hash {

  # This section is for Pound
  hash_data(req.url);

  if (req.http.host) {
    hash_data(req.http.host);
  }
  else {
    hash_data(server.ip);
  }

  # Use special internal SSL hash for https content
  # X-Forwarded-Proto is set to https by Pound
  if (req.http.X-Forwarded-Proto ~ "https") {
    hash_data(req.http.X-Forwarded-Proto);
  }
}

Another change you have to make in Varnish's vcl is this line:

set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]+|NO_CACHE)=", "; \1=");

And replace it with this line:

set req.http.Cookie = regsuball(req.http.Cookie, ";(S?SESS[a-z0-9]+|NO_CACHE)=", "; \1=");

This is done to ensure that Varnish will pass through the secure session cookies to the web server.

Change Apache's Configuration

If you had SSL enabled in Apache, you have to disable it so that only Pound (or Nginx) are listening on port 443. If you do not do this, Pound and Nginx will refuse to start with an error: Address already in use.

First disable the Apache SSL module.

a2dismod ssl 

We also need to make Apache listen on port 8080, which Varnish will use to forward traffic to.

  
Listen 8080

And finally, your VirtualHost directives should listen on port 8080, as follows. It is also best if you restrict the listening on the localhost interface, so outside connections cannot be made to the plain text virtual hosts.

<VirtualHost 127.0.0.1:8080>
...
</VirtualHost>

The rest of Apache's configuration is detailed in an earlier article on Apache MPM Worker threaded server, with PHP-FPM.

Configure Drupal for Varnish and SSL Termination

We are not done yet. In order for Drupal to know that it should only use SSL for this page request, and not allow connections from plain HTTP, we have to add the following to settings.php:

// Force HTTPS, since we are using SSL exclusively
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO'])) {
  if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {
    $_SERVER['HTTPS'] = 'on';
  }
}

If you have not already done so, you also have to enable page cache, and set the external cache age for cached pages. This is just a starting point, assuming Drupal 7.x, and you need to modify these accordingly depending on your specific setup.

// Enable page caching
$conf['cache'] = 1;
// Enable block cache
$conf['block_cache'] = 1;
// Make sure that Memcache does not cache pages
$conf['cache_lifetime'] = 0;
// Enable external page caching via HTTP headers (e.g. in Varnish)
// Adjust the value for the maximum time to allow pages to stay in Varnish
$conf['page_cache_maximum_age'] = 86400;
// Page caching without bootstraping the database, nor invoking hooks
$conf['page_cache_without_database'] = TRUE;
// Nor do we invoke hooks for cached pages
$conf['page_cache_invoke_hooks'] = FALSE;

// Memcache layer
$conf['cache_backends'][]    = './sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['memcache_servers']    = array('127.0.0.1:11211' => 'default');
$conf['memcache_key_prefix'] = 'live';

And that is it for the configuration part.

You now need to clear all caches:

drush cc all

Then restart all the daemons:

service pound restart
service nginx restart # If you use nginx instead of pound
service varnish restart
service apache2 restart

Check that all daemons have indeed restarted, and that there are no errors in the logs. Then test for proper SSL recognition in the browser, and for correct redirects.

For The Extreme Minimalist: Eliminating Various Layers

The above solution stack works trouble free, and has been tested with several sites. However, there is room for eliminating different layers. For example, instead of having Apache as the backend web server, this can be replaced with Nginx itself, listening on both port 443 (SSL), and 8080 (backend), with Varnish in between. In fact, it is possible to even remove Varnish altogether, and use Ngnix FastCGI Cache instead of it. So Nginx listens on port 443, decrypts the connection, and passes the request to its own cache, which decides what is served from cache versus what gets passed through to Nginx itself on port 8080, which hands it over to PHP and Drupal.

Don't let the words 'spaghetti' and 'incest' take over your mind! Eventually, all the oddities will be ironed out, and this will be a viable solution. There are certain things that are much better known in Apache for now in regards to Drupal, like URL rewriting for clean URLs. There are also other things that are handled in .htaccess for Apache that needs to gain wider usage within the community before an Nginx only solution becomes the norm for web server plus cache plus SSL.

Apache MPM Worker Multithreaded with PHP-FPM is a very low overhead, high performance solution, and we will continue to use it until the Nginx only thing matures into a wider used solution, and has wider use and support within the Drupal community to remain viable for the near future.