A key part of Continuous Integration is being able to deploy quickly, safely and with minimal impact to production traffic. Sites use various deploy automation tools like Capistrano, Fabric and a large number of homegrown rsync-based ones. At Etsy we use a tool we built and open-sourced called Deployinator.
What all these tools have in common is that they get files onto multiple servers and are able to run commands on those servers. What ends up varying a lot is what those commands are. Do you clear your caches, graceful your web server, prime your caches, or even stagger your deploys to groups of servers at a time and remove them from your load balancer while they are being updated? There are good reasons to do all of those things, but the less you have to do, the better. It should be possible to atomically update a running server without restarting it and without clearing any caches.
The problem with deploying new code to a running server is quite simple to understand. A request that starts on one version of the code might access other files during the request and if those files are updated to a new version during the request you end up with strange side effects. For example, in PHP you might autoload files on demand when instantiating objects and if the code has changed mid-request the caller and the implementation could easily get out of synch. At Etsy, it was quite normal to have to split significant code changes over 3 deploys before implementing atomic deploys. One deploy to push the new files. A second deploy to push the changes to make use of the new code and a final deploy to clean up any outdated code that isn’t needed anymore.
The problem is simple enough, and the solution is actually quite simple as well. We just need to make sure that we can run concurrent requests on two versions of our code. While the problem statement is simple, actually making it possible to run different versions of the code concurrently isn’t necessarily easy.
When trying to address this problem in the past, I’ve made use of PHP’s realpath cache and APC (a PHP opcode cache, which uses inodes as keys). During a deploy the realpath cache retains the inodes from the previous version, and the opcode cache would retain the actual code from the previous version. This means that requests that are currently in progress during a deploy can continue to use the previous version’s code as they finish. WePloy is an implementation of this approach which works quite well.
With PHP 5.5 there is a new opcode cache called Opcache (which is also available for PHP 5.2-5.4). This cache is not inode-based. It uses realpaths as the cache keys, so the inode trick isn’t going to work anymore. Relying on getting the inodes from a cache also isn’t a terribly robust way of handling the problem because there are still a couple of tiny race windows related to new processes with empty caches starting up at exactly the wrong time. It is also too PHP-oriented in that it relies on very specific PHP behaviour.
Instead of relying on PHP-specific caching, we took a new look at this problem and decided to push the main responsibility to the web server itself. The base characteristic of any atomic deploy mechanism is that existing requests need to continue executing as if nothing has changed. The new code should only be visible to new requests. In order to accomplish this in a generic manner we need two document roots that we toggle between and a new request needs to know which docroot it should use. We wrote a simple Apache module that calls realpath() on the configured document root. This allows us to make the document root a symlink which we can toggle between two directories. The Apache module sets the document root to this resolved path for the request so even if a deploy happens in the middle of the request and the symlink is changed to point at another directory the current request will not be affected. This avoids any tricky signaling or IPC mechanisms someone might otherwise use to inform the web server that it should switch its document root. Such mechanisms are also not request-aware so a server with multiple virtual hosts would really complicate such methods. By simply communicating the docroot change to Apache via a symlink swap we simplify this and also fit right into how existing deploy tools tend to work.
We called this new Apache module mod_realdoc.
If you look at the code closely you will see that we are hooking into Apache first thing in the post_read_request hook. This is run as soon as Apache is finished reading the request from the client. So, from this point on in the request, the document root will be set to the target of the symlink and not the symlink itself. Another thing you will notice is that the result of the realpath() is cached. You can control the stat frequency with the RealpathEvery Apache configuration directive. We have it set to 2s here.
Note that since we have two separate docroots and our opcode cache is realpath-based, we have to have enough space for two complete copies of our site in the cache. By having two docroots and alternating between them on successive deploys we reuse entries that haven’t changed across two deploys and avoid “thundering herd” cache issues on normal deploys.
If you understand things so far and have managed to compile and install mod_realdoc you should be able to simply deploy to a second directory and when the directory is fully populated just flip the docroot symlink to point to it. Don’t forget to flip the symlink atomically by creating a temporary one and renaming it with “mv -T“. Your deploys will now be atomic for simple PHP, CGI, static files and any other technology that makes use of the docroot as provided by Apache.
However, you will likely have a bit more work to do for more complex scenarios. You need to make sure that nothing during your request uses the absolute path to the document_root symlink. For example, if you configure Apache’s DOCUMENT_ROOT for your site to be /var/www/site/htdocs and then you have /var/www/site be a symlink to alternatingly /var/www/A and /var/www/B you need to check your code for any hardcoded instances of /var/www/site/htdocs. This includes your PHP include_path setting. One way of doing this is to set your include_path as the very first thing you do if you have a front controller in your application. You can use something like this:
ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include');
That means once mod_realdoc has resolved /var/www/site/htdocs to /var/www/A/htdocs your include_path will be /var/www/A/htdocs/../include for the remainder of this PHP request and even if the symlink is switched to /var/www/B halfway through the request it won’t be visible to this request.
At Etsy we don’t actually have a front controller where we could easily make this app-level ini_set() call, so we wrote a little PHP extension to do it for us. It is called incpath.
This extension is quite simple. It has three ini settings. incpath.docroot_sapi_list specifies which SAPIs should get the docroot from the SAPI itself. incpath.realpath_sapi_list lists the SAPIs which should do the realpath() call natively. When the extension does the realpath() itself it is essentially a PHP version of the mod_realpath module resolving the symlink in the extension itself. And finally, incpath.search_replace_pattern specifies the string to replace in the existing include_path. It is easier to understand with an example. At Etsy we have it configured something like this:
incpath.docroot_sapi_list = apache2handler
incpath.realpath_sapi_list = cli
incpath.search_replace_pattern = /var/www/site/htdocs
This means that when running PHP under Apache we will get the document root from Apache (apache2handler) and we will look for “/var/www/site/htdocs” in the include_path and replace it with the document root we got from Apache. For cli we will do the realpath() in the extension and use that to substitute into the include_path. Our PHP configuration then has the include_path set to:
/var/www/site/htdocs/../include:.
which the incpath extension will modify to be either /var/www/A/htdocs/../include or /var/www/B/htdocs/../include.
This include_path substitution is done in the RINIT PHP request hook which runs at the beginning of every request before any PHP code has run. The original include_path is restored at the end of the request in the RSHUTDOWN PHP hook. You can, of course, specify different search_replace_pattern values for different virtual hosts and everything should work fine. You can also skip this extension entirely and do it at the app-level or even through PHP’s auto_prepend functionality.
Some caveats. First and foremost this is about getting atomicity for a single web request. This will not address multi-request atomicity issues. For example, if you have a request that triggers AJAX requests back to the server, the initial request and the AJAX request may be handled by different versions of the code. It also doesn’t address changes to shared resources. If you change your database schema in some incompatible way such that the current and the new version of the code cannot run concurrently then this won’t help you. Any shared resources need to stay compatible across your deploy versions. For static assets this means you need proper asset versioning to guarantee that you aren’t accessing incompatible js/css/images.
If you are currently using Apache and a symlink-swapping deploy tool like Capistrano, then mod_realdoc should come in handy for you, and it is likely to let you remove a graceful restart from your deploy procedure. For non-Apache, like nginx, it shouldn’t be all that tricky to write a similar plugin which does the realpath() call and fixates the document root at the top of a request insulating that request from a mid-request symlink change. If you use this Apache module or write your own for another server, please let us know in the comments.
Log in
