First, install system dependencies (you need to be root):

# sudo su -
# apt-get install build-essential python-dev libxml2-dev python-pip lxml

Next, install Python dependencies (don’t install via apt-get as the versions are too old):

# pip install wsgidav
# pip install uwsgi

Next, let’s setup our directory hierarchy and a test file:

# mkdir -p /var/www/dav/server && mkdir -p /var/www/dav/data
# echo "hi there" > /var/www/dav/data/test.txt

Now we need to create our WebDAV app.  Create a file named /var/www/dav/server/dav.py:

#!/usr/bin/python
from wsgidav.version import __version__
from wsgidav.wsgidav_app import DEFAULT_CONFIG, WsgiDAVApp

config = DEFAULT_CONFIG.copy ()
config.update ({
 "provider_mapping": { "/": r"/var/www/dav/data" },
 "user_mapping": { '/': { 'cliff': { 'password': 'secret', 'description': '', 'roles': [] } } },
 "verbose": 1,
 "enable_loggers": [],
 "propsmanager": True, # True: use property_manager.PropertyManager 
 "locksmanager": True, # True: use lock_manager.LockManager 
 "domaincontroller": None, # None: domain_controller.WsgiDAVDomainController(user_mapping)
})

app = WsgiDAVApp (config)

Now, we need our uWSGI configuration file.  Create a file /var/www/dav/server/dav.ini:

[uwsgi]
socket = 127.0.0.1:3031
master = true
processes = 4

Next, we need an Nginx configuration.  Create the file /etc/nginx/sites-enabled/dav.conf:

server {
    listen 80;
    server_name _;
    access_log /var/log/nginx/dav.access.log;
    error_log /var/log/nginx/dav.error.log;
    keepalive_timeout 70;
    client_max_body_size 2G;
    client_body_buffer_size 1M;

    location / {
        include uwsgi_params;
        uwsgi_param UWSGI_SCRIPT dav:app;
        uwsgi_param SCRIPT_NAME /;
        uwsgi_pass 127.0.0.1:3031;
    }
}

Finally, restart Nginx and start uWSGI server:

# service nginx restart
# cd /var/www/dav/server ; uwsgi --ini dav.ini

Point your WebDAV client to dav://localhost/ and see your files.

In my unscientific, ad-hoc tests, I was able to copy a 400MB video (using Nautilus/GVFS) to the DAV share at a rate of about 45MB/s on my laptop.  A 1.6GB file was just as fast up until about 1.1GB of the way in, at which point I’m guessing Linux ran out of buffers and my disk started thrashing. Given that this was on localhost, my disk was almost certainly the bottleneck.  Some optimization is certainly called for as Nginx would be buffering the body to temp files, so the file gets read and written twice, all to the same disk.  Streaming that same video showed uwsgi and nginx using almost no CPU or RAM.  Overall, WsgiDAV performed admirably and I received no errors during my testing (copy, delete, rename, etc).

I have to admit that at the end of all of this, I feel that using, e.g. gunicorn to serve the WSGI app (you can use the same one as above) and Nginx in HTTP proxy mode would provide equivalent performance in this case (little HTTP parsing), and is certainly much simpler to configure and understand.

In August of 2006, Bob Ippolito replied to a discussion between myself and another person about whether to use Pound or Lighttpd as a reverse-proxy in front of TurboGears applications. I held that Pound is the correct solution as it is a proxy, whereas Lighttpd is a web server that can act as one. Further, I expressed my frustration regarding the state of Lighttpd development and the unmaintainability of its config files.

Bob offered up the following information:

One problem with Lighty is that it leaks memory like a sieve [1]. I audited it for a little bit and I gave up, it’s a mess. I’d steer clear of it, it will quickly ruin your day if you throw a lot of traffic at it.

The only solution I know of that’s extremely high performance that offers all of the features that you want is nginx [2], but its documentation is largely in Russian. I can’t read Russian, but I was able to figure it out (the configuration language isn’t Russian, neither is C source). I currently have nginx doing reverse proxy of over tens of millions of HTTP requests per day (thats a few hundred per second) on a *single server*. At peak load it uses about 15MB RAM and 10% CPU on my particular configuration (FreeBSD 6).

Under the same kind of load, apache falls over (after using 1000 or so processes and god knows how much RAM), pound falls over (too many threads, and using 400MB+ of RAM for all the thread stacks), and lighty *leaks* more than 20MB per hour (and uses more CPU, but not significantly more).

[1] http://trac.lighttpd.net/trac/ticket/758
[2] http://sysoev.ru/en/

The main thing that’s changed between then and now is that Nginx now has extensive English documentation.

You can visit the uWSGI homepage to learn more.

The uWSGI protocol is directly supported in Nginx since 0.8.4.  You can find instructions for running your application under uWSGI here.

You can download and read more about it on the ngx_supervisord  home page.

The short answer is that .htaccess files are a performance-killer and a primary goal of Nginx is speed.

The long answer requires an explanation of how Apache processes a site that uses .htaccess files.

Whenever Apache handles a request,   it first checks and processes any .htaccess file in the same directory as the file it is serving.  No surprises there.  Next Apache traverses all the parent directories of the current one.  For instance, if a visitor requests the URL /pages/about.html and your document root is /var/www/htdocs/mysite, this forces Apache to attempt to locate and process .htaccess files in all these directories:

/
/var
/var/www
/var/www/htdocs
/var/www/htdocs/mysite/
/var/www/htdocs/mysite/pages

It’s important to note that even if you don’t have any .htaccess files, Apache must still check all of these directories on every request to see if an .htaccess file has appeared since the last time it looked.  Some of this can be mitigated by careful use of the AllowOverride directive to only allow .htaccess processing in particular directories, but I doubt this is widely utilized and is only a partial solution in any case.

A second issue with .htaccess is that, while they are convenient to use, they are a rather hackish solution.  They spread configuration of a single site into a large hierarchy of files.  If you want to know what rewrite rules might affect a particular URL, you may end up looking at multiple files (including http.conf).

The final issue with .htaccess files is that they provide another attack vector that must be defended against.  Unlike httpd.conf or nginx.conf, .htaccess files live within the site’s home directory.  This means the barrier to overwriting one is much lower.  If someone can manage to overwrite a .htaccess file, the compromise can be quite extensive.

At the end of the day, .htaccess provides a convenient shortcut for the lazy admin or developer, but also kills performance, makes configuration issues more difficult to track down, and opens yet another attack vector on your site.

I believe the idea that Apache is “general purpose” and Nginx is “specialized” boils down to the following idea: Apache has hundreds of modules that enable additional functionality, whereas Nginx focuses on the core functionality of the web.  The definition of general purpose is “designed for or suitable to more than one use; broadly useful”.  It seems that Apache fits this bill quite well. But what about labeling Nginx as ”specialized”?  The definition of  specialized is “concentrating on a small area of a subject”.  I think this is where I have to disagree.  Nginx is an HTTP server.  It does not cover “a small area” of the HTTP protocol, nor does it somehow fail to provide all the features an HTTP server is expected to.  In fact, it provides quite a few features that lie outside that scope (AJP, uWSGI, etc).

To me, “general purpose” means the opposite of “specialized”.  Is Apache’s mod_svn general purpose, or is it specialized?  Obviously, it’s specialized.  Even mod_php is specialized, since it is a feature to enable a single programming language.  In the domain of an HTTP server, general purpose features would be things like serving static files, HTTP proxying, FastCGI support, etc.  So Apache’s multitude of modules don’t necessarily make it general purpose, they make it easy to specialize Apache in various ways.

Apache certainly contains a general purpose web server, but that server is functionally equivalent to Nginx. If this were the end of the story, then Apache would seem to be the clear winner, since it also provides the ability to specialize in a hundred different ways via modules.  Unfortunately, the general purpose web server that Apache contains is far less efficient than the one Nginx provides.  So this brings us to the question: if you need a general purpose web server, and don’t need to specialize, which is preferable?  A server that makes highly efficient use of resources, or one that doesn’t?  Is a HTTP server that focuses on specialization at the expense of core HTTP performance truly “general purpose”? Is a knife that features a spatula and a corkscrew, but has an inferior blade more “general purpose” than a traditional knife with a fully functional blade, or is it “specialized”?

If I need to host a Subversion repository, Apache would seem to be the best answer.  If I need to host a website, I cannot think of a single reason why I would choose Apache over Nginx.

First and foremost, they are making plenty of noise about how 2.4 puts Apache on par with Nginx in terms of speed.  First off, I’ve never thought Apache was particularly slow.  What I have thought about Apache is that it consumes a metric ton of RAM as it spawns threads like spiders on LSD and running out of RAM makes Apache slow. Now, this isn’t really Apache’s fault: you should have bought more RAM.   Anyway, it’s always possible that 2.4 is teh awesome, so I went in search of the benchmarks the Apache Foundation and its members would undoubtedly be flooding the internet with (they did do benchmarks during testing, right?). I could only find one, and it’s not terribly impressive. In fact, it rates amongst the poorest examples of benchmarking HTTP servers that I can imagine.  Here’s the issues I see right off the bat:

1) No significant “tuning” efforts (out of the box configs)

While this sounds reasonable (hey, who configures server software?), with Nginx, the default configuration ships with worker_processes 1; What this means is that no matter how many CPU cores your system has, Nginx will use one of them.  Now, I’ll grant you, changing this number to something like “2″ sounds an awful lot like tuning, but for Nginx this is part of the standard configuration, not quasi-black-art tuning options like Apache’s MaxClients, MinSpareThreads, MaxSpareThreads, ThreadsPerChild and friends.

Anyway, the slides don’t specify how many cores were available, or if they changed this value for Nginx, but assuming a modern CPU and taking the slides at face value, I’d have to come away with the conclusion that Apache was using at least four cores and Nginx only one.  Not off to a good start here, since both servers were essentially performing on par with each other.

2) Where’s the utilization graphs?

The entire benchmark consists of how fast requests are being served.  A decent job is done, as we are shown various metrics on how Apache and Nginx perform under various loads.  The problem is we aren’t shown how much CPU, memory, etc, are used by each server. This is important as it would reveal whether or not either server was pushing the system to its limits.  Given past experience and other people’s testing, I’d venture that Apache was utilizing most of the 1GB of RAM and 90% of the four cores while Nginx used only 20MB of RAM and 20% of one core.  Of course, that’s only speculation, but hey, benchmarks are supposed to clear up speculation, not create more.

3) Caveat emptor

In the section “Main Caveats”, it is noted without any sense of irony that “Apache is never resource starved”.  This is the bit that irks me the most. Like most people, I didn’t start using Nginx because of some inability to discover that 75% of the web runs on Apache.  I started using Nginx because Apache was difficult to tune in such a way that it wouldn’t fall over when resources became thin. So testing these servers to the brink of destruction is the main thing I would have liked to have seen tested.  Really.  I don’t give a damn how Apache or Nginx perform when they have unlimited resources and traffic is exactly the amount I predicted when I installed the DIMMs, I care how they perform when those resources get scarce.

When memory and CPU become scarce, any server will become slower.  Nginx will get slower, Apache will get slower.  No mysteries here.  The problem with Apache is that consumes a large amount of RAM to handle each request.  If memory is tight, it will eventually send the VM into swap.  This will further slow the system.  If this weren’t bad enough, a slower system means that each request will “live” longer, since it takes longer to process.  This means using more RAM for longer.  Unless traffic suddenly lets off, this is usually the beginning of the end for an Apache server.

Now, 2.4 might address this problem, but the apparent lack of willingness on the part of the Apache folks to even discuss it (except in vague, hand-wavy marketing talk) leaves me doubtful.

Anyway, I’m hoping to create a test setup around here real soon now to do some real benchmarks.  Stay tuned.