foobarlab

random thoughts on arts, coding and open source

Accelerating UpStage using Nginx and Varnish

Two months ago I was asked to setup a mir­ror of the UpStage server for the 11:11:11 fes­ti­val, in case some­thing goes wrong with the main server. As I have some insight into the struc­ture, code and deploy­ment of the server appli­ca­tion I agreed and set it up in a few days. Dur­ing the fes­ti­val there have been some fail­ures of the main server, so the orga­niz­ers were quite happy to be able to switch to the mir­ror and to run the fes­ti­val with­out note­wor­thy break­downs. For the mir­ror I have mod­i­fied most set­tings, scripts and the over­all deploy­ment to fit my servers needs. Addi­tion­ally I have used Nginx in com­bi­na­tion with Var­nish to improve the over­all per­for­mance. So here you will find all infor­ma­tion about how I have mod­i­fied the mir­ror server.

Default deploy­ment after fresh install

At first, let’s take a look at the default deploy­ment sce­nario, which is a stan­dard result after a fresh install of UpStage. In the con­fig­u­ra­tion files, respec­tively in the pro­vided scripts, one can indi­vid­u­ally adjust the val­ues for WEB_PORT, SWF_PORT and POLICY_FILE_PORT in upstage/config.py. These stan­dard TCP net­work ports are required to get the plat­form up and run­ning and each of them pro­vides a ser­vice with spe­cial respon­si­bil­ity. All these ports have to be allowed for incom­ing and out­go­ing traf­fic in the fire­wall on the server side. Usu­ally they do not need to be opened on the client side, except in case where you can not access them (e.g. test it on the client side by using the tel­net com­mand on each port on the server).

The orig­i­nal deploy­ment usu­ally looks like this (default ports are used for illustration):

The WEB_PORT is used by the server to deliver all HTML data, all sta­tic media and dynam­i­cally gen­er­ated text-to-speech files by using HTTP. The admin­is­tra­tive back­end is also cov­ered by this port. Also the fron­tend makes use of it for serv­ing the main swf and all media files dur­ing the ini­tial request of a stage. The stan­dard install uses port 8081.

The SWF_PORT is used by the server to exchange all real­time mes­sages with the Flash client. On the client-side a XML­Socket con­nec­tion is opened, on the server side a sim­ple low-level pro­to­col is imple­mented (using LineOn­lyRe­ceiver from Twisted). For every inter­ac­tion, like e.g. chat­ting, draw­ing or avatar move­ments, mes­sages are sent back and forth between client and server. By default, UpStage uses port 7230.

The POLICY_FILE_PORT is needed by the Flash client to get access to the above described SWF_PORT. The pol­icy file named crossdomain.xml is served on this port and holds infor­ma­tion which IP addresses and ports on the server are allowed to be accessed from a Flash client. Typ­i­cally port 843 is pro­posed, but for UpStage port 3000 is rec­om­mended. Espe­cially in net­work envi­ron­ments, where restric­tive fire­wall rules deny uncom­mon priv­iledged ports (below port num­ber 1024), the use of an unpriv­iledged port comes in handy and in effect can pre­vent some network-related prob­lems for some clients. You find more infor­ma­tion about the cross­do­main pol­icy fea­ture of Adobe Flash in the spec­i­fi­ca­tion and also in this blog entry from the Adobe devel­oper con­nec­tion.

Besides the UpStage server an addi­tional FTP server is needed when one wants to use web­cam avatars. The FTP server pro­vides access to a file direc­tory where the web­cam images can be con­stantly uploaded. Each file­name rep­re­sents an image of a web­cam avatar. The uploaded image is polled con­tin­u­ously by the server and also updated in the Flash client in short inter­vals. This way a sim­ple sim­u­la­tion of a video fea­ture can be achieved, but sadly no live audio is pos­si­ble by using this technique.

So what is the weak­ness of  the default deploy­ment scheme?

I would sum­ma­rize as fol­lows: If you have a server run­ning sev­eral domains and other server dae­mons, the default deploy­ment may have some pit­falls. For exam­ple in a multi-domain envi­ron­ment it is not pos­si­ble to restrict the ports to a spe­cific domain, as in gen­eral ports are bound to IP addresses. By default the port is avail­able on every hosted domain, some­thing usu­ally not desired unless your UpStage deploy­ment has its own exclu­sive IP address and you are not run­ning other applications.

Another hur­dle is the over­all per­for­mance loss of the server if lots of clients are try­ing to access the server at the same time: every client request trig­gers the under­ly­ing Twisted process to gen­er­ate the response. Espe­cially dur­ing the ini­tial request of the stage, when all media files are pre­loaded by the client, each request will cause Twisted to send huge amounts of binary data itself, allo­cat­ing pre­cious server resources. While many clients are con­nect­ing this has a strong impact on the total server load and might lead to net­work time­outs for some clients. Other ser­vices on the server may degrade and suf­fer from the high sys­tem load.

Another small issue I stum­bled upon is the “mar­riage” of the pol­icy file server with the UpStage instance in the startup-scripts, which does inter­fere when run­ning mul­ti­ple instances of the UpStage server. This is some­thing to fix quite easy. In gen­eral only a sin­gle instance of a pol­icy file server is needed, because port access for sev­eral UpStage instances can be con­fig­ured at once.

The mod­i­fied and accel­er­ated deployment

The dia­gram below shows the mod­i­fied deploy­ment scheme I have used on the mir­ror. It shows some sig­nif­i­cant changes regard­ing the sep­a­ra­tion of the services:

The biggest dif­fer­ence is the rerout­ing of HTTP requests to Nginx. This does not only allow to bind the requests to a domain name instead of hav­ing to use the server IP address, but also allows to for­ward the requests to Var­nish as a reverse-proxy. Var­nish itself only fetches the requests once from the UpStage server and stores the results in its cache (which can be on disk or in mem­ory). When requests can then be sat­is­fied by the cache, UpStage can idle or do more impor­tant work than serv­ing binary data.

For sure other com­bi­na­tions than Nginx and Var­nish could be used: for exam­ple Nginx as stand­alone reverse proxy, Lighttpd or Squid (e. g. see Caching Sta­tic Con­tent with Reverse Prox­ies). In effect Nginx and Var­nish are both well known as fast and resource friendly. In the end it was a gut deci­sion based on quite good prac­ti­cal expe­ri­ences with Nginx and Var­nish so far.

Another dif­fer­ence in the deploy­ment is the “divorce” of the flash pol­icy server and the UpStage server, as the flash pol­icy is being served using a stand­alone daemon.

Dur­ing eval­u­a­tion of this mod­i­fied deploy­ment I came across a seri­ous issue, which pro­hib­ited to deny access to the WEB_PORT (marked as red in the dia­gram) in the fire­wall rules. Dur­ing authen­ti­ca­tion of a back­end user (play­ers or admins) one is for­warded auto­mat­i­cally to the login screen. This relates to the auth/session han­dling of Twisted by the under­ly­ing woven guard mech­a­nism. While being sent to the login screen, the UpStage server does not allow the port to be spec­i­fied, instead the default port on which the server is run­ning locally is used (WEB_PORT). Maybe there is an alter­na­tive solu­tion apart from rewrit­ing the whole session/auth mech­a­nism, which I def­i­nitely did not want to do, but i have not found it yet. So as a remain­ing weak­ness the WEB_PORT has still to be acces­si­ble through the fire­wall, although for the audi­ence it is not rel­e­vant, as they do not have to authenticate.

Nginx con­fig­u­ra­tion details

The basic install of Nginx can be done on Debian and Ubuntu by exe­cut­ing in the terminal:

sudo apt-get install nginx

If you want a more recent ver­sion on Debian, you can install the pack­ages from Debian back­ports or com­pile from source. There is a great “how to” about con­fig­ur­ing Nginx and also about com­pil­ing from source, and it also explains the most impor­tant set­tings. Also keep an eye on the Nginx wiki for an expla­na­tion of all directives.

Below you see a sim­pli­fied basic server con­fig­u­ra­tion you can include in the main nginx.conf:

# example upstage configuration for nginx for a single upstage instance
# duplicate this for more upstage instances ... 

server {

        # we want to listen on the standard HTTP port
        listen 80;

        # we like to use a specific domain name (here: www.example.com)
        server_name www.example.com;

        # limit the number of connections to a reasonable value
        limit_conn  gulag 100;

        # just a default directory (not really used)
        root /var/www/vhost/www.example.com/htdocs;

        # host specific log & error file
        error_log /var/log/nginx/error_example.log error;
        access_log /var/log/nginx/access_example.log;

        # proxy forward all requests
        location / {

                # set max body size (POST) - 16 MB should be far sufficient
                client_max_body_size 16m;

                # change this out if you want to try it without Varnish
                #proxy_pass http://localhost:8081;	# pass directly to UpStage
                proxy_pass http://localhost:8080;	# pass to Varnish
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

If you decide not to involve Var­nish (e. g. pass directly to UpStage) you can addi­tion­ally set other loca­tion tar­gets point­ing to the file direc­to­ries con­tain­ing data to be served (like the media direc­tory of your UpStage instal­la­tion). This will also reduce dra­mat­i­cally the sys­tem load of your UpStage instance, because Nginx and not UpStage will serve the files.

Var­nish con­fig­u­ra­tion details

Var­nish is installed on Debian or Ubuntu by exe­cut­ing in the terminal:

sudo apt-get install varnish

Prefer­able on Debian you should install a more recent ver­sion from Debian back­ports. You can find more infor­ma­tion on the offi­cial Var­nish doc­u­men­ta­tion page.

To get Var­nish run­ning ensure to enable it in /etc/default/varnish and con­fig­ure the appro­pri­ate dae­mon options, for example:

# Should we start varnishd at boot?  Set to "yes" to enable.
START=yes

# Maximum number of open files (for ulimit -n)
NFILES=131072

# Maximum locked memory size (for ulimit -l)
# Used for locking the shared memory log in memory.  If you increase log size,
# you need to increase this number as well
MEMLOCK=82000

# Default varnish instance name is the local nodename.  Can be overridden with
# the -n switch, to have more instances on a single server.
INSTANCE=$(uname -n)

# Listen on port 8080, administration on localhost:8180, and forward to
# one content server selected by the vcl file, based on the request.  Use a 1GB
# fixed-size cache file.
#
DAEMON_OPTS="-a :8080 \
             -T localhost:8180 \
             -f /etc/varnish/upstage.vcl \
             -S /etc/varnish/secret \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

 

Below is a sim­ple upstage.vcl con­tain­ing all direc­tives for caching:

#-e This is a basic VCL configuration file for varnish.
# See the vcl(7) man page for details on VCL syntax and semantics.

# We define a backend for the UpStage application.
backend upstage {
	.host = "localhost";
	.port = "8081";             # this is the port UpStage is running
        # adjust the following values to your needs or comment out:
        #.max_connections = 100;
	#.connect_timeout = 1s;
	#.first_byte_timeout = 15s;
	#.between_bytes_timeout = 2s;
}

# if you have more than one instance of UpStage add
# further backends here ...

sub vcl_recv {

    # set the backend for domain www.example.com
    if (req.http.host ~ "^www.example.com$") {
       set req.backend = upstage;
    }

    # add more pointers to corresponding backends like above
    # if you have more than one UpStage instance...

    # Allow purging of cache using shift + reload
    if (req.http.Cache-Control ~ "no-cache") {
        purge_url(req.url);
    }

    # do not cache /admin/ or /swf/
    if (req.request == "GET" && req.url ~ "^/admin/" ||
        req.request == "GET" && req.url ~ "^/swf/") {
        return(pass);
    }

    # Unset any cookies and autorization data for static media
    # and fetch from cache
    if (req.request == "GET" && req.url ~ "^/media/" ||
        req.request == "GET" && req.url ~ "^/speech/") {
        unset req.http.cookie;
        unset req.http.Authorization;
        return(lookup);
    }

    # Look for static media files in the cache (images, audio, ...)
    if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js|mp3)$") {
        unset req.http.cookie;
        return(lookup);
    }

    # Do not cache any POST'ed data
    if (req.request == "POST") {
        return(pass);
    }

    # Do not cache any non-standard requests
    if (req.request != "GET" && req.request != "HEAD" &&
        req.request != "PUT" && req.request != "POST" &&
        req.request != "TRACE" && req.request != "OPTIONS" &&
        req.request != "DELETE") {
        return(pass);
    }

    return(lookup);
}

sub vcl_fetch {

    # Remove cookies and cache static content for 12 hours
    if (req.request == "GET" && req.url ~ "^/media/" ||
        req.request == "GET" && req.url ~ "^/speech/") {
        unset beresp.http.Set-Cookie;
        set beresp.ttl = 12h;
        return(deliver);
    }

    # Remove cookies and cache images for 12 hours
    if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
        unset beresp.http.Set-cookie;
        set beresp.ttl = 12h;
        return(deliver);
    }

    # Do not cache anything that does not return a value in the 200's
    if (beresp.status >= 300) {
        return(pass);
    }

    # Do not cache content which varnish has marked uncachable
    if (!beresp.cacheable) {
        return(pass);
    }

    if (beresp.http.Cache-Control ~ "max-age") {
        unset beresp.http.Set-Cookie;
        return(deliver);
    }

    return(deliver);
}

sub vcl_hash {
    set req.hash += req.url;
    if (req.http.host) {
        set req.hash += req.http.host;
    } else {
        set req.hash += server.ip;
    }

    # we do not want to always set the cookie in cache
    # (meaning this would be a per user cache)
    # see: https://www.varnish-cache.org/trac/wiki/VCLExampleCacheCookies
    #set req.hash += req.http.cookie;

    return (hash);
}

Option­ally run­ning a stand­alone flash pol­icy file daemon

To install the flash pol­icy server as stand­alone appli­ca­tion please refer to the Adobe blog entry. There you can down­load a zip-file con­tain­ing all required files.

I have put the file flashpolicyd.py into /usr/local/sbin (use chmod 755 to make it executable).

In /usr/local/etc resides the flashpolicy.xml, where you set the domain name(s) from which the Flash client requires access to the SWF_PORT:

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "/xml/dtds/cross-domain-policy.dtd">

<!-- Policy file for xmlsocket://www.example.com.
     For multiple instances of UpStage just add another port or add
     another allow-access-from directive for another domain. -->
<cross-domain-policy>

<!-- This is a master socket policy file:
     No other socket policies on the host will be permitted -->
<site-control permitted-cross-domain-policies="master-only" />

<!-- Instead of setting to-ports="*", administrator's can
     use ranges and commas.
     This will allow access to ports 123, 456, 457 and 458: -->
<!-- <allow-access-from domain="swf.example.com" to-ports="123,456-458" /> -->
<allow-access-from domain="www.example.com" to-ports="7230" />

</cross-domain-policy>

 

To have a nice start and stop func­tion­al­ity I have cre­ated a script in /etc/init.d/flashpolicyd accord­ing to Debian stan­dards (but you can also make use of the pro­vided scripts in the zip-file):

#!/bin/sh
### BEGIN INIT INFO
# Provides:          flashpolicyd
# Required-Start:    networking
# Required-Stop:     networking
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: flash policy file daemon
### END INIT INFO

#PORT=843       # usual default port
PORT=3000       # we use port 3000 for UpStage
DAEMON=/usr/local/sbin/flashpolicyd.py
POLICY=/usr/local/etc/flashpolicy.xml
LOGFILE=/var/log/flashpolicyd.log
NAME=flashpolicyd
DESC="serve crossdomain requests"

DAEMON_OPTS="--port $PORT --file $POLICY"

PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
SSD="/sbin/start-stop-daemon"

test -x $DAEMON || exit 1

set -e

 . /lib/lsb/init-functions

case "$1" in
  start)
        log_daemon_msg "Starting $DESC" $NAME
        if ! $SSD --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $DAEMON -- $DAEMON_OPTS ; then
            log_end_msg 1
        else
            log_end_msg 0
        fi
    ;;
  stop)
        log_daemon_msg "Stopping $DESC" $NAME
        if $SSD --stop --retry 30 --pidfile $PIDFILE ; then
            rm -f $PIDFILE
            log_end_msg 0
        else
            log_end_msg 1
        fi
        ;;
  restart|force-reload)
        $0 stop
        [ -r  $PIDFILE ] && while pidof -x $NAME | grep -q `cat $PIDFILE 2>/dev/null` 2>/dev/null ; do sleep 1; done
        $0 start
        ;;
  *)
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 1
        ;;
esac

exit 0

As usual you can then start, stop or restart by exe­cut­ing the ade­quate com­mand in the terminal:

/etc/init.d/flashpolicyd <command>

Because now we have a stand­alone pol­icy dae­mon we do not need to instan­ti­ate it from inside the UpStage startup script (upstage-server). You can safely com­ment in this sec­tion to pre­vent exe­cut­ing the pol­icy dae­mon mul­ti­ple times:

#    ############ Policy File Server - Startup ############
#
#    pid = os.fork()
#    if pid == 0:
#    	try:
#            # Starts the server for serving policy files
#            # (for latest version of flash player)
#            policy_server(config.POLICY_FILE_PORT, config.POLICY_FILE).run()
#        except Exception, e:
#            log.msg('Error Policy FILE SERVER!')
#            print >> sys.stderr, e
#            sys.exit(1)
#        sys.exit()
#
#    ############ Policy File Server - Startup ############

Bench­mark results

To ver­ify per­for­mance improve­ments I have com­pared the default and the accel­er­ated deploy­ment in a  benchmark:

sin­gle client, total 50.000 requests10 con­cur­rent clients, total 50.000 requests25 con­cur­rent clients, total 50.000 requests
Default
deploy­ment
(direct access to UpStage)
Small file (~ 75 kB):
668.49 req/s
1.496 ms/req
UpStage load: ~80%
Sys­tem load: ~80%

Big file (~ 1300 kB):
151.72 req/s
6.591 ms/req
UpStage load: ~80%
Sys­tem load: ~80%

Small file (~ 75 kB):
673.31 req/s
1.485 ms/req
UpStage load: ~85%
Sys­tem load: ~85%

Big file (~ 1300 kB):
105.49 req/s
9.480 ms/req
UpStage load: ~70%
Sys­tem load: ~70%

Small file (~ 75 kB):
628.08 req/s
1.592 ms/req
UpStage load: ~85%
Sys­tem load: ~85%

Big file (~ 1300 kB):
112.86 req/s
8.860 ms/req
UpStage load: ~65%
Sys­tem load: ~65%

Accel­er­ated deploy­ment
(using Nginx and Varnish)
Small file (~ 75 kB):
1518.39 req/s
0.659 ms/req
UpStage load: < 1%
Sys­tem load: ~70%

Big file (~ 1300 kB):
205.61 req/s
4.864 ms/req
UpStage load: < 1%
Sys­tem load: ~70%

Small file (~ 75 kB):
1497.19 req/s
0.668 ms/req
UpStage load: < 1%
Sys­tem load: ~65%

Big file (~ 1300 kB):
218.94 req/s
4.567 ms/req
UpStage load: <1%
Sys­tem load: ~65%

Small file (~ 75 kB):
1618.57 req/s
0.618 ms/req
UpStage load: < 1%
Sys­tem load: ~65%

Big file (~ 1300 kB):
207.26 req/s
4.825 ms/req
UpStage load: < 1%
Sys­tem load: ~60%

Over­all improve­ment
(req/s and ms/req)
Small file (~ 75 kB):
Fac­tor: 2,27

Big file (~ 1300 kB):
Fac­tor: 1,36

Small file (~ 75 kB):
Fac­tor: 2,22

Big file (~ 1300 kB):
Fac­tor: 2,08

Small file (~ 75 kB):
Fac­tor: 2,58

Big file (~ 1300 kB):
Fac­tor: 1,84

req/s = num­ber of requests per sec­ond (mean) , ms/req = time in mil­lisec­ons per request (mean, across all con­cur­rent requests)

The above table shows the bench­mark results for the default and the accel­er­ated deploy­ment. Each test was run sev­eral times so that blips are avoided. You can see there is a per­for­mance gain and the direct load on UpStage is reduced dra­mat­i­cally. I have tested it by using the Apache bench­mark tool (ApacheBench) directly on the host­ing server with the fol­low­ing parameters:

ab -c <number of clients> -n <number of requests> <url>

The url was a sin­gle link to a swf media file, because the intial pre­load­ing of the Flash client is hard to test (pre­load­ing is com­pletely done in Flash, but uses HTTP requests). The sys­tem load was observed with the htop com­mand. In effect the sys­tem load was always around 100%, as ApacheBench itself was also run­ning on the sys­tem (pro­duc­ing about 20 to 40% of sys­tem load). The given val­ues in the table on sys­tem load are cal­cu­lated by sub­tract­ing the observed load of ApacheBench from the total sys­tem load. Other dae­mons were mostly sus­pended dur­ing the test and did not use more than 1% CPU time.

In real­ity it would be bet­ter to test with mul­ti­ple clients from dif­fer­ent geo­graphic loca­tions, as the results above defin­i­tivly dif­fer from real-world-usage. For exam­ple I have observed improve­ments dur­ing the fes­ti­val of fac­tor 20–30. But this is hard to ver­ify and mea­sure. Nonethe­less, in the con­crete results above we can see an improve­ment of fac­tor 1.3 to 2.5, which means faster deliv­ery of media files and serv­ing more simul­ta­ne­ous clients at once.

Con­clu­sion

There is a lot of poten­tial to accel­er­ate a web appli­ca­tion like UpStage. The per­for­mance of a webap­pli­ca­tion heav­ily depends on the pos­si­ble num­ber of requests and pres­ence of caching mach­a­nisms. Know­ing this it seems rec­om­mend­able to inves­ti­gate all fur­ther pos­si­bil­i­ties in this area. Espe­cially because only min­i­mal to no change to the source code of the appli­ca­tion is needed. Finally the over­all per­for­mance and sta­bil­ity is some­thing, that every user obvi­ously per­ceives while using an appli­ca­tion. And this is definitly worth to be always improved.

The pro­posed accel­er­ated deploy­ment for sure is still not the opti­mal solu­tion yet. If you have any fur­ther ideas, ques­tions or cor­rec­tions, please feel free to comment.

Author: Martin

Hi! I am a developer and designer with a passion for user experience, software architecture and interdisciplinary topics. Please post a comment below or contact me for questions, ideas or suggestions. You find more information on the about page.

One Comment

  1. Recently I found a lit­tle bug in the nginx con­fig­u­ra­tion which had the effect that all logins get redi­rected to the main URL with the port num­ber (although this should be suppressed).

    There­fore I had to add a line in the nginx.conf:

    location / {
    ...
    # prevent redirection to port numbered URL after login:
    proxy_redirect http://virtual.hostname:8081/ http://virtual.hostname/;
    }

    Instead of virtual.hostname enter the domain name of your vir­tual host (con­fig­ured in the option server_name).

    In addi­tion to this change enabling gzip com­pres­sion for var­i­ous mime-types seems reasonable.