Notes on performance and resource consumption in Akara

General notes

When considering the performance of Akara, start by thinking clearly about its purpose. It's designed for "composable" data services, and not, e.g. for hosting static files. Still, just for reference, here are some useful resources.

Benchmarks of Asynchronous Servers in Python (Akara is not an asynchronous server)
The C10K problem (Which Akara does not aspire to solve)

Tips

In case of memory leak or general resource overconsumption in Akara modules, you might opt to retire processes after they've served a certain number of requests, which would also reclaim their associated resources. The config setting is:

max_requests_per_server

It defaults to 1000, so you could try setting it to 100 or 500 (note: recycling and creating a new server process is a bit expensive in itself).

Dealing with performance issues

This is often a matter for debugging. See: Akara/Debugging.

Benchmarking

See:

Benchmark reports

17NOV2009 by Uche

Run against the end-points defined here: serializer_perf.py

uche@malatesta:~/src/akara$ httperf --server localhost --port 8880 --num-calls=100 --uri=/perf-x1
httperf --client=0/1 --server=localhost --port=8880 --uri=/perf-x1 --send-buffer=4096 --recv-buffer=16384 --num-conns=1 --num-calls=100
Maximum connect burst length: 0

Total: connections 1 requests 100 replies 100 test-duration 4.657 s

Connection rate: 0.2 conn/s (4657.5 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 4657.5 avg 4657.5 max 4657.5 median 4657.5 stddev 0.0
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 100.000

Request rate: 21.5 req/s (46.6 ms/req)
Request size [B]: 69.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 6.5 transfer 40.1
Reply size [B]: header 122.0 content 397.0 footer 0.0 (total 519.0)
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.28 system 3.74 (user 6.0% system 80.3% total 86.3%)
Net I/O: 12.3 KB/s (0.1*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
uche@malatesta:~/src/akara$ 
uche@malatesta:~/src/akara$ 
uche@malatesta:~/src/akara$ httperf --server localhost --port 8880 --num-calls=100 --uri=/perf-x2
httperf --client=0/1 --server=localhost --port=8880 --uri=/perf-x2 --send-buffer=4096 --recv-buffer=16384 --num-conns=1 --num-calls=100
Maximum connect burst length: 0

Total: connections 1 requests 100 replies 100 test-duration 5.758 s

Connection rate: 0.2 conn/s (5757.9 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 5757.9 avg 5757.9 max 5757.9 median 5757.5 stddev 0.0
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 100.000

Request rate: 17.4 req/s (57.6 ms/req)
Request size [B]: 69.0

Reply rate [replies/s]: min 17.2 avg 17.2 max 17.2 stddev 0.0 (1 samples)
Reply time [ms]: response 16.6 transfer 41.0
Reply size [B]: header 123.0 content 1721.0 footer 0.0 (total 1844.0)
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.31 system 5.08 (user 5.4% system 88.2% total 93.6%)
Net I/O: 32.4 KB/s (0.3*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
uche@malatesta:~/src/akara$ 
uche@malatesta:~/src/akara$ 
uche@malatesta:~/src/akara$ httperf --server localhost --port 8880 --num-calls=100 --uri=/perf-x3
httperf --client=0/1 --server=localhost --port=8880 --uri=/perf-x3 --send-buffer=4096 --recv-buffer=16384 --num-conns=1 --num-calls=100
Maximum connect burst length: 0

Total: connections 1 requests 100 replies 100 test-duration 5.259 s

Connection rate: 0.2 conn/s (5259.1 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 5259.1 avg 5259.1 max 5259.1 median 5259.5 stddev 0.0
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 100.000

Request rate: 19.0 req/s (52.6 ms/req)
Request size [B]: 69.0

Reply rate [replies/s]: min 19.0 avg 19.0 max 19.0 stddev 0.0 (1 samples)
Reply time [ms]: response 11.8 transfer 40.8
Reply size [B]: header 123.0 content 1829.0 footer 0.0 (total 1952.0)
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.16 system 4.63 (user 3.0% system 88.0% total 91.1%)
Net I/O: 37.5 KB/s (0.3*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

For more on the machine:

uche@malatesta:~$ uname -a
Linux malatesta 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009 x86_64 GNU/Linux
uche@malatesta:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 75
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 4600+
stepping        : 2
cpu MHz         : 1000.000
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips        : 2010.20
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 75
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 4600+
stepping        : 2
cpu MHz         : 1000.000
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips        : 2010.20
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

uche@malatesta:~$ cat /proc/meminfo
MemTotal:        4058920 kB
MemFree:           26892 kB
Buffers:          364380 kB
Cached:           873516 kB
SwapCached:       426252 kB
Active:          2372832 kB
Inactive:        1331308 kB
Active(anon):    1883496 kB
Inactive(anon):   595484 kB
Active(file):     489336 kB
Inactive(file):   735824 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      11719376 kB
SwapFree:       10622116 kB
Dirty:            622248 kB
Writeback:          1840 kB
AnonPages:       2362480 kB
Mapped:            92152 kB
Slab:             165148 kB
SReclaimable:     139640 kB
SUnreclaim:        25508 kB
PageTables:        39080 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    13748836 kB
Committed_AS:    4333532 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      122408 kB
VmallocChunk:   34359613435 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       32704 kB
DirectMap2M:     4161536 kB

3/25/09 on gonzaga.ogbuji.net

Server specs:

$ uname -a
Linux gonzaga 2.6.18-4-686-bigmem #1 SMP Thu May 10 00:23:00 UTC 2007 i686 GNU/Linux
You have mail in /var/mail/uogbuji

Load check (prior to test):

$ uptime
 11:20:58 up 69 days,  3:33,  1 user,  load average: 0.47, 0.31, 0.21

AB run from MacBookPro:

ab -n 1000 -c 5 http://gonzaga.ogbuji.net:8880/scopus.discovery

Load check (after test):

$ uptime
 11:27:58 up 69 days,  3:40,  1 user,  load average: 0.18, 0.20, 0.18

AB report:

$ ab -n 1000 -c 5 http://labs.zepheira.com:
8880/scopus.discovery
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking labs.zepheira.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        WSGIServer/0.1
Server Hostname:        labs.zepheira.com
Server Port:            8880

Document Path:          /scopus.discovery
Document Length:        1911 bytes

Concurrency Level:      5
Time taken for tests:   134.236 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2079000 bytes
HTML transferred:       1911000 bytes
Requests per second:    7.45 [#/sec] (mean)
Time per request:       671.179 [ms] (mean)
Time per request:       134.236 [ms] (mean, across all concurrent requests)
Transfer rate:          15.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       41   54  34.5     48    1017
Processing:   227  616 503.0    567    5628
Waiting:      162  541 477.3    499    5553
Total:        273  670 502.8    615    5672

Percentage of the requests served within a certain time (ms)
  50%    615
  66%    622
  75%    627
  80%    631
  90%    648
  95%    683
  98%    747
  99%   5637
 100%   5672 (longest request)

Akara/Performance