IScale

#
# Readme
#
# I re-ran this benchmark and got new results in another post, which are more reasonable.
# Thanks to Willy Tarreau for pointing out the inadequacy in this benchmark.
# It’s recommended that you skimming this post and then referring to this fresh one.
#
——–

Today, I finished a benchmark to compare the caching performance and status between Varnish and Squid, which get widely focused on as reverse proxies.
Here we go.

platform

The test-network is made up of:

    * D-Link 1024R, a 24-port Gigabit Switch
    * http_server:
          o OS: Linux 2.6.21.5-smp i686 (Slackware 12.0)
          o CPU: Intel(R) Xeon(TM) CPU 2.80GHz x 2
          o MEM: 1024M x 6
          o DISK: SEAGATE ST373405LC SCSI Disk
          o Ethernet controller: Intel 82546EB PRO/1000 MT Dual Port Server Adapter
    * proxy_server:
          o OS: Linux 2.6.21.5-smp i686 (Slackware 12.0)
          o CPU:  Intel(R) Pentium(R) III CPU family 1133MHz GenuineIntel
          o MEM: 1024M
          o DISK: SEAGATE ST318406LC SCSI Disk
          o Ethernet controller:  Intel 82557/8/9 PRO/100+ Server Adapter
    * client
          o OS: Linux 2.4.31 i686 (Slackware 10.2)
          o CPU: Intel(R) Xeon(TM) CPU 2.80GHz x 2
          o MEM: 1024M x 6
          o Ethernet controller: Intel 82546EB Gigabit Ethernet Controller

file pool

#!/bin/sh
#
# Written for generating 2 sets of 100Mbyte files,
# one is 1,000 files of 100kbyte size, and
# the other is 10 files of 10MByte size.
# by Cherife Li
#

docroot=/home/wwwroot/

# generate the first set
mkdir -p $docroot/100k
cd $docroot/100k
for i in `seq 1 10`; do
mkdir -p files-$i;
for j in `seq 1 100`; do
dd if=/dev/zero of=files-$i/$j bs=100K count=1 2> /dev/null;
done;
done

# generate the second set
mkdir -p $docroot/10m
cd $docroot/10m
for i in `seq 1 5`; do
mkdir -p files-$i;
for j in `seq 1 2`; do
dd if=/dev/zero of=files-$i/$j bs=10M count=1 2> /dev/null;
done;
done

$ cd /home/wwwroot/
$ find ./100k/ | grep 'files.*/.' | sed 's#./#http://benchmark.lo/#' > 100k.urls
$ find ./10m/ | grep 'files.*/.' | sed 's#./#http://benchmark.lo/#' > 10m.urls

benchmark

 ______        ____________        ___________
|      |--A-->|            |--B-->|           |
|client|      |proxy server|      |http server|
|______|<--D--|____________|<--C--|___________|

    * client: http_load
    * proxy server: varnish 1.1.2, squid 2.6.STABLE18, and squid 3.0.STABLE2.
    * http server: nginx/0.6.28

The Nginx http server only ran the stand alone http service.
The proxy server ran the proxy service only the one been benchmarking at a time.

http_load
It’s a good load-generator as it

    * allows random fetches from a list of URLs
    * allows a large number of parallel requests
    * is portable.

There are also other test tools, check this page for detail.

+++ 100KByte +++

$ http_load -verbose -parallel 100 -fetches 100000 ./100k.urls

* Varnish 1.1.2:

— 60 secs, 6868 fetches started, 6768 completed, 100 current
— 120 secs, 13720 fetches started, 13620 completed, 100 current
— 180 secs, 20570 fetches started, 20470 completed, 100 current
— 240 secs, 27432 fetches started, 27332 completed, 100 current
— 300 secs, 34285 fetches started, 34185 completed, 100 current
— 360 secs, 41140 fetches started, 41040 completed, 100 current
— 420 secs, 47996 fetches started, 47896 completed, 100 current
— 480 secs, 54854 fetches started, 54754 completed, 100 current
— 540 secs, 61709 fetches started, 61609 completed, 100 current
— 600 secs, 68565 fetches started, 68465 completed, 100 current
— 660 secs, 75419 fetches started, 75319 completed, 100 current
— 720 secs, 82279 fetches started, 82179 completed, 100 current
— 780 secs, 89131 fetches started, 89031 completed, 100 current
— 840 secs, 95989 fetches started, 95889 completed, 100 current
100000 fetches, 100 max parallel, 1.024e+10 bytes, in 876.794 seconds
102400 mean bytes/connection
114.052 fetches/sec, 1.16789e+07 bytes/sec
msecs/connect: 116.758 mean, 9111.16 max, 0.256 min
msecs/first-response: 120.03 mean, 2494.66 max, 0.614 min
HTTP response codes:
code 200 — 100000

* Squid 2.6.STABLE18:

— 60 secs, 6898 fetches started, 6798 completed, 100 current
— 120 secs, 13748 fetches started, 13648 completed, 100 current
— 180 secs, 20595 fetches started, 20495 completed, 100 current
— 240 secs, 27439 fetches started, 27339 completed, 100 current
— 300 secs, 34287 fetches started, 34187 completed, 100 current
— 360 secs, 41136 fetches started, 41036 completed, 100 current
— 420 secs, 47983 fetches started, 47883 completed, 100 current
— 480 secs, 54829 fetches started, 54729 completed, 100 current
— 540 secs, 61675 fetches started, 61575 completed, 100 current
— 600 secs, 68523 fetches started, 68423 completed, 100 current
— 660 secs, 75371 fetches started, 75271 completed, 100 current
— 720 secs, 82221 fetches started, 82121 completed, 100 current
— 780 secs, 89065 fetches started, 88965 completed, 100 current
— 840 secs, 95909 fetches started, 95809 completed, 100 current
100000 fetches, 100 max parallel, 1.024e+10 bytes, in 878.411 seconds
102400 mean bytes/connection
113.842 fetches/sec, 1.16574e+07 bytes/sec
msecs/connect: 116.02 mean, 9114.65 max, 0.224 min
msecs/first-response: 115.596 mean, 619.381 max, 0.86 min
HTTP response codes:
code 200 — 100000

* Squid 3.0.STABLE2:

— 60 secs, 6885 fetches started, 6785 completed, 100 current
— 120 secs, 13720 fetches started, 13620 completed, 100 current
— 180 secs, 20577 fetches started, 20477 completed, 100 current
— 240 secs, 27418 fetches started, 27318 completed, 100 current
— 300 secs, 34266 fetches started, 34166 completed, 100 current
— 360 secs, 41111 fetches started, 41011 completed, 100 current
— 420 secs, 47957 fetches started, 47857 completed, 100 current
— 480 secs, 54812 fetches started, 54712 completed, 100 current
— 540 secs, 61656 fetches started, 61556 completed, 100 current
— 600 secs, 68503 fetches started, 68403 completed, 100 current
— 660 secs, 75346 fetches started, 75246 completed, 100 current
— 720 secs, 82198 fetches started, 82098 completed, 100 current
— 780 secs, 89040 fetches started, 88940 completed, 100 current
— 840 secs, 95891 fetches started, 95791 completed, 100 current
100000 fetches, 100 max parallel, 1.024e+10 bytes, in 876.658 seconds
102400 mean bytes/connection
114.07 fetches/sec, 1.16807e+07 bytes/sec
msecs/connect: 115.858 mean, 9111.16 max, 0.26 min
msecs/first-response: 116.423 mean, 3318.18 max, 29.481 min
HTTP response codes:
code 200 — 100000

+++ 10MByte +++

$ http_load -verbose -parallel 100 -fetches 1000 ./10m.urls

* Varnish 1.1.2:

— 60 secs, 100 fetches started, 0 completed, 100 current
— 120 secs, 196 fetches started, 96 completed, 100 current
— 180 secs, 231 fetches started, 131 completed, 100 current
— 240 secs, 304 fetches started, 204 completed, 100 current
— 300 secs, 389 fetches started, 289 completed, 100 current
— 360 secs, 434 fetches started, 334 completed, 100 current
— 420 secs, 509 fetches started, 409 completed, 100 current
— 480 secs, 585 fetches started, 485 completed, 100 current
— 540 secs, 637 fetches started, 537 completed, 100 current
— 600 secs, 714 fetches started, 614 completed, 100 current
— 660 secs, 786 fetches started, 686 completed, 100 current
— 720 secs, 844 fetches started, 744 completed, 100 current
— 780 secs, 915 fetches started, 815 completed, 100 current
— 840 secs, 987 fetches started, 887 completed, 100 current
1000 fetches, 100 max parallel, 1.04858e+10 bytes, in 899.718 seconds
1.04858e+07 mean bytes/connection
1.11146 fetches/sec, 1.16545e+07 bytes/sec
msecs/connect: 129.428 mean, 3128.09 max, 0.291 min
msecs/first-response: 1003.12 mean, 9619.7 max, 120.662 min
HTTP response codes:
code 200 — 1000

* Squid 2.6.STABLE18:

— 60 secs, 103 fetches started, 3 completed, 100 current
— 120 secs, 178 fetches started, 78 completed, 100 current
— 180 secs, 255 fetches started, 155 completed, 100 current
— 240 secs, 322 fetches started, 222 completed, 100 current
— 300 secs, 379 fetches started, 279 completed, 100 current
— 360 secs, 458 fetches started, 358 completed, 100 current
— 420 secs, 518 fetches started, 418 completed, 100 current
— 480 secs, 583 fetches started, 483 completed, 100 current
— 540 secs, 661 fetches started, 561 completed, 100 current
— 600 secs, 721 fetches started, 621 completed, 100 current
— 660 secs, 786 fetches started, 686 completed, 100 current
— 720 secs, 863 fetches started, 763 completed, 100 current
— 780 secs, 926 fetches started, 826 completed, 100 current
— 840 secs, 984 fetches started, 884 completed, 100 current
1000 fetches, 100 max parallel, 1.04858e+10 bytes, in 894.062 seconds
1.04858e+07 mean bytes/connection
1.11849 fetches/sec, 1.17282e+07 bytes/sec
msecs/connect: 137.644 mean, 9128.16 max, 0.283 min
msecs/first-response: 131.811 mean, 3509.99 max, 28.891 min
HTTP response codes:
code 200 — 1000

* Squid 3.0.STABLE2:

— 60 secs, 105 fetches started, 5 completed, 100 current
— 120 secs, 190 fetches started, 90 completed, 100 current
http://benchmark.lo/10m/files-2/2: timed out
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-1/1: timed out
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-5/2: timed out
http://benchmark.lo/10m/files-5/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-2/1: timed out
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-4/2: timed out
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
— 180 secs, 260 fetches started, 160 completed, 100 current
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
— 240 secs, 315 fetches started, 215 completed, 100 current
— 300 secs, 395 fetches started, 295 completed, 100 current
— 360 secs, 464 fetches started, 364 completed, 100 current
— 420 secs, 521 fetches started, 421 completed, 100 current
— 480 secs, 596 fetches started, 496 completed, 100 current
— 540 secs, 664 fetches started, 564 completed, 100 current
— 600 secs, 725 fetches started, 625 completed, 100 current
— 660 secs, 798 fetches started, 698 completed, 100 current
— 720 secs, 865 fetches started, 765 completed, 100 current
— 780 secs, 929 fetches started, 829 completed, 100 current
— 840 secs, 997 fetches started, 897 completed, 100 current
1000 fetches, 100 max parallel, 1.03922e+10 bytes, in 886.272 seconds
1.03922e+07 mean bytes/connection
1.12832 fetches/sec, 1.17258e+07 bytes/sec
msecs/connect: 130.639 mean, 3127.72 max, 0.21 min
msecs/first-response: 186.15 mean, 3578.41 max, 34.601 min
5 timeouts
53 bad byte counts
HTTP response codes:
code 200 — 1000

proxy server status

+++ 100KByte +++

100k.varnish.iostat_xm5.log
100k.varnish.vmstat5.log
100k.squid2.iostat_xm5.log
100k.squid2.vmstat5.log
100k.squid3.iostat_xm5.log
100k.squid3.vmstat5.log

+++ 10MByte +++

10m.varnish.iostat_xm5.log
10m.varnish.vmstat5.log
10m.squid2.iostat_xm5.log
10m.squid2.vmstat5.log
10m.squid3.iostat_xm5.log
10m.squid3.vmstat5.log

After finished each benchmark, the status is as follow:

conclusion

The data here, the reference in your mind.
That which is the better choice as a caching proxy, depends on your own judgement.

notes

1. Varnish can cache files immediately.
I found this while comparing the fetching of 10MB files.
Varnish just request the nginx server for the file only once, but Squid will requests them for about
10 times of varnish does.

2. Varnish provides easier access to proxy status.

3. Squid is more than a web proxy, while Varnish is more than an http accelerator (cache proxy).
The Varnish web site claims that Varnish is ten to twenty times faster than the popular Squid
cache on the same hardware.