What to do and how to do, this was a question to me at the beginning. I have no idea about isolinux. Fortunately I found that there is an answer called Cooking Up Some Slack(CUSS). After reading over that, I got to know something and began the process.

1 Prepare the packages
I pulled the -Current repository from rsync://rsync.osuosl.org/slackware/slackware-current to my workstation. Some directories and files are not necessary for the installation, to save bandwidth and time, it is sensible to –exclude them, like source/, pasture/, testing/, and so on.

2 Customize initrd.img
The initrd.img contains the system and scripts for installation. Most things will be customized here.

First, uncompress the initrd.img file to a directory. The initrd.img in -Current now is a gzipped cpio archive, rather than a gzipped image as in former versions(before and include 11.0).

$ gzip -cd initrd.img|cpio -i

Most of the installation scripts are stored in usr/lib/setup directory.

Second, to reduce the size of initrd.img, I deleted some directories, e.g., lib/modules/’kernel-version’(retain the -smp directory, I just use -smp to boot), usr/lib/setup(you need installpkg script at least), usr/man, and some files, e.g., scripts related to LVM in sbin/ directory(I don’t use LVM), odd files in etc/ directory, and so on.

Third, customize the environment and script during installation. The files related are etc/profile, etc/rc.d/rc.S, and also etc/inittab, etc/issue and so on, if you want.

This is some pieces of my setup script:

#!/bin/sh
#
# Copyright (c) 2008, Cherife Li .
# All rights reserved.
#
# Redistribution and use of this script, with or without modification,
# is permitted, and must retain the copyright above.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR “AS IS” AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# Notice:
# - This script is used for one logical disk(not LVM).
# - The disk will be formatted to reiserfs file system.
# - I don’t care about the keyboard map setting.
#
# Bugs, advise, suggestion, or others, if any, please mail me.
#

TMP=/var/log/setup/tmp
SRC=/var/log/setup/src
DEST=/mnt

KERNEL=”`cut -f 5 -d ‘ ‘ /proc/cmdline|cut -f 2 -d =`”
KNAME=”huge-smp”
KVERSION=”`uname -r`”

if [ -d $TMP ]; then
rm -rf $TMP/* 1>/dev/null 2>&1
else
mkdir $TMP
fi

for i in $SRC $DEST; do
if [ -d $i ]; then
umount -f $i 1>/dev/null 2>&1
else
mkdir -p $i 1>/dev/null 2>&1
fi
done

echo
echo “+==============================================+”
echo ” Preparing the hard disk for installation …”
echo
sleep 3

DISK=`fdisk -l|grep ‘^Disk\ \/’|cut -f 2 -d ‘ ‘|cut -f 1 -d :|head -n1`

# clear existing partition(s), if any
PARTNUM=”`fdisk $DISK -l|tail -n1|cut -f 1 -d ‘ ‘|cut -c 9`”
while [ $PARTNUM -gt 0 ]; do
if [ $PARTNUM -gt 1 ]; then
echo -ne “d\n$PARTNUM\n” >> $TMP/fdisk.del
else
echo -ne “d\nw\n” >> $TMP/fdisk.del
fi
let PARTNUM=PARTNUM-1
done
cat $TMP/fdisk.del|fdisk $DISK

# Partition the hard disk into two partitions.
# one for swap with 2000M, the other for root(/) with the left of disk.
echo -ne “n\np\n1\n\n+2000M\n\nn\np\n2\n\n\nt\n1\n82\nw\n”|fdisk $DISK

# Active swap partition and add it to fstab temp
SWAP_PART=${DISK}1
mkswap -v1 $SWAP_PART
swapon $SWAP_PART
printf “%-16s %-16s %-11s %-16s %-3s %s\n” “$SWAP_PART” “swap” “swap” “defaults” “0″ “0″ >> $TMP/fstab.tmp

# Format root (/) partition, mount it, and add it to fstab temp
ROOT_PART=${DISK}2
echo “y”|mkreiserfs $ROOT_PART 1>/dev/null 2>&1
mount $ROOT_PART $DEST || exit 1
printf “%-16s %-16s %-11s %-16s %-3s %s\n” “$ROOT_PART” “/” “reiserfs” “defaults,noatime” “1″ “1″ >> $TMP/fstab.tmp

echo
echo ” The hard disk is ready for installation.”
echo “+=============================================+”
echo
#sleep 3

echo
echo “+=============================================+”
echo ” Processing the package installation …”
echo
sleep 3

# Mount the Slackware source
for MEDIA in \
/dev/hdc /dev/hdd /dev/hda /dev/hdb \
/dev/hde /dev/hdf /dev/hdg /dev/hdh \
/dev/hdi /dev/hdj /dev/hdk /dev/hdl \
/dev/sr0 /dev/sr1 /dev/sr2 /dev/sr3 \
/dev/pcd0 /dev/pcd1 /dev/pcd2 /dev/pcd3;
do
mount -t iso9660 -o ro $MEDIA $SRC 1>/dev/null 2>&1
if [ $? = 0 ]; then
break
fi
done

if [ "`mount|grep "$SRC"`" = "" ]; then
echo
echo “Couldn’t found the installation source!”
echo “Abort …”
echo
exit 1
else
# Process the installation
echo
echo “Found the installation source.”
echo “Now going to install the packages …”
echo
sleep 3
for series in a ap d l n; # you may do some change here
do
installpkg -root $DEST -priority ADD $SRC/slackware/$series/*.tgz
done

echo
echo ” All packages have been installed.”
echo “+=============================================+”
echo
sleep 3
fi

# config system
echo
echo “+==============================================+”
echo ” Configuring the system, wait for a while …”
echo

# Permission issue
chmod 755 $DEST $DEST/var/spool/mail 1>dev/null 2>&1

# Kernel setting
(cd $DEST/boot
rm -rf vmlinuz config System.map 1>/dev/null 2>&1
cp -a $SRC/kernels/$KERNEL/System.map.gz ./System.map-$KNAME-$KVERSION.gz
cp -a $SRC/kernels/$KERNEL/bzImage ./vmlinuz-$KNAME-$KVERSION
cp -a $SRC/kernels/$KERNEL/config ./config-$KNAME-$KVERSION
gzip -df System.map-$KNAME-$KVERSION.gz
ln -sf vmlinuz-$KNAME-$KVERSION vmlinuz
ln -sf config-$KNAME-$KVERSION config
ln -sf System.map-$KNAME-$KVERSION System.map
)

# Boot loader
cat > $DEST/etc/lilo.conf << EOF
boot = $DISK
timeout = 0
change-rules
reset
vga = normal
image = /boot/vmlinuz
root = $ROOT_PART
label = default
read-only
EOF
lilo -r $DEST -C /etc/lilo.conf

# Network
for nicm in \
e1000 tg3 eepro100 epic100 8139too 8139cp pcnet32 hp100 ne2k-pci olympic rcpci 3c59x acenic de4x5 dgrs sktr skge tulip via-rhine yellowfin dl2k ns83820;
do
modprobe $nicm 2> /dev/null
grep -q eth /proc/net/dev
if [ $? = 0 ]; then
echo “/sbin/modprobe $nicm” > $DEST/etc/rc.d/rc.netdevice
chmod 755 $DEST/etc/rc.d/rc.netdevice
break
else
rmmod $nicm 2> /dev/null
fi
done

if [ ! -f $DEST/etc/rc.d/rc.netdevice ]; then
echo
echo “Warning: No network interface card found.”
echo “Maybe, you need to check the card(s) or the modules.”
echo
fi

[... Network setups, e.g., $DEST/etc/rc.d/rc.inet1.conf, $DEST/etc/networks, \
$DEST/etc/hosts, $DEST/etc/HOSTNAME, $DEST/etc/resolv.conf, here \
snip...]

# Time config
cat $DEST/usr/share/zoneinfo/Asia/Shanghai > $DEST/etc/localtime
cat > $DEST/etc/hardwareclocl << EOF
localtime
EOF

# The /etc/fstab file
cat $TMP/fstab.tmp > $DEST/etc/fstab
printf “%-16s %-16s %-11s %-16s %-3s %s\n” “#/dev/cdrom” “/mnt/cdrom” “auto” “noauto,owner,ro” “0″ “0″ >> $DEST/etc/fstab
printf “%-16s %-16s %-11s %-16s %-3s %s\n” “#/dev/fd0″ “/mnt/floppy” “auto” “noauto,owner” “0″ “0″ >> $DEST/etc/fstab
printf “%-16s %-16s %-11s %-16s %-3s %s\n” “devpts” “/dev/pts” “devpts” “gid=5,mode=620″ “0″ “0″ >> $DEST/etc/fstab
printf “%-16s %-16s %-11s %-16s %-3s %s\n” “proc” “/proc” “proc” “defaults” “0″ “0″ >> $DEST/etc/fstab
printf “%-16s %-16s %-11s %-16s %-3s %s\n” “tmpfs” “/dev/shm” “tmpfs” “defaults” “0″ “0″ >> $DEST/etc/fstab

# Misc
cat > $DEST/etc/sysctl.conf << EOF
[... snip ...]
EOF

rm -f $DEST/var/spool/mail/root
sed -i '/^c[2-6].*agetty/s/^/#/' $DEST/etc/inittab
sed -i '/^\/sbin\/modprobe.*[parport_pc|lp|agpgart]/s/^/#/' $DEST/etc/rc.d/rc.modules
echo "blacklist ipv6" >> $DEST/etc/modprobe.d/blacklist
chroot $DEST /sbin/ldconfig
chroot $DEST /usr/bin/passwd root

# All finished
echo
echo “+=========================================================+”
echo ” Wooow, all done.”
echo ” Now you may reboot with CTRL-ALT-DELETE, enjoy. ;-)”
echo “+=========================================================+”
echo
cat /etc/wel.come

Last, rebuild the initrd.img.

$ find . | cpio -o -H newc | gzip -9 > /PATH/TO/isolinux/initrd.img

3 Customize the kernel
Re-build the kernel with your own config, then copy System.map.gz, bzImage, config, to the kernels/huge-smp directory, or an new directory in kernels/. I removed the ‘kernel-version’ subdirectory of kernels/, as I just use the -smp kernel.
Then modify isolinux/isolinux.cfg to fit the changes.

4 Specify the packages to install
In the slackware/ directory, I just put the packages needed here, and removed the unneeded packages. (The setup script will install all the packages in the series directories.) It is possible to use tagfile to install packages. But I just want a small ISO to build the installation CD/DVD. Which method to use depends on the need. The former is preferred to my situation. If using the later, slackbasics.org has an intro to the tagfiles.
I also added some packages built by myself. :-)

5 Build the ISO image
At the top directory of the ISO source,
$ mkisofs -o /tmp/slackware-install-cd.iso -R -J -V "Slackware Install CD" -hide-rr-moved -v -d -N -no-emul-boot -boot-load-size 4 -boot-info-table -sort isolinux/iso.sort -b isolinux/isolinux.bin -c isolinux/isolinux.boot -A Slack-CD .(do not forget this dot)
then this ISO image could be used to burn an install CD/DVD.

That’s all.

Notes:

Today Willy Tarreau gave me some advise. So I took a short time re-ran this benchmark. Here follows the detail.

platform
Everything kept unchanged except for the proxy_server, I replaced it with this new one:

    o OS: Linux 2.6.21.5-smp i686 (Slackware 12.0)
    o CPU:  Intel(R) Xeon(TM) CPU 3.06GHz x 2
    o MEM: 1024M x 6
    o DISK: RAID 5
    o Ethernet controller:  Intel Corporation 82546EB Gigabit Ethernet Controller

file pool
This time, I generated 2 sets of 10Mbyte files, one is 1,000 files of 10kbyte size, and the other is 10 files of 1MByte size.

benchmark

    * client: http_load
    * proxy server: varnish 1.1.2, squid 2.6.STABLE18, and squid 3.0.STABLE2.
    * http server: nginx/0.6.28

Using the same configurations of nginx, varnish, and squid 2/3, got the results below:

+++ 10KByte +++

$ http_load -verbose -parallel 100 -fetches 100000 ./10k.urls

100000 fetches, 100 max parallel, 1.024e+09 bytes, in 14.7505 seconds
10240 mean bytes/connection
6779.43 fetches/sec, 6.94213e+07 bytes/sec
msecs/connect: 0.400918 mean, 11.452 max, 0.067 min
msecs/first-response: 14.0161 mean, 1779.32 max, 0.24 min
HTTP response codes:
code 200 — 100000

* Squid 2.6.STABLE18:

100000 fetches, 100 max parallel, 1.024e+09 bytes, in 26.1771 seconds
10240 mean bytes/connection
3820.13 fetches/sec, 3.91181e+07 bytes/sec
msecs/connect: 0.497665 mean, 2990.79 max, 0.055 min
msecs/first-response: 21.0663 mean, 3018.84 max, 4.071 min
HTTP response codes:
code 200 — 100000

* Squid 3.0.STABLE2:

— 60.0027 secs, 100000 fetches started, 96249 completed, 0 current
100000 fetches, 100 max parallel, 9.85651e+08 bytes, in 102.375 seconds
9856.51 mean bytes/connection
976.8 fetches/sec, 9.62785e+06 bytes/sec
msecs/connect: 2.56114 mean, 91.428 max, 0.061 min
msecs/first-response: 27.8048 mean, 94.563 max, 1.288 min
3751 timeouts
3745 bad byte counts
HTTP response codes:
code 200 — 96255

+++ 1MByte +++

$ http_load -verbose -parallel 100 -fetches 1000 ./1m.urls

* Varnish 1.1.2:

— 60 secs, 6640 fetches started, 6540 completed, 100 current
10000 fetches, 100 max parallel, 1.04858e+10 bytes, in 91.1719 seconds
1.04858e+06 mean bytes/connection
109.683 fetches/sec, 1.15011e+08 bytes/sec
msecs/connect: 36.9187 mean, 9019.28 max, 0.08 min
msecs/first-response: 26.7986 mean, 475.462 max, 18.781 min
HTTP response codes:
code 200 — 10000

* Squid 2.6.STABLE18:

— 60 secs, 5856 fetches started, 5756 completed, 100 current
10000 fetches, 100 max parallel, 1.04858e+10 bytes, in 103.829 seconds
1.04858e+06 mean bytes/connection
96.3126 fetches/sec, 1.00991e+08 bytes/sec
msecs/connect: 2.4862 mean, 2994.65 max, 0.063 min
msecs/first-response: 15.9743 mean, 134.817 max, 8.222 min
HTTP response codes:
code 200 — 10000

* Squid 3.0.STABLE2:

— 60 secs, 6083 fetches started, 5983 completed, 100 current
10000 fetches, 100 max parallel, 1.04858e+10 bytes, in 103.054 seconds
1.04858e+06 mean bytes/connection
97.0367 fetches/sec, 1.0175e+08 bytes/sec
msecs/connect: 6.6513 mean, 3022.23 max, 0.089 min
msecs/first-response: 16.0642 mean, 787.308 max, 0.741 min
HTTP response codes:
code 200 — 10000

proxy server system status

+++ 10KByte +++

10k-varnish-iostat_xm5
10k-squid2-iostat_xm5
10k-squid3-iostat_xm5
10k-varnish-vmstat5
10k-squid2-vmstat5
10k-squid3-vmstat5

+++ 1MByte +++

1m-varnish-iostat_xm5
1m-squid2-iostat_xm5
1m-squid3-iostat_xm5
1m-varnish-vmstat5
1m-squid2-vmstat5
1m-squid3-vmstat5

proxy status

)-: Hmm, I’m still not satisfied with these results, especially the something about squid3.
Maybe I need to optimize the configuration of the proxies?

I just use the DNS forward(cache) function here, which only serves with the systems of internal network, as the configuration file of the dnsmasq server below:

$ cat /etc/dnsmasq.conf

no-hosts
neg-ttl=3600
#log-queries
log-facility=/var/log/dnsmasq.log
pid-file=/var/run/dnsmasq.pid
user=nobody
group=nogroup
port=53
edns-packet-max=1280
interface=eth0 # internal network
except-interface=eth1 # external network
#no-dhcp-interface=eth1
listen-address=192.168.0.10
bind-interfaces
resolv-file=/etc/dnsmasq.resolv.conf
strict-order
#all-servers
stop-dns-rebind
#no-poll
cache-size=2048
no-negcache

$ cat /etc/resolv.conf

nameserver 127.0.0.1

On another system:

$ cat /etc/resolv.conf

nameserver 192.168.0.10
nameserver 192.168.0.11

I think it’s a good choice to effect fault tolerance with setting up two dnsmasq servers. As the two lines above.

the first time:

$ dig dotimes.com

;; Query time: 258 msec
;; SERVER: 192.168.0.10#53(192.168.0.10)

the second time:

$ dig dotimes.com

;; Query time: 1 msec
;; SERVER: 192.168.0.10#53(192.168.0.10)

As you can see the difference in the above example.
Yes, this is only one request, but what about surfing, or sending mails on mail server? It’s really worth the effort.

djbdns, pdnsd are also good DNS cache (proxy) programs, which yep worth a try.
The only thing I’m wondering that whether djbdns is still under maintainence.

I really got impressed with its stability, scalability and high performance. It doesn’t cost any system resource except memory usage, which depends on the connection number. And yet it’s very low-cost, for each connection entry consumes about 128bytes or so. It’s easy for you to figure it out that 100,000 unique connections/s, only 12.8M memory will be used. On the other hand, as a 4th-layer implement, IPVS just direct packages.

So, a question raised, what if I need layer-7 switching?

Yeah, IPVS no longer sucks. Here goes HAProxy, the reliable, high performance TCP/HTTP load balancer. Although there are other solutions, e.g., F5, pound, XLB, pen, and so on, I would like to chose HAProxy and introduce it here.

HAProxy is written by Willy TARREAU, the new Linux Kernel 2.4 maintainer since August 2006, who also maintains patches against the 2.6.20 branch since August of 2007.

Today I glanced my eye over the HAProxy Architecture Guide. I’d like to take a note here.

This note is so annoyed that you would better go for a drink, then YouTube. Yeah, I’m serious. :-) Otherwise, it’s better to check the link in the line above for original detail if you are interested in the architecture. Okay, here I go, see you…

1. Simple HTTP load-balancing with cookie insertion

  192.168.1.1    192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----------+-----+-----+-----+--------+----
        |           |     |     |     |       _|_db
     +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | A | | B | | C | | D |    (___)
     +-----+      +---+ +---+ +---+ +---+    (___)
     haproxy        4 cheap web servers

Flows :

(client)                           (haproxy)                         (server A)
  >-- GET /URI1 HTTP/1.0 ------------> |
               ( no cookie, haproxy forwards in load-balancing mode. )
                                       | >-- GET /URI1 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
               ( the proxy now adds the server cookie in return )
  <-- HTTP/1.0 200 OK ---------------< |
      Set-Cookie: SERVERID=A           |
  >-- GET /URI2 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
      ( the proxy sees the cookie. it forwards to server A and deletes it )
                                       | >-- GET /URI2 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
   ( the proxy does not add the cookie in return because the client knows it )
  <-- HTTP/1.0 200 OK ---------------< |
  >-- GET /URI3 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
                                    ( ... )

If clients use keep-alive (HTTP/1.1), only the first response will have a cookie inserted, and only the first request of each session will be analyzed. The added server cookie will not be removed from the requests forwarded to the servers, so the server must not be sensitive to unknown cookies. If this causes trouble, keep-alive can be disabled by adding the following option :

option httpclose

If for some reason the clients cannot learn more than one cookie, and the application already produces a cookie, “prefix” mode could be used(see below).

Backing LB1 up using keepalived for healthcheck & failover(see below)

If the application needs to log the original client’s IP, use the “forwardfor” option which will add an “X-Forwarded-For” header with the original client’s IP address. Also use “httpclose” to rewrite every requests and not only the first one of each session :

option httpclose
option forwardfor

The web server will have to be configured to use this header instead. For example, on apache:

LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b ” combined
CustomLog /var/log/httpd/access_log combined

In the situation of clients disable cookies on their browser, use the “source” balancing algorithm instead of the “roundrobin”. So a given IP address will always reaches the same server. (as long as the number of servers remains unchanged.) Never use this behind a proxy or in a small network, because the distribution will be unfair. However, in large internal networks, and on the internet, it works quite well. Clients
which have a dynamic address will not be affected as long as they accept the cookie, because the cookie always has precedence over load balancing.

2. HTTP load-balancing with cookie prefixing and high availability

Backed load-balancer up with a second one in VRRP mode using keepalived.

http://www.keepalived.org/

Allow the proxy to bind to the shared IP:

# echo 1 >/proc/sys/net/ipv4/ip_nonlocal_bind

    shared IP=192.168.1.1
  192.168.1.3  192.168.1.4    192.168.1.11-192.168.1.14   192.168.1.2
 -------+------------+-----------+-----+-----+-----+--------+----
        |            |           |     |     |     |       _|_db
     +--+--+      +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | LB2 |      | A | | B | | C | | D |    (___)
     +-----+      +-----+      +---+ +---+ +---+ +---+    (___)
     haproxy      haproxy        4 cheap web servers
     keepalived   keepalived

Set the “httpclose” option to disable keep-alive (HTTP/1.1), so the proxy can access to all cookies in all requests for each session, because the proxy will modify EVERY cookie sent by the client and the server.

Flows :

(client)                           (haproxy)                         (server A)
  >-- GET /URI1 HTTP/1.0 ------------> |
               ( no cookie, haproxy forwards in load-balancing mode. )
                                       | >-- GET /URI1 HTTP/1.0 ---------->
                                       |     X-Forwarded-For: 10.1.2.3
                                       | <-- HTTP/1.0 200 OK -------------<
                        ( no cookie, nothing changed )
  <-- HTTP/1.0 200 OK ---------------< |
  >-- GET /URI2 HTTP/1.0 ------------> |
    ( no cookie, haproxy forwards in lb mode, possibly to another server. )
                                       | >-- GET /URI2 HTTP/1.0 ---------->
                                       |     X-Forwarded-For: 10.1.2.3
                                       | <-- HTTP/1.0 200 OK -------------<
                                       |     Set-Cookie: JSESSIONID=123
    ( the cookie is identified, it will be prefixed with the server name )
  <-- HTTP/1.0 200 OK ---------------< |
      Set-Cookie: JSESSIONID=A~123     |
  >-- GET /URI3 HTTP/1.0 ------------> |
      Cookie: JSESSIONID=A~123         |
       ( the proxy sees the cookie, removes the server name and forwards
          to server A which sees the same cookie as it previously sent )
                                       | >-- GET /URI3 HTTP/1.0 ---------->
                                       |     Cookie: JSESSIONID=123
                                       |     X-Forwarded-For: 10.1.2.3
                                       | <-- HTTP/1.0 200 OK -------------<
                        ( no cookie, nothing changed )
  <-- HTTP/1.0 200 OK ---------------< |
                                    ( ... )

Setting “weight”(values between 1 and 256) to inform haproxy to spread the load of backends the most smoothly possible respecting those ratios:

server webA 192.168.1.11:80 cookie A weight 12 check
server webC 192.168.1.13:80 cookie C weight 26 check

2.1 Variations involving external layer 4 load-balancers

Can the haproxies be load-balanced?
Yeah, by a layer4 load-balancer (eg: Alteon) which will check the services:

              | VIP=192.168.1.1
         +----+----+
         | Alteon  |
         +----+----+
              |
 192.168.1.3  |  192.168.1.4  192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----+------+-----------+-----+-----+-----+--------+----
        |            |           |     |     |     |       _|_db
     +--+--+      +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | LB2 |      | A | | B | | C | | D |    (___)
     +-----+      +-----+      +---+ +---+ +---+ +---+    (___)
     haproxy      haproxy        4 cheap web servers

What if the Alteon fails? Want relay generic TCP protocols (SMTP, TSE, VNC, etc…)? (see below)

2.2 Generic TCP relaying and external layer 4 load-balancers

Using the “monitor-net” keyword to specify a network which will be dedicated to monitoring systems and must not lead to a forwarding connection nor to any log. This expects a version of haproxy greater than or equal to 1.1.32 or 1.2.6.

                |  VIP=172.16.1.1   |
           +----+----+         +----+----+
           | Alteon1 |         | Alteon2 |
           +----+----+         +----+----+
 192.168.1.252  |  GW=192.168.1.254 |  192.168.1.253
                |                   |
          ------+---+------------+--+-----------------> TSE farm : 192.168.1.10
       192.168.1.1  |            | 192.168.1.2
                 +--+--+      +--+--+
                 | LB1 |      | LB2 |
                 +-----+      +-----+
                 haproxy      haproxy

3. Simple HTTP/HTTPS load-balancing with cookie insertion

  192.168.1.1    192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----------+-----+-----+-----+--------+----
        |           |     |     |     |       _|_db
     +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | A | | B | | C | | D |    (___)
     +-----+      +---+ +---+ +---+ +---+    (___)
     apache         4 cheap web servers
     mod_ssl
     haproxy

Do not cache inserted cookies for security measures.

If the cookie works in “prefix” mode, there is no need to add the “nocache” option because it is an application cookie which will be modified, and the application flags will be preserved.

If apache 1.3 is used as a front-end before haproxy, it always disables HTTP keep-alive on the back-end, so the “httpclose” is needn’t.

To log client’s IP, configure apache to set the X-Forwarded-For header not on haproxy.

Flows :

(apache)                           (haproxy)                         (server A)
  >-- GET /URI1 HTTP/1.0 ------------> |
               ( no cookie, haproxy forwards in load-balancing mode. )
                                       | >-- GET /URI1 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
               ( the proxy now adds the server cookie in return )
  <-- HTTP/1.0 200 OK ---------------< |
      Set-Cookie: SERVERID=A           |
      Cache-Control: private           |
  >-- GET /URI2 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
      ( the proxy sees the cookie. it forwards to server A and deletes it )
                                       | >-- GET /URI2 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
   ( the proxy does not add the cookie in return because the client knows it )
  <-- HTTP/1.0 200 OK ---------------< |
  >-- GET /URI3 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
                                    ( ... )

What if only SSL is required and cache is not needed? (see below)

3.1. Alternate solution using Stunnel

Stunnel is a cheaper solution than Apache+mod_ssl. It doesn’t process HTTP or add X-Forwarded-For header by default. (there is a patch on the official haproxy site to provide this feature to recent stunnel versions.)

Stunnel will only process HTTPS. Haproxy will get all HTTP traffic, so add the X-Forwarded-For header for HTTP traffic in haproxy, but not for HTTPS traffic since stunnel will already have done it.

Use the “except” keyword to tell haproxy that connections from local host already have a valid header.

  192.168.1.1    192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----------+-----+-----+-----+--------+----
        |           |     |     |     |       _|_db
     +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | A | | B | | C | | D |    (___)
     +-----+      +---+ +---+ +---+ +---+    (___)
     stunnel        4 cheap web servers
     haproxy

Description :
- stunnel on LB1 will receive clients requests on port 443
- it forwards them to haproxy bound to port 80
- haproxy will receive HTTP client requests on port 80 and decrypted SSL
requests from Stunnel on the same port.
- stunnel will add the X-Forwarded-For header
- haproxy will add the X-Forwarded-For header for everyone except the local
address (stunnel).

4. Soft-stop for application maintenance
4.1 Soft-stop using a file on the servers

Put a file on the server which will be checked by the proxy. Remove this file so that the proxy will treat this server as dead, and won’t send any new sessions, only old ones if the “persist” option is used. Wait a bit then stop the server when there isn’t http connection anymore. And then it’s time to do backend server maintenance.

This solution will effect the clients, not so good.

4.2 Soft-stop using backup servers

Set two different names to one server checked on different port(one is 80), they share the exact same cookies. Those servers will only be used when no other server is available for the same cookie.

When the web servers are started, only one named server is seen as available. On the web server, redirect the other different port to local port 80(e.g., use iptables).

When need maintenance, simply stop the server from responding on port 81 so that its standard instance will be seen as failed, but the other will still work. This won’t effect the clients.

4.2.1 Variations for operating systems without any firewall software

Beside the iptables solution above, this redirection can also be handled by a simple haproxy in tcp mode :

global
daemon
quiet
pidfile /var/run/haproxy-checks.pid
listen 0.0.0.0:81
mode tcp
dispatch 127.0.0.1:80
contimeout 1000
clitimeout 10000
srvtimeout 10000

Starting an haproxy instance with this configuration to start the web service, and killing this instance will make the port 81 stopping responding so that the web service could be stopped.

4.2.2 Centralizing the server management

It’s also an solution to do the port redirection on the load-balancer.

Another solution is to use the “COMAFILE” patch provided by Alexander Lazic, which is available for download here :

http://w.ods.org/tools/haproxy/contrib/

4.3 Hot reconfiguration

Send a SIGTTOU signal to the proxy and it will release the ports so that a new instance can be started.

If the new instance fails to start, sending a SIGTTIN signal back to the original processes will restore the listening ports.

Otherwise, sending a SIGUSR1 signal to the old one and it will exit after its last session ends.

If the old process still exists, sending a SIGTERM to the old process.

5. Multi-site load-balancing with local preference
5.1 Network diagram

Note : offices 1 and 2 are on the same continent as site 1, while office 3 is on the same continent as site 3. Each production site can reach the second one either through the WAN or through a dedicated link.

        Office1         Office2                          Office3
         users           users                            users
192.168  # # #   192.168 # # #                            # # #
.1.0/24  | | |   .2.0/24 | | |             192.168.3.0/24 | | |
  --+----+-+-+-   --+----+-+-+-                   ---+----+-+-+-
    |      | .1     |      | .1                      |      | .1
    |    +-+-+      |    +-+-+                       |    +-+-+
    |    |OP1|      |    |OP2|                       |    |OP3|  ...
  ,-:-.  +---+    ,-:-.  +---+                     ,-:-.  +---+
 (  X  )         (  X  )                          (  X  )
  `-:-'           `-:-'             ,---.          `-:-'
  --+---------------+------+----~~~(  X  )~~~~-------+---------+-
                           |        `---'                      |
                           |                                   |
                 +---+   ,-:-.                       +---+   ,-:-.
                 |SD1|  (  X  )                      |SD2|  (  X  )
   ( SITE 1 )    +-+-+   `-:-'         ( SITE 2 )    +-+-+   `-:-'
                   |.1     |                           |.1     |
   10.1.1.0/24     |       |     ,---. 10.2.1.0/24     |       |
        -+-+-+-+-+-+-+-----+-+--(  X  )------+-+-+-+-+-+-+-----+-+--
         | | | | |   |       |   `---'       | | | | |   |       |
      ...# # # # #   |.11    |.12         ...# # # # #   |.11    |.12
          Site 1   +-+--+  +-+--+              Site 2  +-+--+  +-+--+
          Local    |S1L1|  |S1L2|              Local   |S2L1|  |S2L2|
          users    +-+--+  +--+-+              users   +-+--+  +--+-+
                     |        |	                         |        |
   10.1.2.0/24    -+-+-+--+--++--      10.2.2.0/24    -+-+-+--+--++--
                   |.1       |.4                       |.1       |.4
                 +-+-+     +-+-+                     +-+-+     +-+-+
                 |W11| ~~~ |W14|                     |W21| ~~~ |W24|
                 +---+     +---+                     +---+     +---+
              4 application servers               4 application servers
                    on site 1                           on site 2

5.2 Description
5.2.1 Local users

- Office 1 users connect to OP1 = 192.168.1.1
- Office 2 users connect to OP2 = 192.168.2.1
- Office 3 users connect to OP3 = 192.168.3.1
- Site 1 users connect to SD1 = 10.1.1.1
- Site 2 users connect to SD2 = 10.2.1.1

5.2.2 Office proxies

- Office 1 connects to site 1 by default and uses site 2 as a backup.
- Office 2 connects to site 1 by default and uses site 2 as a backup.
- Office 3 connects to site 2 by default and uses site 1 as a backup.

6. Source balancing

Sometimes it may reveal useful to access servers from a pool of IP addresses instead of only one or two. Some equipments (NAT firewalls, load-balancers) are sensible to source address, and often need many sources to distribute the load evenly amongst their internal hash buckets.

7. Managing high loads on application servers

Limiting the number of connections between the clients and the servers. Setting haproxy to limit the number of connections on a per-server basis. It will then fill all the servers up to the configured connection limit, and will put the remaining connections in a queue, waiting for a connection to be released on a server.

So that,

* all clients can be served whatever their number without crashing the servers, the only impact it that the response time can be delayed.

* the servers can be used at full throttle without the risk of stalling, and fine tuning can lead to optimal performance.

* response times can be reduced by making the servers work below the congestion point, effectively leading to shorter response times even under moderate loads.

* no domino effect when a server goes down or starts up. Requests will be queued more or less, always respecting servers limits.

* it’s easy to achieve high performance even on memory-limited hardware. Indeed, heavy frameworks often consume huge amounts of RAM and not always all the CPU available. In case of wrong sizing, reducing the number of concurrent connections will protect against memory shortages while still ensuring optimal CPU usage.

Today, I finished a benchmark to compare the caching performance and status between Varnish and Squid, which get widely focused on as reverse proxies.
Here we go.

platform

The test-network is made up of:

    * D-Link 1024R, a 24-port Gigabit Switch
    * http_server:
          o OS: Linux 2.6.21.5-smp i686 (Slackware 12.0)
          o CPU: Intel(R) Xeon(TM) CPU 2.80GHz x 2
          o MEM: 1024M x 6
          o DISK: SEAGATE ST373405LC SCSI Disk
          o Ethernet controller: Intel 82546EB PRO/1000 MT Dual Port Server Adapter
    * proxy_server:
          o OS: Linux 2.6.21.5-smp i686 (Slackware 12.0)
          o CPU:  Intel(R) Pentium(R) III CPU family 1133MHz GenuineIntel
          o MEM: 1024M
          o DISK: SEAGATE ST318406LC SCSI Disk
          o Ethernet controller:  Intel 82557/8/9 PRO/100+ Server Adapter
    * client
          o OS: Linux 2.4.31 i686 (Slackware 10.2)
          o CPU: Intel(R) Xeon(TM) CPU 2.80GHz x 2
          o MEM: 1024M x 6
          o Ethernet controller: Intel 82546EB Gigabit Ethernet Controller

file pool

#!/bin/sh
#
# Written for generating 2 sets of 100Mbyte files,
# one is 1,000 files of 100kbyte size, and
# the other is 10 files of 10MByte size.
# by Cherife Li
#

docroot=/home/wwwroot/

# generate the first set
mkdir -p $docroot/100k
cd $docroot/100k
for i in `seq 1 10`; do
mkdir -p files-$i;
for j in `seq 1 100`; do
dd if=/dev/zero of=files-$i/$j bs=100K count=1 2> /dev/null;
done;
done

# generate the second set
mkdir -p $docroot/10m
cd $docroot/10m
for i in `seq 1 5`; do
mkdir -p files-$i;
for j in `seq 1 2`; do
dd if=/dev/zero of=files-$i/$j bs=10M count=1 2> /dev/null;
done;
done

$ cd /home/wwwroot/
$ find ./100k/ | grep 'files.*/.' | sed 's#./#http://benchmark.lo/#' > 100k.urls
$ find ./10m/ | grep 'files.*/.' | sed 's#./#http://benchmark.lo/#' > 10m.urls

benchmark

 ______        ____________        ___________
|      |--A-->|            |--B-->|           |
|client|      |proxy server|      |http server|
|______|<--D--|____________|<--C--|___________|

    * client: http_load
    * proxy server: varnish 1.1.2, squid 2.6.STABLE18, and squid 3.0.STABLE2.
    * http server: nginx/0.6.28

The Nginx http server only ran the stand alone http service.
The proxy server ran the proxy service only the one been benchmarking at a time.

http_load
It’s a good load-generator as it

    * allows random fetches from a list of URLs
    * allows a large number of parallel requests
    * is portable.

There are also other test tools, check this page for detail.

+++ 100KByte +++

$ http_load -verbose -parallel 100 -fetches 100000 ./100k.urls

* Varnish 1.1.2:

— 60 secs, 6868 fetches started, 6768 completed, 100 current
— 120 secs, 13720 fetches started, 13620 completed, 100 current
— 180 secs, 20570 fetches started, 20470 completed, 100 current
— 240 secs, 27432 fetches started, 27332 completed, 100 current
— 300 secs, 34285 fetches started, 34185 completed, 100 current
— 360 secs, 41140 fetches started, 41040 completed, 100 current
— 420 secs, 47996 fetches started, 47896 completed, 100 current
— 480 secs, 54854 fetches started, 54754 completed, 100 current
— 540 secs, 61709 fetches started, 61609 completed, 100 current
— 600 secs, 68565 fetches started, 68465 completed, 100 current
— 660 secs, 75419 fetches started, 75319 completed, 100 current
— 720 secs, 82279 fetches started, 82179 completed, 100 current
— 780 secs, 89131 fetches started, 89031 completed, 100 current
— 840 secs, 95989 fetches started, 95889 completed, 100 current
100000 fetches, 100 max parallel, 1.024e+10 bytes, in 876.794 seconds
102400 mean bytes/connection
114.052 fetches/sec, 1.16789e+07 bytes/sec
msecs/connect: 116.758 mean, 9111.16 max, 0.256 min
msecs/first-response: 120.03 mean, 2494.66 max, 0.614 min
HTTP response codes:
code 200 — 100000

* Squid 2.6.STABLE18:

— 60 secs, 6898 fetches started, 6798 completed, 100 current
— 120 secs, 13748 fetches started, 13648 completed, 100 current
— 180 secs, 20595 fetches started, 20495 completed, 100 current
— 240 secs, 27439 fetches started, 27339 completed, 100 current
— 300 secs, 34287 fetches started, 34187 completed, 100 current
— 360 secs, 41136 fetches started, 41036 completed, 100 current
— 420 secs, 47983 fetches started, 47883 completed, 100 current
— 480 secs, 54829 fetches started, 54729 completed, 100 current
— 540 secs, 61675 fetches started, 61575 completed, 100 current
— 600 secs, 68523 fetches started, 68423 completed, 100 current
— 660 secs, 75371 fetches started, 75271 completed, 100 current
— 720 secs, 82221 fetches started, 82121 completed, 100 current
— 780 secs, 89065 fetches started, 88965 completed, 100 current
— 840 secs, 95909 fetches started, 95809 completed, 100 current
100000 fetches, 100 max parallel, 1.024e+10 bytes, in 878.411 seconds
102400 mean bytes/connection
113.842 fetches/sec, 1.16574e+07 bytes/sec
msecs/connect: 116.02 mean, 9114.65 max, 0.224 min
msecs/first-response: 115.596 mean, 619.381 max, 0.86 min
HTTP response codes:
code 200 — 100000

* Squid 3.0.STABLE2:

— 60 secs, 6885 fetches started, 6785 completed, 100 current
— 120 secs, 13720 fetches started, 13620 completed, 100 current
— 180 secs, 20577 fetches started, 20477 completed, 100 current
— 240 secs, 27418 fetches started, 27318 completed, 100 current
— 300 secs, 34266 fetches started, 34166 completed, 100 current
— 360 secs, 41111 fetches started, 41011 completed, 100 current
— 420 secs, 47957 fetches started, 47857 completed, 100 current
— 480 secs, 54812 fetches started, 54712 completed, 100 current
— 540 secs, 61656 fetches started, 61556 completed, 100 current
— 600 secs, 68503 fetches started, 68403 completed, 100 current
— 660 secs, 75346 fetches started, 75246 completed, 100 current
— 720 secs, 82198 fetches started, 82098 completed, 100 current
— 780 secs, 89040 fetches started, 88940 completed, 100 current
— 840 secs, 95891 fetches started, 95791 completed, 100 current
100000 fetches, 100 max parallel, 1.024e+10 bytes, in 876.658 seconds
102400 mean bytes/connection
114.07 fetches/sec, 1.16807e+07 bytes/sec
msecs/connect: 115.858 mean, 9111.16 max, 0.26 min
msecs/first-response: 116.423 mean, 3318.18 max, 29.481 min
HTTP response codes:
code 200 — 100000

+++ 10MByte +++

$ http_load -verbose -parallel 100 -fetches 1000 ./10m.urls

* Varnish 1.1.2:

— 60 secs, 100 fetches started, 0 completed, 100 current
— 120 secs, 196 fetches started, 96 completed, 100 current
— 180 secs, 231 fetches started, 131 completed, 100 current
— 240 secs, 304 fetches started, 204 completed, 100 current
— 300 secs, 389 fetches started, 289 completed, 100 current
— 360 secs, 434 fetches started, 334 completed, 100 current
— 420 secs, 509 fetches started, 409 completed, 100 current
— 480 secs, 585 fetches started, 485 completed, 100 current
— 540 secs, 637 fetches started, 537 completed, 100 current
— 600 secs, 714 fetches started, 614 completed, 100 current
— 660 secs, 786 fetches started, 686 completed, 100 current
— 720 secs, 844 fetches started, 744 completed, 100 current
— 780 secs, 915 fetches started, 815 completed, 100 current
— 840 secs, 987 fetches started, 887 completed, 100 current
1000 fetches, 100 max parallel, 1.04858e+10 bytes, in 899.718 seconds
1.04858e+07 mean bytes/connection
1.11146 fetches/sec, 1.16545e+07 bytes/sec
msecs/connect: 129.428 mean, 3128.09 max, 0.291 min
msecs/first-response: 1003.12 mean, 9619.7 max, 120.662 min
HTTP response codes:
code 200 — 1000

* Squid 2.6.STABLE18:

— 60 secs, 103 fetches started, 3 completed, 100 current
— 120 secs, 178 fetches started, 78 completed, 100 current
— 180 secs, 255 fetches started, 155 completed, 100 current
— 240 secs, 322 fetches started, 222 completed, 100 current
— 300 secs, 379 fetches started, 279 completed, 100 current
— 360 secs, 458 fetches started, 358 completed, 100 current
— 420 secs, 518 fetches started, 418 completed, 100 current
— 480 secs, 583 fetches started, 483 completed, 100 current
— 540 secs, 661 fetches started, 561 completed, 100 current
— 600 secs, 721 fetches started, 621 completed, 100 current
— 660 secs, 786 fetches started, 686 completed, 100 current
— 720 secs, 863 fetches started, 763 completed, 100 current
— 780 secs, 926 fetches started, 826 completed, 100 current
— 840 secs, 984 fetches started, 884 completed, 100 current
1000 fetches, 100 max parallel, 1.04858e+10 bytes, in 894.062 seconds
1.04858e+07 mean bytes/connection
1.11849 fetches/sec, 1.17282e+07 bytes/sec
msecs/connect: 137.644 mean, 9128.16 max, 0.283 min
msecs/first-response: 131.811 mean, 3509.99 max, 28.891 min
HTTP response codes:
code 200 — 1000

* Squid 3.0.STABLE2:

— 60 secs, 105 fetches started, 5 completed, 100 current
— 120 secs, 190 fetches started, 90 completed, 100 current
http://benchmark.lo/10m/files-2/2: timed out
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-1/1: timed out
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-5/2: timed out
http://benchmark.lo/10m/files-5/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-2/1: timed out
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-4/2: timed out
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
— 180 secs, 260 fetches started, 160 completed, 100 current
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-1/1: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-3/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/1: byte count wrong
http://benchmark.lo/10m/files-4/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
http://benchmark.lo/10m/files-2/2: byte count wrong
— 240 secs, 315 fetches started, 215 completed, 100 current
— 300 secs, 395 fetches started, 295 completed, 100 current
— 360 secs, 464 fetches started, 364 completed, 100 current
— 420 secs, 521 fetches started, 421 completed, 100 current
— 480 secs, 596 fetches started, 496 completed, 100 current
— 540 secs, 664 fetches started, 564 completed, 100 current
— 600 secs, 725 fetches started, 625 completed, 100 current
— 660 secs, 798 fetches started, 698 completed, 100 current
— 720 secs, 865 fetches started, 765 completed, 100 current
— 780 secs, 929 fetches started, 829 completed, 100 current
— 840 secs, 997 fetches started, 897 completed, 100 current
1000 fetches, 100 max parallel, 1.03922e+10 bytes, in 886.272 seconds
1.03922e+07 mean bytes/connection
1.12832 fetches/sec, 1.17258e+07 bytes/sec
msecs/connect: 130.639 mean, 3127.72 max, 0.21 min
msecs/first-response: 186.15 mean, 3578.41 max, 34.601 min
5 timeouts
53 bad byte counts
HTTP response codes:
code 200 — 1000

proxy server status

+++ 100KByte +++

100k.varnish.iostat_xm5.log
100k.varnish.vmstat5.log
100k.squid2.iostat_xm5.log
100k.squid2.vmstat5.log
100k.squid3.iostat_xm5.log
100k.squid3.vmstat5.log

+++ 10MByte +++

10m.varnish.iostat_xm5.log
10m.varnish.vmstat5.log
10m.squid2.iostat_xm5.log
10m.squid2.vmstat5.log
10m.squid3.iostat_xm5.log
10m.squid3.vmstat5.log

After finished each benchmark, the status is as follow:

conclusion

The data here, the reference in your mind.
That which is the better choice as a caching proxy, depends on your own judgement.

notes

1. Varnish can cache files immediately.
I found this while comparing the fetching of 10MB files.
Varnish just request the nginx server for the file only once, but Squid will requests them for about
10 times of varnish does.

2. Varnish provides easier access to proxy status.

3. Squid is more than a web proxy, while Varnish is more than an http accelerator (cache proxy).
The Varnish web site claims that Varnish is ten to twenty times faster than the popular Squid
cache on the same hardware.

<Cherife> hi all, namesys.com couldn't be solved, what happened?
<LinuxyErin> blah i cant sleep
<echelon_> what's namesys and how is it our problem?
<Cherife> it's the official reiserfs site, isn't it?
<echelon_> is it?
<rob0> Domain is still valid for another year.
<rob0> but nameservers are not responding
<echelon_> yup.. same here
* pdw (n=pdw@catv-5984bc75.catv.broadband.hu) has joined ##slackware
<rob0> Yes it is the reiserfs site.
<echelon_> that's what they get for using russian dns servers
<Cherife> lol
<rob0> Probably can find it in google cache or wayback machine.
<sallygal> echelon_, you just kill me
<sallygal> :)
<rob0> It's nominally a Russian-based company, only the owner is in jail in USA.
<echelon_> now what did he do that was so bad?
<sallygal> killed her
<rob0> Accused of murder, you hadn't heard?
<rob0> His estranged wife is missing
<echelon_> nice.. i'm using the fs of a murderer :D
.
.
.

So I married a Kernel Programmer…

Just joking, lol.

An simple intro from the FUSE website:

With FUSE it is possible to implement a fully functional filesystem in a userspace program.  Features include:

    * Simple library API
    * Simple installation (no need to patch or recompile the kernel)
    * Secure implementation
    * Userspace - kernel interface is very efficient
    * Usable by non privileged users
    * Runs on Linux kernels 2.4.X and 2.6.X
    * Has proven very stable over time

The following figure shows the path of a filesystem call:

Operating Systems FUSE Supportes are Linux-2.4.X, Linux-2.6.X, FreeBSD, NetBSD, Mac OS X, Windows, OpenSolaris, GNU/Hurd. Check this link for detail.

There are various types of Filesystems based on FUSE, like:

# ArchiveFileSystems - accessing files inside archives (tar, cpio, zip, etc.)
# CompressedFileSystems - accessing files in a compressed image (gz, zlib, LiveCDs, etc.)
# DatabaseFileSystems - storing files in a relational database (MySQL, BerkeleyDB, etc.) \
    or ones allowing searching using tags or SQL queries
# EncryptedFileSystems - storing files in a more secure way by using a secret key
# MediaFileSystems - storing files on media devices such as cameras and music players \
    or accessing and categorizing media files
# HardwareFileSystems - provide access to weird hardware
# MonitoringFileSystems - provide notification when a file changes
# NetworkFileSystems - storing files on remote computers, including file servers and web sites
# NonNativeFileSystems - traditional disk-based file systems that aren't standard on \
    Linux (NTFS, ZFS, etc.)
# UnionFileSystems - merging multiple file systems into a single tree
# VersioningFileSystems - file systems that remember old versions of files and ones which \
    provide access to version control systems

Check this link for up-to-date detail.

SystemImager is a part of System Installation Suite which automates Linux installs, software distribution, and production deployment.

SystemImager makes it easy to do automated installs (clones), software distribution, content or data distribution, configuration changes, and operating system updates to your network of Linux machines. You can even update from one Linux release version to another!

It can also be used to ensure safe production deployments. By saving your current production image before updating to your new production image, you have a highly reliable contingency mechanism. If the new production enviroment is found to be flawed, simply roll-back to the last production image with a simple update command!

Some typical environments include: Internet server farms, database server farms, high performance clusters, computer labs, and corporate desktop environments.

Benchmark

Check this link for detail.

Useful Resource

    * Article on linux.com: Automate Linux installation and recovery with SystemImager
    * HowTo sur Capweb.biz : Créer des images de votre système Linux avec SystemImager (FR)
    * Clustering Is Not Rocket Science
    * Avatars and Grid 2.0 (article on Grid Today quoting SystemImager as a core component \
        used in the famous virtual world Second Life)
    * Automating Xen Virtual Machine Deployment (how to integrate SystemImager with Xen to \
        create a totally open source deployment framework of virtual machines) 

    * The Linux Enterprise Cluster
    * High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI
    * Linux Clustering: Building and Maintaining Linux Clusters
    * Distributed and Parallel Systems: From Instruction Parallelism to Cluster Computing
    * High Performance Computing Clusters with Intel® Architecture, Part 2 

    * System Installation Suite: Massive Installation for Linux - Sean Dague

    * SystemImager @ Ottawa Linux Symposium 2002 - Sean Dague
    * SystemImager @ LinuxTag 2007 in Berlin - Andrea Righi, Brian Elliott Finley, Erich Focht
          o Get paper and presentation of the talk in PDF 

    * SystemImager at CINECA
          o The IBM BCX/5120 cluster in CINECA uses SystemImager
    * SystemImager at niflheim
          o Wiki about SystemImager used for the NIFLHEIM cluster installation

Before this how to set up replication, take a quickly look at the steps that MySQL goes through to maintain a replicated server. The process is different depending on the version of MySQL. For purposes of this article, my comments will be for version 4.0 or higher, since most systems now are using the later versions.

When replication is running, basically, as SQL statements are executed on the master server, MySQL records them in a binary log (bin.log) along with a log position identification number. The slave server in turn, through an IO thread, regularly and very often reads the master’s binary log for any changes. If it finds a change, it copies the new statements to its relay log (relay.log). It then records the new position identification number in a file (master.info) on the slave server. The slave then goes back to checking the master binary log, using the same IO thread. When the slave server detects a change to its relay log, through an SQL thread the slave executes the new SQL statement recorded in the relay log. As a safeguard, the slave also queries the master server through the SQL thread to compare its data with the master’s data. If the comparison shows inconsistency, the replication process is stopped and an error message is recorded in the slave’s error log (error.log). If the results of the query match, the new log position identification number is recorded in a file on the slave (relay-log.info) and the slave waits for another change to the relay log file.

This process may seem involved and complicated at first glance, but it all occurs quickly, it isn’t a significant drain on the master server, and it ensures reliable replication. Also, it’s surprisingly easy to set up. It only requires a few lines of options to be added to the configuration file (i.e., my.cnf) on the master and slave servers.

How to Set Up Replication

There are a number of different methods for setting up replication, and the exact method that you use will depend on how you are setting up replication, and whether you already have data within your master database.

I. Creating a User for Replication

Each Slave must connect to the Master using a standard username and password. It’s best not to use an existing account for security reasons, for the username and password will be stored in plain text within the master.info file. To do this, enter an SQL statement like the following on the master server, logged in as root or a user that has GRANT OPTION privileges:

mysql> GRANT REPLICATION SLAVE ON *.*
       -> TO 'replic'@'192.168.0.6' IDENTIFIED BY 'passwdhere';

In this SQL statement, the user account replic is granted only what’s needed for replication. The user name can be almost anything. The IP address(or host name) 192.168.0.6 is given in quotes which is allowed to connect for replication. You should enter this same statement on the slave server with the same user name and password, but with the master’s host name or IP address. This way, if the master fails and will be down for a while, you could redirect users to the slave with DNS or by some other method. When the master is back up, you can then use replication to get it up to date by temporarily making it a slave to the former slave server. Incidentally, if you upgraded MySQL to version 4.0 recently, but didn’t upgrade your mysql database, the GRANT statement above won’t work because these privileges didn’t exist in the earlier versions.

II. Configuring Replication Master

For replication to work you must enable binary logging on the master, for it is the binary log that is used to exchange data between the master and slaves.

Each server within a replication group must have a unique server-id. The server-id is used to identify individual servers within the group. You will need to add the following options to the configuration file my.cnf within the [mysqld] section. For example, to enable binary logging, using a log filename prefix of mysql-bin, and setting a server ID of 1:

[mysqld]
log-bin=mysql-bin
server-id=1

Ensure that the skip-networking option has not been enabled on your replication master. If networking has been disabled, then your slave will not able to communicate with the master and replication will fail.

III. Configuring Replication Slave

The only option you must configure on the slave is to set the unique server ID. Ensure that the slave server has the followling effective option:

[mysqld]
server-id=2

If you are setting up multiple slaves, each one must have a unique server-id value that differs from that of the master and from each of the other slaves. You do not have to enable binary logging on the slave for replication to be enabled. However, if you enable binary logging on the slave then you can use the binary log for data backups and crash recovery on the slave, and also use the slave as part of a more complex replication topology.

IV. Obtaining the Master Replication Information

To configure replication on the slave you must determine the masters current point within the master binary log. You will need this information so that when the slave starts the replication process, it is able to start processing events from the binary log at the correct point.

If you have existing data on your master that you want to synchronize on your slaves before starting the replication process, then you must stop processing statements on the master, obtain the current position, and then dump the data, before allowing the master to continue executing statements. If you do not stop the execution of statements then the data dump, the master status information that you use will not match and you will end up with inconsistent or corrupted databases on the slaves.

Start the command line client and flush all tables and block write statements by executing the FLUSH TABLES WITH READ LOCK statement:

mysql> FLUSH TABLES WITH READ LOCK;

Use the SHOW MASTER STATUS statement to determine the current binary log name and offset on the master:

mysql > SHOW MASTER STATUS;
+-----------------+----------+-----------------+---------------------+
| File                | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+----------+-----------------+---------------------+
|mysql-bin.0006| 98         |                      |                          |
+-----------------+----------+-----------------+---------------------+

The File column shows the name of the log and Position shows the offset within the file. In this example, the binary log file is mysql-bin.006 and the offset is 98. Record these values. You need them later when you are setting up the slave.

Keep the databases tables with read lock and follow the next step.

V. Creating a Data Snapshot of Master

There are several different ways to take a databases snapshot. From my experience, I prefer mysqlhotcopy(only for backing up MyISAM tables). For example, on master:

shell> mysqlhotcopy --allowold --noindices -u root -p passwdhere database-names /path-to-destdir

When choosing databases to include in the copy, remember that you will need to filter out databases on each slave that you do not want to include in the replication process. Once you have created the archive or copy of the database, you will need to copy the files to each slave before starting the slave replication process. Now you could free the read lock on master:

mysql> UNLOCK TABLES;

VI. Making Replication Process Running

Importing the databases into slave, then starting the slave, skipping replication by using the –skip-slave option.

To set up the slave to communicate with the master for replication, you must tell the slave the necessary connection information. To do this, execute the following statement on the slave, replacing the option values with the actual values relevant to your system:

mysql> CHANGE MASTER TO
       -> MASTER_HOST='master_host_name',
       -> MASTER_USER='replication_user_name',
       -> MASTER_PASSWORD='replication_password',
       -> MASTER_LOG_FILE='recorded_log_file_name',
       -> MASTER_LOG_POS=recorded_log_position;

After that, start the replication threads:

mysql> START SLAVE;

After you have performed this procedure, the slave should connect to the master and catch up on any updates that have occurred since the snapshot was taken.

There are more replication startup options and variables, check the manual for details.

VII. Checking Replication Status

It’s required that you ensure that replication is taking place and that there have been no errors between the slave and the master. One command for this is SHOW SLAVE STATUS which you must execute on each slave:

mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
                  Slave_IO_State: Waiting for master to send event
                      Master_Host: 192.168.0.16
                     Master_User: replic
                      Master_Port: 3306
                  Connect_Retry: 60
                 Master_Log_File: mysql-bin.000024
         Read_Master_Log_Pos: 4279
                   Relay_Log_File: slave1-relay-bin.000056
                   Relay_Log_Pos: 950
         Relay_Master_Log_File: mysql-bin.000024
               Slave_IO_Running: Yes
             Slave_SQL_Running: Yes
                 Replicate_Do_DB:
            Replicate_Ignore_DB:
             Replicate_Do_Table:
       Replicate_Ignore_Table:
     Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                           Last_Errno: 0
                            Last_Error:
                       Skip_Counter: 0
           Exec_Master_Log_Pos: 4279
                 Relay_Log_Space: 4418
                     Until_Condition: None
                      Until_Log_File:
                      Until_Log_Pos: 0
            Master_SSL_Allowed: No
            Master_SSL_CA_File:
           Master_SSL_CA_Path:
                 Master_SSL_Cert:
              Master_SSL_Cipher:
                  Master_SSL_Key:
       Seconds_Behind_Master: 0
1 row in set (0.01 sec)

The key fields from the status report to examine are:

Slave_IO_State — indicates the current status of the slave.
Slave_IO_Running — shows whether the IO thread for the reading the master’s binary log is running.
Slave_SQL_Running — shows whether the SQL thread for the executing events in the relay log is running.
Last_Error — shows the last error registered when processing the relay log. Ideally this should be blank, indicating no errors.
Seconds_Behind_Master — shows the number of seconds that the slave SQL thread is behind processing the master binary log.
A high number (or an increasing one) can indicate that the slave is unable to cope with the large number of queries from the master.

On the master, you can check the status of slaves by examining the list of running processes. Slaves execute the Binlog Dump command:

*************************** 1. row ***************************
     Id: 112
   User: syn
   Host: 192.168.0.16:56644
     db: NULL
Command: Binlog Dump
   Time: 71268
  State: Has sent all binlog to slave; waiting for binlog to be updated
   Info: NULL
*************************** 2. row ***************************
     Id: 112935
   User: root
   Host: localhost
     db: NULL
Command: Query
   Time: 0
  State: NULL
   Info: SHOW PROCESSLIST
2 rows in set (0.00 sec)

Ganglia is a scalable distributed monitoring system for high-performance computing
systems such as clusters and Grids. It is based on a hierarchical design targeted at
federations of clusters. It leverages widely used technologies such as XML for data
representation, XDR for compact, portable data transport, and RRDtool for data
storage and visualization. It uses carefully engineered data structures and algorithms
to achieve very low per-node overheads and high concurrency. The implementation
is robust, has been ported to an extensive set of operating systems and processor
architectures, and is currently in use on thousands of clusters around the world. It
has been used to link clusters across university campuses and around the world and
can scale to handle clusters with 2000 nodes.

Ganglia is an open-source project that grew out of the University of California,
Berkeley Millennium Project which was initially funded in large part by the
National Partnership for Advanced Computational Infrastructure (NPACI)
and National Science Foundation RI Award EIA-9802069.
NPACI is funded by the National Science Foundation and strives to
advance science by creating a ubiquitous, continuous, and pervasive national
computational infrastructure: the Grid. Current support comes from Planet Lab: an
open platform for developing, deploying, and accessing planetary-scale services.

The ganglia system is comprised of two unique daemons, a PHP-based web frontend and a few other small utility programs.

The two unique daemons are Ganglia Monitoring Daemon (gmond) and Ganglia Meta Daemon (gmetad).
Gmond is a multi-threaded daemon which runs on each cluster node you want to monitor. It monitors changes in host state, announce relevant changes, listen to the state of all other ganglia nodes via a unicast or multicast channel and answer requests for an XML description of the cluster state.
Gmetad periodically polls a collection of child data sources, parses the collected XML, saves all numeric, volatile metrics to round-robin databases and exports the aggregated XML over a TCP sockets to clients. Data sources may be either gmond daemons, representing specific clusters, or other gmetad daemons, representing sets of clusters. Data sources use source IP addresses for access control and can be specified using multiple IP addresses for failover.
Ganglia PHP Web Frontend provides a view of the gathered information via real-time dynamic web pages. It depends on the existence of the gmetad which provides it with data from several Ganglia sources. Specifically, the web frontend will open the local port 8651 (by default) and expects to receive a Ganglia XML tree.

So, how to get ganglia running for you? The official installation and configuration guide and other information can be found here. I just won’t give an iterant and gash guide because ganglia is so easy and express to install and configure. From my experience, compared with other monitoring tools’ introductions such as Nagios, Cacti, Monit, Pandora FMS and so on, I think ganglia is conciser and easier to install and manage.(Honestly speaking, I haven’t used any one of them.) As you may see that, now or later, more functions which the other monitoring tools supplies may not included in gangila, but you can use your own scripts to achieve a few ones by using ganglia’s utility programs.
The official site gives some demo, go and check it if you are interested.

Simplicity is powerful. Lots of famous sites are using it. Yeah, Ganglia is worth a try. Maybe you will love it:-)