Two of the projects of which the systems I’m maintaining currently are running IPVS (IP Virtual Server), an software from the LVS (Linux Virtual Server) Project, which implements transport-layer load balancing inside the Linux kernel, also called Layer-4 switching.
I really got impressed with its stability, scalability and high performance. It doesn’t cost any system resource except memory usage, which depends on the connection number. And yet it’s very low-cost, for each connection entry consumes about 128bytes or so. It’s easy for you to figure it out that 100,000 unique connections/s, only 12.8M memory will be used. On the other hand, as a 4th-layer implement, IPVS just direct packages.
So, a question raised, what if I need layer-7 switching?
Yeah, IPVS no longer sucks. Here goes HAProxy, the reliable, high performance TCP/HTTP load balancer. Although there are other solutions, e.g., F5, pound, XLB, pen, and so on, I would like to chose HAProxy and introduce it here.
HAProxy is written by Willy TARREAU, the new Linux Kernel 2.4 maintainer since August 2006, who also maintains patches against the 2.6.20 branch since August of 2007.
Today I glanced my eye over the HAProxy Architecture Guide. I’d like to take a note here.
This note is so annoyed that you would better go for a drink, then YouTube. Yeah, I’m serious. :-) Otherwise, it’s better to check the link in the line above for original detail if you are interested in the architecture. Okay, here I go, see you…
1. Simple HTTP load-balancing with cookie insertion
192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2
-------+-----------+-----+-----+-----+--------+----
| | | | | _|_db
+--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
| LB1 | | A | | B | | C | | D | (___)
+-----+ +---+ +---+ +---+ +---+ (___)
haproxy 4 cheap web servers
Flows :
(client) (haproxy) (server A)
>-- GET /URI1 HTTP/1.0 ------------> |
( no cookie, haproxy forwards in load-balancing mode. )
| >-- GET /URI1 HTTP/1.0 ---------->
| <-- HTTP/1.0 200 OK -------------<
( the proxy now adds the server cookie in return )
<-- HTTP/1.0 200 OK ---------------< |
Set-Cookie: SERVERID=A |
>-- GET /URI2 HTTP/1.0 ------------> |
Cookie: SERVERID=A |
( the proxy sees the cookie. it forwards to server A and deletes it )
| >-- GET /URI2 HTTP/1.0 ---------->
| <-- HTTP/1.0 200 OK -------------<
( the proxy does not add the cookie in return because the client knows it )
<-- HTTP/1.0 200 OK ---------------< |
>-- GET /URI3 HTTP/1.0 ------------> |
Cookie: SERVERID=A |
( ... )
If clients use keep-alive (HTTP/1.1), only the first response will have a cookie inserted, and only the first request of each session will be analyzed. The added server cookie will not be removed from the requests forwarded to the servers, so the server must not be sensitive to unknown cookies. If this causes trouble, keep-alive can be disabled by adding the following option :
option httpclose
If for some reason the clients cannot learn more than one cookie, and the application already produces a cookie, “prefix” mode could be used(see below).
Backing LB1 up using keepalived for healthcheck & failover(see below)
If the application needs to log the original client’s IP, use the “forwardfor” option which will add an “X-Forwarded-For” header with the original client’s IP address. Also use “httpclose” to rewrite every requests and not only the first one of each session :
option httpclose
option forwardfor
The web server will have to be configured to use this header instead. For example, on apache:
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b ” combined
CustomLog /var/log/httpd/access_log combined
In the situation of clients disable cookies on their browser, use the “source” balancing algorithm instead of the “roundrobin”. So a given IP address will always reaches the same server. (as long as the number of servers remains unchanged.) Never use this behind a proxy or in a small network, because the distribution will be unfair. However, in large internal networks, and on the internet, it works quite well. Clients
which have a dynamic address will not be affected as long as they accept the cookie, because the cookie always has precedence over load balancing.
2. HTTP load-balancing with cookie prefixing and high availability
Backed load-balancer up with a second one in VRRP mode using keepalived.
http://www.keepalived.org/
Allow the proxy to bind to the shared IP:
# echo 1 >/proc/sys/net/ipv4/ip_nonlocal_bind
shared IP=192.168.1.1
192.168.1.3 192.168.1.4 192.168.1.11-192.168.1.14 192.168.1.2
-------+------------+-----------+-----+-----+-----+--------+----
| | | | | | _|_db
+--+--+ +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
| LB1 | | LB2 | | A | | B | | C | | D | (___)
+-----+ +-----+ +---+ +---+ +---+ +---+ (___)
haproxy haproxy 4 cheap web servers
keepalived keepalived
Set the “httpclose” option to disable keep-alive (HTTP/1.1), so the proxy can access to all cookies in all requests for each session, because the proxy will modify EVERY cookie sent by the client and the server.
Flows :
(client) (haproxy) (server A)
>-- GET /URI1 HTTP/1.0 ------------> |
( no cookie, haproxy forwards in load-balancing mode. )
| >-- GET /URI1 HTTP/1.0 ---------->
| X-Forwarded-For: 10.1.2.3
| <-- HTTP/1.0 200 OK -------------<
( no cookie, nothing changed )
<-- HTTP/1.0 200 OK ---------------< |
>-- GET /URI2 HTTP/1.0 ------------> |
( no cookie, haproxy forwards in lb mode, possibly to another server. )
| >-- GET /URI2 HTTP/1.0 ---------->
| X-Forwarded-For: 10.1.2.3
| <-- HTTP/1.0 200 OK -------------<
| Set-Cookie: JSESSIONID=123
( the cookie is identified, it will be prefixed with the server name )
<-- HTTP/1.0 200 OK ---------------< |
Set-Cookie: JSESSIONID=A~123 |
>-- GET /URI3 HTTP/1.0 ------------> |
Cookie: JSESSIONID=A~123 |
( the proxy sees the cookie, removes the server name and forwards
to server A which sees the same cookie as it previously sent )
| >-- GET /URI3 HTTP/1.0 ---------->
| Cookie: JSESSIONID=123
| X-Forwarded-For: 10.1.2.3
| <-- HTTP/1.0 200 OK -------------<
( no cookie, nothing changed )
<-- HTTP/1.0 200 OK ---------------< |
( ... )
Setting “weight”(values between 1 and 256) to inform haproxy to spread the load of backends the most smoothly possible respecting those ratios:
server webA 192.168.1.11:80 cookie A weight 12 check
server webC 192.168.1.13:80 cookie C weight 26 check
2.1 Variations involving external layer 4 load-balancers
Can the haproxies be load-balanced?
Yeah, by a layer4 load-balancer (eg: Alteon) which will check the services:
| VIP=192.168.1.1
+----+----+
| Alteon |
+----+----+
|
192.168.1.3 | 192.168.1.4 192.168.1.11-192.168.1.14 192.168.1.2
-------+-----+------+-----------+-----+-----+-----+--------+----
| | | | | | _|_db
+--+--+ +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
| LB1 | | LB2 | | A | | B | | C | | D | (___)
+-----+ +-----+ +---+ +---+ +---+ +---+ (___)
haproxy haproxy 4 cheap web servers
What if the Alteon fails? Want relay generic TCP protocols (SMTP, TSE, VNC, etc…)? (see below)
2.2 Generic TCP relaying and external layer 4 load-balancers
Using the “monitor-net” keyword to specify a network which will be dedicated to monitoring systems and must not lead to a forwarding connection nor to any log. This expects a version of haproxy greater than or equal to 1.1.32 or 1.2.6.
| VIP=172.16.1.1 |
+----+----+ +----+----+
| Alteon1 | | Alteon2 |
+----+----+ +----+----+
192.168.1.252 | GW=192.168.1.254 | 192.168.1.253
| |
------+---+------------+--+-----------------> TSE farm : 192.168.1.10
192.168.1.1 | | 192.168.1.2
+--+--+ +--+--+
| LB1 | | LB2 |
+-----+ +-----+
haproxy haproxy
3. Simple HTTP/HTTPS load-balancing with cookie insertion
192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2
-------+-----------+-----+-----+-----+--------+----
| | | | | _|_db
+--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
| LB1 | | A | | B | | C | | D | (___)
+-----+ +---+ +---+ +---+ +---+ (___)
apache 4 cheap web servers
mod_ssl
haproxy
Do not cache inserted cookies for security measures.
If the cookie works in “prefix” mode, there is no need to add the “nocache” option because it is an application cookie which will be modified, and the application flags will be preserved.
If apache 1.3 is used as a front-end before haproxy, it always disables HTTP keep-alive on the back-end, so the “httpclose” is needn’t.
To log client’s IP, configure apache to set the X-Forwarded-For header not on haproxy.
Flows :
(apache) (haproxy) (server A)
>-- GET /URI1 HTTP/1.0 ------------> |
( no cookie, haproxy forwards in load-balancing mode. )
| >-- GET /URI1 HTTP/1.0 ---------->
| <-- HTTP/1.0 200 OK -------------<
( the proxy now adds the server cookie in return )
<-- HTTP/1.0 200 OK ---------------< |
Set-Cookie: SERVERID=A |
Cache-Control: private |
>-- GET /URI2 HTTP/1.0 ------------> |
Cookie: SERVERID=A |
( the proxy sees the cookie. it forwards to server A and deletes it )
| >-- GET /URI2 HTTP/1.0 ---------->
| <-- HTTP/1.0 200 OK -------------<
( the proxy does not add the cookie in return because the client knows it )
<-- HTTP/1.0 200 OK ---------------< |
>-- GET /URI3 HTTP/1.0 ------------> |
Cookie: SERVERID=A |
( ... )
What if only SSL is required and cache is not needed? (see below)
3.1. Alternate solution using Stunnel
Stunnel is a cheaper solution than Apache+mod_ssl. It doesn’t process HTTP or add X-Forwarded-For header by default. (there is a patch on the official haproxy site to provide this feature to recent stunnel versions.)
Stunnel will only process HTTPS. Haproxy will get all HTTP traffic, so add the X-Forwarded-For header for HTTP traffic in haproxy, but not for HTTPS traffic since stunnel will already have done it.
Use the “except” keyword to tell haproxy that connections from local host already have a valid header.
192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2
-------+-----------+-----+-----+-----+--------+----
| | | | | _|_db
+--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
| LB1 | | A | | B | | C | | D | (___)
+-----+ +---+ +---+ +---+ +---+ (___)
stunnel 4 cheap web servers
haproxy
Description :
- stunnel on LB1 will receive clients requests on port 443
- it forwards them to haproxy bound to port 80
- haproxy will receive HTTP client requests on port 80 and decrypted SSL
requests from Stunnel on the same port.
- stunnel will add the X-Forwarded-For header
- haproxy will add the X-Forwarded-For header for everyone except the local
address (stunnel).
4. Soft-stop for application maintenance
4.1 Soft-stop using a file on the servers
Put a file on the server which will be checked by the proxy. Remove this file so that the proxy will treat this server as dead, and won’t send any new sessions, only old ones if the “persist” option is used. Wait a bit then stop the server when there isn’t http connection anymore. And then it’s time to do backend server maintenance.
This solution will effect the clients, not so good.
4.2 Soft-stop using backup servers
Set two different names to one server checked on different port(one is 80), they share the exact same cookies. Those servers will only be used when no other server is available for the same cookie.
When the web servers are started, only one named server is seen as available. On the web server, redirect the other different port to local port 80(e.g., use iptables).
When need maintenance, simply stop the server from responding on port 81 so that its standard instance will be seen as failed, but the other will still work. This won’t effect the clients.
4.2.1 Variations for operating systems without any firewall software
Beside the iptables solution above, this redirection can also be handled by a simple haproxy in tcp mode :
global
daemon
quiet
pidfile /var/run/haproxy-checks.pid
listen 0.0.0.0:81
mode tcp
dispatch 127.0.0.1:80
contimeout 1000
clitimeout 10000
srvtimeout 10000
Starting an haproxy instance with this configuration to start the web service, and killing this instance will make the port 81 stopping responding so that the web service could be stopped.
4.2.2 Centralizing the server management
It’s also an solution to do the port redirection on the load-balancer.
Another solution is to use the “COMAFILE” patch provided by Alexander Lazic, which is available for download here :
http://w.ods.org/tools/haproxy/contrib/
4.3 Hot reconfiguration
Send a SIGTTOU signal to the proxy and it will release the ports so that a new instance can be started.
If the new instance fails to start, sending a SIGTTIN signal back to the original processes will restore the listening ports.
Otherwise, sending a SIGUSR1 signal to the old one and it will exit after its last session ends.
If the old process still exists, sending a SIGTERM to the old process.
5. Multi-site load-balancing with local preference
5.1 Network diagram
Note : offices 1 and 2 are on the same continent as site 1, while office 3 is on the same continent as site 3. Each production site can reach the second one either through the WAN or through a dedicated link.
Office1 Office2 Office3
users users users
192.168 # # # 192.168 # # # # # #
.1.0/24 | | | .2.0/24 | | | 192.168.3.0/24 | | |
--+----+-+-+- --+----+-+-+- ---+----+-+-+-
| | .1 | | .1 | | .1
| +-+-+ | +-+-+ | +-+-+
| |OP1| | |OP2| | |OP3| ...
,-:-. +---+ ,-:-. +---+ ,-:-. +---+
( X ) ( X ) ( X )
`-:-' `-:-' ,---. `-:-'
--+---------------+------+----~~~( X )~~~~-------+---------+-
| `---' |
| |
+---+ ,-:-. +---+ ,-:-.
|SD1| ( X ) |SD2| ( X )
( SITE 1 ) +-+-+ `-:-' ( SITE 2 ) +-+-+ `-:-'
|.1 | |.1 |
10.1.1.0/24 | | ,---. 10.2.1.0/24 | |
-+-+-+-+-+-+-+-----+-+--( X )------+-+-+-+-+-+-+-----+-+--
| | | | | | | `---' | | | | | | |
...# # # # # |.11 |.12 ...# # # # # |.11 |.12
Site 1 +-+--+ +-+--+ Site 2 +-+--+ +-+--+
Local |S1L1| |S1L2| Local |S2L1| |S2L2|
users +-+--+ +--+-+ users +-+--+ +--+-+
| | | |
10.1.2.0/24 -+-+-+--+--++-- 10.2.2.0/24 -+-+-+--+--++--
|.1 |.4 |.1 |.4
+-+-+ +-+-+ +-+-+ +-+-+
|W11| ~~~ |W14| |W21| ~~~ |W24|
+---+ +---+ +---+ +---+
4 application servers 4 application servers
on site 1 on site 2
5.2 Description
5.2.1 Local users
- Office 1 users connect to OP1 = 192.168.1.1
- Office 2 users connect to OP2 = 192.168.2.1
- Office 3 users connect to OP3 = 192.168.3.1
- Site 1 users connect to SD1 = 10.1.1.1
- Site 2 users connect to SD2 = 10.2.1.1
5.2.2 Office proxies
- Office 1 connects to site 1 by default and uses site 2 as a backup.
- Office 2 connects to site 1 by default and uses site 2 as a backup.
- Office 3 connects to site 2 by default and uses site 1 as a backup.
6. Source balancing
Sometimes it may reveal useful to access servers from a pool of IP addresses instead of only one or two. Some equipments (NAT firewalls, load-balancers) are sensible to source address, and often need many sources to distribute the load evenly amongst their internal hash buckets.
7. Managing high loads on application servers
Limiting the number of connections between the clients and the servers. Setting haproxy to limit the number of connections on a per-server basis. It will then fill all the servers up to the configured connection limit, and will put the remaining connections in a queue, waiting for a connection to be released on a server.
So that,
* all clients can be served whatever their number without crashing the servers, the only impact it that the response time can be delayed.
* the servers can be used at full throttle without the risk of stalling, and fine tuning can lead to optimal performance.
* response times can be reduced by making the servers work below the congestion point, effectively leading to shorter response times even under moderate loads.
* no domino effect when a server goes down or starts up. Requests will be queued more or less, always respecting servers limits.
* it’s easy to achieve high performance even on memory-limited hardware. Indeed, heavy frameworks often consume huge amounts of RAM and not always all the CPU available. In case of wrong sizing, reducing the number of concurrent connections will protect against memory shortages while still ensuring optimal CPU usage.
Post a Comment