Balancing Connections Over Multiple Links
时间：2009-07-16 来源：齿轮
Contents
- General Idea
- Prerequisites
- Setup
- Results
- TODO
- ChangeLog
General Idea
Say you have access to multiple links to the Internet, such as several wireless networks in range. Wouldn't it be nice to combine all that bandwidth into one big fat pipe?
Unfortunately it's not so easy. You can't just trunk them together because they each have a different public IP address, gateway, etc.
What you can do however, thanks to some nifty Linux NetFilter extensions, is assign outgoing connections to different interfaces. This will allow protocols such as BitTorrent to utilize bandwidth from each of the links.
This document focuses on Linux iptables/NetFilter. You can achieve pretty much the same result with Linux Advanced Routing techniques. One small difference, as the link mentions, is that routes are cached, so connections to frequently used sites will always go over the same link. This may or may not be the behaviour you desire.
Prerequisites
You need a recent Linux kernel patched with support for the ROUTE target and either the "nth" or "random" match module. These patches are available in NetFilter's "patch-o-matic-ng" subversion module. I won't go into how to apply the patches, as more than sufficient documentation is included with them.
Testing I did was on Linux 2.6.14.2 patched with a copy of patch-o-matic-ng checked out with svn on 2005-11-18.
Setup
In the following examples, I use three interfaces:
- eth0: Wired connection, 192.168.1.0/24, gateway 192.168.1.1, default route.
- eth1: Wireless connetion 1, 172.16.0.0/16, gateway 172.16.0.1
- rausb0: Wireless connetion 2, 192.168.0.0/24, gateway 192.168.0.1
Common commands:
# prevent incoming packets on masqueraded connections from being dropped # as "martians" due to the destination address being translated before the # rp_filter check is performed echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter echo 0 > /proc/sys/net/ipv4/conf/rausb0/rp_filter # Load protocol-specific connection tracking modules so that new connections # associated with existing connections have state "RELATED" and inherit the # same connmark. modprobe ip_conntrack_ftp # masquerade outgoing connections on secondary interfaces iptables -t nat -A POSTROUTING -o eth1 -s ! 172.16.0.0/16 -m state --state NEW,RELATED -j MASQUERADE iptables -t nat -A POSTROUTING -o rausb0 -s ! 192.168.0.0/24 -m state --state NEW,RELATED -j MASQUERADE # create a chain for processing new outgoing connetions iptables -t mangle -N NEW_OUT_CONN # Skip connections we want to always go out wired interface iptables -t mangle -A NEW_OUT_CONN -d 192.168.1.0/24 -j RETURN iptables -t mangle -A NEW_OUT_CONN -p tcp -m multiport --destination-ports 21,22,80,443,6667 -j RETURN iptables -t mangle -A NEW_OUT_CONN -p udp --dport 53 -j RETURN # have new outgoing connections pass through the above chain iptables -t mangle -A OUTPUT -o eth0 -m state --state NEW -j NEW_OUT_CONN # send packets out chosen interface iptables -t mangle -A OUTPUT -m connmark --mark 2 -j ROUTE --gw 172.16.0.1 --continue iptables -t mangle -A OUTPUT -m connmark --mark 3 -j ROUTE --gw 192.168.0.1 --continue
The "random" method:
# 34% of the time go out the default interface iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 0 iptables -t mangle -A NEW_OUT_CONN -m random --average 34 -j RETURN # 33% of the time go out eth1 (50% of the remaining probability) iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 2 iptables -t mangle -A NEW_OUT_CONN -m random --average 50 -j RETURN # else (hopefully 33% of the time) go out rausb0 iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 3
The "nth" method:
# 1st of every 3 connections goes out the default interface iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 0 iptables -t mangle -A NEW_OUT_CONN -m nth --counter 1 --every 3 --packet 0 -j RETURN # 2nd of every 3 connections goes out eth1 iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 2 iptables -t mangle -A NEW_OUT_CONN -m nth --counter 1 --every 3 --packet 1 -j RETURN # 3rd of every 3 connections goes out rausb0 iptables -t mangle -A NEW_OUT_CONN -j CONNMARK --set-mark 3 iptables -t mangle -A NEW_OUT_CONN -m nth --counter 1 --every 3 --packet 2 -j RETURN
Handling when an interface goes down:
This script will make sure no packets get routed over a secondary interface that has gone down. Put it in your /etc/network/if-down.d/ (Debian), or equivalent, directory and chmod +x it.
#!/bin/sh if [ "$IFACE" = "eth1" ]; then iptables -t mangle -D OUTPUT -m connmark --mark 2 -j ROUTE --gw 172.16.0.1 --continue 2>/dev/null fi if [ "$IFACE" = "rausb0" ]; then iptables -t mangle -D OUTPUT -m connmark --mark 3 -j ROUTE --gw 192.168.0.1 --continue 2>/dev/null fi exit 0
Handling when an interface comes back up:
This script will allow an interface to be used again when it comes back up. Put it in your /etc/network/if-up.d/ (Debian), or equivalent, directory and chmod +x it.
#!/bin/sh if [ "$IFACE" = "eth1" ]; then iptables -t mangle -A OUTPUT -m connmark --mark 2 -j ROUTE --gw 172.16.0.1 --continue 2>/dev/null fi if [ "$IFACE" = "rausb0" ]; then iptables -t mangle -A OUTPUT -m connmark --mark 3 -j ROUTE --gw 192.168.0.1 --continue 2>/dev/null fi exit 0
Results
Not too shaby I think. Normally with my single DSL connection alone I get somewhere around 150 KB/s. Maybe if there's a smarter way to distribute connections this could be improved upon. Optimally my 3 test links combined would add up to 450 KB/s.
- Write scripts to make setting all this up a snap.
- Figure out a way to translate outgoing FTP 'PORT' commands for all links.
ChangeLog
Mon Jan 2 05:43:47 PST 2006
Michael Heimpold pointed out that --average 33 was wrong for the second -m random rule.
Fri Jan 13 09:37:12 PST 2006
Michael Heimpold figured out that RELATED connections (set as such by modules like ip_conntrack_ftp) inherit the same connmark. Changed the masquerading rules to also match RELATED packets. Now passive FTP works reliably.
相关阅读 更多 +
