Multi homed host, arp flux and source base routing

If you have a linux box and multiple interfaces on the same subnet you may notice that all traffic is really going through the first interface. This is not a problem unless if you want to differentiate services per interface. For example I have a host with 2 interfaces on subnet 192.168.1.0:

eth0 : 192.168.1.1
eth1 : 192.168.1.2

Now on eth0 I want to run smb and on eth1 Alfresco's smb service.

After making sure that each service binds on a specific interface:

i) Alfresco (3.4b):

File: webapps/alfresco/WEB-INF/classes/alfresc o/subsystems/fileServers/default/file-servers.properties

cifs.enabled=true
cifs.serverName={Your hostname on that interface}
cifs.domain={a workgroup}
cifs.broadcast=192.168.1.255
cifs.bindto=192.168.1.2
cifs.tcpipSMB.port=4450
cifs.netBIOSSMB.sessionPort=1390
cifs.netBIOSSMB.namePort=1370
cifs.netBIOSSMB.datagramPort=1380

Obviously you need to adjust some values to your specific needs. If you like me run tomcat as an unprivileged user you should set higher than 1024 ports and use iptables to redirect:

iptables -t nat -F PREROUTING
iptables -t nat -A PREROUTING -i eth1 -p tcp -m tcp --dport 445 -j REDIRECT --to-ports 4450
iptables -t nat -A PREROUTING -i eth1 -p tcp -m tcp --dport 139 -j REDIRECT --to-ports 1390
iptables -t nat -A PREROUTING -i eth1 -p udp -m udp --dport 137 -j REDIRECT --to-ports 1370
iptables -t nat -A PREROUTING -i eth1 -p udp -m udp --dport 138 -j REDIRECT --to-ports 1380

(You should also add rules for INPUT if the default policy is not ACCEPT)

ii) Samba

File: /etc/samba/smb.conf

Add to global:

bind interfaces only = yes
interfaces = lo eth0

Now restarting nmb, smb and tomcat/alfresco will apply all changes. Unfortunately this may not be enough. You have to make sure that traffic for interface eth1 is delivered to it. Or more precisely that outgoing traffic is sent through eth1 when the source is 192.168.1.2

The solution is source based routing:

1) Create a table alias, here called surlatable:

echo 200 surlatable >> /etc/iproute2/rt_tables

2) Traffic with source 192.168.1.2 should be routed according with the newly created table:

ip rule add from 192.168.1.2 table surelatable

3) Local and (optionally) Default routes

ip route add 192.168.1.0/24 via 192.168.1.2 dev eth1 table surelatable
ip route add default via {GATEWAY} dev eth1 table surelatable

4) Apply changes

ip route flush cache

You may want to make these changes permanent by adding the commands in /sbin/if-up.local script (for fedora) when eth1 comes up.

Lastly fix the ARP FLUX issue by editing/adding to /etc/sysctl.conf:

net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1

Modern kernels support these flags and are compiled with Advanced routing features. You can apply these without rebooting:

echo 2 > /proc/sys/net/ipv4/conf/default/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/default/arp_ignore

Now you can verify that everything is working. Make sure to clear the arp cache of the computer you will use as a client. Then ping both interfaces and check the cache. Each address should have the correct MAC address listed.

For more details on advanced routing and much more, don't miss Linux Advanced Routing & Traffic Control HOWTO