Netfilter bug ? NAT'ed connections ignore icmp redirect

[Henrik Stoerner]

I have a setup where I use a Linux box with netfilter to forward
tcp connections between a "client" and a "server.

The Linux box has a default gateway defined. However, there are
multiple other routers on the same network, and the default
gateway router sends ICMP redirects to inform the Linux box
which router should be used to reach some destination.

In ascii art (somewhat simplified):


|----| firewall |---| server |
|
|
|----| routerA |-------
| | def.gw |
| Linux NAT | ------------|
|
|----| routerB |----------
|
|----| routerC |----+-----
| |
| |
| |
| client |


This works fine for traffic that originates on the Linux box.
If I connect from the Linux NAT box to the client, the default
gateway sends an ICMP redirect telling the Linux box to use
routerC, and all further traffic then goes via that router.
A lookup with "ip route get <client-IP>" also tells me that
traffic goes via routerC.

If the client connects to a service on the Linux NAT box, the
packets sent from the Linux NAT box to the client also obey
the routing defined by the ICMP redirect, and go via routerC.

But when the connection is initiated by the client for a port that
get's NAT'ed and forwarded to the server, then the packets going
from the Linux NAT box to the client are sent to the default
gateway, completely ignoring the route generated by the ICMP
redirect.

So:
* client sends a SYN to Linux-NAT on port 1984
* Linux-NAT does SNAT+DNAT and forwards packet to server
* server sends SYN+ACK to Linux-NAT box
* Linux-NAT box does the necessary DNAT/SNAT and forwards
SYN+ACK to client using routerA instead of via routerC

Even though routerA sends an ICMP redirect telling Linux-NAT
to use routerC, all of the packets from Linux-NAT to the client
for this connection are sent via routerA.


The iptables rules are simple. I use this in a script:

/sbin/modprobe iptable_nat
/sbin/iptables -t nat -A PREROUTING -p tcp --dport 1984 -j DNAT --to-destination IP.ADDR.OF.SERVER:1984
/sbin/iptables -t nat -A POSTROUTING -p tcp --dport 1984 -j SNAT --to-source IP.ADDR.OF.LINUX-NAT
/sbin/sysctl -w net.ipv4.ip_forward=1
for f in /proc/sys/net/ipv4/conf/*/accept_redirects; do
echo "1" >$f
done


giving me:

# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- anywhere anywhere tcp dpt:1984 to:IP.ADDR.OF.SERVER:1984

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT tcp -- anywhere anywhere tcp dpt:1984 to:IP.ADDR.OF.LINUX-NAT

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

# lsmod
Module Size Used by Not tainted
iptable_nat 15816 1
ip_conntrack 16488 1 [iptable_nat]
ip_tables 14784 3 [iptable_nat]


I've tried kernels 2.4.18 and 2.4.27. The Linux NAT box is running Debian/testing
on an UltraSparc. The 2.4.18 kernel is the default Debian kernel; the 2.4.27 was
compiled locally. The configuration is available if you need it.

[Jason Opperisano]

Post by Henrik Stoerner
I have a setup where I use a Linux box with netfilter to forward
tcp connections between a "client" and a "server.
The Linux box has a default gateway defined. However, there are
multiple other routers on the same network, and the default
gateway router sends ICMP redirects to inform the Linux box
which router should be used to reach some destination.
|----| firewall |---| server |
|
|
|----| routerA |-------
| | def.gw |
| Linux NAT | ------------|
|
|----| routerB |----------
|
|----| routerC |----+-----
| |
| |
| |
| client |

since this is primarily a "firewall guy" mailing list; i believe i am
required by law to state: do not rely on ICMP redirects to route your
network. removing my "firewall" hat for my "routing" hat now: do not
rely on ICMP redirects to route your network. removing my "routing" hat
for my
"in-the-true-spirit-of-linux-everything-i-want-to-do-should-work-the-
way-i-want-no-matter-how-many-RFC's-it-goes-against-or-how-bad-of-an-
idea-it-is" hat...i will try to explain why i think you're seeing this
behavior...

and no--i don't think this is a *bug* in netfilter, i think it's a
symptom of how linux handles ICMP redirects and the routes created by
them. if you read up on how linux treats an ICMP redirect that it
receives--it limits the scope of the resulting route to host
communications. therefore, i believe the "linuxnat" box is ignoring the
route created by the ICMP redirect because the source of the packet is
not local at the time of the routing decision.

follow this:

SYN:
ORIGINAL PACKET:
clientip.unpriv -> linuxnat.1984

PREROUTING (DNAT):
clientip.unpriv -> server.1984

*** ROUTING DECISION ***

POSTROUTING (SNAT):
linuxnat.unpriv -> server.1984

SYN-ACK:
ORIGINAL PACKET:
server.1984 -> linuxnat.unpriv

PREROUTING (UNDO SNAT):
server.1984 -> client.unpriv

*** ROUTING DECISION ***

POSTROUTING (UNDO DNAT):
linuxnat.1984 -> client.unpriv

so--at the time of the routing decision for the reply packet--the source
is the server IP, not the linuxnat machine's IP.

check the output of "ip route list table all" on linuxnat and look at
the "scope" that the route to "client" ends up under. more
importantly--see if it shows up in the output of "ip route list table
all scope global" as my guess is that it does not.

if you've actually read this far--may i humbly suggest using a dynamic
routing protocol to route your environment? zebra/quagga is pretty
painless--especially if you're familiar with cisco IOS configuration
syntax.

-j

[Henrik Stoerner]

[snip]

thanks Jason, your remark that

Post by Jason Opperisano
"in-the-true-spirit-of-linux-everything-i-want-to-do-should-work-the-
way-i-want-no-matter-how-many-RFC's-it-goes-against-or-how-bad-of-an-
idea-it-is" hat...i will try to explain why i think you're seeing this
behavior...

did make my day :-)

I wholeheartedly agree - with any hat I can put on - that relying on
ICMP redirects for this is a very bad idea. Had I been responsible for
the design of this network it would have been different, but I am not.
I work at a place where there are people designing networks who
seriously believe they know best how to set things up, and since I am
just the looney playing around with Linux I should not tell them how
to do things.

So my mail to the list was an attempt to understand why netfilter
behaves the way it does - I find it a lot easier to go into a
discussion about matters when I understand them.

Post by Jason Opperisano
and no--i don't think this is a *bug* in netfilter, i think it's a
symptom of how linux handles ICMP redirects and the routes created by
them. if you read up on how linux treats an ICMP redirect that it
receives--it limits the scope of the resulting route to host
communications.

This is the piece of the puzzle that I was missing. It explains the
behaviour I see. I'm re-reading RFC 1122 now, but would appreciate
some pointers to Linux specific docs.

Post by Jason Opperisano
if you've actually read this far--may i humbly suggest using a dynamic
routing protocol to route your environment? zebra/quagga is pretty
painless--especially if you're familiar with cisco IOS configuration
syntax.

That would be the ideal solution, I'll try if I can persuade our
network guys to implement this at the router end. If not I guess I'll
have to implement an application-layer proxy instead of using
netfilter for it, and so avoid the problem that way.


Regards,

Henrik

[Aleksandar Milivojevic]

Post by Henrik Stoerner
I have a setup where I use a Linux box with netfilter to forward
tcp connections between a "client" and a "server.
The Linux box has a default gateway defined. However, there are
multiple other routers on the same network, and the default
gateway router sends ICMP redirects to inform the Linux box
which router should be used to reach some destination.

Probably a stupid question. But. Why don't you simply define two
static routes on the NAT box pointing to firewall (packets to server)
and routerC (packets to client)? Or let the router do its job of
actually routing packets instead of generating error messages back to
clients (in which case you would really need one router with 5 interfaces).

IMHO, relying on ICMP redirects to create routes for you is inefficient
and error prone. It's kind of asking for trouble.