Discussion:
protocol 50 unreachable
Helge Weissig
2004-12-01 22:51:51 UTC
Permalink
Sorry for the cross-post, but this problem is really nagging me. What I
did not put into the post below is the fact that it only occurred after a
reboot of my linux system due to a short power outage. Here is the routing
table, if that makes any difference:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
xx.xx.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 xx.xx.xx.1 0.0.0.0 UG 0 0 0 eth0

the xx.xx.xx is the first part of my external ip address.

thanks for any advice or insight!
h.



From: ***@grajagan.org (Helge Weissig)
Newsgroups: comp.security.firewalls
Subject: protocol 50 unreachable
NNTP-Posting-Host: 63.196.131.66
Message-ID: <***@posting.google.com>

Hi,

I have been searching for information about this problem high and low
but came up dry. Basically, I am trying to connect to a VPN server via
ipsec from behind a NAT firewall set up on a Linux (kernel 2.4.x) box
with iptables. I have no problem establishing the connection via port
500 as this is initiated by the client. However, I cannot seem to get
protocol 50 (ESP) to work, independent of whether the ipsec tunnel is
established or not. I have tried every incantation of iptables rules I
could find, to no avail. When I set up tcdump on both interfaces on my
server as well as on the client behind it, a port I have opened for
forwarding responds as expected. If I run 'nmap -sO' from somewhere
outside however, it will report protocol 50 as open although the
external interface reports a 'icmp: xx.xx.xx.xx protocol 50
unreachable' response and the two other interfaces never see the
traffic.

Here is my current iptables configuration

$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F

echo "Enabling PORTFW Redirection on the external LAN.."
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p esp -j ACCEPT
$IPTABLES -A PREROUTING -t nat -d $VPN_SERVER -p esp -j DNAT \
--to-destination $VPN_CLIENT

echo " FWD: Allow all connections OUT and only existing and related
ones IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG

echo " Enabling SNAT (MASQUERADE) functionality on $EXTIF"
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

here is the tcpdump info I see on $EXTIF:

10:23:09.234937 (vpn server ip) > (my ip): ESP(spi=0x00000000,seq=0x0)
10:23:09.235055 (my ip) > (vpn server ip): icmp: (my ip) protocol 50
unreachable [tos 0xc0]

(these are empty packets sent by nmap but it looks the same for legit
ones coming from the vpn server ip). FWIW, when the ipsec tunnel is
established and I try to ping the a host behind the vpn server, I see
the outgoing packets on all three interfaces, but not response.

thanks for any information or pointers in advance!
h.
John A. Sullivan III
2004-12-01 23:59:51 UTC
Permalink
Post by Helge Weissig
Sorry for the cross-post, but this problem is really nagging me. What I
did not put into the post below is the fact that it only occurred after a
reboot of my linux system due to a short power outage. Here is the routing
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
xx.xx.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 xx.xx.xx.1 0.0.0.0 UG 0 0 0 eth0
the xx.xx.xx is the first part of my external ip address.
thanks for any advice or insight!
h.
Newsgroups: comp.security.firewalls
Subject: protocol 50 unreachable
NNTP-Posting-Host: 63.196.131.66
Hi,
I have been searching for information about this problem high and low
but came up dry. Basically, I am trying to connect to a VPN server via
ipsec from behind a NAT firewall set up on a Linux (kernel 2.4.x) box
with iptables. I have no problem establishing the connection via port
500 as this is initiated by the client. However, I cannot seem to get
protocol 50 (ESP) to work, independent of whether the ipsec tunnel is
established or not. I have tried every incantation of iptables rules I
could find, to no avail. When I set up tcdump on both interfaces on my
server as well as on the client behind it, a port I have opened for
forwarding responds as expected. If I run 'nmap -sO' from somewhere
outside however, it will report protocol 50 as open although the
external interface reports a 'icmp: xx.xx.xx.xx protocol 50
unreachable' response and the two other interfaces never see the
traffic.
Here is my current iptables configuration
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F
echo "Enabling PORTFW Redirection on the external LAN.."
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p esp -j ACCEPT
$IPTABLES -A PREROUTING -t nat -d $VPN_SERVER -p esp -j DNAT \
--to-destination $VPN_CLIENT
echo " FWD: Allow all connections OUT and only existing and related
ones IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG
echo " Enabling SNAT (MASQUERADE) functionality on $EXTIF"
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
10:23:09.234937 (vpn server ip) > (my ip): ESP(spi=0x00000000,seq=0x0)
10:23:09.235055 (my ip) > (vpn server ip): icmp: (my ip) protocol 50
unreachable [tos 0xc0]
(these are empty packets sent by nmap but it looks the same for legit
ones coming from the vpn server ip). FWIW, when the ipsec tunnel is
established and I try to ping the a host behind the vpn server, I see
the outgoing packets on all three interfaces, but not response.
thanks for any information or pointers in advance!
h.
Silly question but, since the problem started after a reboot, are you
sure that ESP is running on your client? Are you using *swan or the
native 2.6 IPSec implementation on the client?
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
***@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
Helge Weissig
2004-12-02 00:07:02 UTC
Permalink
Thanks for getting back to me, John. Here are some more details and an
attempt at clarification:

The client is a laptop running MacOS X and VPN Tracker. It sits on the
private LAN and as I mentioned in the post, I can see outgoing traffic
(e.g. pings of VPN hosts) going *out* through ESP (i.e. all three
interfaces - client, internal and external - report ESP packets going
out). The client s/w log also indicates "ESP tunnel established".
Regardless though, if I just try to get any ESP traffic *into* the client,
from outside the firewall, it only shows up on the external interface and
triggers the described response.

cheers,
h.

On Wed, 1 Dec 2004 at 18:59 -0500, John A. Sullivan III wrote:

[snip]
JASI> Silly question but, since the problem started after a reboot, are
JASI> you sure that ESP is running on your client? Are you using *swan or
JASI> the native 2.6 IPSec implementation on the client?
Jason Opperisano
2004-12-02 00:29:00 UTC
Permalink
On Wed, 2004-12-01 at 17:51, Helge Weissig wrote:
<snip>
Post by Helge Weissig
Here is my current iptables configuration
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F
echo "Enabling PORTFW Redirection on the external LAN.."
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p esp -j ACCEPT
$IPTABLES -A PREROUTING -t nat -d $VPN_SERVER -p esp -j DNAT \
--to-destination $VPN_CLIENT
what on earth is that rule supposed to accomplish? it's says "any esp
packet destined for $VPN_SERVER should be destination translated to
$VPN_CLIENT."

if this is the firewall in front of $VPN_CLIENT (which it sounds like it
is), you have created (for lack of a better term) a packet reflector.
any esp packet sent from VPN client to VPN server will be spit back at
the VPN client.
Post by Helge Weissig
echo " FWD: Allow all connections OUT and only existing and related
ones IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG
echo " Enabling SNAT (MASQUERADE) functionality on $EXTIF"
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
10:23:09.234937 (vpn server ip) > (my ip): ESP(spi=0x00000000,seq=0x0)
10:23:09.235055 (my ip) > (vpn server ip): icmp: (my ip) protocol 50
unreachable [tos 0xc0]
(these are empty packets sent by nmap but it looks the same for legit
ones coming from the vpn server ip). FWIW, when the ipsec tunnel is
established and I try to ping the a host behind the vpn server, I see
the outgoing packets on all three interfaces, but not response.
thanks for any information or pointers in advance!
h.
simplify:

# start fresh
for t in mangle nat filter; do
iptables -t $t -F
iptables -t $t -X
iptables -t $t -Z
done
for c in INPUT FORWARD OUTPUT; do
iptables -P $c ACCEPT
done

# hide-nat outbound traffic
iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# ip forwarding
sysctl -w net.ipv4.ip_forward=1

try and connect with your VPN client to your VPN server with that
script. if you can't connect--it's more likely an IPSec configuration
detail that your missing.

-j

--
"This has purple stuff inside - purple is a fruit."
--The Simpsons
Helge Weissig
2004-12-02 03:29:19 UTC
Permalink
On Wed, 1 Dec 2004 at 19:29 -0500, netfilter-***@lists.netfilter.org wrote:

JO> On Wed, 2004-12-01 at 17:51, Helge Weissig wrote:
JO> <snip>
JO> > Here is my current iptables configuration
JO> >
JO> > $IPTABLES -P INPUT ACCEPT
JO> > $IPTABLES -F INPUT
JO> > $IPTABLES -P OUTPUT ACCEPT
JO> > $IPTABLES -F OUTPUT
JO> > $IPTABLES -P FORWARD DROP
JO> > $IPTABLES -F FORWARD
JO> > $IPTABLES -t nat -F
JO> >
JO> > echo "Enabling PORTFW Redirection on the external LAN.."
JO> > $IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p esp -j ACCEPT
JO> > $IPTABLES -A PREROUTING -t nat -d $VPN_SERVER -p esp -j DNAT \
JO> > --to-destination $VPN_CLIENT
JO>
JO> what on earth is that rule supposed to accomplish? it's says "any esp
JO> packet destined for $VPN_SERVER should be destination translated to
JO> $VPN_CLIENT."
[snip]

my bad... when I obfuscated my script, I should have used "$EXT_IP" or
something like it. That IP and the VPN server's are very similar.

[snip]
JO> simplify:
JO>
JO> # start fresh
JO> for t in mangle nat filter; do
JO> iptables -t $t -F
JO> iptables -t $t -X
JO> iptables -t $t -Z
JO> done
JO> for c in INPUT FORWARD OUTPUT; do
JO> iptables -P $c ACCEPT
JO> done
JO>
JO> # hide-nat outbound traffic
JO> iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
JO>
JO> # ip forwarding
JO> sysctl -w net.ipv4.ip_forward=1
JO>
JO> try and connect with your VPN client to your VPN server with that
JO> script. if you can't connect--it's more likely an IPSec configuration
JO> detail that your missing.

no such luck :(. I should note that the VPN connections works fine when I
hook the client up directly to my DSL line. btw - it looks like your
script does not forward anything from one of my interfaces to the other.

cheers,
h.
Jason Opperisano
2004-12-02 03:46:00 UTC
Permalink
Post by Helge Weissig
my bad... when I obfuscated my script, I should have used "$EXT_IP" or
something like it. That IP and the VPN server's are very similar.
ok...
Post by Helge Weissig
JO>
JO> try and connect with your VPN client to your VPN server with that
JO> script. if you can't connect--it's more likely an IPSec configuration
JO> detail that your missing.
no such luck :(. I should note that the VPN connections works fine when I
hook the client up directly to my DSL line. btw - it looks like your
script does not forward anything from one of my interfaces to the other.
yeah--precisely. you seem obsessed with the desire to "port forward"
esp traffic to your VPN client, which is absolutely not necessary.

look into configuring NAT-T with your VPN client, sometimes called "UDP
Encapsulation" as your VPN server appears unwilling to accept esp
packets that have traversed an intermediate NAT device.

-j

--
"Mmmm...free goo."
--The Simpsons
Helge Weissig
2004-12-02 04:00:48 UTC
Permalink
On Wed, 1 Dec 2004 at 22:46 -0500, Jason Opperisano wrote:

JO> > no such luck :(. I should note that the VPN connections works fine when I
JO> > hook the client up directly to my DSL line. btw - it looks like your
JO> > script does not forward anything from one of my interfaces to the other.
JO>
JO> yeah--precisely. you seem obsessed with the desire to "port forward"
JO> esp traffic to your VPN client, which is absolutely not necessary.
JO>
JO> look into configuring NAT-T with your VPN client, sometimes called "UDP
JO> Encapsulation" as your VPN server appears unwilling to accept esp
JO> packets that have traversed an intermediate NAT device.

hmm... how does a packet know it needs to go from my external NIC to my
internal NIC if it comes through ESP? Maybe I am confused here...

let's leave the VPN client/server out of the picture to simplify. If I
send an ESP packet from somewhere to my external IP address I get the
"protocol 50 unreachable" ICMP response. The underlying problem seems to
be the primary cause of my troubles, no?

h.
John A. Sullivan III
2004-12-02 04:09:25 UTC
Permalink
Post by Helge Weissig
JO> > no such luck :(. I should note that the VPN connections works fine when I
JO> > hook the client up directly to my DSL line. btw - it looks like your
JO> > script does not forward anything from one of my interfaces to the other.
JO>
JO> yeah--precisely. you seem obsessed with the desire to "port forward"
JO> esp traffic to your VPN client, which is absolutely not necessary.
JO>
JO> look into configuring NAT-T with your VPN client, sometimes called "UDP
JO> Encapsulation" as your VPN server appears unwilling to accept esp
JO> packets that have traversed an intermediate NAT device.
hmm... how does a packet know it needs to go from my external NIC to my
internal NIC if it comes through ESP? Maybe I am confused here...
let's leave the VPN client/server out of the picture to simplify. If I
send an ESP packet from somewhere to my external IP address I get the
"protocol 50 unreachable" ICMP response. The underlying problem seems to
be the primary cause of my troubles, no?
h.
Yes, you should be able to get this to work as long as there is only one
station behind the NAT gateway using IPSec. NAT traversal is a valid
way to go and the only way to go if you have more than one IPSec client
using the same public address. I do assume that the NAT gateway is not
running IPSec.

To use NAT traversal, you would forward the appropriate UDP port
(typically 4500 or 500) rather than ip/50.

I do not know why your NAT gateway is refusing to pass the IPSec
packets. That's why I suggest logging in my previous e-mail is
clarifying the DNAT interface does not work. Good luck - John
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
***@nexusmgmt.com
Jason Opperisano
2004-12-02 04:12:25 UTC
Permalink
Post by Helge Weissig
hmm... how does a packet know it needs to go from my external NIC to my
internal NIC if it comes through ESP? Maybe I am confused here...
state tracking. in a VPN client -> VPN server scenario the esp packets
are *always* initiated from the client side:

iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i $INSIDE_IF -p 50 -j ACCEPT

iptables -t nat -A POSTROUTING -o $OUTSIDE_IF -j MASQUERADE

that's it...
Post by Helge Weissig
let's leave the VPN client/server out of the picture to simplify. If I
send an ESP packet from somewhere to my external IP address I get the
"protocol 50 unreachable" ICMP response. The underlying problem seems to
be the primary cause of my troubles, no?
um--no. i have run pretty much every single VPN client/server
combination in existence, and have never even thought of testing it by
sending unsolicited IP Protocol 50 packets through a firewall to the VPN
client behind it. it's the most flawed troubleshooting technique i've
heard in quite some time...save yourself some time and give up on it.

to troubleshoot VPN client/server connection problems--use the debug
logs on either side. use tcpdump. do both the client and server agree
that both Phase 1 and Phase 2 key exchanges complete? is there a
routing problem on either side of the tunnel? are packets successfully
encrypted and sent from the client? successfully received and decrypted
by the server? where do they go from there? do the replies get
encrypted and sent back from the server? received and decrypted by the
client?

debug at each link in the chain--don't poke at the chain from afar.

-j

--
"Dear Baby, Welcome to Dumpsville. Population: You"
--The Simpsons
Helge Weissig
2004-12-02 04:53:55 UTC
Permalink
On Wed, 1 Dec 2004 at 23:12 -0500, Jason Opperisano wrote:

JO> On Wed, 2004-12-01 at 23:00, Helge Weissig wrote:
JO> > hmm... how does a packet know it needs to go from my external NIC to my
JO> > internal NIC if it comes through ESP? Maybe I am confused here...
JO>
JO> state tracking. in a VPN client -> VPN server scenario the esp packets
JO> are *always* initiated from the client side:

my tcpdump on the external interface shows me packets coming on ESP from
the server every ten seconds.

JO>
JO> iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
JO> iptables -A FORWARD -i $INSIDE_IF -p 50 -j ACCEPT
JO>
JO> iptables -t nat -A POSTROUTING -o $OUTSIDE_IF -j MASQUERADE
JO>
JO> that's it...
JO>
JO> > let's leave the VPN client/server out of the picture to simplify. If I
JO> > send an ESP packet from somewhere to my external IP address I get the
JO> > "protocol 50 unreachable" ICMP response. The underlying problem seems to
JO> > be the primary cause of my troubles, no?
JO>
JO> um--no. i have run pretty much every single VPN client/server
JO> combination in existence, and have never even thought of testing it by
JO> sending unsolicited IP Protocol 50 packets through a firewall to the VPN
JO> client behind it. it's the most flawed troubleshooting technique i've
JO> heard in quite some time...save yourself some time and give up on it.

can you be more specific about why you would not expect ESP forwarding to
work without an IPsec tunnel in place? I am really trying to understand
this since most suggestions (including the ipchains rules in the netfilter
documentation) include some FORWARD rules for ip/50.

JO> to troubleshoot VPN client/server connection problems--use the debug
JO> logs on either side. use tcpdump. do both the client and server agree
JO> that both Phase 1 and Phase 2 key exchanges complete? is there a
JO> routing problem on either side of the tunnel? are packets successfully
JO> encrypted and sent from the client? successfully received and decrypted
JO> by the server? where do they go from there? do the replies get
JO> encrypted and sent back from the server? received and decrypted by the
JO> client?
JO>
JO> debug at each link in the chain--don't poke at the chain from afar.

the server and client think that the tunnel is established.

I changed my iptables script to the following now to get some logging
information, but I think I may need help with it:

$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F

$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p 50 -j LOG --log-level info \
--log-prefix 'esp fwd: '
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p 50 -j ACCEPT
$IPTABLES -A PREROUTING -t nat -p 50 -i $EXTIF -j LOG --log-level info \
--log-prefix 'esp route: '
$IPTABLES -A PREROUTING -t nat -p 50 -i $EXTIF -j DNAT \
--to-destination $VPN_CLIENT

$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG --log-level info --log-prefix 'dropped: '
$IPTABLES -A FORWARD -j DROP

$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j LOG --log-level info \
--log-prefix 'masq: '
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

Basically, I only ever see log entries from the postrouting chain. I also
do not see ESP packets going from INTIF to EXTIF contrary to what I
thought. So with the VPN "up" (how do I check for encryption for example)
and packets coming in from the server, they don't get to INTIF. Pinging a
host on VPN shows ESP packets getting to INTIF but no further. If I ping
the server itself from the VPN client, i.e. not through the tunnel, all
three interfaces report outgoing and incoming traffic.

thanks for your input so far.

h.
John A. Sullivan III
2004-12-02 05:15:47 UTC
Permalink
<snip>
Post by Helge Weissig
JO> state tracking. in a VPN client -> VPN server scenario the esp packets
I don't believe that is true. The tunnel is always initiated from the
client side but the traffic in the tunnel can be initiated from the
remote side.
<snip>
Post by Helge Weissig
JO> > let's leave the VPN client/server out of the picture to simplify. If I
JO> > send an ESP packet from somewhere to my external IP address I get the
JO> > "protocol 50 unreachable" ICMP response. The underlying problem seems to
JO> > be the primary cause of my troubles, no?
JO>
JO> um--no. i have run pretty much every single VPN client/server
JO> combination in existence, and have never even thought of testing it by
JO> sending unsolicited IP Protocol 50 packets through a firewall to the VPN
JO> client behind it. it's the most flawed troubleshooting technique i've
JO> heard in quite some time...save yourself some time and give up on it.
Why? Helge's methodology is simplifying matters. He's not testing end
to end connectivity. He's testing only one small part, viz., will the
NAT gateway properly forward the packet. It doesn't matter if the
client won't accept it at this stage of testing.
Post by Helge Weissig
can you be more specific about why you would not expect ESP forwarding to
work without an IPsec tunnel in place? I am really trying to understand
this since most suggestions (including the ipchains rules in the netfilter
documentation) include some FORWARD rules for ip/50.
It should work. It won't be a tunnel but you should see the esp packet
pass through the gateway.
<snip>
Post by Helge Weissig
I changed my iptables script to the following now to get some logging
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p 50 -j LOG --log-level info \
--log-prefix 'esp fwd: '
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p 50 -j ACCEPT
$IPTABLES -A PREROUTING -t nat -p 50 -i $EXTIF -j LOG --log-level info \
--log-prefix 'esp route: '
$IPTABLES -A PREROUTING -t nat -p 50 -i $EXTIF -j DNAT \
--to-destination $VPN_CLIENT
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG --log-level info --log-prefix 'dropped: '
$IPTABLES -A FORWARD -j DROP
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j LOG --log-level info \
--log-prefix 'masq: '
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
Basically, I only ever see log entries from the postrouting chain. I also
do not see ESP packets going from INTIF to EXTIF contrary to what I
thought. So with the VPN "up" (how do I check for encryption for example)
and packets coming in from the server, they don't get to INTIF. Pinging a
host on VPN shows ESP packets getting to INTIF but no further. If I ping
the server itself from the VPN client, i.e. not through the tunnel, all
three interfaces report outgoing and incoming traffic.
<snip>
Interesting. You've showed us what you think you have. Would you
kindly show us what you do have.
Do a
iptables -v -n -t nat -L
iptables -v -n -L FORWARD
iptables -v -n -L INPUT

If I'm following this correctly, the esp packets should hit your NAT
rule and have the destination changed. It should then hit the routing
table which will determine that the packet is not local and send it to
the FORWARD chain. If you are not logging packets on the FORWARD chain,
perhaps the match is wrong. Try a generic log rule for all packets and
see what the packets look like on the FORWARD chain. Do they match the
rules dumped in the above list? - John
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
***@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
Helge Weissig
2004-12-02 05:44:13 UTC
Permalink
On Thu, 2 Dec 2004 at 00:15 -0500, John A. Sullivan III wrote:

[snip]
JASI> Interesting. You've showed us what you think you have. Would you
JASI> kindly show us what you do have.
JASI> Do a
JASI> iptables -v -n -t nat -L
JASI> iptables -v -n -L FORWARD
JASI> iptables -v -n -L INPUT
JASI>
JASI> If I'm following this correctly, the esp packets should hit your NAT
JASI> rule and have the destination changed. It should then hit the routing
JASI> table which will determine that the packet is not local and send it to
JASI> the FORWARD chain. If you are not logging packets on the FORWARD chain,
JASI> perhaps the match is wrong. Try a generic log rule for all packets and
JASI> see what the packets look like on the FORWARD chain. Do they match the
JASI> rules dumped in the above list? - John

ok... I now have:

[***@gollum ~]# iptables -v -n -t nat -L
Chain PREROUTING (policy ACCEPT 1380 packets, 95540 bytes)
pkts bytes target prot opt in out source destination
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 6 prefix `all preroute: '
0 0 LOG esp -- eth0 * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 6 prefix `esp route: '
0 0 DNAT esp -- eth0 * 0.0.0.0/0 0.0.0.0/0 to:10.0.0.200

Chain POSTROUTING (policy ACCEPT 586 packets, 45154 bytes)
pkts bytes target prot opt in out source destination
0 0 LOG all -- * eth0 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 6 prefix `masq: '
0 0 MASQUERADE all -- * eth0 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 1207 packets, 93374 bytes)
pkts bytes target prot opt in out source destination
[***@gollum ~]# iptables -v -n -L FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 6 prefix `all fwd: '
0 0 LOG esp -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 6 prefix `esp fwd: '
0 0 ACCEPT esp -- eth0 eth1 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT all -- eth1 eth0 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 6 prefix `dropped: '
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
[***@gollum ~]# iptables -v -n -L INPUT
Chain INPUT (policy ACCEPT 10 packets, 1176 bytes)
pkts bytes target prot opt in out source destination
[***@gollum ~]#

and the log shows:

VPN tunnel establishment:
Dec 1 21:34:10 gollum kernel: all preroute: IN=eth1 OUT= MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=104 TOS=0x00 PREC=0x00 TTL=64 ID=8317 PROTO=UDP SPT=500 DPT=500 LEN=84
Dec 1 21:34:10 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=104 TOS=0x00 PREC=0x00 TTL=63 ID=8317 PROTO=UDP SPT=500 DPT=500 LEN=84
Dec 1 21:34:10 gollum kernel: masq: IN= OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=104 TOS=0x00 PREC=0x00 TTL=63 ID=8317 PROTO=UDP SPT=500 DPT=500 LEN=84
Dec 1 21:34:16 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=274 TOS=0x00 PREC=0x00 TTL=63 ID=8322 PROTO=UDP SPT=500 DPT=500 LEN=254
Dec 1 21:34:16 gollum kernel: all fwd: IN=eth0 OUT=eth1 SRC=(VPN SERVER IP) DST=10.0.0.200 LEN=356 TOS=0x00 PREC=0x00 TTL=49 ID=19301 PROTO=UDP SPT=500 DPT=500 LEN=336
Dec 1 21:34:17 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=80 TOS=0x00 PREC=0x00 TTL=63 ID=8328 PROTO=UDP SPT=500 DPT=500 LEN=60
Dec 1 21:34:18 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=112 TOS=0x00 PREC=0x00 TTL=63 ID=8332 PROTO=UDP SPT=500 DPT=500 LEN=92
Dec 1 21:34:18 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=256 TOS=0x00 PREC=0x00 TTL=63 ID=8335 PROTO=UDP SPT=500 DPT=500 LEN=236
Dec 1 21:34:18 gollum kernel: all fwd: IN=eth0 OUT=eth1 SRC=(VPN SERVER IP) DST=10.0.0.200 LEN=192 TOS=0x00 PREC=0x00 TTL=49 ID=19304 PROTO=UDP SPT=500 DPT=500 LEN=172
Dec 1 21:34:19 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=88 TOS=0x00 PREC=0x00 TTL=63 ID=8341 PROTO=UDP SPT=500 DPT=500 LEN=68
Dec 1 21:34:19 gollum kernel: all fwd: IN=eth0 OUT=eth1 SRC=(VPN SERVER IP) DST=10.0.0.200 LEN=104 TOS=0x00 PREC=0x00 TTL=49 ID=19305 PROTO=UDP SPT=500 DPT=500 LEN=84

ping of VPN host:
Dec 1 21:38:37 gollum kernel: all preroute: IN=eth1 OUT= MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=64 ID=8977 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:37 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=63 ID=8977 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:37 gollum kernel: masq: IN= OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=63 ID=8977 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:38 gollum kernel: all preroute: IN=eth1 OUT= MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=64 ID=8981 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:38 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=63 ID=8981 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:38 gollum kernel: masq: IN= OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=63 ID=8981 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:39 gollum kernel: all preroute: IN=eth1 OUT= MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=64 ID=8986 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:39 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=63 ID=8986 PROTO=ESP SPI=0x2e9dc0d2
Dec 1 21:38:39 gollum kernel: masq: IN= OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=63 ID=8986 PROTO=ESP SPI=0x2e9dc0d2

ping of VPN server:
Dec 1 21:42:28 gollum kernel: all preroute: IN=eth1 OUT= MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9650 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=0
Dec 1 21:42:28 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=9650 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=0
Dec 1 21:42:28 gollum kernel: masq: IN= OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=9650 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=0
Dec 1 21:42:29 gollum kernel: all fwd: IN=eth0 OUT=eth1 SRC=(VPN SERVER IP) DST=10.0.0.200 LEN=84 TOS=0x00 PREC=0x00 TTL=49 ID=19796 PROTO=ICMP TYPE=0 CODE=0 ID=6817 SEQ=0
Dec 1 21:42:29 gollum kernel: all preroute: IN=eth1 OUT= MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9656 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=1
Dec 1 21:42:29 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=9656 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=1
Dec 1 21:42:29 gollum kernel: masq: IN= OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=9656 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=1
Dec 1 21:42:30 gollum kernel: all fwd: IN=eth0 OUT=eth1 SRC=(VPN SERVER IP) DST=10.0.0.200 LEN=84 TOS=0x00 PREC=0x00 TTL=49 ID=19797 PROTO=ICMP TYPE=0 CODE=0 ID=6817 SEQ=1
Dec 1 21:42:30 gollum kernel: all preroute: IN=eth1 OUT= MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9662 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=2
Dec 1 21:42:30 gollum kernel: all fwd: IN=eth1 OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=9662 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=2
Dec 1 21:42:30 gollum kernel: masq: IN= OUT=eth0 SRC=10.0.0.200 DST=(VPN SERVER IP) LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=9662 PROTO=ICMP TYPE=8 CODE=0 ID=6817 SEQ=2
Dec 1 21:42:31 gollum kernel: all fwd: IN=eth0 OUT=eth1 SRC=(VPN SERVER IP) DST=10.0.0.200 LEN=84 TOS=0x00 PREC=0x00 TTL=49 ID=19798 PROTO=ICMP TYPE=0 CODE=0 ID=6817 SEQ=2

otherwise... silence :( I don't see ping replies through the tunnel or the
server side initiated packets

h.
Jason Opperisano
2004-12-02 15:14:47 UTC
Permalink
<snip>

do you control the VPN server? it looks like the packets from your
client are getting dropped at that end either pre or post decrypting
(most likely post). what is the VPN server?

-j

--
"English - Who needs that? I'm never going to England!"
--The Simpsons
Helge Weissig
2004-12-02 15:13:59 UTC
Permalink
Jason,

my ESP packets do not go from the external interface to the internal
one and vice versa. The connection to the VPN server works when I hook
up directly with no changes other than the IP of the client. I cannot
see how this would be a problem with the VPN network at all.

h.
Post by Jason Opperisano
<snip>
do you control the VPN server? it looks like the packets from your
client are getting dropped at that end either pre or post decrypting
(most likely post). what is the VPN server?
-j
--
"English - Who needs that? I'm never going to England!"
--The Simpsons
Jason Opperisano
2004-12-02 17:25:31 UTC
Permalink
Post by Helge Weissig
Jason,
my ESP packets do not go from the external interface to the internal
one and vice versa. The connection to the VPN server works when I hook
up directly with no changes other than the IP of the client. I cannot
see how this would be a problem with the VPN network at all.
h.
looking at your logs--all your ESP packets are from client->server.
you don't have a single ESP packet from server->client. so when you
say, "my ESP packets do not go from the external interface..." you are
ignoring the fact that there are no ESP packets ever getting to your
external interface.

which brings me back to what i said several replies ago:

your VPN server is discarding the ESP packets from your client as a
result of the mangling of your intermediate NAT device.

either make the VPN server more tolerant, or use NAT-T on your client.

-j

--
"Ah, good ol' trustworthy beer. My love for you will never die."
--The Simpsons
Helge Weissig
2004-12-02 18:22:27 UTC
Permalink
The iptable logs are not complete and as I mentioned, I may need help with
setting that up. I can see the packets coming from the server with tcpdump
as I showed in my original post but then an immediate reply is sent back
and nothing goes through to the internal interface. The same thing happens
when I use nmap to scan ip protocols. Conversely, my internal ESP traffic
ends at the internal interface of my firewall. It never reaches the
external interface or the outside. TCP traffic works fine as you can see
from the ping logs from the internal client. Could this indicate that
there is a problem before anything gets to iptables?

h.

On Thu, 2 Dec 2004 at 12:25 -0500, netfilter-***@lists.netfilter.org wrote:

JO> On Thu, Dec 02, 2004 at 07:13:59AM -0800, Helge Weissig wrote:
JO> > Jason,
JO> >
JO> > my ESP packets do not go from the external interface to the internal
JO> > one and vice versa. The connection to the VPN server works when I hook
JO> > up directly with no changes other than the IP of the client. I cannot
JO> > see how this would be a problem with the VPN network at all.
JO> >
JO> > h.
JO>
JO> looking at your logs--all your ESP packets are from client->server.
JO> you don't have a single ESP packet from server->client. so when you
JO> say, "my ESP packets do not go from the external interface..." you are
JO> ignoring the fact that there are no ESP packets ever getting to your
JO> external interface.
JO>
JO> which brings me back to what i said several replies ago:
JO>
JO> your VPN server is discarding the ESP packets from your client as a
JO> result of the mangling of your intermediate NAT device.
JO>
JO> either make the VPN server more tolerant, or use NAT-T on your client.
JO>
JO> -j
JO>
JO> --
JO> "Ah, good ol' trustworthy beer. My love for you will never die."
JO> --The Simpsons
JO>
John A. Sullivan III
2004-12-02 18:54:36 UTC
Permalink
I'm afraid I'm up to my eyeballs today so I won't be able to help very
much but let me see if I understand what we have found thus far. Am I
correct to assume that there are actually four devices involved and they
lay out like this:

VPN_HOST
|
|
VPN_SERVER
|
|
INTERNET
|
|
NAT_GATEWAY
|
|
VPN_CLIENT

It appears that we do successfully establish a tunnel between VPN_CLIENT
and VPN_SERVER. When we ping from VPN_CLIENT to VPN_HOST, we see the
esp packets pass through PREROUTING, FORWARD and POSTROUTING. I'd
imagine we do but I would confirm that we see them on eth0.

I believe your troubleshooting methodology of examining inbound esp
traffic apart from the tunnel is valid but, like Jason, my instinct
shows a different troubleshooting path. If you have control of the
VPN_SERVER, I would troubleshoot the remote side.

If the packets are leaving NAT_GATEWAY eth0, are they arriving at
VPN_SERVER eth0 (assuming eth0 is the public facing interface)?
If they are, do we see the ping packets going out VPN_SERVER eth1?
If we do, do we see the reply ping packets on VPN_SERVER eth1?
If we do, do we see the reply esp packets on VPN_SERVER eth0?
If we do, do we see the reply esp packets on NAT_GATEWAY eth0?
If we do, then we start following the packet's progress through
netfilter on NAT_GATEWAY.
I would suggest a general log rule in front of the FORWARD rule for esp
to see if any esp packets are making it to the FORWARD chain. If so, do
they match the match criteria for the ALLOW rule, e.g., the interfaces?
Post by Helge Weissig
The iptable logs are not complete and as I mentioned, I may need help with
setting that up. I can see the packets coming from the server with tcpdump
as I showed in my original post but then an immediate reply is sent back
and nothing goes through to the internal interface. The same thing happens
when I use nmap to scan ip protocols. Conversely, my internal ESP traffic
ends at the internal interface of my firewall. It never reaches the
external interface or the outside. TCP traffic works fine as you can see
from the ping logs from the internal client. Could this indicate that
there is a problem before anything gets to iptables?
h.
JO> > Jason,
JO> >
JO> > my ESP packets do not go from the external interface to the internal
JO> > one and vice versa. The connection to the VPN server works when I hook
JO> > up directly with no changes other than the IP of the client. I cannot
JO> > see how this would be a problem with the VPN network at all.
JO> >
JO> > h.
JO>
JO> looking at your logs--all your ESP packets are from client->server.
JO> you don't have a single ESP packet from server->client. so when you
JO> say, "my ESP packets do not go from the external interface..." you are
JO> ignoring the fact that there are no ESP packets ever getting to your
JO> external interface.
JO>
JO>
JO> your VPN server is discarding the ESP packets from your client as a
JO> result of the mangling of your intermediate NAT device.
JO>
JO> either make the VPN server more tolerant, or use NAT-T on your client.
JO>
JO> -j
JO>
JO> --
JO> "Ah, good ol' trustworthy beer. My love for you will never die."
JO> --The Simpsons
JO>
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
***@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
Jason Opperisano
2004-12-02 20:11:02 UTC
Permalink
Post by Helge Weissig
The iptable logs are not complete
if you're not providing all the details, i'm not sure how we're supposed
to be able to help. the information i was going on was:

Chain PREROUTING (policy ACCEPT 1380 packets, 95540 bytes)
pkts bytes target prot opt in out source
destination
0 0 LOG all -- * * 0.0.0.0/0
0.0.0.0/0 LOG flags 0 level 6 prefix `all preroute: '

and the log entries:

Dec 1 21:38:37 gollum kernel: all preroute: IN=eth1 OUT=
MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN
SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=64 ID=8977 PROTO=ESP
SPI=0x2e9dc0d2

your log rule in PREROUTING of the nat table should catch every single
packet, before any filtering (or even routing) takes place (unless you
have filter rules in mangle). the fact that every log entry you provided
(like the above) shows ESP from client->server means:

1) there are zero ESP packets coming from server->client
2) you removed those log entries from your post
3) packets from server->client disappear somewhere between the pcap
layer and the netfilter PREROUTING hook

are you saying it's #2?
Post by Helge Weissig
and as I mentioned, I may need help with
setting that up. I can see the packets coming from the server with tcpdump
as I showed in my original post but then an immediate reply is sent back
and nothing goes through to the internal interface.
if you can see ESP packets hitting your external interface from the VPN
server with tcpdump, but a log rule in PREROUTING of the nat table
doesn't see them--you have something horribly, horribly wrong with your
firewall machine.
Post by Helge Weissig
The same thing happens
when I use nmap to scan ip protocols. Conversely, my internal ESP traffic
ends at the internal interface of my firewall. It never reaches the
external interface or the outside. TCP traffic works fine as you can see
from the ping logs from the internal client. Could this indicate that
there is a problem before anything gets to iptables?
yes.

if you use these rules:

iptables -t nat -A PREROUTING -p 50 -j LOG \
--log-prefix "PREROUTE ESP: "

iptables -A FORWARD -p 50 -j LOG --log-prefix "FWD ESP: "

iptables -t nat -A POSTROUTING -o $EXT_IF -j MASQUERADE

you should get logs in both directions (client->server and
server->client). if not...well, let's assume for now that you will...

it would also greatly help if you made sure to post all the logs
generated by the above, so as not to mislead us.

-j

--
"Ah, beer, my one weakness. My Achilles heel, if you will."
--The Simpsons
Helge Weissig
2004-12-02 19:26:22 UTC
Permalink
I mean with "incomplete" that the tcpdump traffic I see does not show up
in the logs. I used your rules at the end of your reply and see the same
thing: ESP from VPN_SERVER hits $EXTIF, triggers the "protocol 50
unreachable" icmp response and no log entry ever shows up in the kernel
log from the iptables log rule. I am suspecting that your option 3) is
indeed the problem.

h.

On Thu, 2 Dec 2004 at 15:11 -0500, netfilter-***@lists.netfilter.org wrote:

JO> On Thu, Dec 02, 2004 at 10:22:27AM -0800, Helge Weissig wrote:
JO> > The iptable logs are not complete
JO>
JO> if you're not providing all the details, i'm not sure how we're supposed
JO> to be able to help. the information i was going on was:
JO>
JO> Chain PREROUTING (policy ACCEPT 1380 packets, 95540 bytes)
JO> pkts bytes target prot opt in out source
JO> destination
JO> 0 0 LOG all -- * * 0.0.0.0/0
JO> 0.0.0.0/0 LOG flags 0 level 6 prefix `all preroute: '
JO>
JO> and the log entries:
JO>
JO> Dec 1 21:38:37 gollum kernel: all preroute: IN=eth1 OUT=
JO> MAC=00:c0:4f:22:05:03:00:30:65:1f:ed:0a:08:00 SRC=10.0.0.200 DST=(VPN
JO> SERVER IP) LEN=136 TOS=0x00 PREC=0x00 TTL=64 ID=8977 PROTO=ESP
JO> SPI=0x2e9dc0d2
JO>
JO> your log rule in PREROUTING of the nat table should catch every single
JO> packet, before any filtering (or even routing) takes place (unless you
JO> have filter rules in mangle). the fact that every log entry you provided
JO> (like the above) shows ESP from client->server means:
JO>
JO> 1) there are zero ESP packets coming from server->client
JO> 2) you removed those log entries from your post
JO> 3) packets from server->client disappear somewhere between the pcap
JO> layer and the netfilter PREROUTING hook
JO>
JO> are you saying it's #2?
JO>
JO> > and as I mentioned, I may need help with
JO> > setting that up. I can see the packets coming from the server with tcpdump
JO> > as I showed in my original post but then an immediate reply is sent back
JO> > and nothing goes through to the internal interface.
JO>
JO> if you can see ESP packets hitting your external interface from the
JO> VPN server with tcpdump, but a log rule in PREROUTING of the nat table
JO> doesn't see them--you have something horribly, horribly wrong with your
JO> firewall machine.
JO>
JO> > The same thing happens
JO> > when I use nmap to scan ip protocols. Conversely, my internal ESP traffic
JO> > ends at the internal interface of my firewall. It never reaches the
JO> > external interface or the outside. TCP traffic works fine as you can see
JO> > from the ping logs from the internal client. Could this indicate that
JO> > there is a problem before anything gets to iptables?
JO>
JO> yes.
JO>
JO> if you use these rules:
JO>
JO> iptables -t nat -A PREROUTING -p 50 -j LOG \
JO> --log-prefix "PREROUTE ESP: "
JO>
JO> iptables -A FORWARD -p 50 -j LOG --log-prefix "FWD ESP: "
JO>
JO> iptables -t nat -A POSTROUTING -o $EXT_IF -j MASQUERADE
JO>
JO> you should get logs in both directions (client->server and
JO> server->client). if not...well, let's assume for now that you will...
JO>
JO> it would also greatly help if you made sure to post all the logs
JO> generated by the above, so as not to mislead us.
JO>
JO> -j
JO>
JO> --
JO> "Ah, beer, my one weakness. My Achilles heel, if you will."
JO> --The Simpsons
JO>
Jason Opperisano
2004-12-02 20:56:45 UTC
Permalink
Post by Helge Weissig
I mean with "incomplete" that the tcpdump traffic I see does not show up
in the logs. I used your rules at the end of your reply and see the same
thing: ESP from VPN_SERVER hits $EXTIF, triggers the "protocol 50
unreachable" icmp response and no log entry ever shows up in the kernel
log from the iptables log rule. I am suspecting that your option 3) is
indeed the problem.
h.
what kernel are you running on this firewall [*]? the only plausible
explanation at this point (and i'm not even sure that this is possible)
is that you're running an IPsec-enabled kernel that has SPD entries in
it that is scarfing up the ESP packets prior to the PREROUTING netfilter
hooks. i'm not sure that the last part is even possible though, as i
was under the impression that ESP packets fully travel through PREROUTING
and INPUT before they get processed by the kernel IPsec code...

[*] if the answer is 2.6, the output of "setkey -aPD" should be:
No SPD entries.

-j

--
"I hope I didn't brain my damage."
--The Simpsons
Helge Weissig
2004-12-02 20:12:25 UTC
Permalink
My kernel is 2.4.18-24.8.0 - h.

On Thu, 2 Dec 2004 at 15:56 -0500, netfilter-***@lists.netfilter.org wrote:

JO> On Thu, Dec 02, 2004 at 11:26:22AM -0800, Helge Weissig wrote:
JO> > I mean with "incomplete" that the tcpdump traffic I see does not show up
JO> > in the logs. I used your rules at the end of your reply and see the same
JO> > thing: ESP from VPN_SERVER hits $EXTIF, triggers the "protocol 50
JO> > unreachable" icmp response and no log entry ever shows up in the kernel
JO> > log from the iptables log rule. I am suspecting that your option 3) is
JO> > indeed the problem.
JO> >
JO> > h.
JO>
JO> what kernel are you running on this firewall [*]? the only plausible
JO> explanation at this point (and i'm not even sure that this is possible)
JO> is that you're running an IPsec-enabled kernel that has SPD entries in
JO> it that is scarfing up the ESP packets prior to the PREROUTING netfilter
JO> hooks. i'm not sure that the last part is even possible though, as i
JO> was under the impression that ESP packets fully travel through PREROUTING
JO> and INPUT before they get processed by the kernel IPsec code...
JO>
JO> [*] if the answer is 2.6, the output of "setkey -aPD" should be:
JO> No SPD entries.
JO>
JO> -j
JO>
JO> --
JO> "I hope I didn't brain my damage."
JO> --The Simpsons
JO>
Jason Opperisano
2004-12-02 21:30:08 UTC
Permalink
Post by Helge Weissig
My kernel is 2.4.18-24.8.0 - h.
as much as it pains me to say this ('cause i sorta dig figuring this
stuff out): you've stumped me.

-j

--
"Weaseling out of things is important to learn. It's what separates
us from the animals...except the weasel."
--The Simpsons
Philip Craig
2004-12-03 06:35:59 UTC
Permalink
Post by Helge Weissig
I mean with "incomplete" that the tcpdump traffic I see does not show up
in the logs. I used your rules at the end of your reply and see the same
thing: ESP from VPN_SERVER hits $EXTIF, triggers the "protocol 50
unreachable" icmp response and no log entry ever shows up in the kernel
log from the iptables log rule. I am suspecting that your option 3) is
indeed the problem.
h.
It is possible that a conntrack already exists, or the packet can't be
conntracked, so the packet doesn't pass through nat PREROUTING.

Try putting the log rule in the mangle PREROUTING chain.
If they do match a log rule here, check if they are invalid
with -m conntrack --ctstate INVALID.

Also check if there are any esp conntracks in /proc/net/ip_conntrack
--
Philip Craig - SnapGear, A CyberGuard Company - http://www.SnapGear.com
Helge Weissig
2004-12-03 17:11:27 UTC
Permalink
ahhh... finally I see something... but what does it mean???

added the following two log rules:
$IPTABLES -A PREROUTING -t mangle -j LOG --log-level info --log-prefix 'all mangle preroute: '
$IPTABLES -A PREROUTING -t mangle -m conntrack --ctstate INVALID -j LOG --log-level info --log-prefix 'contrack mangle preroute: '

the second generates the following error:
iptables v1.2.6a: Couldn't load match `conntrack':/lib/iptables/libipt_conntrack.so: cannot open shared object file: No such file or directory

the ESP's however now show up in the log (these are nmap generated):
Dec 3 09:07:23 gollum kernel: all mangle preroute: IN=eth0 OUT= MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00 SRC=vpn.server.ip DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=56785 PROTO=ESP INCOMPLETE [0 bytes]
Dec 3 09:07:23 gollum kernel: all mangle preroute: IN=eth0 OUT= MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00 SRC=vpn.server.ip DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=7732 PROTO=ESP INCOMPLETE [0 bytes]

h.

On Fri, 3 Dec 2004 at 16:35 +1000, Philip Craig wrote:

PC> Helge Weissig wrote:
PC> > I mean with "incomplete" that the tcpdump traffic I see does not show up
PC> > in the logs. I used your rules at the end of your reply and see the same
PC> > thing: ESP from VPN_SERVER hits $EXTIF, triggers the "protocol 50
PC> > unreachable" icmp response and no log entry ever shows up in the kernel
PC> > log from the iptables log rule. I am suspecting that your option 3) is
PC> > indeed the problem.
PC> >
PC> > h.
PC>
PC> It is possible that a conntrack already exists, or the packet can't be
PC> conntracked, so the packet doesn't pass through nat PREROUTING.
PC>
PC> Try putting the log rule in the mangle PREROUTING chain.
PC> If they do match a log rule here, check if they are invalid
PC> with -m conntrack --ctstate INVALID.
PC>
PC> Also check if there are any esp conntracks in /proc/net/ip_conntrack
PC>
PC>
Alistair Tonner
2004-12-04 02:20:44 UTC
Permalink
Post by Helge Weissig
ahhh... finally I see something... but what does it mean???
$IPTABLES -A PREROUTING -t mangle -j LOG --log-level info --log-prefix 'all
mangle preroute: ' $IPTABLES -A PREROUTING -t mangle -m conntrack --ctstate
INVALID -j LOG --log-level info --log-prefix 'contrack mangle preroute: '
iptables v1.2.6a: Couldn't load match
`conntrack':/lib/iptables/libipt_conntrack.so: cannot open shared object
file: No such file or directory
The above error indicates you did not build the conntrack match module and
related iptables code.
Post by Helge Weissig
Dec  3 09:07:23 gollum kernel: all mangle preroute: IN=eth0 OUT=
MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00 SRC=vpn.server.ip
DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=56785 PROTO=ESP
INCOMPLETE [0 bytes]
IN=eth0 OUT= MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00
SRC=vpn.server.ip DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=7732
PROTO=ESP INCOMPLETE [0 bytes]
It would be nice to have the other packet(s) that went out to initiate this
connection. But it doesn't look good to me -- I *think* that ipt_LOG.c is
saying that the packet structure for the ESP packet is incomplete.

eh = skb_header_pointer(skb, iphoff+ih->ihl*4,
sizeof(_esph), &_esph);
if (eh == NULL) {
printk("INCOMPLETE [%u bytes] ",
skb->len - iphoff - ih->ihl*4);
break;

Not sure how the packet is arriving in the LOG routine without the relevant
data.


Alistair Tonner
RSO HP Unix admin.
Post by Helge Weissig
h.
Jason Opperisano
2004-12-04 02:35:36 UTC
Permalink
Post by Alistair Tonner
Post by Helge Weissig
ahhh... finally I see something... but what does it mean???
$IPTABLES -A PREROUTING -t mangle -j LOG --log-level info --log-prefix 'all
mangle preroute: ' $IPTABLES -A PREROUTING -t mangle -m conntrack --ctstate
INVALID -j LOG --log-level info --log-prefix 'contrack mangle preroute: '
iptables v1.2.6a: Couldn't load match
`conntrack':/lib/iptables/libipt_conntrack.so: cannot open shared object
file: No such file or directory
The above error indicates you did not build the conntrack match module and
related iptables code.
Post by Helge Weissig
Dec 3 09:07:23 gollum kernel: all mangle preroute: IN=eth0 OUT=
MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00 SRC=vpn.server.ip
DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=56785 PROTO=ESP
INCOMPLETE [0 bytes]
IN=eth0 OUT= MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00
SRC=vpn.server.ip DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=7732
PROTO=ESP INCOMPLETE [0 bytes]
LEN=20 means the IP packet is only 20 bytes--which would lead one to
believe that the packet contains only an IP header and no data. which
is probably all nmap is generating. not sure what more you would expect
from such a test.

-j

--
"I have thought this through. First, I will send Bart the money to
fly home. Then I will murder him."
--The Simpsons
Helge Weissig
2004-12-04 03:03:21 UTC
Permalink
On Fri, 3 Dec 2004 at 21:35 -0500, netfilter-***@lists.netfilter.org wrote:

JO> > > the ESP's however now show up in the log (these are nmap generated):
JO> >
JO> > > Dec 3 09:07:23 gollum kernel: all mangle preroute: IN=eth0 OUT=
JO> > > MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00 SRC=vpn.server.ip
JO> > > DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=56785 PROTO=ESP
JO> > > INCOMPLETE [0 bytes]
JO> >
JO> > > Dec 3 09:07:23 gollum kernel: all mangle preroute:
JO> > > IN=eth0 OUT= MAC=00:90:27:ca:39:56:00:10:67:00:b4:3e:08:00
JO> > > SRC=vpn.server.ip DST=ext.if.ip LEN=20 TOS=0x00 PREC=0x00 TTL=32 ID=7732
JO> > > PROTO=ESP INCOMPLETE [0 bytes]
JO>
JO> LEN=20 means the IP packet is only 20 bytes--which would lead one to
JO> believe that the packet contains only an IP header and no data. which
JO> is probably all nmap is generating. not sure what more you would expect
JO> from such a test.

would this match the following log rule though?

$IPTABLES -A PREROUTING -p esp -t mangle -m state --state INVALID -j LOG

I am seeing log entries from that that say "INCOMPLETE" (which would make
sense). FWIW, I was aware of the nmap limitation in this case, but could
not test it with the VPN connection yet today... stay tuned!

h.
John A. Sullivan III
2004-12-02 04:03:01 UTC
Permalink
Post by Jason Opperisano
<snip>
Post by Helge Weissig
Here is my current iptables configuration
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F
echo "Enabling PORTFW Redirection on the external LAN.."
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p esp -j ACCEPT
$IPTABLES -A PREROUTING -t nat -d $VPN_SERVER -p esp -j DNAT \
--to-destination $VPN_CLIENT
what on earth is that rule supposed to accomplish? it's says "any esp
packet destined for $VPN_SERVER should be destination translated to
$VPN_CLIENT."
if this is the firewall in front of $VPN_CLIENT (which it sounds like it
is), you have created (for lack of a better term) a packet reflector.
any esp packet sent from VPN client to VPN server will be spit back at
the VPN client.
Post by Helge Weissig
echo " FWD: Allow all connections OUT and only existing and related
ones IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG
echo " Enabling SNAT (MASQUERADE) functionality on $EXTIF"
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
10:23:09.234937 (vpn server ip) > (my ip): ESP(spi=0x00000000,seq=0x0)
10:23:09.235055 (my ip) > (vpn server ip): icmp: (my ip) protocol 50
unreachable [tos 0xc0]
(these are empty packets sent by nmap but it looks the same for legit
ones coming from the vpn server ip). FWIW, when the ipsec tunnel is
established and I try to ping the a host behind the vpn server, I see
the outgoing packets on all three interfaces, but not response.
thanks for any information or pointers in advance!
h.
# start fresh
for t in mangle nat filter; do
iptables -t $t -F
iptables -t $t -X
iptables -t $t -Z
done
for c in INPUT FORWARD OUTPUT; do
iptables -P $c ACCEPT
done
# hide-nat outbound traffic
iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# ip forwarding
sysctl -w net.ipv4.ip_forward=1
try and connect with your VPN client to your VPN server with that
script. if you can't connect--it's more likely an IPSec configuration
detail that your missing.
<snip>
I don't think this is quite right, Jason. It is possible that the other
side of the tunnel will need to initiate traffic once the tunnel is
established. Thus, there may be inbound packets on ip/50 that are not
associated with any entry in the connection tracking table. Thus he
does need the specific port forwarding DNAT rule.

I believe you are correct that he does need to qualify the interface,
i.e., the DNAT rule should only apply to traffic where -i $EXTIF lest he
have the reflector effect. In fact, I'm surprised that he saw the ESP
pings going through. I would have thought they would have been
reflected. Nonetheless, Helge, you should qualify the interface. See
if that helps.

If that doesn't work, I would suggest placing log rules at strategic
points in the rule set to find out what the packet looks like just
before the -p esp -j ACCEPT rule to find out why it doesn't match. I
assume this is not the entire rule set, e.g., somewhere I would imagine
you change the INPUT and OUTPUT policies to DROP.

Is there any chance that one of the variables is set incorrectly, e.g.,
$VPN_CLIENT or $VPN_SERVER?
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
***@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
Helge Weissig
2004-12-04 17:07:22 UTC
Permalink
what do you know?? As mysteriously as the problem appeared, it seems to
have resolved itself. I am stumped!

Thanks to all of you for your input and patience.

h.
Post by Helge Weissig
Sorry for the cross-post, but this problem is really nagging me. What I
did not put into the post below is the fact that it only occurred after a
reboot of my linux system due to a short power outage. Here is the routing
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
xx.xx.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 xx.xx.xx.1 0.0.0.0 UG 0 0 0 eth0
the xx.xx.xx is the first part of my external ip address.
thanks for any advice or insight!
h.
Newsgroups: comp.security.firewalls
Subject: protocol 50 unreachable
NNTP-Posting-Host: 63.196.131.66
Hi,
I have been searching for information about this problem high and low
but came up dry. Basically, I am trying to connect to a VPN server via
ipsec from behind a NAT firewall set up on a Linux (kernel 2.4.x) box
with iptables. I have no problem establishing the connection via port
500 as this is initiated by the client. However, I cannot seem to get
protocol 50 (ESP) to work, independent of whether the ipsec tunnel is
established or not. I have tried every incantation of iptables rules I
could find, to no avail. When I set up tcdump on both interfaces on my
server as well as on the client behind it, a port I have opened for
forwarding responds as expected. If I run 'nmap -sO' from somewhere
outside however, it will report protocol 50 as open although the
external interface reports a 'icmp: xx.xx.xx.xx protocol 50
unreachable' response and the two other interfaces never see the
traffic.
Here is my current iptables configuration
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -F OUTPUT
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD
$IPTABLES -t nat -F
echo "Enabling PORTFW Redirection on the external LAN.."
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p esp -j ACCEPT
$IPTABLES -A PREROUTING -t nat -d $VPN_SERVER -p esp -j DNAT \
--to-destination $VPN_CLIENT
echo " FWD: Allow all connections OUT and only existing and related
ones IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j LOG
echo " Enabling SNAT (MASQUERADE) functionality on $EXTIF"
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
10:23:09.234937 (vpn server ip) > (my ip): ESP(spi=0x00000000,seq=0x0)
10:23:09.235055 (my ip) > (vpn server ip): icmp: (my ip) protocol 50
unreachable [tos 0xc0]
(these are empty packets sent by nmap but it looks the same for legit
ones coming from the vpn server ip). FWIW, when the ipsec tunnel is
established and I try to ping the a host behind the vpn server, I see
the outgoing packets on all three interfaces, but not response.
thanks for any information or pointers in advance!
h.
Loading...