conntrackd, internal cache keeps filling up

[Martin Kraus]

Hi.
I'm running conntrackd 1.2.1 on debian wheezy between two routers with
conntrackd external cache disabled.

After running for some time the states stop replicating due to conntrackd hitting
HashLimit in the internal cache. When I looked at it it's basically broadcast
and multicast connections that never time out even though they are no longer
in the conntrack tables on both routers. I filtered some of it out using
NOTRACK in iptables. That helped for a while but now it's back with
tcp LAST_ACK state.

tcp 6 LAST_ACK src=172.20.6.74 dst=172.20.20.23 sport=48818 dport=445 src=172.20.20.23 dst=172.20.6.74 sport=445 dport=48818 [ASSURED] [active since 249398s]
tcp 6 LAST_ACK src=172.20.6.119 dst=172.20.20.21 sport=55843 dport=80 src=172.20.20.21 dst=172.20.6.119 sport=80 dport=55843 [ASSURED] [active since 15458s]
tcp 6 LAST_ACK src=172.20.6.119 dst=172.20.20.21 sport=58573 dport=80 src=172.20.20.21 dst=172.20.6.119 sport=80 dport=58573 [ASSURED] [active since 7426s]
tcp 6 LAST_ACK src=172.20.6.119 dst=172.20.20.21 sport=57923 dport=80 src=172.20.20.21 dst=172.20.6.119 sport=80 dport=57923 [ASSURED] [active since 9946s]
tcp 6 LAST_ACK src=172.20.6.74 dst=172.20.20.23 sport=47560 dport=445 src=172.20.20.23 dst=172.20.6.74 sport=445 dport=47560 [ASSURED] [active since 252927s]

There's thousands of these entries and in a few days they'll fill up the
internal cache and break internal routing.

My understanding was that the internal cache is a copy of the kernel conntrack
table on each router so it should not be larger then the current conntrack
table plus some information about terminated connections that need to be
synced to the other router.

How are the entries in internal cache purged? Most of them seem to be deleted
when the connection is terminated but some of the just keep hanging around.

thanks for help
Martin
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Pablo Neira Ayuso]

Post by Martin Kraus
Hi.
I'm running conntrackd 1.2.1 on debian wheezy between two routers with
conntrackd external cache disabled.
After running for some time the states stop replicating due to conntrackd hitting
HashLimit in the internal cache. When I looked at it it's basically broadcast
and multicast connections that never time out even though they are no longer
in the conntrack tables on both routers. I filtered some of it out using
NOTRACK in iptables. That helped for a while but now it's back with
tcp LAST_ACK state.
tcp 6 LAST_ACK src=172.20.6.74 dst=172.20.20.23 sport=48818 dport=445 src=172.20.20.23 dst=172.20.6.74 sport=445 dport=48818 [ASSURED] [active since 249398s]
tcp 6 LAST_ACK src=172.20.6.119 dst=172.20.20.21 sport=55843 dport=80 src=172.20.20.21 dst=172.20.6.119 sport=80 dport=55843 [ASSURED] [active since 15458s]
tcp 6 LAST_ACK src=172.20.6.119 dst=172.20.20.21 sport=58573 dport=80 src=172.20.20.21 dst=172.20.6.119 sport=80 dport=58573 [ASSURED] [active since 7426s]
tcp 6 LAST_ACK src=172.20.6.119 dst=172.20.20.21 sport=57923 dport=80 src=172.20.20.21 dst=172.20.6.119 sport=80 dport=57923 [ASSURED] [active since 9946s]
tcp 6 LAST_ACK src=172.20.6.74 dst=172.20.20.23 sport=47560 dport=445 src=172.20.20.23 dst=172.20.6.74 sport=445 dport=47560 [ASSURED] [active since 252927s]
There's thousands of these entries and in a few days they'll fill up the
internal cache and break internal routing.

Could you retry with lastest conntrackd version? 1.4.2.

You didn't specify your Linux kernel version either. Thanks.
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Martin Kraus]

Post by Pablo Neira Ayuso

Post by Martin Kraus
There's thousands of these entries and in a few days they'll fill up the
internal cache and break internal routing.

Could you retry with lastest conntrackd version? 1.4.2.

will try 1.4.2. we just need to package it.

Post by Pablo Neira Ayuso
You didn't specify your Linux kernel version either. Thanks.

current kernel is 3.13.7.

we already hit a bug in the official 3.2 kernel packaged with wheezy where
our scan for heartbleed vulnerability would cause conntrackd to kernel panic
the router.

mk
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Pablo Neira Ayuso]

Post by Martin Kraus

Post by Pablo Neira Ayuso

Post by Martin Kraus
There's thousands of these entries and in a few days they'll fill up the
internal cache and break internal routing.

Could you retry with lastest conntrackd version? 1.4.2.

will try 1.4.2. we just need to package it.

OK.

Post by Martin Kraus

Post by Pablo Neira Ayuso
You didn't specify your Linux kernel version either. Thanks.

current kernel is 3.13.7.
we already hit a bug in the official 3.2 kernel packaged with wheezy where
our scan for heartbleed vulnerability would cause conntrackd to kernel panic
the router.

Please, provide more information on how to reproduce the problem that
you're noticing. Thank you.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Martin Kraus]

Post by Pablo Neira Ayuso

Post by Martin Kraus
current kernel is 3.13.7.
we already hit a bug in the official 3.2 kernel packaged with wheezy where
our scan for heartbleed vulnerability would cause conntrackd to kernel panic
the router.

Please, provide more information on how to reproduce the problem that
you're noticing. Thank you.

regarding the kernel panic on 3.2 a colleague of mine was using nmap with it's
heartbleed plugin

nmap --script ssl-heartbleed -sT -oX logfile.log 10.0.0.0/20

http://nmap.org/nsedoc/scripts/ssl-heartbleed.html

it took about 30 minutes to trigger the problem.

regarding the internal cache fill up. we have two routers and some vlans using
one and some vlans using the other router as the default gateway.

this is the conntrackd config on both routers.

Sync {
Mode FTFW {
ResendQueueSize 131072
ACKWindowSize 300
DisableExternalCache On
}
UDP {
IPv4_address 192.168.100.200
IPv4_Destination_Address 192.168.100.100
Port 3780
Interface eth0
Checksum on
}
Options {
TCPWindowTracking On
}
}

General {
Nice -20

HashSize 65536
HashLimit 262144

Syslog on
LockFile /var/lock/conntrack.lock
UNIX {
Path /var/run/conntrackd.ctl
Backlog 20
}

NetlinkBufferSize 2097152
NetlinkBufferSizeMaxGrowth 8388608
NetlinkEventsReliable On
NetlinkOverrunResync Off

Filter From Kernelspace {
Address Ignore {
IPv4_address 127.0.0.1 # loopback
}
}
}

We have about 80 users, some of them running window or macs, so there is
plenty of multicasts and broadcasts that fill the conntrack table. some of
these then get stuck in the conntrackd internal cache. We can see the
LAST_ACK tcp states stuck in the internal cache as well, but I think these are
related to TCPWindowTracking On.

mk
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Florian Westphal]

Post by Martin Kraus

Post by Pablo Neira Ayuso

Post by Martin Kraus
current kernel is 3.13.7.
we already hit a bug in the official 3.2 kernel packaged with wheezy where
our scan for heartbleed vulnerability would cause conntrackd to kernel panic
the router.

Please, provide more information on how to reproduce the problem that
you're noticing. Thank you.

regarding the kernel panic on 3.2 a colleague of mine was using nmap with it's
heartbleed plugin
nmap --script ssl-heartbleed -sT -oX logfile.log 10.0.0.0/20
http://nmap.org/nsedoc/scripts/ssl-heartbleed.html
it took about 30 minutes to trigger the problem.

[..]

Post by Martin Kraus
NetlinkEventsReliable On

known broken until at least Linux 3.6, see f.e.

5b423f6a40a0327f9d40bc8b97ce9be266f74368
("netfilter: nf_conntrack: fix racy timer handling with reliable events")
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Pablo Neira Ayuso]

Post by Florian Westphal

Post by Martin Kraus

Post by Pablo Neira Ayuso

Post by Martin Kraus
current kernel is 3.13.7.
we already hit a bug in the official 3.2 kernel packaged with wheezy where
our scan for heartbleed vulnerability would cause conntrackd to kernel panic
the router.

Please, provide more information on how to reproduce the problem that
you're noticing. Thank you.

regarding the kernel panic on 3.2 a colleague of mine was using nmap with it's
heartbleed plugin
nmap --script ssl-heartbleed -sT -oX logfile.log 10.0.0.0/20
http://nmap.org/nsedoc/scripts/ssl-heartbleed.html
it took about 30 minutes to trigger the problem.

[..]

Post by Martin Kraus
NetlinkEventsReliable On

known broken until at least Linux 3.6, see f.e.
5b423f6a40a0327f9d40bc8b97ce9be266f74368
("netfilter: nf_conntrack: fix racy timer handling with reliable events")

If they are using latest 3.2, that patch is already there.
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Pablo Neira Ayuso]

Post by Martin Kraus

Post by Pablo Neira Ayuso

Post by Martin Kraus
current kernel is 3.13.7.
we already hit a bug in the official 3.2 kernel packaged with wheezy where
our scan for heartbleed vulnerability would cause conntrackd to kernel panic
the router.

Please, provide more information on how to reproduce the problem that
you're noticing. Thank you.

regarding the kernel panic on 3.2 a colleague of mine was using nmap with it's
heartbleed plugin
nmap --script ssl-heartbleed -sT -oX logfile.log 10.0.0.0/20
http://nmap.org/nsedoc/scripts/ssl-heartbleed.html
it took about 30 minutes to trigger the problem.

Did you annotate the kernel oops backtrace? Without that information,
this is pretty much like looking for the needle in the stack.

Post by Martin Kraus
regarding the internal cache fill up. we have two routers and some vlans using
one and some vlans using the other router as the default gateway.
this is the conntrackd config on both routers.
Sync {
Mode FTFW {
ResendQueueSize 131072
ACKWindowSize 300
DisableExternalCache On
}
UDP {
IPv4_address 192.168.100.200
IPv4_Destination_Address 192.168.100.100
Port 3780
Interface eth0
Checksum on
}
Options {
TCPWindowTracking On
}
}
General {
Nice -20
HashSize 65536
HashLimit 262144
Syslog on
LockFile /var/lock/conntrack.lock
UNIX {
Path /var/run/conntrackd.ctl
Backlog 20
}
NetlinkBufferSize 2097152
NetlinkBufferSizeMaxGrowth 8388608
NetlinkEventsReliable On
NetlinkOverrunResync Off
Filter From Kernelspace {
Address Ignore {
IPv4_address 127.0.0.1 # loopback
}
}
}
We have about 80 users, some of them running window or macs, so there is
plenty of multicasts and broadcasts that fill the conntrack table. some of
these then get stuck in the conntrackd internal cache. We can see the
LAST_ACK tcp states stuck in the internal cache as well, but I think these are
related to TCPWindowTracking On.

Did you retry with lastest conntrack-tools version? If so, please
collect as much information as you can via all -s options, moreover
check the logs. If you didn't retry with lastest, please upgrade.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Martin Kraus]

Post by Pablo Neira Ayuso

Post by Martin Kraus
it took about 30 minutes to trigger the problem.

Did you annotate the kernel oops backtrace? Without that information,
this is pretty much like looking for the needle in the stack.

Unfortunately we don't have any other information. I could reboot to the old
kernel and trigger it again but I'll need to do that in the evening.

Is there a better way to get the oops backtrace than a redirect to the serial
console?

mk
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Martin Kraus]

Post by Pablo Neira Ayuso

Post by Martin Kraus
will try 1.4.2. we just need to package it.

Hi.

we've been running 1.4.2 since the end of May but the problem persists. We've
managed to keep it down to a restart once a month through NOTRACK rules but
that's not a nice solution.

Post by Pablo Neira Ayuso
Please, provide more information on how to reproduce the problem that
you're noticing. Thank you.

We have two office routers (router1, router2) which are connected to our internal
vlans and then to the uplinks. Both routers are active and there is a dedicated
line used for conntrackd synchronization.

An example of state that keeps filling the conntrackd internal cache is dns
traffic to our nat64 proxy.

A user from vlan6 goes to our nat64 proxy in vlan20. vlan6 has a default
gateway to router2, vlan20 has a default gateway to vlan1.

User ipv6 address is 2001:1488:fffe:6:c941:fae:4505:7a22.
Nat64 ipv6 address is 2001:1488:fffe:20::34.

The dns packet from the user goes to router2 and then directly to vlan20 where the
nat64 host is located.

The dns reply packet from nat64 then goes to router1 and then directly to
vlan6 back to the user.

Now on router2 when I run conntrackd -i I can see

udp 17 src=2001:1488:fffe:6:c941:fae:4505:7a22 dst=2001:1488:fffe:20::34 sport=6728 dport=53 [UNREPLIED] src=2001:1488:fffe:20::34 dst=2001:1488:fffe:6:c941:fae:4505:7a22 sport=53 dport=6728 [active since 192159s]
udp 17 src=2001:1488:fffe:6:c941:fae:4505:7a22 dst=2001:1488:fffe:20::34 sport=6961 dport=53 [UNREPLIED] src=2001:1488:fffe:20::34 dst=2001:1488:fffe:6:c941:fae:4505:7a22 sport=53 dport=6961 [active since 191949s]
udp 17 src=2001:1488:fffe:6:c941:fae:4505:7a22 dst=2001:1488:fffe:20::34 sport=6977 dport=53 [UNREPLIED] src=2001:1488:fffe:20::34 dst=2001:1488:fffe:6:c941:fae:4505:7a22 sport=53 dport=6977 [active since 191962s]
udp 17 src=2001:1488:fffe:6:c941:fae:4505:7a22 dst=2001:1488:fffe:20::34 sport=6979 dport=53 [UNREPLIED] src=2001:1488:fffe:20::34 dst=2001:1488:fffe:6:c941:fae:4505:7a22 sport=53 dport=6979 [active since 168352s]

and another 126000 entries like this. router1 is similar except that the state
is not UNREPLIED. kernel conntrack table doesn't have any of these entries.

Interesting thing is that it's usually 1 ipv[46] address that generates most
of these stale entries.

This is the config file used

Sync {
Mode FTFW {
ResendQueueSize 131072
ACKWindowSize 300
DisableExternalCache On
}
UDP {
IPv4_address 192.168.100.100
IPv4_Destination_Address 192.168.100.200
Port 3780
Interface eth0
Checksum on
}
Options {
TCPWindowTracking On
}

}

General {
Nice -20

HashSize 65536
HashLimit 262144

Syslog on
LockFile /var/lock/conntrack.lock
UNIX {
Path /var/run/conntrackd.ctl
Backlog 20
}

NetlinkBufferSize 2097152
NetlinkBufferSizeMaxGrowth 8388608
NetlinkEventsReliable Off
NetlinkOverrunResync On

Filter From Kernelspace {
Address Ignore {
IPv4_address 127.0.0.1 # loopback
}
}
}

I always assumed that the internal cache is a copy of the kernel conntrack table
plus entries that have not yet been synchronized to the other router so I
don't understand why is it getting this huge.

mk
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html