too many INVALID packets

Hello,

Post by Vladislav Kurz
I have a couple of questions about netfilter conntrack and INVALID packets.
I'm administrator of couple of servers, some of them still running 2.4
kernels, some running (upgraded or new install) to 2.6.18 (Debian packaged
iptables -t filter -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t filter -A INPUT -m state --state INVALID -j DROP
and the same in FORWARD chain.
Machines with 2.4 kernel see almost no INVALID packets in the long run.
Machines with 2.6 kernel see hundreds of INVALID packets every day.
I suppose that the logic deciding what is INVALID has changed a lot a now
catches more stuff than before.

Yes, the main difference (from conntrack point of view) is that TCP window
tracking was added to netfilter in 2.6. However if I recall correctly,
2.6.18 has got some TCP window tracking related bugs. So you should
upgrade your kernel.

Post by Vladislav Kurz
However on two servers I had to pass INVALID packets through as the users
complained about hanging connections or being unable to connect at all.
Part of the problem involved Xen, and checksum offloading and was solved by
turning tx-checksumming off on virtual interfaces. But even after that, this
server and another one (without xen) see lots (thousands a day on workdays,
hundreds a day on weekends) of INVALID packets. I do not think they are just
some weird portscans as they come also from local network and there is
significant drop on weekends. Most of the invalid packets have ACK FIN or ACK
RST flags, and seem to be part (end) of regular connections. What can I do
about these? and what happens if I just drop them?

ACK FIN and ACK RST packets are (probably) packets which arrive too late
relative to the bookkeeping of conntrack. So you can drop them (but
you should instead upgrade ;-).

Post by Vladislav Kurz
echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
BTW it should be documented somwhere - not only in source code - that you have
to enter the protocol number you want get logged. Many sources on internet
suggest echo 1 > /proc/...log_invalid, and it sounds logical that this file
just swithces logiing on/off. But no, it logs only the protocol you ask
(1-ICMP, 6-TCP, 17-UDP .. see ./include/linux/in.h)

Yes, the proc files should be documented better.

Post by Vladislav Kurz
ip_ct_tcp: invalid packet ignored IN= OUT= SRC=a.b.c.d DST=x.y.z.q LEN=48
TOS=0x00 PREC=0x00 TTL=118 ID=57462 DF PROTO=TCP SPT=2745 DPT=110
SEQ=1241333208 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402)
(most of them have SYN flag)
Interesting thing is that these log entries do not match those logged by
iptables -t filter -A INPUT -m state --state INVALID -j LOG
Can someone tell why these two logs do not match, and what happens with
packets logged by (ip_ct_tcp) ip_conntrack_log_invalid?

TCP conntrack tries very hard to keep track of the connections.

The packets which are logged as 'invalid packet ignored' looks to
conntrack as invalid, because it holds a connections and the packet does
not fit into the stream. But at the same time it might be that conntrack
holds a dead connection and the packet is a real, new,
connection-initiating packet (SYN flag is on). So conntrack ignores the
seemingly invalid packet, flags it as 'ESTABLISHED' (not 'INVALID') and
lets it through. The reply packet will tell conntrack what should it do:
delete the old dead connection or keep it as real.

ip_conntrack_log_invalid logs the "magic", which can help to identify
possible problems in TCP connection tracking: ignored packets or packets
dropped by conntrack itself. All else is flagges as NEW, ESTABLISHED,
RELATED or INVALID.

Post by Vladislav Kurz
I noticed that iptables offers new match: -m conntrack --ctstate
which has also NEW, RELATED, ESTABLISHED and INVALID states. Are they somehow
different from those matched by -m state --state ?

The conntrack match offers more internal flags (like SNAT, DNAT) above
the state match.

Post by Vladislav Kurz
I suspected that problems might be with intel giabit ethernet cards (e1000)
and their checksum offloading (these are the only servers with such cards) so
I tried to disable checksum offloading, but anyway there were just a few
logged packets with wrong checksum. Even when I turned ip_conntrack_checksum
off situation didn't change much. Anyway, I was wondering if checksum
offloading has any sense on firewall, as netfilter will calculate and check
the checkums anyway. Or is there any smart communication between the driver
and netfilter saying that checksum is ok, or will be calculated later so
netfilter does not have to care about checksums?

Checksum offloading is useful on firewalls and netfilter can take it into
account. So you should enable it if the card supports it.

Post by Vladislav Kurz
Is it possible that disabling ip_conntrack_checksums would solve the problem
with xen machines in the same way as disabling cheksum offloading did?

No, I don't think so, unless it's a defected card.

Post by Vladislav Kurz
At last I would like to know if conntrack cares about TCP flags? Does it check
if NEW (and RELATED) TCP packets have SYN flag set and if not treats them as
iptables -t filter -A INPUT -m state --state NEW -p tcp ! --syn -j LOG
and it catches quite a lot of packets, but still it seems that conntrack knows
about TCP connections being opened (SYN) and closed (FIN). Is there any
reason for NEW packets without SYN to be allowed through?

Yes, it's a (well known) feature of netfilter. This way we can catch up
connections already established. If you do not want to support it, use
the last rule you wrote.

Back to the INVALID packets: enable ip_conntrack_tcp_be_liberal:

# echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

If the INVALID packets disappear, it can indicate the TCP window tracking
bug and then you should use a newer kernel release (or run with
ip_conntrack_tcp_be_liberal enabled).

Best regards,
Jozsef
-
E-mail : ***@blackhole.kfki.hu, ***@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Vladislav Kurz

2008-06-19 10:31:56 UTC

Post by Jozsef Kadlecsik
Hello,

Thanks for quick reply.

Post by Vladislav Kurz
I have a couple of questions about netfilter conntrack and INVALID packets.
I'm administrator of couple of servers, some of them still running 2.4
kernels, some running (upgraded or new install) to 2.6.18 (Debian
packaged kernels). On all of them I have firewall beginning with these
iptables -t filter -A INPUT -m state --state ESTABLISHED,RELATED -j
ACCEPT iptables -t filter -A INPUT -m state --state INVALID -j DROP
and the same in FORWARD chain.
Machines with 2.4 kernel see almost no INVALID packets in the long run.
Machines with 2.6 kernel see hundreds of INVALID packets every day.
I suppose that the logic deciding what is INVALID has changed a lot a now
catches more stuff than before.

Is there any site where can I find more info on this issue?

Post by Vladislav Kurz
ip_ct_tcp: invalid packet ignored IN= OUT= SRC=a.b.c.d DST=x.y.z.q LEN=48
TOS=0x00 PREC=0x00 TTL=118 ID=57462 DF PROTO=TCP SPT=2745 DPT=110
SEQ=1241333208 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B401010402) (most of them have SYN flag)
Interesting thing is that these log entries do not match those logged by
iptables -t filter -A INPUT -m state --state INVALID -j LOG
Can someone tell why these two logs do not match, and what happens with
packets logged by (ip_ct_tcp) ip_conntrack_log_invalid?

TCP conntrack tries very hard to keep track of the connections.
The packets which are logged as 'invalid packet ignored' looks to
conntrack as invalid, because it holds a connections and the packet does
not fit into the stream. But at the same time it might be that conntrack
holds a dead connection and the packet is a real, new,
connection-initiating packet (SYN flag is on). So conntrack ignores the
seemingly invalid packet, flags it as 'ESTABLISHED' (not 'INVALID') and
delete the old dead connection or keep it as real.
ip_conntrack_log_invalid logs the "magic", which can help to identify
possible problems in TCP connection tracking: ignored packets or packets
dropped by conntrack itself. All else is flagges as NEW, ESTABLISHED,
RELATED or INVALID.

So does that mean that some packets are dropped before they enter iptables?
Which ones?

Post by Vladislav Kurz
I noticed that iptables offers new match: -m conntrack --ctstate
which has also NEW, RELATED, ESTABLISHED and INVALID states. Are they
somehow different from those matched by -m state --state ?

The conntrack match offers more internal flags (like SNAT, DNAT) above
the state match.

Ok, thats documented,what I wanted to know is if the four states NEW, RELATED,
ESTABLISHED and INVALID are equal in -m conntrack and -m state.

Post by Vladislav Kurz
At last I would like to know if conntrack cares about TCP flags? Does it
check if NEW (and RELATED) TCP packets have SYN flag set and if not
iptables -t filter -A INPUT -m state --state NEW -p tcp ! --syn -j LOG
and it catches quite a lot of packets, but still it seems that conntrack
knows about TCP connections being opened (SYN) and closed (FIN). Is there
any reason for NEW packets without SYN to be allowed through?

Yes, it's a (well known) feature of netfilter. This way we can catch up
connections already established. If you do not want to support it, use
the last rule you wrote.

I thought it might be useful to block some weird portscans (e.g. nmap xmas
scan).

Post by Jozsef Kadlecsik
# echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
If the INVALID packets disappear, it can indicate the TCP window tracking
bug and then you should use a newer kernel release (or run with
ip_conntrack_tcp_be_liberal enabled).

Yep, I had this enabled on those servers. Now I turned it off on one of them
and get more logs of this type:

ip_ct_tcp: SEQ is under the lower bound (already ACKed data retransmitted) IN=
OUT= SRC=192.168.1.10 DST=192.168.2.122 LEN=40 TOS=0x00 PREC=0x00 TTL=64
ID=44558 DF PROTO=TCP SPT=3128 DPT=2140 SEQ=2998644047 ACK=4173779699
WINDOW=62780 RES=0x00 ACK FIN URGP=0 UID=13

ip_ct_tcp: ACK is under the lower bound (possible overly delayed ACK) IN= OUT=
SRC=192.168.2.122 DST=192.168.1.10 LEN=40 TOS=0x
00 PREC=0x00 TTL=128 ID=46970 DF PROTO=TCP SPT=2140 DPT=3128 SEQ=4173779699
ACK=2998644048 WINDOW=64313 RES=0x00 ACK URGP=0

INVALID: IN=eth0 OUT= MAC=00:1a:64:6d:96:3f:00:13:d3:ed:6e:c7:08:00
SRC=192.168.2.122 DST=192.168.1.10 LEN=40 TOS=0x00 PREC=0x00
TTL=128 ID=46970 DF PROTO=TCP SPT=2140 DPT=3128 WINDOW=64313 RES=0x00 ACK
URGP=0

Last line is logged by iptables. What I wonder is that there is corresponding
log from iptables to "ACK is under the lower bound" but not to "SEQ is under
the lower bound". I get also quite a few "ACK/SEQ is over the upper bound"
bith with corresponding log from iptables. Does that mean that packets
with "SEQ under the lower bound" are ESTABLISHED ? Thinking more about it,
they should - ACK might be lost, so retransmitting already ACKed data is ok.

Where can I find which ip_ct_tcp log entries get flagged as INVALID and which
as other states?

Jozsef Kadlecsik

2008-06-19 10:45:41 UTC

Post by Jozsef Kadlecsik
Yes, the main difference (from conntrack point of view) is that TCP window
tracking was added to netfilter in 2.6. However if I recall correctly,
2.6.18 has got some TCP window tracking related bugs. So you should
upgrade your kernel.

Is there any site where can I find more info on this issue?

The TCP window tracking code is based on the article 'Real Stateful TCP
Packet Filtering in IP Filter' by Guido van Rooij. You can find it at
http://www.nluug.nl/events/sane2000/papers.html.

The functions evolved a lot and there are a handful modification compared
to the algorithm written in the article, but it's undocumented (not
counting the comments in the source).

Post by Jozsef Kadlecsik
ip_conntrack_log_invalid logs the "magic", which can help to identify
possible problems in TCP connection tracking: ignored packets or packets
dropped by conntrack itself. All else is flagges as NEW, ESTABLISHED,
RELATED or INVALID.

So does that mean that some packets are dropped before they enter iptables?

You mean mangle, nat and filter tables by 'iptables, isn't it?
(All packets are seen in the raw table, then comes conntrack.)

Post by Vladislav Kurz
Which ones?

SYN/ACK packets sent as reply to SYN packets which was ignored as invalid.
(Thus we can delete the stale conntrack entry and force the client to
resend the initial SYN.)

Post by Jozsef Kadlecsik
The conntrack match offers more internal flags (like SNAT, DNAT) above
the state match.

Ok, thats documented,what I wanted to know is if the four states NEW, RELATED,
ESTABLISHED and INVALID are equal in -m conntrack and -m state.

They are identical.

Yep, I had this enabled on those servers. Now I turned it off on one of them
ip_ct_tcp: SEQ is under the lower bound (already ACKed data retransmitted) IN=
OUT= SRC=192.168.1.10 DST=192.168.2.122 LEN=40 TOS=0x00 PREC=0x00 TTL=64
ID=44558 DF PROTO=TCP SPT=3128 DPT=2140 SEQ=2998644047 ACK=4173779699
WINDOW=62780 RES=0x00 ACK FIN URGP=0 UID=13
ip_ct_tcp: ACK is under the lower bound (possible overly delayed ACK) IN= OUT=
SRC=192.168.2.122 DST=192.168.1.10 LEN=40 TOS=0x
00 PREC=0x00 TTL=128 ID=46970 DF PROTO=TCP SPT=2140 DPT=3128 SEQ=4173779699
ACK=2998644048 WINDOW=64313 RES=0x00 ACK URGP=0

These packets are marked as INVALID as they are out of window ones.

Post by Vladislav Kurz
INVALID: IN=eth0 OUT= MAC=00:1a:64:6d:96:3f:00:13:d3:ed:6e:c7:08:00
SRC=192.168.2.122 DST=192.168.1.10 LEN=40 TOS=0x00 PREC=0x00
TTL=128 ID=46970 DF PROTO=TCP SPT=2140 DPT=3128 WINDOW=64313 RES=0x00 ACK
URGP=0
Last line is logged by iptables. What I wonder is that there is corresponding
log from iptables to "ACK is under the lower bound" but not to "SEQ is under
the lower bound".

That is strange. If you log INVALID packets, then you should get the
correspondig log lines.

Post by Vladislav Kurz
I get also quite a few "ACK/SEQ is over the upper bound" bith with
corresponding log from iptables. Does that mean that packets with "SEQ
under the lower bound" are ESTABLISHED ? Thinking more about it, they
should - ACK might be lost, so retransmitting already ACKed data is ok.

If they were not INVALID, then would be ESTABLISHED.

Post by Vladislav Kurz
Where can I find which ip_ct_tcp log entries get flagged as INVALID and which
as other states?

I don't quite understand what do you mean here. All packets logged with
"ip_ct_tcp: text" are flagged as INVALID.

Best regards,
Jozsef
-
E-mail : ***@blackhole.kfki.hu, ***@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Vladislav Kurz

2008-06-19 11:46:14 UTC

Post by Jozsef Kadlecsik
# echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
If the INVALID packets disappear, it can indicate the TCP window
tracking bug and then you should use a newer kernel release (or run
with ip_conntrack_tcp_be_liberal enabled).

Yep, I had this enabled on those servers. Now I turned it off on one of
ip_ct_tcp: SEQ is under the lower bound (already ACKed data
retransmitted) IN= OUT= SRC=192.168.1.10 DST=192.168.2.122 LEN=40
TOS=0x00 PREC=0x00 TTL=64 ID=44558 DF PROTO=TCP SPT=3128 DPT=2140
SEQ=2998644047 ACK=4173779699 WINDOW=62780 RES=0x00 ACK FIN URGP=0 UID=13
ip_ct_tcp: ACK is under the lower bound (possible overly delayed ACK) IN=
OUT= SRC=192.168.2.122 DST=192.168.1.10 LEN=40 TOS=0x
00 PREC=0x00 TTL=128 ID=46970 DF PROTO=TCP SPT=2140 DPT=3128
SEQ=4173779699 ACK=2998644048 WINDOW=64313 RES=0x00 ACK URGP=0

These packets are marked as INVALID as they are out of window ones.

Post by Vladislav Kurz
INVALID: IN=eth0 OUT= MAC=00:1a:64:6d:96:3f:00:13:d3:ed:6e:c7:08:00
SRC=192.168.2.122 DST=192.168.1.10 LEN=40 TOS=0x00 PREC=0x00
TTL=128 ID=46970 DF PROTO=TCP SPT=2140 DPT=3128 WINDOW=64313 RES=0x00
ACK URGP=0
Last line is logged by iptables. What I wonder is that there is
corresponding log from iptables to "ACK is under the lower bound" but not
to "SEQ is under the lower bound".

That is strange. If you log INVALID packets, then you should get the
correspondig log lines.

Well, my bad in this case, I forgot to log INVALID in OUTPUT chain.
But anyway I see lots of "ip_ct_tcp: invalid packet ignored" without
corresponding log from iptables. And vice-versa, iptables log much more
invalid TCP packets then ip_ct_tcp does.

If they were not INVALID, then would be ESTABLISHED.

Post by Vladislav Kurz
Where can I find which ip_ct_tcp log entries get flagged as INVALID and
which as other states?

I don't quite understand what do you mean here. All packets logged with
"ip_ct_tcp: text" are flagged as INVALID.

If all are flagged as invalid then they would be logged by iptables but not
all of them are, and vice versa quite a lot is logged as INVALID but not with
ip_ct_tcp: text.
--
S pozdravem
Vladislav Kurz

=== WebStep, s.r.o. (Ltd.) ========= a step to the Web ===
address: Mezirka 1, 602 00 Brno, CZ, tel: +420 548 214 711
=== www.webstep.net ======= ***@webstep.net ===

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jozsef Kadlecsik

2008-06-19 13:13:26 UTC

Post by Vladislav Kurz
Last line is logged by iptables. What I wonder is that there is
corresponding log from iptables to "ACK is under the lower bound" but not
to "SEQ is under the lower bound".

That is strange. If you log INVALID packets, then you should get the
correspondig log lines.

If iptables logs much more INVALID TCP packets than what is logged by
ip_conntrack_log_invalid, that's normal. The latter simply gives more
information on why the packet was flagged as INVALID, for selected cases.
Not for all INVALID TCP packets. But you should always find the
corresponding log lines if you log INVALID packets too.

Best regards,
Jozsef
-
E-mail : ***@blackhole.kfki.hu, ***@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jan Engelhardt

2008-06-19 10:50:48 UTC

Post by Vladislav Kurz
I noticed that iptables offers new match: -m conntrack --ctstate
which has also NEW, RELATED, ESTABLISHED and INVALID states. Are they
somehow different from those matched by -m state --state ?

The conntrack match offers more internal flags (like SNAT, DNAT) above
the state match.

Ok, thats documented,what I wanted to know is if the four states NEW, RELATED,
ESTABLISHED and INVALID are equal in -m conntrack and -m state.

They are the same, but conntrack has more matching options.

Post by Vladislav Kurz
iptables -t filter -A INPUT -m state --state NEW -p tcp ! --syn -j LOG
and it catches quite a lot of packets, but still it seems that conntrack
knows about TCP connections being opened (SYN) and closed (FIN). Is there
any reason for NEW packets without SYN to be allowed through?

Yes, it's a (well known) feature of netfilter. This way we can catch up
connections already established. If you do not want to support it, use
the last rule you wrote.

I thought it might be useful to block some weird portscans (e.g. nmap xmas
scan).

Xmas scans can be matched much more easily - using INVALID, see
http://jengelh.medozas.de/projects/chaostables/

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jan Engelhardt

2008-06-19 10:54:13 UTC

Post by Jan Engelhardt