PPPoE and obscure packet dropping (some websites not working)...

I have been having a mare with packets just disappearing or packets being marked as invalid and dropped, then resulting in established and related packets being dropped in response to mangled requests.  I spent ages trying to debug this, logging anything and everything with iptables log rules.  I couldn't determine what on earth was going on, until I started reading about MSS exceeding the MTU of 1500 set by the ISP.  This is only apparent with large amounts of data and even more apparent on devices connecting from behind the firewall, with mangling going on.  Essentially, it boils down to packets being mangled, increasing the segment size, which eventually leads to a packet that exceeds the MTU when the ISP reroutes it.  At the firwall level, the packet is fine and fits just within the 1500 limit.  Proposed solutions seem to be centred around adjusting the PPPoE interface MTU to something that would not exceed the limit of 1500 when mangled.  This is a good idea to have set.  For me, I modified my PPP configuration and changed the MTU:

$ sed -i.bak 's/^mtu [0-9]\+$/mtu 1454/' /etc/ppp/peers/dsl-provider

This is not always guaranteed to work alone though, as I found out.  This is because you don't know what additional mangling is going on at the ISP's end.  In the end, I started reading the iptables man page for the 3.4 kernel I am running.  I found something very interesting...


TCPMSS
       This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your  outgoing  inter-
       face's MTU minus 40 for IPv4 or 60 for IPv6, respectively).  Of course, it can only be used in conjunction with -p tcp.

       This  target is used to overcome criminally braindead ISPs or servers which block "ICMP Fragmentation Needed" or "ICMPv6 Packet Too Big" packets.  The symp-
       toms of this problem are that everything works fine from your Linux firewall/router, but machines behind it can never exchange large packets:
        1) Web browsers connect, then hang with no data received.
        2) Small mail works fine, but large emails hang.
        3) ssh works fine, but scp hangs after initial handshaking.
       Workaround: activate this option and add a rule to your firewall configuration like:

               iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN
                           -j TCPMSS --clamp-mss-to-pmtu

       --set-mss value
              Explicitly sets MSS option to specified value. If the MSS of the packet is already lower than value, it will not  be  increased  (from  Linux  2.6.25
              onwards) to avoid more problems with hosts relying on a proper MSS.

       --clamp-mss-to-pmtu
              Automatically  clamp  MSS  value  to (path_MTU - 40 for IPv4; -60 for IPv6).  This may not function as desired where asymmetric routes with differing
              path MTU exist -- the kernel uses the path MTU which it would use to send packets from itself to the source and destination IP  addresses.  Prior  to
              Linux  2.6.25,  only  the  path MTU to the destination IP address was considered by this option; subsequent kernels also consider the path MTU to the
              source IP address.

       These options are mutually exclusive.

So I wasn't going mad after all and it does appear to be a widely known issue.  So I gave it a try...

-A FORWARD -p tcp --tcp-flags SYN,RST SYN -j LOG --log-prefix "CLAMP-MSS-TO-PMTU" --log-tcp-options --log-ip-options --log-level 7

-A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

I am only specifying the LOG action here to ensure that it works.  Indeed it does.  After enabling it, I instantly started seeing entries appear in the logs and all the websites that weren't previously accessible  suddenly started working.  I don't need to explain why this works, since it's explained enough in the above manual excerpt.  But if you share my blight, then hopefully this helps you too!