We ran across a weirdness in one of our customers internet lately. This weirdness is that it had packet loss even though they have 12mbits/sec up and down and we had our Tomato router installed with everything set up nicely (this setup works for hundreds of other customers). Except he uses a fiber connection and I ended up being able to pinpoint the problem to this :
His provider pushes traffic at speeds up to 100mbps to the providers switch at the customers facility and that switch then does traffic shaping to 12mbits/sec on the outbound port to the customers equipment thus dropping some of the packets.
What can you do in such a case ? Well I discussed this with a network engineer and ended up seeing the following option : police the incoming traffic to a lower bandwidth than the one the customer is supposed to have (say 10mbps/sec) that way the TCP windows will shrink enough not to saturate the 12mbps.
So how do I do that in practice ?
Well I connected through ssh to the tomato router, edited /etc/qos (that's where the qos happens) and changed the ingress rules to :
$TQA handle ffff: ingress
$TFA parent ffff: prio 1 protocol ip u32 match ip src 1.2.3.0/24 police rate 11000kbit burst 10kbit drop flowid ffff:1
$TFA parent ffff: prio 5 protocol ip u32 match ip src 0.0.0.0/0 police rate 10000kbit burst 10kbit drop flowid ffff:5
What does this do ?
It limits incoming traffic to 10mbits/s with a 10k burst and allows everything coming in from 1.2.3.0/24 to use up to 11mbits/s of traffic.
Result ? No more packet loss and a happy customer.