Lumen help
Are you experiencing strange throughput issues? It could be related to your MAC address. If your MAC address begins with a 4 or a 6, this could contribute to packet reordering over your Lumen service—which may cause slow speeds. This issue can appear intermittently and make it difficult to troubleshoot.
For L2VPN VPLS or E-Line, the OSI Layer 3 endpoints are on the CE devices only. Therefore, it’s only the CEs’ MAC addresses that would impact whether packets are reordered. Anywhere there are Ethernet PWs, there is potential for packet reordering. The reordering occurs with traffic destined to MAC addresses with a leading 4 or 6. In the example below, both directions would experience packet reordering because both CEs’ MACs lead with 4 or 6.
For L3VPN MPLS, the OSI Layer 3 endpoints are on the CEs and PEs. Therefore, the MAC addresses of both PEs and both CEs would impact whether the packets are reordered. Anywhere there are Ethernet PWs there, is potential for packet reordering. The reordering occurs with traffic destined to MAC addresses with a leading 4 or 6. In the example below, both directions would experience packet reordering because the PEs’ and CEs’ MACs lead with 4 or 6.
For DIA, the OSI Layer 3 endpoints are on the CE and PE. Therefore, the MAC addresses of the CE and PE would impact whether the packets are reordered. Anywhere there are Ethernet PWs, there is potential for packet reordering. The reordering occurs with traffic destined to MAC addresses with a leading 4 or 6. In the exanple below, only the upload traffic would be reordered.
You can check for this issue by looking at your CE’s ARP table. If either of the ARP entries for the CE-WAN interface begin with a 4 or 6 you could be experiencing this issue. You can open a ticket and we’ll confirm if you’re impacted by this issue and what the next steps are.
Cisco IOS [CE-WAN interface is GigabitEthernet0/0]
CE-ROUTER#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.233.1 131 6c18.a042.24c4 ARPA GigabitEthernet0/0
Internet 192.168.233.2 - 4403.a792.29e0 ARPA GigabitEthernet0/0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Juniper [CE-WAN interface is ge-0/0/0]
User@CE-ROUTER> show arp
MAC Address Address Name Interface Flags
40:19:aa:01:b1:79 192.168.1.1 192.168.1.1 ge-0/0/0 none
64:b5:2f:f1:8f:c5 192.168.1.2 192.168.1.2 ge-0/0/0 none
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Cisco IOS [CE-WAN interface is GigabitEthernet0/0]
CE-DEVICE#sho int GigabitEthernet0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 4403.a798.29e1 (bia 4403.a798.29e1)
CE-DEVICE #config t
Enter configuration commands, one per line. End with CNTL/Z.
CE-DEVICE (config)#interface GigabitEthernet0/0
CE-DEVICE (config-if)#mac-address 5403.a798.29e1
CE-DEVICE #sho int GigabitEthernet0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 5403.a798.29e1 (bia 4403.a798.29e1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Juniper [CE-WAN interface is ge-0/0/0]
User@CE-ROUTER> show interfaces ge-0/0/0
Physical interface: ge-0/0/0, Enabled, Physical link is Up
Interface index: 134, SNMP ifIndex: 508
Current address: 6c:5e:ab:e5:f5:80, Hardware address: 6c:5e:ab:e5:f5:80
User@CE-ROUTER> configure private
User@CE-ROUTER# set interfaces ge-0/0/0 mac 5c:5e:ab:e5:f5:80
User@CE-ROUTER# commit and-quit
User@CE-ROUTER> show interfaces ge-0/0/0
Physical interface: ge-0/0/0, Enabled, Physical link is Up
Interface index: 134, SNMP ifIndex: 508
Current address: 5c:5e:ab:e5:f5:80, Hardware address: 6c:5e:ab:e5:f5:80
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When traversing an MPLS network, if the traffic is carried using an Ethernet PW (pseudowire) there is a possibility that packets will be reordered due to ECMP (equal-cost multipath) mechanisms within the MPLS network. This reordering of packets can have a negative impact on your traffic (voice, video, or data). Lumen has observed that this issue often impacts data traffic, specifically TCP throughput.
Normally, each protocol header will have an identifier so that it understands the payload. Ethernet II has the ethertype, IPv4 has the protocol ID, IPv6 has the next header, etc. MPLS has no such identifier in its header, so it doesn’t know what the MPLS payload will be. Normally, ECMP will examine the first nibble (4 bits) of the IP header to determine its version. If it’s a 4, it’s an IPv4 packet, if it’s a 6, it’s an IPv6 packet. With this issue, the MPLS router assumes that the packet carries a IP payload if the first nibble following the MPLS stack contains a 4 or 6. For an Ethernet PW, this first nibble following the MPLS stack is the first character of the destination MAC address inside the Ethernet II header. This means the MPLS router may mistake the MPLS packet as an IPv4 packet if the destination MAC begins with a 4 or as an IPv6 packet if the destination MAC begins with a 6. Because the MPLS router incorrectly identifies the packet as an IP packet rather than an MPLS packet, the normal tuple fields used for hashing will be out of alignment and traffic belonging to the same flow is load balanced over multiple paths causing out-of-order packets. Normally, traffic belonging to the same flow is pinned to the same path to ensure packets remain in order. For more details on the MPLS/ECMP hashing issue, please see the following RFCs:
Every time a TCP device receives an out-of-order packet, it replies with a duplicate ACK. If enough duplicate ACKs are generated (TCP duplicate ACK threshold: 3 “Dup ACKs”), it causes TCP to retransmit the packet. This threshold is also used for packet loss recovery as packet loss will appear to TCP as out-of-order packets. Anytime the TCP duplicate ACK threshold is exceeded, the missing packet (set of bytes) is retransmitted. TCP wasn’t designed to retransmit every time packets arrive out-of-order as TCP can put them back in the right order and there’s no need to retransmit a packet that has successfully been delivered. However, the TCP duplicate ACK threshold is in place to strike a balance between not taking too long to recover from packet loss and not assuming that every out-of-order packet was due to packet loss. When a packet is retransmitted, TCP will reduce its congestion window according to the congestion algorithm in use. This results in slow throughput as the TCP performance is a function of the TCP window size divided by the RTT (round-trip time). A smaller TCP window equates to a slower speed, for example:
(TCP Window (bytes) * 8) / (RTT (ms) / 1000) = TCP throughput (Mbps)
(500,000 bytes) * 8) / (50 ms / 1000) = 80 Mbps
(250,000 bytes) * 8) / (50 ms / 1000) = 40 Mbps
For more details on the TCP behavior, please see the following RFC: