Teaching Points
In this article, you will learn:
- Cause of asymmetric upload and download throughput
- Learn Network speed interface mismatch concept
- What is Flow Control
Chief Complaints
“My upload speed is much faster than the download.”
Signs/Symptoms
- Using iPerf local speed test, my local WiFi download speed caps in 300 Mbps range while upload can reach to ~600 Mbps.
- Using internet speed test, 10 Gigabit wired client can get beyond 1 Gigabit throughput while 1 Gigabit client caps at under 600 Mbps download.
Above symptoms are both real world examples I myself encountered due to “one single” same problem. However, before digging further about this, let’s ensure simple issues are checks.
ISP restriction
Xfinity Gigabit Internet service has advanced, next generation technology, with WiFi download speeds of up to 1000 Mbps (up to 1200 Mbps in some areas) and upload speeds of up to 35 Mbps.
https://www.xfinity.com/support/articles/get-most-from-xfinity-gigabit-internet-service
Internet throughput testing has many variables that are beyond control of an end-user. So it is not a reliable way of testing network speed. However, after testing with different speed testing sites, different time of the day and still getting consistently asymmetric internet speed result, one main reason may be ISP restriction.
For example, Xfinity/Comcast internet service is highly asymmetric. My gigabit plan is 1 Gbps download and 35 Mbps upload. This is simply the service restriction and actually expected result.
Asymmetric Traffic Congestion
Wired part of network is full duplex and wireless part is half duplex. This means upload and download path are independent on wired parts of the network. If you are streaming Netflix, you may be using a part of download bandwidth while this itself have no effect on the upload bandwidth. So if you run download and upload speed test, your download may be asymmetrically reduced due to download path congestion. In general, this type of asymmetry may be reduced/alleviated by taking out network congestion i.e. stop streaming while doing speed test for more accurate measure. So for maximum throughput test result, one must ensure turning off all other processes.
Real Asymmetric Throughput issue
Now let’s take a closer look at individual cases I had been facing.
Scenario 1: Using iPerf local speed test, my local WiFi download speed caps in 300 Mbps range while upload can reach to ~600 Mbps.
Since the issue is only seen on the 10G pathway, one may think the issue is the difference between the two pathways i.e. USW Aggregation switch. However, it is worth noting that I can get full 940 Mbps up/down via both 10G and 1G pathways with wired client connected to the USW Pro 24 Gen 2 switch.
Scenario 2: On internet speed test, 10 Gigabit wired client can get beyond 1 Gigabit throughput while 1 Gigabit client caps at under 600 Mbps download.
This is yet another odd result at the first glance. Above numbers demonstrate I can get beyond 1 Gigabit throughput when 10Gbps pathway is utilized; however, changing 10Gbps link to 1 Gbps link on my iMac 2020 client reaches almost 50% of full gigabit bandwidth. One may wonder this is now due to USW Pro 24 Gen 2 switch’s issue.
In this particular case, I can say my USW Pro 24 Gen 2 Pro switch is working fine and not defective because the issue started after I swapped my modem.
With original setup i.e. Netgear CM 1100 modem, I had full 940 Mbps up and down either via 1Gbps or 10Gbps links, which are essentially identical setup and topology except for the modem.
Now expanding above two scenario to a bit more systematic, testing setup, I have done followings:
3 senders:
- Local Qnap NAS running Iperf Server with 10 Gigabit Ethernet interface
- Local Qnap NAS running Iperf Server with 1 Gigabit Ethernet interface
- Internet Xfinity gigabit (1.2Gbps) service through Surfboard S33 modem 2.5 Gigabit Ethernet interface
3 clients/receivers:
- iMac 2020 via 1 gigabit ethernet port
- iMac 2020 via 10 gigabit ethernet to thunderbolt 3 adapter
- iPhone 12 Pro (866 Mbps wireless link) via UniFi Access Point HD
3 senders, 3 clients created 9 test conditions and results are as following.
Assessment/Diagnosis
Network interface mismatch
Interface mismatch issue
Without a knowledge about this potential, this is a very tricky one to figure out and may even be extremely frustrating issue because those who encounter this issue are usually ones who had spent more time, effort and perhaps money to construct higher bandwidth network. Throughput suffers when there is significant mismatch with high rate sender to lower rate receiver.
Throughput suffers when there is significant mismatch with high rate sender to lower rate receiver
One fact about network data transfer is network interface card is designed to operate in either “on” or “off” mode. When it is “on” the rate of data transfer is based on the line rate (ref).
This binary operation means that a smooth “average” speed is never possible, and as such devices in the path must be ready to accept the line rate burst of data, or risk dropping some due to a lack of buffer space. The speed mismatch happens when there is a lack of available network buffering to handle the influx of data and drop packet […]
https://fasterdata.es.net/performance-testing/troubleshooting/interface-speed-mismatch/
This situation has been shown to be true in several environments:
– 10G to 1G host
– 10G host to a 2G virtual circuit
– 40G to 10G host
– Fast host to slower host (based on CPU speed)Note that not all 10G to 1G mismatches will produce this behavior. If there is sufficient buffering in the network (e.g. enough to handle the 10Gbps bursts that the faster network card may produce), the problem will not exist.
https://fasterdata.es.net/performance-testing/troubleshooting/interface-speed-mismatch/
Analogy
It’s time for analogy. This time, let’s use fast food restaurant.
- # of customers coming in at once = data send rate
- # of casher at fast-food restaurant = receiver accepting data rate
- # of customers can line up inside restaurant = buffer size
Matching Case
When # of cashers and # of customers entering restaurant are matching, there is no issue. This is an optimal situation with customers will be happy that they do not need to wait and able to continue flow. As a side note, restaurant is also happy as they are not waisting cacher. So in the above case, 3 customers can come in and order then next 3 customers immediately after without any wait/interruption.
In a full gigabit network, this is analogous to downloading a huge file from internet hosted server at full 1 Gbps rate on your client computer. This can happen because along the pathway there are full 1 Gigabit throughout including 1 Gigabit internet service to modem/router, (router to switch) and to end client device (PC).
Low to high mismatching: Too many cashers
This is a mismatch case #1 where there are more cashers than customer. There isn’t any issue here from customer side. Each customer comes in and one of casher takes the order and customer continues to flow without any wait.
In the network, this is a case where you have a 10 Gbps capable client device downloading file at 1 Gbps rate because that’s what your modem/server is receiving (internet service provider limit). You may be not fully utilizing your network bandwidth but you will get full 1 Gigabit (that you are paying for).
High to low mismatching: Too many customers
This is a mismatch case #2 and is the conundrum. When there is only one casher but 3 customers coming in at once, customers have to start forming a line as only one can order at a time. Now while customers can line up and wait inside a restaurant for awhile, if 3 customers rate continues, the restaurant will eventually be full due to the waiting line filling up entire restaurant. At that point, customers will see the restaurant is too crowded and stop coming in. This results in interruption of continuous customer flow and rate actually goes below 1 customer at a time.
In the network world, this is sender/upstream sending data at higher rate than what client/downstream can receive. While buffer remains available, full speed of the client/downstream rate can be maintained, once the buffer overflows, sender completely stops sending data. Hence, rate will go below full client/downstream rate.
It is important to note that network flow is bidirectional (upstream and downstream). So depending on the task, what’s acting as sender becomes recipient. Therefore, the symptomatic high to low mismatch on one direction becomes asymptomatic low to high mismatch on the other direction.