March 2014 Broadband Report
Live Website Download Comparison Exposes ISPs
TrueNet's new focus is on the time taken to download 8 live websites. The smaller ISPs came out on top with a standout performance from Voyager VDSL measured at 2.5 seconds download speed for all websites. Vodafone ADSL had the biggest discrepancy between test website (6.1s), and live website (2.8s) download speeds.
Voyager VDSL also came out on top for best Time of Day performance achieving 98% of their best speed throughout the day. Flip achieved the best ADSL performance maintaining 99% of their best speed.
Chorus states that 51% of connections can be VDSL, while only 8% can be Fibre. The 8% is chnaging slowly, so if you are unable to connect with Fibre, then VDSL is a great option.
- Webpage download time
- Download Speed
- Upload Speed
- Want NZ to have faster broadband? Become a TrueNet Volunteer
Summary of Performance Measures - All ISPs
TrueNet selected a new set of live webpages for tests during March. These were selected from the next group of popular websites from alexa.com. By changing live webpages we ensure ISPs must improve all of their network to be ranked top in webpage downloads rather than just the test sites we choose.
In March, the best ISP for webpage downloads using ADSL is a small subsidiary of Slingshot called Flip at 3.1 seconds. Slingshot was second at 4.1 seconds, taking 30% longer. The worst ISP was Vodafone ADSL, at 6.1 seconds, taking almost twice that of Flip.
VDSL is a lot quicker, with all suppliers taking about the same time as the best ADSL, ranging between 2.5 to 3.4 seconds. The best VDSL supplier was Voyager.
If you have access to Orcon Fibre or Snap Fibre, this is by far the best option, taking a lightning fast 2.0 and 2.1 seconds respectively to download all webpages.Table 1: Summary of All Performance Measures
The time it takes to download a webpage is inherently related to a user's internet experience. TrueNet measures webpage download performance in two ways - using Test webpages, and Live Public Webpages.
Test Webpages are kept at the same size, and are easily cached by the ISPs. At the beginning of March, TrueNet refreshed their testing regime with all new Live Public Webpages.
Each chart shows the time to download all webpages, measured in seconds, with total download time shown as stacked bars in the charts.
Live Webpage Download Times
Eight popular public web sites (NZ Post, Warehouse, etc.) are used in this test. For several ISPs and their Services, the NZ Post site proves the most challenging.Chart 1a: February 2014 Live Webpage Results
This month we report 5 VDSL suppliers, with fairly similar results although Voyager is the best by a small margin. Flip is outstanding in the 7 strong ADSL group with a performance matching the VDSL suppliers.Chart 1: Live Webpage Download Time in Seconds
Test Page Download Times
TrueNet test pages are downloaded from our servers in Wellington, Auckland, Sydney and Dallas. All testpages pages are identical.
Test pages are relatively easy to cache, so on international links we expect to see caching making both international, and national downloads almost equal.
Test Page times are very similar to last month, but it is clear that some ISPs are not yet caching, with some very slow downloads of our Sydney & Dallas sites. Compare Slingshot VDSL & Vodafone Cable with Snap ADSL & Telecom ADSL.Chart 2: Test Page Download Time in Seconds - National & International
Chart 3 - compares Live webpages with Test webpages over the last 6 months and enables a better understanding of the impact of caching, and managing pages that change a lot. Chart 3 includes a lot of information. Study it, the effort is worth it. The bars are the total time to download all webpages, averaged over the period. Points to note:
- Slingshot & Vodafone's Live pages download times are almost equal before and after March, the other 3 ISPs take longer in March.
- Vodafone's total Live pages download times is much larger than all other ISPs.
- While Telecom's Test pages download times are amongst the worst, their Livepages results are competitive with others.
- Snap's earlier lead in Live pages has gone, with the new sites they are roughy equal with 3 others.
We think the Live webpage method of measurement is more reliable than the Test webpages, however the Testpages provide a common size for comparing what is possible. For those with interest in the detail, we include a summary of performance by month at the end of this report.Chart 3: Download Comparison - Live Webpages & Test Webpages
Fibre - Cable - VDSL Comparison
This section looks at the relative performance of Fibre, Cable and VDSL technology, reporting on actual speeds achieved by Volunteers' probes, by time of day.
The TrueNet Speed test downloads a 1MB file from both Auckland and Wellington every 5 hours per probe. The best download speed from Auckland or Wellington is used from each test run, and the data points below are the medians of these. We compare the worst speed during the day to the best to get a Time of Day (TOD) comparison.
ISP File Download Comparison
While Vodafone Cable performance has improved to match that of Snap's Fibre during the day at almost 90Mb/s, the performance drops to 50Mb/s when most customers wish to use it. Note, this drop in performance is also evident in the 15Mb/s cable product, and appears as a feature of Cable.
While Orcon's 100Mb/s Fibre service lags behind Snap, their 30Mb/s service is consistently above 30Mb/s.
VDSL speeds seem to be rising as more ISPs introduce the product and TrueNet collects more volunteers. We had 86 VDSL probes report during March.Chart 4: Fibre, Cable, and VDSL File Download Speeds
Technology File Download Comparison
TrueNet compares all probes to demonstrate the performance of each technology, including those from minor ISPs.
We have 54 fibre probes from various ISPs, and are currently recruiting Telecom fibre volunteers.
VDSL speed is dependent on the distance between the volunteer modem and the DSLAM, and with cabinetisation, we have 50% of our probes on FTTN. FTTN VDSL has an average speed of 25Mb/s, whereas Exchange VDSL is just 15Mb/s due to the longer distances to Volunteers' connections.
Some ISPs are struggling to achieve the 100Mb/s fibre speed to our servers, so when we include all probes, the average speed drops to 70Mb/s. Interestingly, the 30Mb/s speed remains the same across all ISPs.
The cable options we test (100Mb/s and 15Mb/s) both show a large speed dip during the peak demand period.Chart 5: Comparison of File Download Speeds by Technology
Time of Day Comparison
TrueNet has used this measure as a standard for performance since we began publishing reports. ISPs have improved performance consistently in the last 3 years as seen in last week's report where we compared NZ with the UK, and USA.
VDSL ToD performance
Results are based on a 1MB file downloaded by each probe every 5 hours, from both Auckland and Wellington servers. The best speed, from Auckland or Wellington, for each probe is used, and the median of these is reported.
All ISPs offering VDSL are performing at better than 93% of their best speed throughout the day. ADSL performance is less consistent, with the poorest result at 87% of an ISPs best speed.
With 86 VDSL probes, we now have more ISPs, and growing reliability of results. The new ISPs are up to the speed performance of the first ISPs to migrate to VDSL, with Voyager topping the list and Slingshot not far away.Chart 6: VDSL File Download Speed by Hour
Telecom VDSL performance dropped during the busy hour, although this was not the case for ADSL. Conversely, Slingshot had a very high performing VDSL, while their ADSL dropped significantly.
ADSL ToD performance
While many volunteers are migrating to VDSL, we continue to have almost 250 ADSL probes. We believe it is relevant to separate the results of these technologies at present.
Flip again came out on top in March, for the second month, after just two months' reporting. Slingshot and Vodafone results are noticably worse right through the day and evening periods, with Vodafone dropping below 90% for the first time in 3 months.Chart 7: ADSL File Download Speed by Hour
TrueNet uploads a 1MB file from all volunteers' networks to our server in Wellington. We measure the best decile speed of these uploads.
File Upload Performance by Technology
Upload speed is almost always the same speed as that advertised. The only exception is 100Mb/s fibre advertised as 50Mb/s upload, with our measurements showing between 35 and 40Mb/s upload. To some extent this is due to the specifications of fibre speeds supplied by the LFC's as explained here - 50Mb/s is supplied to the ISP but cannot be passed on at that speed to the user. Until the upload speed supplied is 55Mb/s at the LFC end, the user will never get 50Mb/s. This is true of 10Mb/s fibre upload also, but harder to see on this scale.Chart 8: File Upload Speed by Time of Day: Fibre - Cable - VDSL
ADSL File Upload Performance by ISP
Upload speed on ADSL is not so dependent on distance, but is dependent on provision using Unbundled Local Loop (ULL).
Chart 9, shows Flip is almost exclusively using ULL to obtain much higher upload speeds for ADSL (bigger is better).Chart 9: ADSL File Upload Speed by Time of Day - ISP Comparison
Latency is the time it takes for a packet of data to be returned by a remote server to a Volunteer's probe.
Latency can impact many internet activities. Very poor latency will make browsing the internet difficult if page requests fail due to timeouts. High latency can make gaming impossible, adding to reaction time, meaning if someone else has lower latency they may see a game change, and react well before a slower connection is able to show the change (ie you can be shot and killed in a game before your computer shows the shooter)
Cable has the highest latency based on the total of all three measurements (refer Summary Table at the start of the monthly report). Surprisingly, the 100/10M Service has higher latency than the 15/2M service.
There is only a small difference in Latency across Technologies, but Fibre and VDSL have marginally lower latency than cable and ADSL.Chart 10: Latency Performance - by ISP and Technology
A user's Internet Browsing experience can be affected by slow DNS response times, as well as Latency and slow line speed.
TrueNet measures the time taken to receive a response from all of the ISPs DNS servers, and records the best result for each hourly test.
By ISP (Urban only)
Snap stands out in the South Island and is best in Auckland. Vodafone performance in Wellington and Canterbury is likely due to the number of cable probes included along with Vodafone DSL probes.Chart 11: Regional Domain Name Server Response Time by ISP
Technology is a strong predictor of Performance in this test - Cable and Fibre are clearly the best. VDSL in the South is much the same as ADSL, and may be due to lower South Island investment levels.
Otago also had the slowest response times in every Technology, implying that the distance to the DNS server - hence Latency - is likely to be the limiting factor.Chart 12: Domain Name Server Response Times by Technology
TrueNet measurement service shows that ISP performance varies over time. In the past year some top performing ISPs have started to slip in the rankings, as others have risen. we expect to see this change in lead continue as ISPs investment in their networks strives to, but does not always match customer demand.
Regular reporting by TrueNet is providing the incentive for ISPs to strive for the premium position in rankings, helping all Internet users to gain the best performance possible.
Volunteer to test in Australia here - (any technology)Glossary
Details on how we measure are available on our Technical page.
ADSL, VDSL, DSL - the standard broadband service provided over a telephone line from an exchange or a cabinet (FTTN), VDSL is a faster version than ADSL. They use similar technology and backhaul, so sometimes DSL is used when referring to both.
Capped Plans - the most common ADSL service, where you have a monthly plan having a GigaByte (GB) limit of usage each month before your speed is slowed or you must pay more.
Unlimited Plans - ADSL service where there is no monthly limit on the amount of data used. Specifications for this service include that it may be "Managed" and have "performance reductions applied during peak demand periods."
Cable - Cable is offered by Telstra & Optus, and is available in a limited number of cities.
DSLAM - the exchange or cabinet based equipment that your modem is connected to, over the pair of copper wires that are exclusively allocated to your premises.
Ethernet - The wiring used to connect computers to a network, typically an Ethernet cable is coloured (often blue), with small square connectors at each end.
ISPs - Internet Service Providers. TrueNet has probes measuring almost 20 ISPs but only reports on those where there are 5 or more probes working during any particular month.
Latency - The time for a packet of data to be returned by a remote server to the probe when a "Ping" command is issued. TrueNet sets targets for maximum median latency that are known to be achievable.
Median - The Median is found for each probe and this is input to any analysis to calulate the average performance. This means that any result represents the “middle” performance measure applicable for that probe. Using median ensures that the result is more representative due to the often skewed nature of measurements by probe.
Speed - Throughput or the median peak connection speed achieved during our standard test downloading an image from our test servers. TrueNet normally reports speed as a comparison at low vs high demand times to show any capacity constraints evident in speed performance, often called the Time of Day analysis.
UFB Fibre (NZ) - Ultra Fast Broadband connections are the service offered by some ISPs over the Fibre to the Home (FTTH) network built by LFCs over government subsidised fibres. Services now being offered include 100Mbps and 30Mbps.
NBN Fibre (AUS) - "NBN co is a single entity rolling out fibre nationwide and then wholesaling it to ISP’s" from a good comparison here.
FTTN - is based on fiber-optic cables run to a cabinet serving a neighborhood. It uses existing coaxial or twisted-pair infrastructure to provide connections from the cabinet to the home.
FTTH - Premises are connected using a gigabit passive optical network (GPON). A fibre cable, known as the "drop fibre", goes from the premises to the street. The "drop fibre" cable joins a "local network" which links a number of premises to a splitter in the fibre distribution hub.
LFC - Local Fibre Company. These companies are rolling out FTTH connections subsidised by the government, but must sell services through ISPs.
Webpage Download - TrueNet maintains a Standard Test page which is used for measuring the time to download the entire page. This page is visible here, we use a copy located on our test servers for test downloads. The time to download excludes the time for a browser to generate the page on a screen, some are faster than others.