I recently received a report entitled 'Website testing and ranking of
Major Amateur Sports Bodies and Olympic related websites' from a company
called Business2ww/SiteMorse.
I had never heard of them previously, but I now understand that they do
publish various website league tables from time to time.

Anyway, our site came 24th in the tables, so obviously I wanted to find
out how we could improve this - as the published report did not have any
detailed technical information I went to the SiteMorse website and saw
that I could submit a site for a 'quick test', which I did.

Now, the report came back with the various issues, which included items
such as accessibility, HTML validation, transfer speed and response
time. Well I'm not arguing with the accessibility and validation issues
as the specs for these are published in the relevant standards
documentation, so I can see how our pages comply (or not!).
What I do think is incorrect however is the reported data transfer rate
and response times.

Heres an extract from the Sitemorse quick test:

Ten slowest URLs by speed
URL                                     Transfer speed (bytes/sec)
http://www.british-gymnastics.org/milano.jpg    2,727
http://www.british-gymnastics.org/gymnova.jpg    2,832
http://www.british-gymnastics.org/gymaid.jpg    3,164
http://www.british-gymnastics.org/awards1.jpg    4,088
http://www.british-gymnastics.org/venturelli.jpg    6,816
http://www.british-gymnastics.org/rover.jpg    10,366
http://www.british-gymnastics.org/sub_whatpapersay.gif    10,407
http://www.british-gymnastics.org/spring1.jpg    11,827
http://www.british-gymnastics.org/lottery1.jpg    16,928
http://www.british-gymnastics.org/uksport.jpg    18,067

Now, those speeds seem extremely slow considering the website is hosted
on a dedicated server with very little load, in an ISP hosting facility,
and I was a little concerned that there may be a problem.
So I did some of my own tests using wget on those files, as well as some
large files (7MB), and it shows very fast speeds of around 200 Kilobytes
per second. I also got a colleague to test it on his University
connection elsewhere and showed even faster speeds. So I can't see a
problem with our server or bandwidth. I did repeat the Sitemorse test to
ensure it wasn't just an anomaly. (You can repeat tests after clearing
your cookies!)

So, obviously I contacted Sitemorse to try and clarify why this may be
the case. Maybe it's due to their method of testing?
They replied and gave me some marketing blurb about testing overall
website performance and not individual files, and blamed the poor speed
of the site on poor HTML.

So I responded and pointed out that they do quote speeds for individual
files, and the content of the HTML documents on the site have no
relevance to the transfer rate of an HTTP GET request on a specific
file. I suggested that maybe they could give me some information as to
how they test the speed, e.g. does the system add any calculations for
'simulated' transfer speed, does it fetch the same object multiple times
simultaneously, or consecutively then average, what hardware does the
system run on, does it use standard transfer rate measurement tools or
custom code, what type and speed is the Internet connection serving the
testing hardware etc. etc.

Anyway, several emails went back and forth where I explained the tests I
had carried out, but with no useful information coming back from
Sitemorse except for marketing blurb, and to cut a long story short,I
eventually received this response:

----
The speed is only a small part of the overall score, if you are trying
to find a way to discredit our work I would suggest your time is far
better spent sorting out the very poor HTML on your site.
----

Not a very professional reply. I was simply trying to establish how they
arrive at the speed results!
Anyway, I thought it would be interesting to submit for a quick-test the
site that came top in their tables for transfer rate (note, I am NOT
critisicing any other site, just using this as a benchmark):
The test came back with 3 FAILs under transfer rate, and the following
detail against one of the files:

Sitemorse test: http://www.ihf.info/gfx/four_dots.gif    55bytes/sec
My test :  wget http://www.ihf.info/gfx/four_dots.gif (361.33 KB/s)

I repeated the tests, just to make sure. Results were very similar.
I sent this result to SiteMorse again asking to clarify this, and again
asking 'how do you measure the transfer rate?'

The response was:
----
How about, the speed that data is transmitted at.
----

OK so I gave up trying to get any sense out of them at this point. It
seems that they are unable to give any kind of reference to how their
speeds are calculated, and as they obviously are wildly different from
real-world tests it would appear that the results are meaningless, and
just plain wrong!

Is there another explanation that I may not have thought of?