How we compute a score

What rules govern every score?

Eight rules govern every score on the site. They are generated from the code that computes the scores, so if a rule changes this text changes with it:

1. The score is the weighted sum of the sub-scores; the weights add up to 100%.
2. Rounding: each measurement’s points to two decimals, each sub-score to one decimal, then the score to one decimal, half up.
3. An unmeasured criterion is left out, the remaining weights are re-scaled, and the score is labelled provisional everywhere it appears.
4. Order comes from the score alone; a tie after rounding goes to the higher unrounded sum. No position is set by hand, and no position in any list is for sale.
5. Every service on the list carries a visit link to its own page, whatever its score. The score is our reading; the link leaves the decision to you.
6. A superlative is written only for a sole winner in the stated scope; ties earn none.
7. Each measurement’s points are held between 0 and 10.

How do we know what we publish?

Before any formula comes the question underneath it: where did the figure come from? Every figure we publish has one of six provenance levels. Once real provider figures start going up, the level is printed beside the figure itself and in the sources table at the foot of the page:

• Measured by us: the plan is bought and the measurement made. No figure on the site is of this kind today.
• Captured from their page: we read it on the company’s page and kept a dated capture of it.
• The company states it: published on its page with no capture kept; the link and the date are given.
• Users report it: a repeated pattern in user reports, never a single complaint. We do not write it without three independent reports across two separate venues.
• Arithmetic on their figures: a number we computed from published figures. Its provenance is governed by the frailest figure inside it.
• We could not find it: we looked and did not find it, and we say where we looked. A missing figure is information too.

A sub-score is computed only from figures at the first three levels. A figure past its refresh window drops off the page until we read it again: prices are read again every few weeks, policies every few months, and description pages about once every six months.

How does the work run?

The work runs in a fixed order, because the order is itself part of the protection: whoever reads the figures does not know the score they will produce.

• We fix the list of services in the category before reading a single figure.
• We read each company’s pages and record every figure with its link, its date and its provenance level.
• We put the figures into the formulas published below; the sub-scores come out, then the score.
• The editor checks every figure and its source, and redoes the arithmetic.
• The page is published with its date, and any later change is logged in the changelog.

What are the weights?

The weights are fixed and published, and they add up to one hundred percent. They do not change from one service to another or from one month to the next, and any amendment appears under method versions with its computed effect on the ranking.

How is each criterion computed?

Each criterion has its formula written out: what goes into it, how a raw figure becomes points, and what share each measurement has of the sub-score. Points are held between zero and ten, so a high result on one measurement cannot indefinitely offset a poor one elsewhere.

Before you read on: no measurement round has run yet. The protocol written under each criterion is what will apply when the first one does, and the scores shown on the site today rest on placeholder data: figures put in to build the pages and test the arithmetic, not readings about real companies. The site does not go public while that is true.

A fully worked example

The clearest test of a method is redoing it yourself. Here is a complete example on a named service: each sub-score times its weight, the sum, then the rounding.

Verpex

When is a score provisional?

A criterion we have no figure for is not filled in with a guess or an average. It is left out, the remaining weights are re-scaled, and the score is called provisional everywhere it appears until the check is complete.

where s is a sub-score and w its criterion’s weight; both sums run over the measured criteria only.

How do we handle prices, currency and VAT?

Every price is in US dollars, because that is how these companies usually invoice. Local-currency figures are converted at a dated rate and are approximate: the bank that issued your card may give you a different rate and add a conversion fee.

Tax is not included in table figures unless stated, because it follows your country rather than the company. A promotional price never enters a table: the table carries the standard price and the renewal price, and a temporary offer is mentioned in prose with its expiry date.

When do we write a superlative?

A phrase that sets one service above another is not an opinion we write; it is a computed result. A claim goes only to a sole winner in a stated scope, and a tie awards nothing. The table shows the available claims and who holds each one today:

Claim types with no sole winner today: 5

When do we check again?

Every service carries the date it was last checked and the date of the next check, both shown on its page. A figure past its window does not sit on the page waiting for the next round.

What have we not checked yet?

That this section exists at all is part of the answer. The list below is generated from the records themselves, and counts what went unchecked in each review:

Sample on record: 16 services, of which 1 excluded.

Where is this method weak?

Every method chooses what to weigh, and the choice is itself a judgement. These are the weak points as we see them today:

• The criteria that need a purchase have not been measured. While that is true the picture is incomplete, and the score is labelled provisional.
• What a company says about itself is not evidence of what it does. A written refund policy can differ from the refund policy applied, and we cannot prove the gap until we try it ourselves.
• The weights are a choice. Change them and the order changes, which is why we publish them, hold them still, and log every amendment with its computed effect.
• A company that publishes less looks worse than it may be, because absence counts against it. That is deliberate, but it is not neutral.
• The list of services is not the whole market. A company missing from it is missing from the ranking, not from the market.

What about the other categories?

We build the categories one at a time. A category whose measurement model is not finished shows stored scores, labelled as such, and is not mixed in with a measured one.

How do we change the method?

We do not change the method quietly. Every amendment has a date and a reason, and the scores are recomputed both the old way and the new one so the effect appears as a number.

Can you use our data?

Each category’s data is available as a single CSV file from its comparison page, holding the same figures that compute the scores on the site.

You may use and quote it free of charge, provided you give the source, the date it was read, and a link to the page. We ask for no permission and charge no fee, and we grant nobody the right to use our scores in an advertisement without their date.

How does money affect the ranking?

The site is funded by commission from the companies listed, when a reader signs up through a “Visit site” button. No agreement has started yet.

What matters here is that commission enters none of the formulas on this page. No figure in the score is one a company can move by paying, and the visit button follows a rule that can withhold it from a company that pays us. The full detail is on the income and disclosure pages.