Aurum wrote:
If I'm not mistaken, the questions are randomized, which makes me wonder how reliable any such test is.
Only the order in which the questions appear is randomized. The reason for this is to get more reliable question-question correlations (they tend to increase if similar questions are placed after each others).
Another reason is that the five control question pairs should be randomly ordered so people don't know which questions are used for consistency checks.

A third reason is to "mix-up" the ordinary questions that counts with experimental question that are being evaluated. Currently, 17 questions are experimental.
Yeah but that makes it hard to see what is fully reliable when someone has taken the quiz many times. At least that's what I make of it. I've done it several times and I've gotten a range of about 20pts or so difference over the past like 4 years. *shrugs*