Open source testing and peer review

Another key component of open source testing is the process of peer review. Currently, examination papers are secret. This is a source of constant tension, as claims and counter-claims are made about diminishing standards and increasing achievement.

Having an open source repository of test questions opens up the entire process to peer review. Individual questions can be discussed, edited, accepted or rejected by experts in the field. This will inevitably result in a superior end product that is both better suited to discriminating between potential students and employees, and more accurately reflects the test takers mastery of the subject.

Because the repository of questions is significantly larger than the number of questions on an exam, there is no danger in allowing students to study the repository. If they are capable of mastering every question in the repository, that in itself is a useful measure of their mastery of the subject. Because every exam is a random collection of questions from the repository (rather than every student receiving the same questions), there is also no danger of leaked papers.

Peer review is the system that universities already use to evaluate scholarly work and research. Applying the same ideas to testing will produce exams that are useful academic and vocational tools, rather than political footballs.

Government and open source testing

Open source testing nearly eliminates the government's ongoing role in standardised testing. Once the legal criteria for establishing a governing board are established, there's no need for continuing government oversight. Since the tests are created and administered by the consumers, rather than the producers, there is no pressure to make schools look good (or bad), to make tests easier (or harder), or to raise scores arbitrarily (or lower them). The only pressure is for the governing boards to offer a test that is useful to themselves in discriminating between potential students and employees.

In this environment, test takers can vote with their feet. If there are competing governing boards in a subject, students can choose which they wish to take and schools which they wish to offer. If a governing board's test becomes out of date or irrelevant, they will simply collect less in examination fees until they update the test or abandon it. But since the test exists solely for the purpose of allowing the governing board (and other interested consumers) to evaluate test takers, their interest in maintaining an up to date and relevant test is intrinsic, rather than predicated solely on exam fees.

Open source testing

Education is extremely well suited to open source methodology. There are already a number of successful open source curriculum projects, including:

1. MIT OpenCourseWare
2. Connexions
3. Open Content Curriculum Project
4. OpenEducator
5. Open Source Learning Project

Using the same methodology for testing would involve setting up a repository of exam questions for a subject. If the exam should consist of, say, a hundred questions, the repository might consist of a thousand or more questions. Every student would then receive a random collection of those questions; given the number of possible tests, it's unlikely that any two students would receive exactly the same exam paper.

The process of deciding which subjects should be offered would be left to open source methodology as well. Each subject would require a governing board of at least three universities. Any accredited university would be allowed to join this governing board. In addition, any private company, charity or individual could join the governing board with the consent of the universities on the board. If ever a subject no longer had at least three universities on its governing board, it would cease to be offered as a standardised test.

No governing board would have a monopoly on a subject. If a group of at least three universities wished to form a competing governing board, offering a test in the same subject, they would be free to do so.

These questions could be contributed and edited by anyone, regardless of qualifications. However, these contributions and edits would not be applied to the visible repository until approved by a volunteer editor. Volunteer editors would be appointed by the governing board, generally from the faculty of member universities, but also including anyone from the general public that the board found acceptable. In traditional open source terms, contributors submit patches, but only editors have commit access to the repository.

Standardised testing

In the UK, there is no debate about whether or not standardised testing in schools is a good idea or not. Exam boards were introduced starting in 1857, and the first national standardised tests came in 1918 during Lloyd George's Liberal coalition government - the same government that introduced women's suffrage that same year. National testing was considered a progressive policy, in contrast to the US where standardised testing is still considered a largely conservative idea and is much opposed.

The debate in the UK revolves around whether or not the current tests are as rigorous as previous ones. To some degree this is missing the point: the requirement for standardised tests is not that they are rigorous, but that the results are meaningful to consumers of test results.

When I say 'consumers of test results', I mean the people that use the results to choose between students: universities and employers. This is in contrast to the 'producers of test results', which are the schools, local education authorities, whichever Whitehall department has control over schools this week, the Joint Council for Qualifications (JCQ) and its six component examination boards, and the Qualifications and Curriculum Authority (QCA) - soon to be replaced by Ofqual.

This somewhat bewildering array of organisations create, administer and mark the two most important standardised tests: GCSEs and A-levels. Schools and LEAs are judged by these results, but only by the same organisations that produce the tests in the first place. Students are judged by these results as well, but by a group that has no input on the nature of the tests: universities and employers. The unintended consequence of this system is that the organisations responsible for testing create tests more suited for schools than students. This isn't because they are evil or trying to destroy education, it's simply because the schools, not the students, are effectively their customers.

The current system therefor doesn't meet the requirement of standardised testing: to give universities and employers a means to choose usefully between students. Can this be done without creating another closed system, this time with universities and employers as both producers and consumers of test results? Yes, it can, using open source methodology and peer review.