Open source testing and peer review

Another key component of open source testing is the process of peer review. Currently, examination papers are secret. This is a source of constant tension, as claims and counter-claims are made about diminishing standards and increasing achievement.

Having an open source repository of test questions opens up the entire process to peer review. Individual questions can be discussed, edited, accepted or rejected by experts in the field. This will inevitably result in a superior end product that is both better suited to discriminating between potential students and employees, and more accurately reflects the test takers mastery of the subject.

Because the repository of questions is significantly larger than the number of questions on an exam, there is no danger in allowing students to study the repository. If they are capable of mastering every question in the repository, that in itself is a useful measure of their mastery of the subject. Because every exam is a random collection of questions from the repository (rather than every student receiving the same questions), there is also no danger of leaked papers.

Peer review is the system that universities already use to evaluate scholarly work and research. Applying the same ideas to testing will produce exams that are useful academic and vocational tools, rather than political footballs.

Government and open source testing

Open source testing nearly eliminates the government's ongoing role in standardised testing. Once the legal criteria for establishing a governing board are established, there's no need for continuing government oversight. Since the tests are created and administered by the consumers, rather than the producers, there is no pressure to make schools look good (or bad), to make tests easier (or harder), or to raise scores arbitrarily (or lower them). The only pressure is for the governing boards to offer a test that is useful to themselves in discriminating between potential students and employees.

In this environment, test takers can vote with their feet. If there are competing governing boards in a subject, students can choose which they wish to take and schools which they wish to offer. If a governing board's test becomes out of date or irrelevant, they will simply collect less in examination fees until they update the test or abandon it. But since the test exists solely for the purpose of allowing the governing board (and other interested consumers) to evaluate test takers, their interest in maintaining an up to date and relevant test is intrinsic, rather than predicated solely on exam fees.

Open source testing

Education is extremely well suited to open source methodology. There are already a number of successful open source curriculum projects, including:

1. MIT OpenCourseWare
2. Connexions
3. Open Content Curriculum Project
4. OpenEducator
5. Open Source Learning Project

Using the same methodology for testing would involve setting up a repository of exam questions for a subject. If the exam should consist of, say, a hundred questions, the repository might consist of a thousand or more questions. Every student would then receive a random collection of those questions; given the number of possible tests, it's unlikely that any two students would receive exactly the same exam paper.

The process of deciding which subjects should be offered would be left to open source methodology as well. Each subject would require a governing board of at least three universities. Any accredited university would be allowed to join this governing board. In addition, any private company, charity or individual could join the governing board with the consent of the universities on the board. If ever a subject no longer had at least three universities on its governing board, it would cease to be offered as a standardised test.

No governing board would have a monopoly on a subject. If a group of at least three universities wished to form a competing governing board, offering a test in the same subject, they would be free to do so.

These questions could be contributed and edited by anyone, regardless of qualifications. However, these contributions and edits would not be applied to the visible repository until approved by a volunteer editor. Volunteer editors would be appointed by the governing board, generally from the faculty of member universities, but also including anyone from the general public that the board found acceptable. In traditional open source terms, contributors submit patches, but only editors have commit access to the repository.

Standardised testing

In the UK, there is no debate about whether or not standardised testing in schools is a good idea or not. Exam boards were introduced starting in 1857, and the first national standardised tests came in 1918 during Lloyd George's Liberal coalition government - the same government that introduced women's suffrage that same year. National testing was considered a progressive policy, in contrast to the US where standardised testing is still considered a largely conservative idea and is much opposed.

The debate in the UK revolves around whether or not the current tests are as rigorous as previous ones. To some degree this is missing the point: the requirement for standardised tests is not that they are rigorous, but that the results are meaningful to consumers of test results.

When I say 'consumers of test results', I mean the people that use the results to choose between students: universities and employers. This is in contrast to the 'producers of test results', which are the schools, local education authorities, whichever Whitehall department has control over schools this week, the Joint Council for Qualifications (JCQ) and its six component examination boards, and the Qualifications and Curriculum Authority (QCA) - soon to be replaced by Ofqual.

This somewhat bewildering array of organisations create, administer and mark the two most important standardised tests: GCSEs and A-levels. Schools and LEAs are judged by these results, but only by the same organisations that produce the tests in the first place. Students are judged by these results as well, but by a group that has no input on the nature of the tests: universities and employers. The unintended consequence of this system is that the organisations responsible for testing create tests more suited for schools than students. This isn't because they are evil or trying to destroy education, it's simply because the schools, not the students, are effectively their customers.

The current system therefor doesn't meet the requirement of standardised testing: to give universities and employers a means to choose usefully between students. Can this be done without creating another closed system, this time with universities and employers as both producers and consumers of test results? Yes, it can, using open source methodology and peer review.

Voting at age 16

Earlier, I wrote:

While there is nothing inherently wrong with any of these proposals (except perhaps lowering the voting age to 16, although I'm willing to be convinced if someone wants to try)

Neil Harding has taken me up on this and offered an excellent and well-considered support of the proposal. The point I found most compelling was this:

By reducing the lower age to 16, no-one would have to wait past 21 to be able to vote.

However, I'm afraid I remain unconvinced. Not against, mind you, just not convinced. Obviously, if Anytime Voting were to be implemented, all voters would be able to lodge their vote the moment they turned 18. This would address Neil's concerns in a much more direct way, and without unintended consequences.

Without Anytime Voting, the proposal has to be evaluated in terms of the requirements and unintended consequences. Is the requirement that all voters be allowed to lodge a vote in a general election by the age of 21? No: Neil later advocates general elections every 3 years, guaranteeing a vote by the age of 19. This is actually two requirements at once: improving accountability by reducing the election period, and improving enfranchisement by guaranteeing a vote by an earlier age. These are both goals I find laudable (after all, Anytime Voting guarantees a vote by the age of 18, and makes accountability instantaneous).

But are they achieved without significant unintended consequences? My argument is that they are not.

Reducing the election period without moving to Anytime Voting increases the time representatives (in this case, MPs) must spend campaigning and fund-raising. Every general election is preceded by a period of governmental inaction as the candidates and parties concentrate on re-election rather than legislation, and it's possible for the civil service to be paralysed by a predicted (but not inevitable) change of government. This is a delicate balance of course - without Anytime Voting, some measure of election fever has to be accepted, since elections achieve the far more important goal of accountability. The problem isn't that elections every 3 years are worse than every 5 (they aren't), but that they don't achieve the goal of increased accountability without unintended and negative consequences.

Reducing the voting age to 16 does indeed achieve of goal of guaranteeing a vote by the age of 21 (or 19 if the election period is also reduced). Following Neil's line of thought (a good one), the unintended consequence is allowing some voters to vote before the age of 18. Neil correctly points out:

Remember also, that only a quarter or fifth of new voters will get to vote at 16. Most will still be over 18 by the time they get to vote in a general election.

That figure of course changes to a half or a third of new voters if the election period is reduced. Does the unchallenged benefit of allowing all voters to vote by the age of 21 (or 19) outweigh the unintended consequence of allowing 16 and 17 year old kids to vote? Possibly, but this is where I remain unconvinced. I would prefer to solve the problem without the consequence.

Basically, I'm in full support of Neil's goals, but I think Anytime Voting is a better way to achieve them. But then, I would say that, wouldn't I?

Satire and system design

Mark Thompson has an excellent satire of the voting system today. While part of it is a call for proportional representation, the rest raises issues that can only be solved with Anytime Voting.

Satire is a useful tool for examining system design. Ferreting out the ridiculous and the absurd can help to expose the problems in a self-consistent system that is nevertheless producing unintended consequences; in this case, a theoretically representative system that actually only asks for the views of a small number of swing voters in marginal seats every four or five years.

Moving averages

In a previous post here, I mentioned moving averages:

In a first-past-the-post election, a moving average can be applied to this result in order to eliminate fast fluctuations.

This raises two questions:

1. What sort of moving average, and how should it be applied?
2. Why is a moving average not necessary with proportional representation?

For voting, the best moving average is a simple moving average. This means the data points aren't weighted - that is, the voting result for any given day is just as important as the voting result for any other day. Applying this involves simply choosing a number of days to include in the moving average. The result is to smooth out rapid fluctuations in voting results.

Another, even simpler, method is to announce results for a first-past-the-post district less often than every day. Results could be announced weekly or monthly, for example. There is an important balance to be considered, however: longer reporting periods smooth out fluctuations when they happen to occur, but they also reduce the immediate accountability of the representative.

Elections using some form of proportional representation requires no smoothing or extended reporting periods because any fluctuation would have to be at least as large as the number of votes needed to secure a seat in order to have any effect. Such a large change is by definition significant. This holds true whether the election uses party-list proportional representation (like the European elections) or the Additional Member System (like Welsh and Scottish elections).

Fluctuation in a first-past-the-post district is, in itself, an important indicator. It occurs when:

1. The electorate is highly polarised. That is, the two leading parties have nearly the same number of votes.
2. Neither party is able to gain and hold the support of the swing voters in the district.
3. Those swing voters do not abandon the two leading parties for a third party, but instead continually switch their votes between the two leading parties.

Knowing that fluctuation is occurring in a district should act as a major motivating factor not only for the two leading parties but for all other parties with candidates in that district.

Voting 'none of the above'

Bloo raises an interesting point:

[I]f I voted for my rep., can I remove my vote without placing it elsewhere?

The simple answer is yes, you can. You can submit an unmarked ballot and, just as now, that ballot will not register a vote for any candidate.

A more radical approach is to allow a voter to mark a ballot for 'none of the above'. If 'none of the above' wins a seat, no candidate takes the seat, and that seat is considered to always vote no to every proposal. In a first-past-the-post system, voting 'none of the above' is of limited utility, since it would be difficult (though certainly not impossible) to achieve a plurality in a district. However, in any system that includes any form of proportional representation (including the party list system used in European elections and the Additional Member System used in Welsh, Scottish and London elections), voting 'none of the above' carries more weight. This is because 'none of the above' functions similarly to a minor party, and one of the design elements of proportional representation is to allow minor parties to win seats.

Allowing 'none of the above' votes is likely to have several consequences:

1. Increased voter turnout. It's not unusual for a non-voter to say that dissatisfaction with all the available parties led them to not bother voting.
   
2. Reduced support for fringe or extremist parties. Fringe parties can gather votes from people who don't necessarily support all of their policies, but wish to protest against the major parties.
3. An improved measure of political dissatisfaction. Currently, non-voters are an amalgam of those who don't care what the results of an election are and those who don't wish to support any of the candidates. A 'none of the above' vote makes it possible to distinguish between these two very different groups.

Anytime Voting and 'none of the above' votes are independent measures. Either can be implemented without the other.