Monday 20 July 2009

Open source testing and peer review

Another key component of open source testing is the process of peer review. Currently, examination papers are secret. This is a source of constant tension, as claims and counter-claims are made about diminishing standards and increasing achievement.

Having an open source repository of test questions opens up the entire process to peer review. Individual questions can be discussed, edited, accepted or rejected by experts in the field. This will inevitably result in a superior end product that is both better suited to discriminating between potential students and employees, and more accurately reflects the test takers mastery of the subject.

Because the repository of questions is significantly larger than the number of questions on an exam, there is no danger in allowing students to study the repository. If they are capable of mastering every question in the repository, that in itself is a useful measure of their mastery of the subject. Because every exam is a random collection of questions from the repository (rather than every student receiving the same questions), there is also no danger of leaked papers.

Peer review is the system that universities already use to evaluate scholarly work and research. Applying the same ideas to testing will produce exams that are useful academic and vocational tools, rather than political footballs.

Government and open source testing

Open source testing nearly eliminates the government's ongoing role in standardised testing. Once the legal criteria for establishing a governing board are established, there's no need for continuing government oversight. Since the tests are created and administered by the consumers, rather than the producers, there is no pressure to make schools look good (or bad), to make tests easier (or harder), or to raise scores arbitrarily (or lower them). The only pressure is for the governing boards to offer a test that is useful to themselves in discriminating between potential students and employees.

In this environment, test takers can vote with their feet. If there are competing governing boards in a subject, students can choose which they wish to take and schools which they wish to offer. If a governing board's test becomes out of date or irrelevant, they will simply collect less in examination fees until they update the test or abandon it. But since the test exists solely for the purpose of allowing the governing board (and other interested consumers) to evaluate test takers, their interest in maintaining an up to date and relevant test is intrinsic, rather than predicated solely on exam fees.

Thursday 16 July 2009

Open source testing

Education is extremely well suited to open source methodology. There are already a number of successful open source curriculum projects, including:
  1. MIT OpenCourseWare
  2. Connexions
  3. Open Content Curriculum Project
  4. OpenEducator
  5. Open Source Learning Project
Using the same methodology for testing would involve setting up a repository of exam questions for a subject. If the exam should consist of, say, a hundred questions, the repository might consist of a thousand or more questions. Every student would then receive a random collection of those questions; given the number of possible tests, it's unlikely that any two students would receive exactly the same exam paper.

The process of deciding which subjects should be offered would be left to open source methodology as well. Each subject would require a governing board of at least three universities. Any accredited university would be allowed to join this governing board. In addition, any private company, charity or individual could join the governing board with the consent of the universities on the board. If ever a subject no longer had at least three universities on its governing board, it would cease to be offered as a standardised test.

No governing board would have a monopoly on a subject. If a group of at least three universities wished to form a competing governing board, offering a test in the same subject, they would be free to do so.

These questions could be contributed and edited by anyone, regardless of qualifications. However, these contributions and edits would not be applied to the visible repository until approved by a volunteer editor. Volunteer editors would be appointed by the governing board, generally from the faculty of member universities, but also including anyone from the general public that the board found acceptable. In traditional open source terms, contributors submit patches, but only editors have commit access to the repository.

Wednesday 15 July 2009

Standardised testing

In the UK, there is no debate about whether or not standardised testing in schools is a good idea or not. Exam boards were introduced starting in 1857, and the first national standardised tests came in 1918 during Lloyd George's Liberal coalition government - the same government that introduced women's suffrage that same year. National testing was considered a progressive policy, in contrast to the US where standardised testing is still considered a largely conservative idea and is much opposed.

The debate in the UK revolves around whether or not the current tests are as rigorous as previous ones. To some degree this is missing the point: the requirement for standardised tests is not that they are rigorous, but that the results are meaningful to consumers of test results.

When I say 'consumers of test results', I mean the people that use the results to choose between students: universities and employers. This is in contrast to the 'producers of test results', which are the schools, local education authorities, whichever Whitehall department has control over schools this week, the Joint Council for Qualifications (JCQ) and its six component examination boards, and the Qualifications and Curriculum Authority (QCA) - soon to be replaced by Ofqual.

This somewhat bewildering array of organisations create, administer and mark the two most important standardised tests: GCSEs and A-levels. Schools and LEAs are judged by these results, but only by the same organisations that produce the tests in the first place. Students are judged by these results as well, but by a group that has no input on the nature of the tests: universities and employers. The unintended consequence of this system is that the organisations responsible for testing create tests more suited for schools than students. This isn't because they are evil or trying to destroy education, it's simply because the schools, not the students, are effectively their customers.

The current system therefor doesn't meet the requirement of standardised testing: to give universities and employers a means to choose usefully between students. Can this be done without creating another closed system, this time with universities and employers as both producers and consumers of test results? Yes, it can, using open source methodology and peer review.

Tuesday 14 July 2009

Voting at age 16

Earlier, I wrote:
While there is nothing inherently wrong with any of these proposals (except perhaps lowering the voting age to 16, although I'm willing to be convinced if someone wants to try)
Neil Harding has taken me up on this and offered an excellent and well-considered support of the proposal. The point I found most compelling was this:
By reducing the lower age to 16, no-one would have to wait past 21 to be able to vote.
However, I'm afraid I remain unconvinced. Not against, mind you, just not convinced. Obviously, if Anytime Voting were to be implemented, all voters would be able to lodge their vote the moment they turned 18. This would address Neil's concerns in a much more direct way, and without unintended consequences.

Without Anytime Voting, the proposal has to be evaluated in terms of the requirements and unintended consequences. Is the requirement that all voters be allowed to lodge a vote in a general election by the age of 21? No: Neil later advocates general elections every 3 years, guaranteeing a vote by the age of 19. This is actually two requirements at once: improving accountability by reducing the election period, and improving enfranchisement by guaranteeing a vote by an earlier age. These are both goals I find laudable (after all, Anytime Voting guarantees a vote by the age of 18, and makes accountability instantaneous).

But are they achieved without significant unintended consequences? My argument is that they are not.

Reducing the election period without moving to Anytime Voting increases the time representatives (in this case, MPs) must spend campaigning and fund-raising. Every general election is preceded by a period of governmental inaction as the candidates and parties concentrate on re-election rather than legislation, and it's possible for the civil service to be paralysed by a predicted (but not inevitable) change of government. This is a delicate balance of course - without Anytime Voting, some measure of election fever has to be accepted, since elections achieve the far more important goal of accountability. The problem isn't that elections every 3 years are worse than every 5 (they aren't), but that they don't achieve the goal of increased accountability without unintended and negative consequences.

Reducing the voting age to 16 does indeed achieve of goal of guaranteeing a vote by the age of 21 (or 19 if the election period is also reduced). Following Neil's line of thought (a good one), the unintended consequence is allowing some voters to vote before the age of 18. Neil correctly points out:
Remember also, that only a quarter or fifth of new voters will get to vote at 16. Most will still be over 18 by the time they get to vote in a general election.
That figure of course changes to a half or a third of new voters if the election period is reduced. Does the unchallenged benefit of allowing all voters to vote by the age of 21 (or 19) outweigh the unintended consequence of allowing 16 and 17 year old kids to vote? Possibly, but this is where I remain unconvinced. I would prefer to solve the problem without the consequence.

Basically, I'm in full support of Neil's goals, but I think Anytime Voting is a better way to achieve them. But then, I would say that, wouldn't I?

Monday 13 July 2009

Satire and system design

Mark Thompson has an excellent satire of the voting system today. While part of it is a call for proportional representation, the rest raises issues that can only be solved with Anytime Voting.

Satire is a useful tool for examining system design. Ferreting out the ridiculous and the absurd can help to expose the problems in a self-consistent system that is nevertheless producing unintended consequences; in this case, a theoretically representative system that actually only asks for the views of a small number of swing voters in marginal seats every four or five years.

Sunday 12 July 2009

Moving averages

In a previous post here, I mentioned moving averages:
In a first-past-the-post election, a moving average can be applied to this result in order to eliminate fast fluctuations.
This raises two questions:
  1. What sort of moving average, and how should it be applied?
  2. Why is a moving average not necessary with proportional representation?
For voting, the best moving average is a simple moving average. This means the data points aren't weighted - that is, the voting result for any given day is just as important as the voting result for any other day. Applying this involves simply choosing a number of days to include in the moving average. The result is to smooth out rapid fluctuations in voting results.

Another, even simpler, method is to announce results for a first-past-the-post district less often than every day. Results could be announced weekly or monthly, for example. There is an important balance to be considered, however: longer reporting periods smooth out fluctuations when they happen to occur, but they also reduce the immediate accountability of the representative.

Elections using some form of proportional representation requires no smoothing or extended reporting periods because any fluctuation would have to be at least as large as the number of votes needed to secure a seat in order to have any effect. Such a large change is by definition significant. This holds true whether the election uses party-list proportional representation (like the European elections) or the Additional Member System (like Welsh and Scottish elections).

Fluctuation in a first-past-the-post district is, in itself, an important indicator. It occurs when:
  1. The electorate is highly polarised. That is, the two leading parties have nearly the same number of votes.
  2. Neither party is able to gain and hold the support of the swing voters in the district.
  3. Those swing voters do not abandon the two leading parties for a third party, but instead continually switch their votes between the two leading parties.
Knowing that fluctuation is occurring in a district should act as a major motivating factor not only for the two leading parties but for all other parties with candidates in that district.

Friday 10 July 2009

Voting 'none of the above'

Bloo raises an interesting point:
[I]f I voted for my rep., can I remove my vote without placing it elsewhere?
The simple answer is yes, you can. You can submit an unmarked ballot and, just as now, that ballot will not register a vote for any candidate.

A more radical approach is to allow a voter to mark a ballot for 'none of the above'. If 'none of the above' wins a seat, no candidate takes the seat, and that seat is considered to always vote no to every proposal. In a first-past-the-post system, voting 'none of the above' is of limited utility, since it would be difficult (though certainly not impossible) to achieve a plurality in a district. However, in any system that includes any form of proportional representation (including the party list system used in European elections and the Additional Member System used in Welsh, Scottish and London elections), voting 'none of the above' carries more weight. This is because 'none of the above' functions similarly to a minor party, and one of the design elements of proportional representation is to allow minor parties to win seats.

Allowing 'none of the above' votes is likely to have several consequences:
  1. Increased voter turnout. It's not unusual for a non-voter to say that dissatisfaction with all the available parties led them to not bother voting.
  2. Reduced support for fringe or extremist parties. Fringe parties can gather votes from people who don't necessarily support all of their policies, but wish to protest against the major parties.
  3. An improved measure of political dissatisfaction. Currently, non-voters are an amalgam of those who don't care what the results of an election are and those who don't wish to support any of the candidates. A 'none of the above' vote makes it possible to distinguish between these two very different groups.
Anytime Voting and 'none of the above' votes are independent measures. Either can be implemented without the other.

Thursday 9 July 2009

Anytime voting can be permanent voting

An interesting consequence of Anytime Voting is that it's possible for a voter to vote just once in their life and still be represented. Suppose a voter reaches voting age (or becomes a citizen) and proceeds to vote for their preferred party's candidates on the various ballots (parliament, national assemblies, local elections, European elections, etc.). If they move to a different district, CORE may allow their votes to be inherited, obviating the need to cast a new ballot. As long as their preferred party continues to contest all of the seats in their district, they need never vote again.

It's not that permanent voting is something to be striven for, but that it illustrates the impact of Anytime Voting. It isn't just about being able to dismiss or recall a representative that no longer has the support of their constituents (although that's a critical component), it's also about altering the relationship between voters and political parties.

What happens when a voter moves?

Currently, in the UK, registering to vote is accomplished by filling out the annual canvass form that the local electoral registration office distributes. In this way, the electoral roll for a district is recompiled from scratch each year.

This system can be left in place with Anytime Voting. The only change necessary is to ensure that a voter that was on the roll the previous year retains the same voter ID number. When a voter is removed from the electoral roll, theirexisting ballots are invalidated. When a voter is added to the electoral roll, they are sent a letter and a ballot asking them to please vote at their leisure.

If the Co-ordinated On-line Record of Electors (CORE) project is ever completed, it will even be possible for vote inheritance to be applied when a voter moves. When a voter is added to a register and simultaneously removed from an old register, vote inheritance can be applied to their ballot, requiring them to vote again only if inheritance isn't possible (due to, for example, their favoured party not contesting seats in their new district).

Wednesday 8 July 2009

What about when ballots change?

Ballots are not permanent things. Every so often, the ballot itself changes, whether it's because a party changes their candidate, a new party begins to contest the seat, an old party stops contesting the seat, or an independent candidate adds or removes themselves from the ballot. When this happens, it's possible for a vote that's been cast to become invalid.

Rather than invalidate all votes when the ballot changes, forcing every voter to cast a new ballot, it's simple to determine the set of invalid ballots.
  1. When a party changes candidates, there are two options. The complicated option is to invalidate any ballot that voted for the old candidate. The much simpler option is to acknowledge that party affiliation is meaningful, and count ballots with votes for the old candidate as ballots for the new one. We'll call this 'vote inheritance'.
  2. When a new party begins to contest a seat, it's not necessary to invalidate any ballots at all. It's up to the new party to convince voters to change their existing votes. The same applies when a new independent candidate is added to the ballot.
  3. When an old party stops contesting a seat, every ballot with a vote for that party's candidate must be invalidated. The same applies when an old independent candidate is removed from a ballot.
By acknowledging party affiliation and allowing vote inheritance, invalid ballots only occur when a party or independent candidate stops contesting a seat. But what do we do with invalid ballots? Simple: the vote counting agency posts the list of invalid ballot numbers to the agency in charge of the electoral roll. The electoral roll agency then correlates these ballot numbers with the voter ID and posts a notice to every voter that's had their ballot invalidated, asking them to please vote again.

Tuesday 7 July 2009

Anytime voting technology

So what is this great technology that enables Anytime Voting? The Internet? Public key cryptography? Magic beans? No: it's the post.

Currently, in the UK, when a person registers to vote they are issued with a voter ID number, and each ballot paper is individually numbered. When a voter is given a ballot, their voter ID number is written on the ballot stub (or counterfoil), which also has the ballot paper number on it. This is an important fraud prevention scheme, since it allows an Elections Court to link a voter to a ballot if necessary. While in theory this violates the principle of a secret ballot, in practice this scheme works extraordinarily well, and has done for decades.

This link between voter ID and ballot paper number makes Anytime Voting easy to implement. The process works like this:
  1. The voter is given a ballot paper. This can either be in person at a town hall, or mailed on request to the voter's registered address.
  2. The civic minded individual handing over the ballot or mailing it out records the voter ID on the ballot stub, and posts the stub to the agency that controls the electoral roll for the voter's district. In England and Wales, this is the local council; in Scotland it's a separate local electoral registration office; in Northern Ireland there is a single central Electoral Office.
  3. Upon receipt of the ballot stub, the office in charge of the electoral roll determines if the voter ID in question has a previous ballot (in other words, they've voted in that election previously). If so, the previous ballot is recalled. In any case, the new ballot is approved.
  4. The electoral roll agency periodically posts a list of recalled and approved ballots to the vote counting agency. They stop counting ballots on the recall list, and start counting ballots on the approved list. It's possible to have a ballot approved that the vote counting agency has not yet received (because the voter hasn't posted it yet, for example), but that's not a problem: the vote counting agency now knows to start counting that ballot as soon as it's received.
  5. Meanwhile, the voter fills out their ballot at their leisure. It can be posted to the vote counting agency or returned to the town hall in person (at which point it will be posted from there to the vote counting agency).
  6. Each day, the vote counting agency declares the current result of the election, based on ballots that are both approved and received. In a first-past-the-post election, a moving average can be applied to this result in order to eliminate fast fluctuations.
This simple procedure allows voters to vote when they want, whenever they want, and to change their vote as often as they like. Every representative would, in essence, face re-election every day, making them instantly and permanently accountable to the electorate.

'Anytime Voting'

The core idea behind Anytime Voting is to allow voters to cast their votes whenever they like, rather than on specific election days. This also means a voter can change their vote whenever they like, which means the outcome in a district can change at any time. This drastically increases the immediate impact of voting: it's possible for voters to react to events the same day rather than being disenfranchised until the next election.

There are other practical effects as well: election specific advertising is eliminated (replaced with the need to maintain a party's image at all times), opinion polls are largely devalued (since actual electoral changes can be monitored instead), and core voters become a far more reliable base for a party (since a voter can lodge their vote once and not change it for many years, rather than having to be convinced to vote in every election).

In a first-past-the-post election, a situation can arise with Anytime Voting wherein a small group of voters that change their vote regularly can cause the result in a marginal constituency to fluctuate quickly. The simplest way to address this is to use a moving average to stabilise the result. The type (simple, weighted, exponential, etc.) and period of the moving average can be selected to achieve the desired balance between stability and responsiveness.

This isn't just pie in the sky thinking, either. Anytime Voting can be practically achieved with existing technology.

Voting and accountability

One of the key issues being discussed these days is accountability. Once a representative is in office, what can the electorate do if they're dissatisfied? When applied to entire governments, this becomes an issue of legitimacy as well as accountability. Does a party manifesto establish what policies can or should be pursued? What about policies not in the manifesto, or the failure to achieve a promised goal?

Referendums are often proposed as solutions, but they suffer from a host of unintended consequences. Referendums have led to the paralysis of government in Florida, near-bankruptcy in California, and it's very difficult to defend the 'vote until you get it right' series of referendums on the EU in Ireland. This doesn't mean that all referendums are bad, but it does mean that referendums do not solve the accountability problem.

Recall elections should theoretically result in greater accountability, but in practice there's no greater accountability in US states that have them than in those that don't. The process is so expensive and politically charged that it's rarely used (there have only been 14 attempts at recalls in the US in the hundred or so years there have been recall elections, including everything from mayors and county supervisors to two state governors).

The real problem is that elections occur only periodically. It is the election itself that grants legitimacy to both an individual representative and to a government, and that legitimacy is only questioned when a difficult to pin down balance between public satisfaction and the time elapsed since the last election is upset. The actual requirement, from a system design point of view, is to always have zero time elapsed since the last election, such that every representative is reelected every day.

Permanent elections? Is this even possible? It is, and I call it 'Anytime Voting'.

Let's start with voting

There's been a lot of debate in the UK recently about the process of voting. Low voter turnout and a general sense of disenfranchisement and unfairness have led all sorts of proposals to be made, including:
  • Lowering the voting age to 16 (Labour).
  • Adopting 'Alternative Vote Plus' (AV+) in general elections (Labour).
  • Adopting party-list proportional representation in general elections (LibDems).
  • Referendums on demand (Tories).
  • Allowing individual MPs to be recalled from office (Tories).
  • An elected or partially-elected upper chamber (various).
Not all of these are official party policy, of course. The policies labeled 'Tory', for example, come from Hannan and Carswell's book 'The Plan' (although Cameron seems to be adopting 'The Plan' bit by bit).

While there is nothing inherently wrong with any of these proposals (except perhaps lowering the voting age to 16, although I'm willing to be convinced if someone wants to try), they all suffer somewhat from being patches on the existing system rather than coherently designed wholes.

And that's what I'll be writing about, in general: applying system design to politics and policy. The idea is to determine requirements first, then design a system that meets those requirements while avoiding unintended consequences. Quite often, politics and policy work in the other direction: ideology dictates a system, which is then applied without reference to requirements, with the result being a lot of unintended consequences.