Monday 20 July 2009

Open source testing and peer review

Another key component of open source testing is the process of peer review. Currently, examination papers are secret. This is a source of constant tension, as claims and counter-claims are made about diminishing standards and increasing achievement.

Having an open source repository of test questions opens up the entire process to peer review. Individual questions can be discussed, edited, accepted or rejected by experts in the field. This will inevitably result in a superior end product that is both better suited to discriminating between potential students and employees, and more accurately reflects the test takers mastery of the subject.

Because the repository of questions is significantly larger than the number of questions on an exam, there is no danger in allowing students to study the repository. If they are capable of mastering every question in the repository, that in itself is a useful measure of their mastery of the subject. Because every exam is a random collection of questions from the repository (rather than every student receiving the same questions), there is also no danger of leaked papers.

Peer review is the system that universities already use to evaluate scholarly work and research. Applying the same ideas to testing will produce exams that are useful academic and vocational tools, rather than political footballs.

1 comment:

  1. When I attended law school, I was pleasantly surprised to find that the every year, a number of exams were published and made available in the library. Inclusion was optional for each professor.

    This didn't make tests any easier, especially if previous test for your courses weren't included (like most of mine), but it was very useful.

    For the bar exam itself, which I think all would agree is a very important exam, as those go, nearly every question is available before hand in previous exams and sample tests from bar exam study companies. In some cases, only the proper nouns in an essay test when are changed.

    And the multi-state portion of the bar exam is a multiple-choice test with formulaic questions and answers, though it isn't easy to dismiss incorrect answers through logic as it is with S.A.T. type tests because the answers are heavily based on knowledge. But if one learns to recognize the limited patterns in which the test attempts to mislead you, the time required for the exam can be dramatically reduced.

    Having the previous tests available helped prepare for the taking of the test, but did not much reduce the need to have the necessary knowledge to answer correctly. If that makes sense.

    I once had to write the exam for a law school class of students from overseas and it was a torturous process. I can only imagine that many professors, at least those without assistants to pawn the work off on, would like this.

    ReplyDelete