SAT happens

The new, improved SAT exam has arrived. The test takes longer, includes an essay-writing section, and "enhanced" math. It also makes even more money for people who offer to prepare students to take the exam – up to $310 million per year.

And those who have objected to the test continue to do so.

Its designers have claimed it meet the objections to the old test: that it was basically flawed, unfair to poorer students and overly relied upon by universities that have found it a convenient way to thin out applicants. But is that claim of improvement accurate for the new version? The same objections persist and those making them – including major test-preparation companies like the Princeton Review – contend the exam remains unhealthy in many respects.

...continued in full post...

And of course, there's the objection that the SAT may not measure what it purports to measure.

The Princeton Review people have contended forever that the SAT can be manipulated by those who know how to take it. In other words, it isn't knowledge that is being measured but student test-taking ability.

Although its designers claim the SAT is immune to the use of tricks designed to boost scores, people continue to use them.

More threatening to the College Board, which owns the SAT, is that those taking it are beginning to regard it as more a game played on the way to a college education than a serious predictor of the ability to do college work.

"More a game"... When I was preparing for the GRE, back in 1981, I found a book, titled How to Beat the SAT and Other Standardized Tests. It looked at the whole testing process from the viewpoint of game theory. Game theory is the branch of mathematics in which you have two parties with conflicting goals, and a set of strategies that can affect the pay-offs for both sides.

In the SAT (and GRE) "game", the pay-off is the numerical score from the test. The test-taker wants to obtain the highest possible score, and the test designer wants to keep the test-taker from obtaining a high score without knowing the answers to the test questions.

My high-school physics teacher used a random number generator to generate the answer keys for his tests. He then arranged the choices on his multiple-choice questions to match the numbers that came up. The designers of the SAT and GRE don't do this.

They design tests based on assumptions about how test-takers will behave when they don't know the answers to questions. Among other things, this leads to deviations from randomly assigned answers, and these can be detected using statistical analysis.

For one thing, people who don't know the answer to a question, and are just marking a guess, tend to mark one of the end choices, or dead center. In a five-item choice, they will pick "A", "C", or "E". Sure enough, those letters turn out to be the right answer significantly less often than chance would allow.

People who don't know the answers in a block of questions will tend to mark the same answer in successive questions. Since test designers know this, they will tend to break up runs of the same letter being the right answer. And again, a statistical analysis shows that the same letter is less likely to be the right answer twice in a row than chance would allow – far less likely to be right three times in a row.

There were a number of other techniques, some formal, others informal. (In the same sense of formal and informal logic, described here.)

Did they work? I can't say with absolute certainty, but I took the SAT in 1977, and got a combined score of 1350. After going through college and getting a B.S. in physics, I took the GRE in 1981 and got a combined score of 1450. The GRE is supposed to be tougher than the SAT, but a physics curriculum can't help but build up your math skills.

My verbal score on the SAT put me in the 99th percentile, and my math score put me in the 98th percentile; on the GRE, my scores were in (if I recall correctly) 96th percentile and 99th percentile, respectively.

So the answer is, how the hell should I know? At least it doesn't seem to have hurt.

(And it could have. A test preparation method that was a total waste of time could have taken time away from stuff that worked, even if it were only resting up beforehand.)