No Easy Answers

Are you getting ready to take an FAA pilot knowledge test, or helping a student prepare for test day? If so, you might want to sharpen your pencils and take notes, because the FAA has made some changes to the tests. You'll be relieved to learn that the tests aren't any longer than they were-nor are the questions to be any harder or cover topics that you haven't already studied. Instead, the FAA wants to change the way we study for our tests.

Most every pilot knows that the FAA traditionally made available all of the questions that can potentially appear on the FAA knowledge tests. In turn, most aviation publishers offer test preparation books that organize the FAA questions by topic area, and conveniently list the correct answer next to each question. For some time, the FAA has questioned these practices.

The FAA's concern is that pilot applicants may be spending too much time memorizing test questions and not enough time learning the required aeronautical knowledge and skills.

"We need to distinguish between pilots who memorize questions and answers, and pilots who are learning the material," says Martin Weaver, head of the FAA's Airman Testing Standards Branch in Oklahoma City, the branch of the agency that creates and oversees knowledge tests as well as practical tests. "A safe pilot is a master of the subject matter, not a regurgitator of information."

How did the FAA come to suspect that there might be a problem with the tests? It turns out that the computers that we use to take our FAA tests are capable of recording the amount of time that we require to complete our tests. "We've had cases of applicants completing the private pilot knowledge test in as little as two minutes," Weaver said. This wasn't the work of just one super-genius; there were many instances of applicants finishing the test in less time than it would take a mere mortal to even read the questions. The same two minutes were the fastest completion times for the instrument airplane test, as well. Even the high-flying airline transport pilot test saw some speed demons crossing the finish line in 10 minutes. Test-takers routinely answer the lengthy cross-country flight planning, aircraft performance, and weight-and-balance problems in as little as five seconds!

Observations like these prompted the FAA to make some changes to the tests.

The changes

When you walk in to take your next pilot knowledge test, you're going to notice two changes that were implemented by the FAA in July.

The first difference is that the answer choices for each question will be shuffled-what was answer choice A in the test prep books might be answer choice C on your test. This change is aimed at eliminating the possibility of using simple memorization tactics that associate letters with questions. For example, there's a private pilot cross-country flight planning question that starts at Addison Airport. The answer to the question has traditionally been letter A-A for Addison. No E6B flight computer needed for that one!

The second (and more striking) difference concerns the "math" type questions: those that ask you to perform flight-planning, performance, or weight-and-balance calculations. The FAA now will only release one or two examples of each type of question that can appear on the test. For example, consider a flight-planning question that requires you to use a sectional chart to perform a time en route calculation between two airports. Your test will contain a question of this type, but one that will require you to perform the same calculation using different airports on a different sectional chart. If you know how to do the calculation, rest assured-you won't have any problem answering the new questions. You just won't have the opportunity to see the questions in advance.

What people think

The practice of publishing test questions has also been a topic of some discussion among pilots and flight instructors. What do they think of these changes?

Mikkel Grandjean-Thomsen, a flight instructor in San Carlos, California, sees the move as a positive step. "I think it'll force people to hit the books more," he said. He argues that flying skills and book knowledge are different things, and says that he often see students whose "knowledge is lagging behind their practical skill, or vice versa." His bottom line: "We need both. Get better at either one, and you're a safer pilot."

Flight instructor Steve Philipson takes the opposite viewpoint. "Anyone who takes the time to memorize a thousand FAA questions has demonstrated enough familiarity with the material to meet the standards of the test," argues Philipson. "As for finishing the test in two minutes, there will always be people who game the system. We should report the test completion time on the score sheet for the examiner to see." Philipson adds, "Not publishing the questions will prevent people from providing feedback to improve the test."

"As I'm studying for this test, having the questions and answers in front of me almost seems like cheating," says Karen Jones, who is a student pilot as well as an aviation human factors researcher. "On the other hand, just because pilots are looking at the questions and answers doesn't necessarily mean that they haven't studied the books and don't know their stuff."

Science to the rescue

So which viewpoint is right? Are the changes to the test a step in the right direction, or are the feds just trying to make things harder for us? To help answer these questions, our research group at NASA got the idea to do an experiment.

We created a paper-and-pencil version of the computerized FAA knowledge test and gave it to student pilots who had just completed the actual private pilot knowledge test. Our experimental test changed the FAA questions in the ways that the FAA was proposing. We shuffled the answers for some questions and swapped in different charts and examples for the calculation questions. We included an equal number of unaltered FAA questions so we could make a comparison. If pilots knew their stuff, they should do just as well on any of the questions, right?

Forty-eight student pilots completed our test, emerging after about 30 minutes, and were presented with a NASA Aviation T-shirt, payment equal to the cost of their FAA test, and a slap on the back for being such a good sport.

So how do you think they did?

It turns out that shuffling the answers didn't fool anyone. Scores were about the same on shuffled questions and questions whose answers were left alone.

For the questions in which we posed the same kind of problem but swapped in a different chart, however, scores were significantly lower. Everyone still passed, but scores dropped from an average of about 86 percent to an average of 74 percent. That's still a passing score, but not one to brag about.

Even though the FAA plans to continue to release the non-math questions, we decided to get a sneak preview of what might happen if pilots had to answer these questions without seeing them first. Yet another group of good-natured student pilots graciously agreed to face this challenge in the name of science and aviation safety. For this test, we created questions that covered the same material as FAA questions, only in different ways. We were careful not to create test questions that were harder than the existing FAA questions.

The results for these questions were particularly worrisome. Student pilots scored an average of 66 percent on our non-published questions versus an average of 86 percent on the published FAA questions-quite a big drop.

Overall, the results of the study suggest that the FAA's concern about question memorization isn't off the mark. Releasing all test questions in advance may indeed help defeat the very purpose of any test-to help instructors and students figure out what has been learned and what still needs a little more work.

So how should we study?

So what does all this mean for pilots (like me) who are preparing for an upcoming knowledge test? For now, it probably means we're going to have to work a few extra flight-planning, performance, and weight-and-balance problems. A little extra effort here should nicely prepare us to put our skills to work regardless of which charts the FAA decides to slide under our noses come test day.

Start with the FAA questions, and swap in some different numbers. For weight and-balance-problems, try moving passengers around in the airplane, add some more fuel or baggage, and then crunch the numbers to see what you get. Dig up a pilot's operating handbook for some other airplanes. Review the sections on flight planning, performance, and weight and balance in other textbooks or test-prep products written for your certificate or rating. Good ones will include problems at the end of each chapter for you to get some practice. Textbook authors should step forward and meet the demand for extra questions for student practice.

Even though the FAA will, for the time being, continue to release the non-math questions, start fortifying your understanding of these aeronautical topics, too. Reading other textbooks and reviewing other computer-based courses is one good way to get different perspectives. Talking to fellow pilots is another. Perhaps the best exercise is to look for ways to relate what you are learning in the books to what you see and do when you fly. Martin Weaver at the FAA emphasizes this point: "Rote memorization is only the first step. The goal is to be able to apply what you know and to relate it to other things you know." I remember once watching my flight instructor colleague, Homi Irani, staring at a fan sitting on the floor of our flight school. After a few minutes, he walked over and picked it up with one finger, by the handle on the top of the running fan. He then poked the side of the fan with his other finger and watched the fan pitch up. He looked over and uttered two words: "Gyroscopic precession." Yeah!

How can CFIs make sure that their students are learning the right stuff? One idea comes from veteran flight instructor Howard Fried, who explains how he gives students his own, more difficult knowledge test before signing them off to take the FAA test. "Only those who scored 80 percent or better on our final were...allowed to take the FAA test." Fried adds that when the student knows the material, the exam takes care of itself.

Lessons learned

I remember how confident I felt walking into those test rooms, on 11 different occasions, breezing through the familiar-looking questions and then air-drying the ink on the notarized sheet that documented my impressive score. Looking at those scores, I couldn't help feeling like I really knew what I was doing.

I learned a lot of things during the time I spent working on this article, the usual outcome whenever anyone sets out to do anything in aviation. I learned that it sure is hot in Oklahoma during the summertime, and that there will be times during my flying career-past, present, and future-when I probably don't know as much as I think I do.

Steve Casner is an aviation safety researcher at NASA. His work covers a variety of pilot topics for both general aviation and air transport. He is an ATP, a Gold Seal CFI, and the author of The Pilot's Guide to the Modern Airline Cockpit and Cockpit Automation for General Aviators.

Want to know more?

The following resources offer additional information on topics discussed in this article:

• Review airman knowledge test guides and test questions at AOPA Online.
• Review the FAA's knowledge test standards and test banks.
• Find a testing center; download a coupon for a $10 AOPA member discount at any CATS location.

Links to these resources are available on AOPA Online.

By Steve Casner