In December, Harvard joined many other universities to eliminate ACT and SAT requirements—the standardized tests that these institutions have acknowledged won’t tell the full story about applicants. Is it time for the aviation industry to follow?
The trend of colleges abandoning SAT admission protocols—while mostly for social reasons to encourage equity amongst applicants—illustrates that sometimes when we look at things with fresh eyes, we realize there are multiple ways to solve a problem.
The FAA knowledge exam—generally a requirement to pass before applying to take a check ride—is in my mind an enduring and faulty system that all stakeholders from pilots to examiners and even up to policymakers have acknowledged needs fixing.
Or, in my estimation, eliminating.
Tradition says that it weeds out certain candidates, but I would argue that this system provides very little proof of actual knowledge. So why are we hanging on to it?
A Flawed System
Just like college test scores, proponents arguing for keeping knowledge exams might say examiners need a yardstick to compare applicants according to a certain standard. Examiners are charged with questioning applicants on any weak areas revealed by missed questions on the knowledge test. With the recent transition to the airmen certification standards, this connection is made even more explicit by the alignment of task codes.
Getting rid of the tests would mean revising the whole mechanism for check rides—and rethinking an entire methodology. It would surely be a costly endeavor, but I counter that keeping the tests takes a toll on the entire system.
The Knowledge Testing Industry
Consider what students now have to go through to take a knowledge test. In 2018, the FAA awarded a 10-year contract to PSI Services LLC (PSI), a testing services provider across North America.
PSI (formerly LaserGrade) offers proctored computer-based test administration for professional and occupational licensure and certification examinations, and FAA certification using its own secure network and software. PSI is one of only two computer-based testing companies authorized by the federal government to administer all FAA airman knowledge tests. PSI delivers more than 60 required exams for commercial and private pilots, as well as airframe and powerplant technicians.
As more and more universities are finding ways to consider applicants without SATs and ACTs scores, it proves that options exist.
On the surface, it would appear that this move to standardize things meant more efficiency. All you had to do was register on the FAA’s website for the specific knowledge test you wish to take, and visit your local PSI testing center.
Except, pilots know the entire process of taking a test—from the prep to check ride—is anything but simple.
Because knowledge tests serve such an important role as a precursor to most check rides—then as a component of the check ride itself—applicants know they have to make the best of them. Like the SAT and ACT, these knowledge tests have led to their own market, where an entire industry of test preparation companies now coach students through the tests, charging hefty fees. In fact, a March 2019 report from IBISWorld valued the tutoring and test preparation industry at $1.1 billion, with exam prep services making up 25 percent of the industry.
To be clear, this relates to the entire U.S. test prep market for all levels of schooling, including college admissions and other industries outside aviation. The aviation industry has its own commercial players such as King School, ASA, Sporty’s Jeppesen, Boldmethod, and many more.
While it is hard to measure how many pilots use these commercial services for prep, a 2020 webinar by the FAA gives an eye into the scope of the industry, explaining that it administers over 150,000 exams annually. In fact, the presentation calls it one of the “largest and most complex testing programs in the world from any industry.”
Pilots might recognize this process: It’s almost time to take your check ride, and you need to pass a knowledge exam prior to applying to take the check ride with a score of at least 70 percent. Your well-meaning instructor might encourage you to pay a subscription fee to a popular test prep company that advertises high pass rates and provide tips on how to study the question bank. Additionally, when you get to the actual check ride and the examiner has to review your knowledge test to quiz you on any deficiencies, the pilot is incentivized to score as high as possible on the knowledge test. This makes the entire process even more rigorous for pilots, who spend a lot of time memorizing answers rather than fundamentally understanding question problems—and paying more to retake the test if they don’t succeed the first time.
Furthermore, as an example to prove that the process has become a proxy for the actual result—pilot knowledge—the FAA Airman Knowledge Testing Matrix, which the administration released in May 2021 attempting to clarify parts of the process, adds to the complexity. In other words, in pursuit of making the process as efficient as possible, the system misses the fact that pilots still view the process as pass-through.
The Cost for Already Cash-Strapped Pilots
What’s more? The costs of testing applicants are onerous. The fee for all FAA airmen knowledge exams is now $175. Considering that the journey of pilots from private pilot to flight instructor includes taking at least six knowledge tests, by the end, they have expended a lot of capital.
The case can be made that this model discourages would-be pilots. Sure, compared to the direct cost of flight training it is much less, but to a lot of people, it is something.
Do the numbers prove anything?
Is there any correlation between knowledge tests and initial check-ride pass rates? Do pilots who do well on knowledge tests pass their check rides on the initial attempts?
Here is the 2020 data on initial check ride pass/fail rates in 2020. Across the board—at the private, commercial, and even flight instructor level—the average initial pass rate is 77.5 percent. Can we argue that is sufficient?
Meanwhile, here are the 2020 Airman Knowledge Tests statistics. The interesting thing to note is the higher pass rate and average score of the knowledge tests compared to the initial check ride pass/fail statistics. Is it sufficient to continue making the case that knowledge test scores are a good measure of whether or not someone will do well on their checkride?
Test Questions Lack Nuance for Aviation’s Global Cohort
The FAA has been pushing for more comprehensive testing, with more focus on the pilot’s competency level. What does a limited test question bank really prove of a candidate that a check pilot shouldn’t be able to figure out?
Another thing to consider is that more and more foreign students, with English as a secondary language, are signing up for training globally. While English is the standard language of the industry, the rigid multiple-choice structure of these doesn’t provide any wiggle room for students who might want to clarify something. Invariably, this reinforces the test-prep cycle where more memorization takes place. A part of revamping the system is creating room for a broader base of multicultural contributors at the table to encourage better inclusivity.
It’s time to rethink our approach to standardize knowledge tests.
Is it time for the aviation industry to take cues from what is going on in the broader college testing industry? As more and more universities are finding ways to consider applicants without SATs and ACTs scores, it proves that options exist. Likewise, if our industry were to admit the limits of the knowledge tests and the lack of perceived value, we might realize that options exist. While the FAA has been making efforts—to their credit—more can be done, in my view.
