You can imagine our deep sense of grief when occasionally we’d return to a city on our ground-school circuit and someone would say, “Did you hear about Bill?” “No,” we’d say, “what happened to Bill?” They would then explain the aircraft crash that had killed Bill. It happened all too often. We began to realize the things that were killing these learning pilots were things the FAA was not asking on the knowledge tests.
The problem was, the FAA wanted a bell-shaped distribution of test scores on the test results. It was difficult for them to get a bell-shaped curve because our highly motivated students would come back to our classes and discuss with us what they had been asked on their knowledge tests. We would immediately change our courses to do a better job of covering anything they had missed. Consequently, our next students would not miss many questions—frustrating the test givers. To get applicants to miss questions, the FAA had to make the questions trickier and more difficult—and less and less relevant to the risk-management issues pilots face in real life.
Martha and I began to realize that we were spending our weekends covering tricky questions and trivia that was irrelevant to actual flight, while the things actually causing fatalities were not being asked and, therefore, not being taught.
In their efforts to ensure that an adequate number of test questions would be missed, the FAA testing folks moved to make the test questions secret from the aviation-training community—the people who were actually teaching pilots. They saw us as adversaries. I brought up this adversarial relationship with Nick Sabatini, then the FAA’s associate administrator for aviation safety, at a “Meet the FAA” session at EAA AirVenture in Oshkosh, Wisconsin. Sabatini was concerned, and he mandated that the airman-testing folks meet with the aviation community annually. The result was an adversarial meeting once a year.
Eventually, in 2011, an aviation rule-making committee on the subject was formed. An ARC consists of both FAA and aviation community members and meets periodically to provide advice and information to the FAA.
The atmosphere initially between the two groups was hostile. But the purpose of the meetings was for us to talk. And talk we did. The ARC lasted two years and made recommendations to the FAA about how to reform testing. Then it evolved into a working group which has lasted another seven years. The result has been nothing short of a miracle. Most significantly, the hostility slowly went away. As we communicated, we began to develop mutual respect and understanding. The FAA began to refer to members of the “aviation community” rather than the “industry.” Members of that community began to appreciate the competence and goodwill of the FAA.
Read More from John King: Sky Kings
We gradually made significant progress on fixing important problems. We became aware that while there were existing standards for the skills that could be required of an applicant, there were no standards for the knowledge that could be asked of an applicant—on either the knowledge test or the practical test. The knowledge tests could and did ask an applicant anything no matter how trivial. As the discussions continued, we agreed it was a good idea to have standards for the knowledge so applicants would know to be studying meaningful and relevant concepts.
But then we started discussing the idea that although applicants would have to demonstrate skills and knowledge, applicants were not required to demonstrate the ability to identify and mitigate risks. Yet it was not lack of skills that was the biggest cause of fatalities; it was that pilots had not developed the habit of risk management. Remembering my friends who had come to grief, I became a champion for the idea that applicants should be required to demonstrate the ability to identify and mitigate risks. Through the airman certification standards, risk management became part of pilot check rides.
The biggest regret I have is that the FAA never did accept the recommendation of the ARC that the test questions be returned to the public domain—depriving the test writers of valuable input from the aviation community and robbing applicants, their instructors and examiners of the specific details on any knowledge-test questions they miss (plus creating a business for any company who is covertly purchasing the questions from test-takers and selling them to applicants).
But all in all, I have to say that these nine years have been the most inspiring example of developing respect and collaboration I have ever seen. The ongoing input to the working group from its community members is a feedback process that serves to ensure the ACS will evolve as needed to improve our risk management over time. Rather than merely having practical test standards, we now have complete standards for the certification of pilots. And pilots who are learning the practice of identifying the risks in a flight and developing the habit of mitigating them. I am hopeful that this will save thousands of lives.
This story appeared in the December 2020 issue of Flying Magazine
