The problem with flying is not that it is flat out unaffordably expensive - because, as you point out, it's not - but that it is much more expensive than pretty much everything else a normal person might spend money on.
Most people - especially people with some disposable income to spend on flying in the first place - expect a certain level of quality and "intactness" from things they buy. The massive improvement in consumer goods quality in the past 30 years contributes to this.
So, when they look at the kind of automobile they can buy for $40,000 - say, a 2008 Porsche Boxster S with 20,000 miles, in which everything is solid, well put together, sleek and modern, etc., and compare that with the quality of an airplane they can buy for $40,000 - perhaps a 30-year-old Cessna with cracked, warped plastic paneling the color of various types of baby vomit and a set of avionics that would look at home in the ENIAC - they balk. They just can't justify spending that much money on an ancient relic that appears to be falling apart around them, even if rationally they know it is perfectly airworthy. If they saw a car in the same kind of cosmetic shape they probably wouldn't pay more than $1,000 for it.
The minimum amount someone has to spend to purchase an airplane that does not resemble their dad's beat-up old Buick in one way or another is about $100,000. This is for a Skyhawk made after the production restart (although they still have a pretty iffy interior to my eyes, just with better colors), or better yet an early Diamond DA40, or maybe a lightly-used LSA. This will not be a luxurious interior by any means, but it will at least not make potential passengers run in fear for their lives.
Yes, you can technically fly for less money. But a lot of people don't want to fly around in a 40-year-old airplane. And I don't think that's unreasonable.
