I'm a pilot with 3500 hours.
Of course flying isn't safer than driving... at 3 times the speed, there is 9 times the kinetic energy to deal with in a crash. Aircraft by design are lightweight, and therefore much less crashworthy than an automobile even if the energy were the same.
The failure modes in an automobile are much more benign also. Unless you happen to be driving through a very bad neighborhood at just the wrong moment, the consequences of an engine failure, even well managed are much riskier in an aircraft than in a car. An engine failure 30 seconds after liftoff in an airplane is exponentially more dangerous than the same event in a car.
Factor in weather and the relative complexity of flying as compared to driving and it's a wonder that the stastics aren't much worse than they are.
This should not be surprising to anyone, airplanes leave the ground and cars do not.
Most of the risk in driving is uncontrollable by the driver, such as the risk of colliding with another driver who is intoxicated, or distracted. In flying we get to choose our level of risk for the most part -- given a good attitude, good training, and a little experience, and much (but not all) of the risk can be mitigated -- this is not possible to teh same extent in an automobile.
Finally, let me comment on the passenger miles vs hours flown vs miles flown discussion. For the individual on the aircraft, it seems to me that the sensible denominator is "per departure". In other words, if I, today, buckle myself into this vehicle, what are the odds that I will DIE before I get out of it. This is the most personal and immutable of the stastics, and really it is the only one that an individual cares about. GA doesn't fare particularly well in this category. If 400 people get on an airplane and it flys 3000 miles in one hop, you can see that your actual risk of NOT walking out of the airplane you just walked into could be 800,000 (400*3000) times as much as this "passenger miles" value.
Fly because it's convenient, flexible, and fun, but don't kid yourself into thinking you're safer than driving.
