My tl:dr point of all this is: The pitcher did something good, the batter did something bad, and it's a weird quirk that would allow the batter to reach in that situation, including the randomness of playing surfaces, backstop distance and bounciness, etc. We should reward the greater athletic feat.
You typed a lot when you clearly don't get it.
You just want to change something because you don't like it, but you haven't made a single valid argument for why it logically should not exist. I'm not arguing that I like it. I'm arguing that the rule is consistent with the rest of the rules and they ought to remain consistent.
It's not a "weird quirk." That's how I know you don't get it. The pitcher making the batter swing and miss is only completing half the job! The catcher cleanly receiving the ball is the second half. You must do both. That's what's consistent with the entire rest of the game. Deviating to appease your sensibilities is what breaks the game.
Btw, the Infield Fly Rule exists to protect the offense, not help the defense. It's not breaking any guideline because of the reason it exists.