It is a judgement call and in general it is one of those 'you know it when you see it" calls you make fairly easily once you have umpired a bunch of games. I was looking for an online description of how I was taught to judge it and found this. It is pretty much how it was described to me:
"If I can look up as the ball rises, then look down quickly to see if a fielder is handling it, and then look up again as the ball reaches its apex, then I have an infield fly."
