If it isn't called immediately then it is not in effect. Ump is wrong.
With the exception of NCAA, it is the situation that makes it an infield fly not the umpire calling it. It may be the umpires judgement as to if an infielder had the opportunity to catch it with normal effort, but other than that it is not a requirement for the umpire to verbally call it. Yes they should but if they dont it doesnt negate the situation being an infield fly. There are even case plays stating the infield fly can be invoked retroactively if the umpires miss calling it and the result puts either team in jeopardy due to the non call.