You're mistaken - teams are allowed 3 charged defensive conferences and the pitcher is removed on the 4th. It is a common misunderstanding due to the way the rule is phrased.
Ugh, this happened to us this weekend. My DD's 14u team is playing, she's pitching, and the HC has already had one defensive conference that inning. He asks for time for another one, and GETS DENIED by the umpire and is told that he's only allowed one defensive conference per inning. Whut? I had just read this particular post the day before and had looked up the rule, and there was no special tournament rule superseding it, so I told our HC he has 3 per 7 innings, which he goes to argue with the umpire. He allows him to have his defensive conference, and as he's walking away, the umpire tells him he has to change the pitcher because he's had two defensive conferences in one inning. HC asks the umpire to ask the UIC for help with that one. UIC had been observing all this, finally came down to consult with the umpire. After their discussion, the umpire turns around and says "play ball" without offering a resolution. Our HC asks what the outcome of the discussion was, and the umpire tells him he can leave the pitcher in.
For the rest of the tournament, the umpires were very clear about when a defensive conference was being charged.