Personally, I've never gotten to the point where I walked off the field. But what you say is so much easier said than done.
All of us have a threshold that, once it's reached, we will react. It's simply human nature. Either we let it get to us personally, we see that it's affecting the players, coaches or other umpires, and/or we feel that if we don't react, things will get completely out of hand. And how we react is just as important, if not more so, than doing nothing.
What I described in my high school game this past spring got to that point because it was affecting my partner. And I could tell from some of the players' body language, it was getting to them as well. Should I have lashed out at the official scorekeeper? Probably not. I probably should have just cleared the field and let things get back to some semblance of normal. But because it was the scorekeeper, I felt she needed a lesson on what it means to serve in that role, and hoped that my letting her know that would somehow get the crowd to understand that they were being idiots, too.
I can count on two hands when my threshold was reached. What I usually do is clear the field of all players, and then bring both coaches to the plate. I'll let them know that the game cannot continue until the fans settle down. I won't do what other umpires do and tell the coaches they have to take care of the problem or I'll eject them. Sometimes, coaches have little control of their fans, and holding them accountable to the point of ejection isn't fair. I just tell them I won't restart the game until the problem gets fixed. They can go talk to the parents, find the site administrator, call the cops, do nothing until it fixes itself, whatever. In those situations I've had, we were able to get the game restarted with no more real issues.
