I don't think it's rude. Then again I don't have a team to root for; I am a floater. People stay in the shade, once they get it. Obnoxious cheering that's another story.
Thats the point. It was obnoxious cheering, and got worse when they realized they were getting to us. Sitting with a few people from other teams is normal and acceptable. We often watch other games while waiting for our own - but wouldn't think of taking choice seating, and don't take sides and cheer for the "other side", even though quietly, we may have a preference as to who wins. Our team, we think, has a good reputation and care what others think of us. This team, due to a few bad apples, has ruined their reputation in our eyes.