Most of the stronger college pitchers I've seen are bigger than average. OU's Paige Parker (71") is one example.
It's also fairly obvious that bigger stronger kids, as long as they have a decent swing, are typically going to hit the ball harder and further than a smaller one. However, unless HRs are coming with freakish regularity, speed and defense are more important, and the most successful teams are built for that.
One look a last year's National Champion (OU) shows that the average and median height was 5'7", with only four players taller than 5'8" and five players shorter than 5'6". While that's taller than the "average" female, it certainly isn't huge.