I wear appropriately coloured cargo shorts; which also happen to be my normal non-work attire; year round (fortunately, despite being Canadian, we live in the Pacific Northwest - winter isn't exactly harsh here, really wet yes, but we dont get many days below freezing).
Yes, you could try to claim they aren't particularly athletic, I can counter by saying I wear them for playing slo-pitch too so, if i can and do play in them, they're good enough for coaching in, for now at least; so, nyah! LOL