The Downside of Diversity