What have I heard.....
Don't throw out the baby with the bath water.
Don't throw out the baby with the bath water.
is an idiomatic expression and a concept[1] used to suggest an avoidable error in which something good is eliminated when trying to get rid of something bad,[2] or in other words, rejecting the essential along with the inessential.[3]