No, both ASA and FED have clarifications that dirt is not a foreign substance and there is no need to wipe the hand after touching dirt. But, it is illegal to pick up a handful and put it directly on the ball, rub it directly on the ball, rub the ball in the dirt, or put dirt in the glove and then the ball into the glove.
To clarify and/or expand, here is the change NCAA made this year.Just as an FYI, college does require the hand to be wiped after touching any drying agent.