Men have been steadily dropping out of the workforce — especially men ages 25 to 54, who are considered to be in their prime working years.