An ugly week for housing stocks includes news that Fannie Mae and Freddie Mac would delist from the New York Stock Exchange as well as a selloff in shares of residential builders.