Nonbank mortgage lenders are making up a greater share of the market but don’t have the same resources banks do, leading to worries about liquidity.