Depth　maps　acquired　by　3D　cameras　usually　suffer　from　low　resolution　and　insufficient　quality,　which　makes　it　difficult　to　be　directly　used　in　visual　depth　perception　and　3D　reconstruction.　To　handle　this　problem,　we　propose　a　novel　multi-direction　dictionary　and　joint　regularization　model　for　high　quality　depth　recovery.　To　enhance　the　sparsity,　image　is　divided　into　classified　patches　according　to　the　same　geometrical　direction　and　a　compact　dictionary　is　trained　within　each　class.　Then　for　a　patch　to　be　coded,　the　most　relevant　dictionary　can　be　selected　according　to　its　geometrical　direction.　We　further　introduce　two　regularization　terms　into　the　reconstruction　model.　One　is　the　anisotropic　total　variation　(TV)　defined　by　the　local　gradients　of　color/depth　pair.　The　other　is　the　nonlocal　similarity　to　provide　nonlocal　constraint　to　the　local　structure.　Experimental　results　demonstrate　that　our　method　outperforms　other　state-of-the-art　methods　in　terms　of　both　subjective　quality　and　objective　quality.