Semantic　segmentation　based　on　the　complementary　information　from　RGB　and　depth　images　has　recently　gained　great　popularity,　but　due　to　the　difference　between　RGB　and　depth　maps,　how　to　effectively　use　RGB-D　information　is　still　a　problem.　In　this　paper,　we　propose　a　novel　RGB-D　semantic　segmentation　network　named　RAFNet,　which　can　selectively　gather　features　from　the　RGB　and　depth　information.　Specifically,　we　construct　an　architecture　with　three　parallel　branches　and　propose　several　complementary　attention　modules.　This　structure　enables　a　fusion　branch　and　we　add　the　Bi-directional　Multi-step　Propagation　(BMP)　strategy　to　it,　which　can　not　only　retain　the　feature　streams　of　the　original　RGB　and　depth　branches　but　also　fully　utilize　the　feature　flow　of　the　fusion　branch.　There　are　three　kinds　of　complementary　attention　modules　that　we　have　constructed.　The　RGB-D　fusion　module　can　effectively　extract　important　features　from　the　RGB　and　depth　branch　streams.　The　refinement　module　can　reduce　the　loss　of　semantic　information　and　the　context　aggregation　module　can　help　propagate　and　integrate　information　better.　We　train　and　evaluate　our　model　on　NYUDv2　and　SUN-RGBD　datasets,　and　prove　that　our　model　achieves　state-of-the-art　performances.