Point　clouds　and　RGB　images　are　both　critical　data　for　3D　object　detection.　While　recent　multi-modal　methods　combine　them　directly　and　show　remarkable　performances,　they　ignore　the　distinct　forms　of　these　two　types　of　data.　For　mitigating　the　influence　of　this　intrinsic　difference　on　performance,　we　propose　a　novel　but　effective　fusion　model　named　LI-Attention　model,　which　takes　both　RGB　features　and　point　cloud　features　into　consideration　and　assigns　a　weight　to　each　RGB　feature　by　attention　mechanism.　Furthermore,　based　on　the　LI-Attention　model,　we　propose　a　3D　object　detection　method　called　image　attention　transformer　network　(IAT-Net)　specialized　for　indoor　RGB-D　scene.　Compared　with　previous　work　on　multi-modal　detection,　IAT-Net　fuses　elaborate　RGB　features　from　2D　detection　results　with　point　cloud　features　in　attention　mechanism,　meanwhile　generates　and　refines　3D　detection　results　with　transformer　model.　Extensive　experiments　demonstrate　that　our　approach　outperforms　state-of-the-art　performance　on　two　widely　used　benchmarks　of　indoor　3D　object　detection,　SUN　RGB-D　and　NYU　Depth　V2,　while　ablation　studies　have　been　provided　to　analyze　the　effect　of　each　module.　And　the　source　code　for　the　proposed　IAT-Net　is　publicly　available　at　https://github.com/wisper181/IAT-Net.