In　H.264/AVC,　a　novel　criterion　named　the　rate　distortion　optimization(RDO)　is　employed　to　select　the　optimal　coding　modes　for　each　macroblock(MB)　within　the　intra　prediction　coding　pictures,　which　can　achieve　a　high　compression　ratio　while　leading　to　a　great　increase　in　the　complexity　and　computational　load　unfortunately.　In　this　paper,　a　DCT-based　energy　function　is　proposed,　which　is　used　to　estimate　the　textural　complexity　of　residual　blocks　of　intra　prediction.　Then　not　more　than　4　modes　instead　of　9　intra　prediction　modes　for　4x4　block　can　be　chosen　to　perform　RDO　according　to　the　textural　complexity　of　residual　block.　Consequently,　the　complexity　decreases　significantly.　Experiments　show　that　the　proposed　algorithm　can　save　about　48%　of　the　computation　time　stably　while　maintaining　the　PSNR-Bitrate　performance　of　H.264/AVC.