The　rate-distortion　optimization　(RDO)　technique　is　employed　in　H.264/AVC　intra　frame　coding　to　achieve　the　best　rate-distortion　performance.　But　the　computational　complexity　is　very　high　because　it　encodes　the　current　block　by　exhaustively　searching　all　possible　modes　to　select　the　best　one.　To　reduce　the　computational　complexity　we　propose　a　new　fast　algorithm　based　on　edge　direction　detection　(EDD)　to　reduce　the　number　of　candidate　modes　for　further　RDO　calculation.　The　proposed　algorithm　can　accurately　detect　the　edge　direction　of　different　blocks　to　reduce　the　number　of　candidate　modes　for　further　RDO　calculation.　The　simulation　results　show　that　the　proposed　algorithm　contributes　to　the　bit-rate　reduction　compared　to　that　of　previous　algorithms　and　saves　around　62%　of　the　encoding　time　with　negligible　PSNR　degradation.　©2008　The　Institution　of　Engineering　and　Technology.