Self-attention　based　Transformer　has　been　successfully　introduced　in　the　encoder-decoder　framework　of　image　captioning,　which　is　superior　in　modeling　the　inner　relations　of　inputs,　i.e.,　image　regions　or　semantic　words.　However,　relations　in　self-attention　are　usually　too　dense　to　be　fully　optimized,　which　may　result　in　noisy　relations　and　attentions.　Meanwhile,　the　prior　relations,　e.g.,　visual　relation　and　semantic　relation　between　objects,　which　are　essential　for　understanding　and　describing　an　image,　are　ignored　by　current　self-attention.　Thus,　the　relation　learning　of　self-attention　in　image　captioning　is　biased,　which　leads　to　a　dilution　of　the　concentration　of　attentions.　In　this　paper,　we　propose　a　Relation　Constraint　Self-Attention　(RCSA)　model　to　enhance　the　relation　learning　of　self-attention　in　image　captioning　by　constraining　self-attention　with　prior　relations.　RCSA　exploits　the　prior　visual　and　semantic　relation　information　from　scene　graph　as　constraint　factors.　And　then　it　builds　constraints　for　self-attention　through　two　sub-modules:　an　RCSA-E　encoder　module　and　an　RCSA-D　decoder　module.　RCSA-E　introduces　the　visual　relation　information　to　self-attention　in　encoder,　which　helps　generate　a　sparse　attention　map　by　omitting　the　attention　weights　of　irrelevant　regions　to　highlight　relevant　visual　features.　RCSA-D　extends　the　keys　and　values　of　self-attention　in　decoder　with　the　semantic　relation　information　to　constrain　the　learning　of　semantic　relation,　and　improve　the　accuracy　of　generated　semantic　words.　Intuitively,　RCSA-E　endows　model　with　an　ability　to　figure　out　which　region　to　omit　and　which　region　to　focus　by　visual　relation　information;　RCSA-D　then　strengthens　the　relation　learning　of　the　focused　regions　and　improves　the　sentence　generation　with　semantic　relation　information.　Experiments　on　the　MSCOCO　dataset　demonstrate　the　effectiveness　of　our　proposed　RCSA.(c)　2022　Elsevier　B.V.　All　rights　reserved.