We　propose　a　novel　Text-to-Image　Generation　Network,　Adaptive　Layout　Refinement　Generative　Adversarial　Network　(ALR-GAN),　to　adaptively　refine　the　layout　of　synthesized　images　without　any　auxiliary　information.　The　ALR-GAN　includes　an　Adaptive　Layout　Refinement　(ALR)　module　and　a　Layout　Visual　Refinement　(LVR)　loss.　The　ALR　module　aligns　the　layout　structure　(which　refers　to　locations　of　objects　and　background)　of　a　synthesized　image　with　that　of　its　corresponding　real　image.　In　ALR　module,　we　proposed　an　Adaptive　Layout　Refinement　(ALR)　loss　to　balance　the　matching　of　hard　and　easy　features,　for　more　efficient　layout　structure　matching.　Based　on　the　refined　layout　structure,　the　LVR　loss　further　refines　the　visual　representation　within　the　layout　area.　Experimental　results　on　two　widely-used　datasets　show　that　ALR-GAN　performs　competitively　at　the　Text-to-Image　generation　task.