Brain　CT　report　generation　is　significant　to　aid　physicians　in　diagnosing　cranial　diseases.　Recent　studies　concentrate　on　handling　the　consistency　between　visual　and　textual　pathological　features　to　improve　the　coherence　of　report.　However,　there　exist　some　challenges:　1)　Redundant　visual　representing:　Massive　irrelevant　areas　in　3D　scans　distract　models　from　representing　salient　visual　contexts.　2)　Shifted　semantic　representing:　Limited　medical　corpus　causes　difficulties　for　models　to　transfer　the　learned　textual　representations　to　generative　layers.　This　study　introduces　a　Pathological　Clue-driven　Representation　Learning　(PCRL)　model　to　build　cross-modal　representations　based　on　pathological　clues　and　naturally　adapt　them　for　accurate　report　generation.　Specifically,　we　construct　pathological　clues　from　perspectives　of　segmented　regions,　pathological　entities,　and　report　themes,　to　fully　grasp　visual　pathological　patterns　and　learn　cross-modal　feature　representations.　To　adapt　the　representations　for　the　text　generation　task,　we　bridge　the　gap　between　representation　learning　and　report　generation　by　using　a　unified　large　language　model　(LLM)　with　task-tailored　instructions.　These　crafted　instructions　enable　the　LLM　to　be　flexibly　fine-tuned　across　tasks　and　smoothly　transfer　the　semantic　representation　for　report　generation.　Experiments　demonstrate　that　our　method　outperforms　previous　methods　and　achieves　SoTA　performance.　Our　code　is　available　at　https://github.com/ChaunceyJheng/PCRL-MRG.　©　2024　Association　for　Computational　Linguistics.