Image　emotion　classification　is　an　important　computer　vision　task　to　extract　emotions　from　images.　The　state-of-the-art　methods　for　image　emotion　classification　are　primarily　based　on　proposing　new　architectures　and　fine-tuning　them　on　pre-trained　Convolutional　Neural　Networks.　Recently,　learning　transferable　visual　models　from　natural　language　supervision　has　shown　great　success　in　zero-shot　settings　due　to　the　easily　accessible　web-scale　training　data,　i.e.,　CLIP.　In　this　paper,　we　present　a　conceptually　simple　while　empirically　powerful　framework　for　supervised　image　emotion　classification,　SimEmotion,　to　effectively　leverage　the　rich　image　and　text　semantics　entailed　in　CLIP.　Specifically,　we　propose　a　prompt-based　fine-tuning　strategy　to　learn　task-specific　representations　while　preserving　knowledge　contained　in　CLIP.　As　image　emotion　classification　tasks　lack　text　descriptions,　sentiment-level　concept　and　entity-level　information　are　introduced　to　enrich　text　semantics,　forming　knowledgeable　prompts　and　avoiding　considerable　bias　introduced　by　fixed　designed　prompts,　further　improving　the　model’s　ability　to　distinguish　emotion　categories.　Evaluations　on　four　widely-used　affective　datasets,　namely,　Flickr　and　Instagram　(FI),　EmotionROI,　Twitter　I,　and　Twitter　II,　demonstrate　that　the　proposed　algorithm　outperforms　the　state-of-the-art　methods　to　a　large　margin　(i.e.,　5.27%　absolute　accuracy　gain　on　FI)　on　image　emotion　classification　tasks.　©　2022,　The　Author(s),　under　exclusive　license　to　Springer　Nature　Switzerland　AG.