With　the　increasing　number　of　images　containing　rich　emotional　information　in　social　media　and　the　urgent　demand　for　faster　and　more　accurate　image　emotional　information　mining,　some　researchers　have　begun　to　pay　attention　to　image　emotion　classification　research.　However,　most　of　the　work　focuses　on　the　complex　model　design,　neglecting　the　proper　consideration　of　the　loss　function,　which　is　common　in　the　research　of　image　emotion　classification　task.　Simultaneously,　the　widely　used　loss　function,　such　as　the　Softmax　Loss,　ignores　the　difference　in　the　concentration　of　the　inner-class　features　in　image　emotion　and　object　classification,　which　causes　the　problem　of　lacking　inner-class　feature　distance　converging　data　imbalance　leading　to　more　misclassifications　of　affective　images.　We　explored　the　problem　of　inner-class　feature　constraints　in　the　loss　function　design　for　image　emotion　classification　tasks.　Based　on　the　existing　loss　improvement,　we　propose　a　method　with　the　Emotion　Class-wise　Aware　(ECWA)　loss　to　get　better　accuracy　and　robustness　on　more　occasions.　Results　show　that　the　method　we　proposed　is　more　effective　in　the　image　emotion　classification　task,　especially　in　the　emotion　category　with　few　samples.　©　2021,　Springer　Nature　Switzerland　AG.