The　objective　of　group　activity　recognition　is　to　identify　behaviors　performed　by　multiple　individuals　within　a　given　scene.　However,　current　weakly　supervised　approaches　often　rely　on　object　detectors　or　use　self-attention　mechanisms.　The　former　approach　is　susceptible　to　background　clutter　and　entails　high　computational　costs,　while　the　latter　method　learns　weights　from　the　input　video　and　assigns　them　to　key　targets　which　is　not　reliable　enough　to　find　the　key　person.　To　address　these　limitations,　we　present　a　novel　weakly　supervised　framework.　Our　proposed　framework　eliminates　the　need　for　ground-truth　bounding　boxes　or　object　detectors.　Meanwhile,　it　incorporates　the　semantics　of　individual　action　labels　to　replace　self-attention　to　guide　the　learning　process,　enabling　the　extraction　of　more　sophisticated　semantic　features　relevant　to　activity.　This　approach　also　explores　the　interactions　to　promote　group　activity　classification.　Experimental　results　demonstrate　that　our　method　achieves　state-of-the-art　performances　on　both　volleyball　and　collective　datasets.　©　2023,　The　Author(s),　under　exclusive　license　to　Springer　Nature　Singapore　Pte　Ltd.