Action　segmentation　is　vital　for　video　understanding　because　it　heuristically　divides　complex　untrimmed　videos　into　short　semantic　clips.　Real-world　human　actions　exhibit　complex　temporal　dynamics,　encompassing　variations　in　duration,　rhythm,　and　range　of　motions,　etc.　While　deep　networks　have　been　successfully　applied　to　these　tasks,　they　face　challenges　in　effectively　adapting　to　these　complex　variations　due　to　the　inherent　difficulty　in　capturing　semantic　information　from　a　global　perspective.　Merely　relying　on　distinguishing　visual　representations　in　local　regions　leads　to　the　issue　of　over-segmentation.　In　an　attempt　to　address　this　practical　issue,　we　propose　a　novel　approach　named　ASGSA,　which　aims　to　obtain　smoother　segmentation　results　by　extracting　instructive　semantic　information.　Our　core　component,　Global　Semantic-Aware　module,　provides　an　effective　way　to　encode　the　long-range　temporal　relation　in　the　long　untrimmed　video.　Specifically,　we　exploit　a　hierarchical　temporal　context　aggregation,　which　is　identified　by　a　gated-mechanism　selection　to　control　the　information　passage　at　different　scales.　In　addition,　an　adaptive　fusion　strategy　is　designed　to　guide　the　segmentation　with　the　extracted　semantic　information.　Simultaneously,　to　obtain　higher-quality　video　representation　without　extra　annotations,　we　resort　to　self-supervised　training　strategy　and　propose　the　Video　Speed　Prediction　module.　Extensive　experiments　demonstrate　that　our　approach　achieves　state-of-the-art　performance　on　all　three　challenging　benchmark　datasets　(Breakfast,　50Salads,　GTEA)　and　significantly　improves　the　F1　score@50,　which　represents　the　reduction　of　over-segmentation.　The　code　is　available　at　https://github.com/ten000/ASGSA.　©　The　Author(s),　under　exclusive　licence　to　Springer-Verlag　London　Ltd.,　part　of　Springer　Nature　2024.