Tracking　an　Unmanned　Aerial　Vehicle　(UAV)　to　obtain　its　locations　and　trajectory　is　a　crucial　task　to　avoid　the　unlawful　use　of　UAVs.　However,　most　existing　UAV　tracking　methods　fail　when　facing　cluster　environments,　out-of-view,　and　occlusions　because　of　their　insufficient　representation　of　global　context　information　capacity.　To　mitigate　these　issues,　we　propose　a　new　tracker,　namely　SiamFusion,　to　innovate　a　dual　fusion　procedure　that　leverages　the　advantages　in　both　the　feature　and　decision　levels.　In　particular,　we　propose　a　novel　feature　fusion　module　named　Modality-Fusion　to　utilize　multi-modal　information,　enhancing　the　perception　of　the　target.　From　the　decision　level,　we　further　develop　a　local-global　converter　based　on　a　multi-modal　fusion　decision-making　mechanism　to　reduce　the　accumulation　during　tracking,　which　significantly　increases　the　robustness　of　the　tracking　process.　Extensive　experiments　demonstrate　the　superiority　of　the　proposed　SiamFusion,　which　achieves　the　best　performance　on　Anti-UAV　in　terms　of　accuracy　and　speed.　In　particular,　we　exceed　the　state-of-the-art　tracking　algorithm　in　the　tracking　accuracy　by　4.2%　at　a　similar　frame　rate.　Our　source　codes,　pre-trained　models,　and　online　demos　will　be　released　upon　acceptance.