Dynamic　displacement　response　is　an　essential　indicator　for　assessing　structural　state　and　performance.　Vision-based　structural　displacement　monitoring　is　considered　as　a　promising　approach.　However,　the　current　vision-based　methods　usually　only　focus　on　certain　application　scenarios.　This　study　introduces　a　Sparse　Bayesian　Learning-based　(SBL)　algorithm　to　enhance　robustness,　accuracy,　and　computational　efficiency　in　target　tracking.　Furthermore,　a　robust　and　versatile　Vision-based　Dynamic　Displacement　Monitoring　System　(VDDMS)　was　developed,　capable　of　monitoring　displacements　of　varying　application　scenarios.　The　robustness　of　the　proposed　algorithm　under　changing　illumination　conditions　is　validated　through　a　specially　designed　indoor　experiment.　The　feasibility　of　field　application　of　VDDMS　is　confirmed　through　an　outdoor　shear　wall　shaking　table　test.　Furthermore,　a　large-scale　bridge　shaking　table　test　is　conducted　to　evaluate　the　reliability　and　versatility　of　VDDMS　in　monitoring　natural　feature　targets　on　large　structures　subjected　to　different　seismic　excitations.　The　root　mean　square　error,　when　compared　to　laser　displacement　sensors,　ranges　from　0.2%　to　2.9%　of　the　peak-to-peak　displacement.　Additionally,　VDDMS　accurately　identifies　multi-order　frequencies　in　bridge　structures.　The　study　investigates　the　influence　of　initial　template　selection　on　accuracy,　highlighting　the　significance　of　distinctive　texture　features.　Moreover,　two　error　evaluation　schemes　are　proposed　to　quickly　assess　the　reliability　of　vision-based　displacement　sensing　technologies　in　various　application　scenarios.