Given　that　buildings　consume　approximately　33%　of　global　energy,　and　HVAC　systems　contribute　nearly　half　of　a　building’s　total　energy　demand,　optimizing　their　efficiency　is　imperative　for　sustainable　energy　use.　Many　existing　buildings　operate　HVAC　systems　inefficiently,　displaying　non-stationary　behavior.　Current　reinforcement　learning　(RL)　training　methods　rely　on　historical　data,　which　is　often　obtained　through　costly　modeling　or　trial-and-error　methods　in　real　buildings.　This　paper　introduces　a　novel　reinforcement　learning　construction　framework　designed　to　improve　the　robustness　and　learning　speed　of　RL　control　while　reducing　learning　costs.　The　framework　is　specifically　tailored　for　existing　office　buildings.　Applying　this　framework　to　control　HVAC　systems　in　real　office　buildings　in　Beijing,　engineering　practice　results　demonstrate:　during　the　data　collection　phase,　energy　efficiency　surpasses　traditional　rule-based　control　methods　from　the　previous　year,　achieving　significantly　improved　energy　performance　(a　17.27%　reduction)　with　minimal　comfort　sacrifices.　The　system　achieves　acceptable　robustness,　learning　speed,　and　control　stability.　Reduced　ongoing　manual　supervision　leads　to　savings　in　optimization　labor.　Systematic　exploration　of　actions　required　for　RL　training　lays　the　foundation　for　RL　algorithm　development.　Furthermore,　by　leveraging　collected　data,　a　reinforcement　learning　control　algorithm　is　established,　validating　the　reliability　of　this　approach.　This　construction　framework　reduces　the　prerequisites　for　historical　data　and　models,　providing　an　acceptable　alternative　for　systems　with　insufficient　data　or　equipment　conditions.