Learning　from　visual　observations　is　a　significant　yet　challenging　problem　in　Reinforcement　Learning　(RL).　Two　respective　problems,　representation　learning　and　task　learning,　need　to　solve　to　infer　an　optimal　policy.　Some　methods　have　been　proposed　to　utilize　data　augmentation　in　reinforcement　learning　to　directly　learn　from　images.　Although　these　methods　can　improve　generation　in　RL,　they　are　often　found　to　make　the　task　learning　unsteady　and　can　even　lead　to　divergence.　We　investigate　the　causes　of　instability　and　find　it　is　usually　rooted　in　high-variance　of　Q-functions.　In　this　paper,　we　propose　an　easy-to-implement　and　unified　method　to　solve　above-mentioned　problems,　Data-augmented　Reinforcement　Learning　with　Ensemble　Exploration　and　Exploitation　(DAR-EEE).　Bootstrap　ensembles　are　incorporated　into　data　augmented　reinforcement　learning　and　provide　uncertainty　estimation　of　both　original　and　augmented　states,　which　can　be　utilized　to　stabilize　and　accelerate　the　task　learning.　Specially,　a　novel　strategy　called　uncertainty-weighted　exploitation　is　designed　for　rational　utilization　of　transition　tuples.　Moreover,　an　efficient　exploration　method　using　the　highest　upper　confidence　is　used　to　balance　exploration　and　exploitation.　Our　experimental　evaluation　demonstrates　the　improved　sample　efficiency　and　final　performance　of　our　method　on　a　range　of　difficult　image-based　control　tasks.　Especially,　our　method　has　achieved　the　new　state-of-the-art　performance　on　Reacher-easy　and　Cheetah-run　tasks.