We　report　high　pulse　energy　generation　from　an　Yb-doped　fiber　laser　mode-locked　or　passively　Q-switched　by　a　graphene　saturable　absorber.　In　the　case　of　mode-locked　operation,　a　ring　cavity　was　used.　The　average　output　power　is　170　mW　and　the　repetition　rate　is　1.04　MHz,　corresponding　to　single　pulse　energy　of　163　nJ.　To　the　best　of　our　knowledge,　this　is　the　highest　pulse　energy　reported　for　fiber　lasers　mode-locked　with　graphene　saturable　absorber.　In　the　case　of　passively　Q-switched　operation,　a　linear　cavity　was　used.　The　average　output　power　is　12　mW　and　the　repetition　rate　is　257　kHz,　corresponding　to　single　pulse　energy　of　46　nJ　with　pulse　width　of　70　ns.　We　beliveve　we　are　among　the　first　to　report　the　stable　passively　Q-switched　operation　by　graphene　saturable　absorber　in　a　Yb-doped　fiber　laser　as　well.　Graphene-based　saturable　absorber　shows　promising　potential　to　replace　SESAM　as　mode-locker　or　Q-switcher　for　pulsed　lasers　in　the　near　future　because　of　the　good　performance,　easy　fabrication,　low　cost　and　wide　availability.