In　order　to　provide　spectrum　and　energy　efficient　communication　for　unmanned　aerial　vehicle　assisted　cellular　network,　the　problem　of　joint　beamforming　and　power　allocation　(JBPA)　in　aerial　multicell　scenario　is　addressed.　The　JBPA　multi-objective　optimization　model　which　would　simultaneously　maximize　the　achievable　spectrum　and　energy　efficiency　is　first　developed.　In　view　of　the　model,　the　centralized　deep　reinforcement　learning　(DRL)　algorithm,　i.e.,　upper　confidence　bound　based　Dueling　deep　Q　network　(UCB　DDQN)　with　Mish　activation　function,　is　proposed　to　solve　the　multi-objective　optimization　problem　and　we　make　use　of　this　learning　algorithm　to　design　JBPA　strategy.　Furthermore,　a　federated　UCB　DDQN　learning　based　JBPA　is　to　proposed　tackle　the　challenge　of　the　centralized　DRL　would　require　excessive　data　exchange.　Simulation　results　validate　that　the　faster　convergence　speed　and　the　total　weighted　energy-spectrum　efficiency　(TWESE)　achieved　by　the　JBPA　based　on　UCB　DDQN　is　greater　than　conventional　DQN　based　resource　allocation　approach,　and　also　indicate　that　the　federated　UCB　DDQN　achieves　better　TWESE　performance　than　the　UCB　DDQN.