The　core　of　the　optimal　tracking　control　problem　for　nonlinear　systems　is　how　to　ensure　that　the　controlled　system　tracks　the　desired　trajectory.　The　utility　functions　in　previous　studies　have　different　properties　which　affect　the　final　tracking　effect　of　the　intelligent　critic　algorithm.　In　this　paper,　we　introduce　a　novel　utility　function　and　propose　a　Q-function　based　policy　iteration　algorithm　to　eliminate　the　final　tracking　error.　In　addition,　neural　networks　are　used　as　function　approximator　to　approximate　the　performance　index　and　control　policy.　Considering　the　impact　of　the　approximation　error　on　the　tracking　performance,　an　approximation　error　bound　for　each　iteration　of　the　novel　Q-function　is　established.　Under　the　given　conditions,　the　approximation　Q-function　converges　to　the　finite　neighborhood　of　the　optimal　value.　Moreover,　it　is　proved　that　weight　estimation　errors　of　neural　networks　are　uniformly　ultimately　bounded.　Finally,　the　effectiveness　of　the　algorithm　is　verified　by　the　simulation　example.　©　2023　Elsevier　B.V.