This　article　investigates　the　stability　of　the　closed-loop　system　under　the　iterative　control　policy　generated　by　various　iterative　Q-learning　algorithms.　First,　a　new　stability　criterion　is　developed　for　the　value-iteration-based　Q-learning　(VIQL)　algorithm,　which　is　initialized　by　a　positive　semi-definite　function.　Through　this　operation,　VIQL　can　provide　an　initial　admissible　control　policy　for　the　policy-iteration-based　Q-learning　(PIQL)　algorithm.　It　is　emphasized　that　evolving　control　policies　generated　by　PIQL　can　stabilize　the　controlled　system.　The　numerical　result　is　provided　to　verify　the　effectiveness　of　the　present　algorithms.　　©　2023　IEEE.