Abstract ：

Optimal control problems are ubiquitous in practical engineering applications and social life with the idea of cost or resource conservation. Based on the critic learning scheme, adaptive dynamic programming (ADP) is regarded as a significant avenue to address the optimal control problems by combining the advanced design ideas such as adaptive control, reinforcement learning, and intelligent control. This survey introduces the recent development of ADP and related intelligent critic control with an emphasis on advanced value iteration (VI) schemes for discrete-time nonlinear systems. The theoretical results focus on convergence and stability properties for general VI, stabilizing VI, integrated VI, evolving VI, adjustable VI schemes and so on. Several significant applications are also elaborated in aspects of optimal regulation, optimal tracking, and zero-sum games. We aim to break through the bottleneck problems for VI algorithms in realizing evolving control, accelerating learning speed, and reducing the calculation expense. In addition, the prospects of new theoretical and technical fields for advanced VI schemes are looked ahead.

Keyword ：

Optimal control Optimal control Adaptive dynamic programming Adaptive dynamic programming Advanced value iteration Advanced value iteration

Cite：

Copy from the list or Export to your reference management。

Abstract ：

This article puts emphasis on the deterministic value-iteration-based Q-learning (VIQL) algorithm with adjustable convergence speed, followed by the application verification on trajectory tracking for completely unknown nonaffine systems. It is worth emphasizing that, under the effect of learning rates, the convergence speed can be adjusted and the new convergence criterion of the VIQL framework is investigated. The merit of the adjustable VIQL scheme is that it can quicken the learning speed and decrease the number of iterations, thereby reducing the computation burden. To carry out the model-free VIQL algorithm, the offline data of system states and reference trajectories are collected to provide the reference control, the tracking error, and the tracking control, which promotes the parameter updating of the adjustable VIQL algorithm via the off-policy learning scheme. By this updating operation, the convergent optimal tracking policy can guarantee that arbitrary initial state tracks the desired trajectory and can completely obviate the terminal tracking error. Finally, numerical simulations are conducted to indicate the validity of the designed tracking control algorithm.

Keyword ：

Q-learning Q-learning convergence speed convergence speed Adaptive critic control Adaptive critic control optimal tracking optimal tracking adaptive dynamic programming (ADP) adaptive dynamic programming (ADP)

Cite：

Copy from the list or Export to your reference management。

Abstract ：

设计了一种基于事件的迭代自适应评判算法,用于解决一类非仿射系统的零和博弈最优跟踪控制问题.通过数值求解方法得到参考轨迹的稳定控制,进而将未知非线性系统的零和博弈最优跟踪控制问题转化为误差系统的最优调节问题.为了保证闭环系统在具有良好控制性能的基础上有效地提高资源利用率,引入一个合适的事件触发条件来获得阶段性更新的跟踪策略对.然后,根据设计的触发条件,采用Lyapunov方法证明误差系统的渐近稳定性.接着,通过构建四个神经网络,来促进所提算法的实现.为了提高目标轨迹对应稳定控制的精度,采用模型网络直接逼近未知系统函数而不是误差动态系统.构建评判网络、执行网络和扰动网络用于近似迭代代价函数和迭代跟踪策略对.最后,通过两个仿真实例,验证该控制方法的可行性和有效性.

Keyword ：

稳定性分析 稳定性分析 自适应评判设计 自适应评判设计 事件触发控制 事件触发控制 神经网络 神经网络 零和博弈 零和博弈 最优跟踪控制 最优跟踪控制

Cite：

Copy from the list or Export to your reference management。

Abstract ：

Inspired by the successive relaxation method, a novel discounted iterative adaptive dynamic programming framework is developed, in which the iterative value function sequence possesses an adjustable convergence rate. The different convergence properties of the value function sequence and the stability of the closed-loop systems under the new discounted value iteration (VI) are investigated. Based on the properties of the given VI scheme, an accelerated learning algorithm with convergence guarantee is presented. Moreover, the implementations of the new VI scheme and its accelerated learning design are elaborated, which involve value function approximation and policy improvement. A nonlinear fourth-order ball-and-beam balancing plant is used to verify the performance of the developed approaches. Compared with the traditional VI, the present discounted iterative adaptive critic designs greatly accelerate the convergence rate of the value function and reduce the computational cost simultaneously.

Keyword ：

Iterative methods Iterative methods Stability criteria Stability criteria adaptive dynamic programming (ADP) adaptive dynamic programming (ADP) Cost function Cost function reinforcement learning reinforcement learning Power system stability Power system stability Adaptive critic designs Adaptive critic designs Closed loop systems Closed loop systems fast convergence rate fast convergence rate Optimal control Optimal control Convergence Convergence value iteration (VI) value iteration (VI) discrete-time nonlinear systems discrete-time nonlinear systems

Cite：

Copy from the list or Export to your reference management。

Abstract ：

This paper is concerned with a novel integrated multi-step heuristic dynamic programming (MsHDP) algorithm for solving optimal control problems. It is shown that, initialized by the zero cost function, MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman (HJB) equation. Then, the stability of the system is analyzed using control policies generated by MsHDP.Also, a general stability criterion is designed to determine the admissibility of the current control policy. That is, the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP. Further, based on the convergence and the stability criterion, the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly. Besides, actor-critic is utilized to implement the integrated MsHDP scheme, where neural networks are used to evaluate and improve the iterative policy as the parameter architecture. Finally, two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

Keyword ：

multi-step reinforcement learning multi-step reinforcement learning Index Terms-Adaptive critic Index Terms-Adaptive critic multi-step heuristic dynamic pro-gramming multi-step heuristic dynamic pro-gramming artificial neural networks artificial neural networks Hamilton-Jacobi-Bellman (HJB) equation Hamilton-Jacobi-Bellman (HJB) equation optimal control optimal control

Cite：

Copy from the list or Export to your reference management。

Abstract ：

In this article, a decentralized event-triggered control mechanism is established to solve the interconnected issue of continuous-time nonlinear systems with asymmetric input constraints and matched interconnections based on the adaptive critic technology. First, by inserting the discount factor, a novel nonquadratic cost function is constructed for the constrained subsystem with nonzero equilibrium point. Meanwhile, the decentralized event-triggered control issue is transformed into a set of optimal control issues. Then, the execution of nominal subsystem is based on the event-triggered mechanism (ETM) with an event-triggering condition which increases the algorithm efficiency. Moreover, we derive the associated event-triggered Hamilton-Jacobi-Bellman (HJB) equation which arising in the discounted-cost optimal event-triggered control issues of nominal subsystems. In the implementation, an adaptive critic framework is employed to approximate the optimal cost function. Later, the experience replay (ER) approach is introduced into a novel weight tuning mechanism, which converts the traditional persistence of excitation (PE) condition into an easy-checked rank condition. Theoretically, the stability of the system and the exclusion of Zeno behavior are demonstrated. Finally, one representative example is simulated to validate the efficacy of the constructed framework.

Keyword ：

Cost function Cost function adaptive dynamic programming (ADP) adaptive dynamic programming (ADP) Adaptive systems Adaptive systems neural networks neural networks Adaptive critic designs Adaptive critic designs Mathematical models Mathematical models event-triggered control event-triggered control asymmetric input constraints asymmetric input constraints interconnected systems interconnected systems experience replay (ER) experience replay (ER) Decentralized control Decentralized control Optimal control Optimal control Interconnected systems Interconnected systems Tuning Tuning

Cite：

Copy from the list or Export to your reference management。

Abstract ：

The majority of the results on constrained nonlinear multi-agent systems (MASs) control focused on output or state constraints without considering the saving of communication resources. In this paper, for a class of uncertain nonlinear MASs, we first present a new adaptive bounded consensus tracking control scheme in which the asymmetric and full-state constraints are jointly synthesized with a switching threshold event-triggered strategy, such that the communication resources are effectively utilized. The key to accomplishing the asymmetric and full-state constraints is that a kind of improved $tan$ -type barrier Lyapunov functions are constructed. The controller constructed for each agent by the switching threshold event-triggered strategy guarantees that the asymmetric and full-state constraints are not violated and the output of each agent can track the leader's trajectory with an adjustable bounded tracking error. Furthermore, to achieve the asymptotic consensus tracking control, we give another kind of novel $tan$ -type barrier Lyapunov functions to design the desired controller for each agent. A simulation example of five single-link robots is proposed to illustrate the effectiveness of our control scheme.

Keyword ：

Nonlinear multi-agent systems Nonlinear multi-agent systems tan-type barrier Lyapunov functions tan-type barrier Lyapunov functions full-state constraints full-state constraints switching threshold event-triggered strategy switching threshold event-triggered strategy consensus track-ing control consensus track-ing control

Cite：

Copy from the list or Export to your reference management。

Abstract ：

In this article, the generalized N-step value gradient learning (GNSVGL) algorithm, which takes a long-term prediction parameter lambda into account, is developed for infinite horizon discounted near-optimal control of discrete-time nonlinear systems. The proposed GNSVGL algorithm can accelerate the learning process of adaptive dynamic programming (ADP)and has a better performance by learning from more than one future reward. Compared with the traditional N-step value gradient learning (NSVGL) algorithm with zero initial functions, the proposed GNSVGL algorithm is initialized with positive definite functions. Considering different initial cost functions, the convergence analysis of the value-iteration-based algorithm is provided. The stability condition for the iterative control policy is established to determine the value of the iteration index, under which the control law can make the system asymptotically stable. Under such a condition, if the system is asymptotically stable at the current iteration, then the iterative control laws after this step are guaranteed to be stabilizing. Two critic neural networks and one action network are constructed to approximate the one-return costate function, the lambda-return costate function, and the control law, respectively. It is emphasized that one-return and lambda-return critic networks are combined to train the action neural network. Finally, via conducting simulation studies and comparisons, the superiority of the developed algorithm is confirmed.

Keyword ：

convergence and stability convergence and stability N-step value gradient learning (NSVGL) N-step value gradient learning (NSVGL) adaptive critic adaptive critic ?-return cost function ?-return cost function neural networks neural networks

Cite：

Copy from the list or Export to your reference management。

Abstract ：

In this thesis, we construct improved value iteration (VI) and online VI structures, in a bid to tackle the optimal tracking control problem for discrete-time nonlinear systems. Note that asymmetric control restraints and the discount factor are considered. First, related properties are discussed for novel VI, involving the monotonicity of the iterative cost function sequence and the admissibility of the iterative tracking control policy. Second, the stability condition for the discount factor is provided to ensure the stability of all iterative tracking control policies, which are created by stabilizing VI. Third, by combining novel VI and stabilizing VI, an improved VI algorithm is developed, where iterative cost function sequences are monotonically nondecreasing and nonincreasing during novel VI and stabilizing VI stages, respectively. Fourth, with the appropriate discount factor, an online VI algorithm is proposed by integrating the attraction domain with improved VI. Also, under the online VI structure, the asymptotic stability proof is performed for the tracking error system. Finally, regarding theoretical contributions are illustrated by a simulation case.

Keyword ：

constrained systems constrained systems online value iteration (VI) online value iteration (VI) stability proof stability proof attraction domain attraction domain discounted optimal tracking control discounted optimal tracking control Index Terms-Actor-critic Index Terms-Actor-critic

Cite：

Copy from the list or Export to your reference management。

Abstract ：

In recent years, the application of function approximators, such as neural networks and polynomials, has ushered in a new stage of development in solving optimal control problems. However, considering the existence of approximation errors, the stability of the controlled system cannot be guaranteed. Therefore, in view of the prevalence of approximation errors, we investigate optimal tracking control problems for discrete-time systems. First, a novel value function is introduced into the intelligent critic framework. Second, an implicit method is utilized to demonstrate the boundedness of the iterative value functions with approximation errors. An explicit method is applied to prove the stability of the system with approximation errors. Furthermore, an evolving policy is designed to iteratively tackle the optimal tracking control problem and demonstrate the stability of the system. Finally, the effectiveness of the developed method is verified through numerical as well as practical examples.

Keyword ：

intelligent control intelligent control Adaptive dynamic programming (ADP) Adaptive dynamic programming (ADP) optimal tracking control optimal tracking control approximation errors approximation errors valueiteration (VI) valueiteration (VI)

Cite：

Copy from the list or Export to your reference management。