当前位置：首页 > news >正文

优势演员-评论家（Advantage Actor-Critic，A2C）算法详解与完成

news 2026/3/27 7:00:47

优势演员-评论家（Advantage Actor-Critic，A2C）算法详解与实现

- 0. 前言
- 1. Advantage Actor-Critic (A2C) 算法原理
- 2. A2C 算法流程
- 3. 实现 A2C

定义可微参数化目标策略网络 $π(at∣st,θ)\pi(a_t|s_t,\theta)$ 与可微参数化价值网络 $V(st,θv)V(s_t,\theta_v)$ ；设定折扣因子 $γ∈[0,1]\gamma∈[0,1]$ ，性能梯度学习率 $α\alpha$ ，价值梯度学习率 $αv\alpha_v$ ，以及熵权重 $β\beta$ ；初始化策略网络参数 $θ0\theta_0$ 与价值网络参数 $θv0\theta_{v_0}$
重复执行
依据策略 $π(at∣st,θ)\pi(a_t|s_t,\theta)$ 生成完整轨迹 $s_0a_0r_1s_1, s_1a_1r_2s_2, ..., s_{t-1}a_{t-1}r_ts_t)$
计算终止状态回报： $Rt={0sT为终止状态V(sT,θv)对于非终止状态sT,从最终状态开始进行自举法估计R_t=\begin{cases}0 & s_T为终止状态\\ V(s_T,θ_v) & 对于非终止状态 s_T, 从最终状态开始进行自举法估计 \end{cases}$
从最后一步 $t = T - 1$ 反向遍历至初始状态 $t = 0$ ：
计算累计回报： $Rt=rt+γRtR_t = r_t + \gamma R_t$
计算价值梯度： $∇V(θv)=∂(Rt−V(s,θv))2∂θv\nabla V(\theta _v) = \frac {\partial (R_t - V(s,\theta _v))²}{\partial \theta _v}$
累积价值梯度更新： $θv=θv+αv∇V(θv)\theta _v = \theta _v + \alpha _v\nabla V(\theta _v)$
计算策略梯度： $∇J(θ)=∇θlnπ(at∣st,θ)(Rt−V(s,θv))+β∇θH(π(at∣st,θ))\nabla J(\theta) = \nabla _\theta ln\pi(a_t|s_t,\theta)(R_t - V(s,\theta _v)) + \beta \nabla _\theta H(\pi(a_t|s_t,\theta))$
执行梯度上升： $θ=θ+α∇J(θ)\theta = \theta + \alpha \nabla J(\theta)$

3. 实现 A2C

接下来，使用 Keras 实现 A2CAgent 类。与两种 REINFORCE 方法不同，本算法从最终经验单元反向计算至初始状态。在每个经验单元处，目标函数网络 logp_model 和价值函数网络 value_model 分别通过调用 fit() 方法进行优化。需要注意的是，在对象实例化时，熵损失权重 beta 设置为 0.9 以启用熵损失函数，且 value_model 采用均方误差损失函数进行训练。

class A2CAgent(PolicyAgent):
def __init__(self,env):
super().__init__(env)
#beta of entropy used in A2C
self.beta = 0.9
#loss function of A2C value_model is mse
self.loss = 'mse'
def train_by_episode(self,last_value=0):
#implements A2C training from the last state
#to the first state
#discount factor
gamma = 0.95
r = last_value
#the memory is visited in reverse
for item in self.memory[::-1]:
[step,state,next_state,reward,done] = item
#compute the return
r = reward + gamma * r
item = [step,state,next_state,r,done]
#train pre step
#a2c reward has been discounted
self.train(item)
def train(self,item,gamma=1.0):
[step,state,next_state,reward,done] = item
#must save state for entropy computation
self.state = state
discount_factor = gamma ** step
#a2c: delta = discounted_reward - value
delta = reward - self.value(state)[0]
discounted_delta = delta * discount_factor
discounted_delta = np.reshape(discounted_delta,[-1,1])
verbose = 1 if done else 0
#train the logp model (implies training of actor model 
# as well) since they share exactly the same set of parameters
self.logp_model.fit(np.array(state),
discounted_delta,
batch_size=1,
epochs=1,
verbose=verbose)
#in A2C, the target value is the return (reward
# replaced by return in train_by_episode function)
discounted_delta = reward
discounted_delta = np.reshape(discounted_delta,[-1,1])
#train the value network (critic)
self.value_model.fit(np.array(state),
discounted_delta,
batch_size=1,
epochs=1,
verbose=verbose)