Mxnet (30): (RMSProp, AdaDelta, Adam)算法

1. RMSProp算法

该种方式具有显著的效果. Adagrad累积梯度的平方到state向量\mathbf{g}_t = \sum (\frac{\partial L}{\partial x})^2. 因为\sum (\frac{\partial L}{\partial x})^2在缺乏正则化的情况下会随算法收敛呈线性增长. RMSProp算法将这些梯度按元素平方做指数加权移动平均:\sum (\frac{\partial L}{\partial x})^2 ← γ\sum (\frac{\partial L}{\partial x})^2 + (1−γ)(\frac{\partial L}{∂x})^2. 参数γ>0

    from d2l import mxnet as d2l
    from mxnet import np, npx, autograd, init, gluon
    from mxnet.gluon import nn
    import plotly.graph_objs as go
    import plotly.express as px
    import pandas as pd
    import math
    npx.set_np()
    
    
    def train_2d