Understanding the Adam Optimizer

Oct 13, 2025 by magicalceylontours.com 131 views

Introduction

The Adam optimizer is a popular stochastic gradient descent (SGD) optimization algorithm used in deep learning to minimize the loss function. It is known for its ability to adapt the learning rate for each parameter based on the magnitude of the gradient. However, the Adam optimizer uses uncentered variance to downscale the parameter updates, which may seem counterintuitive at first. In this article, we will explore why the Adam optimizer uses uncentered variance instead of centered variance to downscale the parameter updates.

Understanding the Adam Optimizer

The Adam optimizer updates the model parameters using the following formula:

$\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$

Where:

$\theta_t$ is the updated parameter at time step $t$
$\theta_{t-1}$ is the previous parameter value
$\alpha$ is the learning rate
$\hat{m}_t$ is the bias-corrected first moment (i.e., the moving average of the gradient)
$\hat{v}_t$ is the bias-corrected second moment (i.e., the moving average of the squared gradient)
$\epsilon$ is a small value added to the denominator to prevent division by zero

Why Uncentered Variance?

So, why does the Adam optimizer use uncentered variance instead of centered variance to downscale the parameter updates? To understand this, let's first define centered and uncentered variance.

Centered Variance

The centered variance is defined as:

$\text{Var}(X) = E[(X - E[X])^2]$

Where $E[X]$ is the expected value of $X$ .

Uncentered Variance

The uncentered variance is defined as:

$\text{Var}(X) = E[X^2] - (E[X])^2$

Now, let's consider the bias-corrected second moment $\hat{v}_t$ used in the Adam optimizer:

$\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$

Where:

$v_t$ is the second moment (i.e., the moving average of the squared gradient)
$\beta_2$ is the decay rate for the second moment

The bias-corrected second moment $\hat{v}_t$ is an estimate of the true second moment, but it is biased because it is calculated using the moving average of the squared gradient. To correct for this bias, we divide the second moment $v_t$ by $1 - \beta_2^t$ .

Why Uncentered Variance is Used

Now, let's consider why uncentered variance is used instead of centered variance to downscale the parameter updates. The centered variance is defined as:

$\text{Var}(X) = E[(X - E[X])^2]$

However, when we calculate the bias-corrected second moment $\hat{v}_t$ , we are using the moving average of the squared gradient, which is biased. If we use centered variance, we would need to subtract the expected value of the squared gradient, which is not available.

On the other hand, the uncentered variance is defined as:

$\text{Var}(X) = E[X^2] - (E[X])^2$

This is exactly what we need to calculate the bias-corrected second moment $\hat{v}_t$ . By using uncentered variance, we can avoid subtracting the expected value of the squared gradient, which is not available.

Conclusion

In conclusion, the Adam optimizer uses uncentered variance instead of centered variance to downscale the parameter updates because it is a more suitable choice for the bias-corrected second moment $\hat{v}_t$ . The uncentered variance is defined as $E[X^2] - (E[X])^2$ , which is exactly what we need to calculate the bias-corrected second moment $\hat{v}_t$ . By using uncentered variance, we can avoid subtracting the expected value of the squared gradient, which is not available.

References

Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Dozat, T. (2016). Incorporating Nesterov Acceleration into Adam. arXiv preprint arXiv:1609.04747.

Code Implementation

Here is an example code implementation of the Adam optimizer using uncentered variance:

import numpy as np

class AdamOptimizer:
    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
        self.learning_rate = learning_rate
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.m = 0
        self.v = 0

    def update(self, gradient):
        self.m = self.beta1 * self.m + (1 - self.beta1) * gradient
        self.v = self.beta2 * self.v + (1 - self.beta2) * gradient ** 2
        self.m_corrected = self.m / (1 - self.beta1 ** (self.t + 1))
        self.v_corrected = self.v / (1 - self.beta2 ** (self.t + 1))
        self.t += 1
        return -self.learning_rate * self.m_corrected / (np.sqrt(self.v_corrected) + self.epsilon)

Q: What is the main reason why Adam optimizer uses uncentered variance instead of centered variance?

A: The main reason why Adam optimizer uses uncentered variance instead of centered variance is that it is a more suitable choice for the bias-corrected second moment. The uncentered variance is defined as $E[X^2] - (E[X])^2$ , which is exactly what we need to calculate the bias-corrected second moment.

Q: What is the difference between centered and uncentered variance?

A: The centered variance is defined as $E[(X - E[X])^2]$ , while the uncentered variance is defined as $E[X^2] - (E[X])^2$ . The centered variance subtracts the expected value of the variable, while the uncentered variance does not.

Q: Why is the centered variance not suitable for the Adam optimizer?

A: The centered variance is not suitable for the Adam optimizer because it requires the expected value of the variable, which is not available in the Adam optimizer. The Adam optimizer uses the moving average of the squared gradient to calculate the bias-corrected second moment, which is biased. If we use centered variance, we would need to subtract the expected value of the squared gradient, which is not available.

Q: What is the bias-corrected second moment in the Adam optimizer?

A: The bias-corrected second moment in the Adam optimizer is defined as $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$ , where $v_t$ is the second moment (i.e., the moving average of the squared gradient) and $\beta_2$ is the decay rate for the second moment.

Q: Why is the bias-corrected second moment important in the Adam optimizer?

A: The bias-corrected second moment is important in the Adam optimizer because it helps to correct for the bias in the second moment. The second moment is biased because it is calculated using the moving average of the squared gradient, which is not a true representation of the second moment.

Q: How does the Adam optimizer use the bias-corrected second moment to downscale the parameter updates?

A: The Adam optimizer uses the bias-corrected second moment to downscale the parameter updates by dividing the gradient by the square root of the bias-corrected second moment. This helps to prevent the parameter updates from being too large and to improve the convergence of the optimizer.

Q: What are the benefits of using uncentered variance in the Adam optimizer?

A: The benefits of using uncentered variance in the Adam optimizer include:

Improved convergence: The Adam optimizer converges faster and more reliably when using uncentered variance.
Better handling of biased gradients: The Adam optimizer is better able to handle biased gradients when using uncentered variance.
Simplified implementation: The Adam optimizer is simpler to implement when using uncentered variance.

Q: What are the limitations of using uncentered variance in the Adam optimizer?

A: The limitations of using uncentered variance in the Adam optimizer include:

Increased variance: The Adam optimizer may have increased variance when using uncentered variance.
Reduced stability: The Adam optimizer may be less stable when using uncentered variance.

Q: Can I use centered variance in the Adam optimizer instead of uncentered variance?

A: Yes, you can use centered variance in the Adam optimizer instead of uncentered variance. However, this may require additional calculations and may not be as efficient as using uncentered variance.

Q: How can I implement the Adam optimizer with uncentered variance in my code?

A: You can implement the Adam optimizer with uncentered variance in your code by using the following formula:

$\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$

Where:

$\theta_t$ is the updated parameter at time step $t$
$\theta_{t-1}$ is the previous parameter value
$\alpha$ is the learning rate
$\hat{m}_t$ is the bias-corrected first moment (i.e., the moving average of the gradient)
$\hat{v}_t$ is the bias-corrected second moment (i.e., the moving average of the squared gradient)
$\epsilon$ is a small value added to the denominator to prevent division by zero

Note that this is a simplified implementation of the Adam optimizer and may not include all the features and optimizations of the original implementation.