What is gradient descent and where is it used?
Filed under Amazon
Free read 1 of 3Gradient descent is an iterative optimisation method that minimises a loss function by repeatedly stepping in the direction opposite to the gradient, because that is the direction of steepest decrease.
The question
What is gradient descent and in which algorithms is it used?
Approach
Gradient descent is an iterative optimisation method that minimises a loss function by repeatedly stepping in the direction opposite to the gradient, because that is the direction of steepest decrease.
theta = theta - eta * gradient of J(theta), where eta is the learning rate. Too large a rate overshoots or diverges; too small is slow. Scaling features helps convergence.
Batch uses all data per step (stable, slow), stochastic uses one sample (noisy, fast), mini-batch uses small groups (the common compromise). Momentum, RMSProp and Adam improve it.
Linear and logistic regression, neural networks (via backpropagation), SVMs trained with gradient methods, and gradient boosting (each tree fits the negative gradient). Limits: local minima and saddle points on non-convex losses, and sensitivity to the learning rate.
Tips
- What they are testing: This question tests the core idea, the update rule, and the role of the learning rate.
- Common mistake: Confusing gradient descent with the model itself, or ignoring the learning rate and feature scaling.
Related Amazon questions
Collected from public candidate posts and coding forums. Updated Nov 2026.