Gradient descent, vanishing gradient, feature selection vs reduction, PCA, overfitting
Filed under Nvidia
Free read 3 of 3Gradient descent updates parameters opposite the gradient of the loss: theta = theta - lr * grad (batch, stochastic or mini-batch). Vanishing gradients occur when many small derivatives multiply in deep networks (sigmoid or tanh saturation), so early layers barely learn. Fixes: ReLU, batch norm, residual connections, good initialisation, LSTM or GRU.
The question
Explain gradient descent, the vanishing gradient problem, feature selection versus feature reduction, the steps of PCA, and ways to reduce overfitting.
Approach
Gradient descent updates parameters opposite the gradient of the loss: theta = theta - lr * grad (batch, stochastic or mini-batch). Vanishing gradients occur when many small derivatives multiply in deep networks (sigmoid or tanh saturation), so early layers barely learn. Fixes: ReLU, batch norm, residual connections, good initialisation, LSTM or GRU.
Feature selection keeps a subset of the original features (filter, wrapper or embedded methods such as L1). Feature reduction creates new features from combinations of originals (PCA, autoencoders), so original meaning is lost.
1) Standardise data. 2) Compute the covariance matrix. 3) Find eigenvectors and eigenvalues. 4) Sort by eigenvalue and keep the top k. 5) Project the data onto those components. Explained variance helps choose k.
Get more data or augment it, use regularisation (L1 or L2), dropout, early stopping, a simpler model, cross-validation and feature selection.
Tips
- What they are testing: Breadth across core ML topics, with the right cause-and-fix reasoning.
- Common mistake: Confusing feature selection with PCA, or forgetting to standardise before PCA.
Related Nvidia questions
Collected from public candidate posts and coding forums. Updated Nov 2026.