Frequently Asked Questions About Socratica PyTorch Neural Network Builder
22 answers covering everything from basics to advanced usage.
// Basics
What is a tensor and why does PyTorch use them?
A tensor is a container for numerical data arranged in an n-dimensional array with a defined shape and data type. PyTorch uses tensors because, unlike ordinary lists or arrays, they're optimised for AI computations and can reside on a GPU for accelerated processing. All data — inputs, weights, and outputs — must be represented as tensors before entering a network.
What is the difference between forward and back propagation?
Forward propagation moves input data through the network layer by layer — input to hidden to output — producing a prediction. Back propagation is the mirror image: gradients are calculated starting from the output and moving backward through the computational graph, revealing how much each weight contributed to the error so the optimizer can make targeted adjustments.
What is autograd and how does it work?
Autograd is PyTorch's automatic differentiation system. It builds a computational graph recording every operation performed on tensors with requires_grad=True, then calculates gradients automatically when you call .backward(). Those gradients are stored in each tensor's .grad attribute and tell the optimizer the direction and magnitude of each weight adjustment needed to reduce loss.
What is the difference between a weight and a bias?
A weight is a number associated with each connection between neurons, representing the importance of that connection. A bias is a value added inside each neuron after the weighted sum of inputs, serving as a baseline adjustment before the activation function. Both weights and biases are the parameters the optimizer adjusts during training to minimise loss.
What is an epoch and how many should I run?
An epoch is one full pass through the entire training dataset. The methodology defaults to 100 epochs if unspecified, but the right number depends on your problem — run until loss reaches an acceptable level or stops improving. Monitor loss at intervals; if it plateaus, more epochs won't help and you should revisit learning rate or architecture.
// How To
How do I design the layer sizes for my network?
In __init__, define nn.Linear layers that map input dimensionality to hidden dimensions to output dimensionality. The first layer's in_features must match your feature count, and the final layer's out_features must match your task — 1 for regression, N for N-class classification. Hidden layer sizes (e.g., 8 neurons) are tunable; start small and increase if the network underfits.
How do I evaluate my model on test data correctly?
Run the trained model on new samples that were not part of the training set, comparing expected output, predicted output, and computed loss. Wrap the evaluation in torch.no_grad() to disable gradient tracking for efficiency. Close alignment between predictions and expected results confirms the network generalised rather than memorised the training examples.
How do I verify autograd is computing gradients correctly?
Create a scalar tensor x with requires_grad=True, define a simple function like y = x³, call y.backward(), then inspect x.grad. It should contain the analytically derived derivative (3x²) evaluated at the current x. This confirms autograd tracked the computational graph correctly — the same mechanism that powers back propagation in full networks.
How do I choose and tune the learning rate?
Start with 0.01 for SGD, then adjust based on loss behaviour. If loss decreases then spikes or oscillates, the learning rate is too high and the optimizer is overshooting — reduce it. If training is painfully slow and loss barely moves, the rate is too low — increase it. Learning rate must be tuned per problem; there's no universal value.
// Troubleshooting
My loss isn't decreasing — what should I check?
First verify your learning rate isn't too low. Then confirm you're calling optimizer.zero_grad(), loss.backward(), and optimizer.step() in the right order each epoch. Check that tensors are float32 with correct shapes, that requires_grad is set where needed, and that your architecture has enough depth and ReLU activations to model the pattern. Poor data quality can also stall learning.
Why does my loss decrease then suddenly spike?
A loss that drops then spikes almost always means the learning rate is too high — the optimizer overshoots the optimal parameters and diverges. Lower the learning rate (try halving it) and rerun. If the problem persists, check for exploding gradients or corrupted data. A stable, steadily declining loss curve is the goal.
Why is my regression model unable to predict negative values?
You likely applied ReLU to the final output layer. ReLU sets all negative values to zero, which artificially caps a regression network's output range and prevents negative predictions. Remove the activation from the final layer — for regression, the output must be able to take any real value. Apply ReLU only after hidden layers.
Why does loss.backward() throw an error about non-scalar output?
PyTorch requires the output to be a scalar before you call backward(). If your output is a vector or matrix, reduce it to a single number first — for example, by summing or averaging its components, which your loss criterion normally does. During training, the loss value returned by criterion() is already scalar, so this error usually appears in manual gradient experiments.
My model performs great on training data but poorly on new data — why?
This is a generalisation failure — the network memorised the training examples instead of learning the underlying pattern. Strong training performance never confirms generalisation; you must always evaluate on a separate held-out test set. If the gap is large, consider more diverse training data, a simpler architecture, or regularisation techniques.
// Comparisons
How does building a network by hand compare to just using model.fit()?
Building the loop by hand — with explicit zero_grad(), backward(), and step() calls — exposes exactly how autograd, back propagation, and optimizer updates work. High-level fit() methods hide these mechanics, which is convenient but leaves you helpless when training breaks. This methodology trades convenience for the deep understanding needed to debug, customise, and trust your models.
How does SGD compare to other optimizers like Adam?
SGD (Stochastic Gradient Descent) is a fundamental, reliable optimizer that updates weights using randomly selected batches — the methodology recommends it as a starting point. Adam adapts the learning rate per parameter and often converges faster on complex problems, but SGD's simplicity makes it ideal for learning the training cycle. Once comfortable, you can swap optimizers with a single line change.
How does MSE compare to Cross Entropy Loss?
MSE (Mean Squared Error) is the standard criterion for regression — it averages the squared differences between predictions and targets, magnifying large errors. Cross Entropy Loss is designed for classification, measuring the divergence between predicted class probabilities and true labels. Using the wrong criterion for your task type produces meaningless gradients and prevents effective learning.
// Advanced
How do I add more hidden layers to make my network deeper?
In __init__, define additional nn.Linear layers, chaining each layer's out_features into the next layer's in_features. In forward, pass X through each new hidden layer and apply F.relu() after it. Deeper networks can model more abstract representations, but too much depth risks slower training and overfitting — increase depth gradually and monitor test performance.
How do I train on a GPU instead of a CPU?
Move your model and tensors to the GPU with .to('cuda') (or .to(device) using a device variable). Because tensors can reside on a GPU for accelerated processing, this dramatically speeds up training on large datasets. Ensure both your data tensors and model parameters are on the same device, or PyTorch will raise a device-mismatch error.
How do I adapt this workflow from regression to classification?
Change three things: set the final layer's output size to N (number of classes), apply an appropriate final activation like softmax or sigmoid instead of leaving the output raw, and swap nn.MSELoss() for nn.CrossEntropyLoss(). The rest of the predict-measure-adjust loop — zero_grad, backward, step — stays identical. Evaluation still uses held-out test data.
What is the role of the computational graph during training?
The computational graph is PyTorch's internal record of every mathematical operation performed on tracked tensors. During back propagation, autograd traverses this graph in reverse to calculate how each parameter contributed to the loss. It's rebuilt on each forward pass, which is why clearing gradients with zero_grad() each epoch is essential to avoid corrupting updates.
Should I always aim for zero loss during training?
No — zero loss is not the goal and often signals overfitting. The aim is generalisation: a network that performs well on data it has never seen. Compare your baseline loss before training to the final loss to confirm clear, measurable improvement, then validate on a held-out test set. Close alignment between predictions and true test values matters more than a perfect training score.