Frequently Asked Questions About Kylie Ying ML for Everyone Framework
22 answers covering everything from basics to advanced usage.
// Basics
What is a features matrix X and labels vector y?
The features matrix X is a 2D array of all input features across all samples — each row is one sample, each column is one feature. The labels vector y is a 1D array of true output values, one per sample, that the model tries to predict. Every model learns the mapping from X to y. For unsupervised learning, you skip y entirely.
What is loss and why does it matter?
Loss is a numerical measure of the difference between a model's prediction and the true label. The entire training loop exists to minimize loss — lower loss means better performance. Common loss functions include L1 loss (absolute difference, penalizes all errors equally), L2 loss (squared difference, penalizes large errors more), and binary cross-entropy (for binary classification).
What does StandardScaler actually do?
StandardScaler normalizes each feature column by subtracting its mean and dividing by its standard deviation, putting all features on a comparable scale. It matters when columns have wildly different ranges — for example one feature spanning 0–1 and another 0–10,000 — which can distort distance-based models like KNN and SVM. Always fit it on training data only, then transform validation and test sets.
What is one-hot encoding and when should I use it?
One-hot encoding converts nominal categorical features — categories with no inherent order, like country or color — into binary columns, one per category. A sample gets a 1 in its category's column and 0 everywhere else. Use it for nominal data. For ordinal data with a natural order (like ratings), map categories to ordered integers instead so the model preserves that ranking.
// How To
How do I encode a binary string label like 'yes'/'no'?
Convert binary string labels to 0 and 1 using a comparison and astype(int) — for example, checking whether a value equals 'yes' and casting the boolean result to an integer. Computers understand numbers, not strings, so feeding raw string labels into a model causes errors or nonsensical results. Always encode all features and labels as numbers before training.
How do I split my data correctly without leaking?
First shuffle the dataset (for example, with df.sample(frac=1)) to remove ordering bias, then split into roughly 60% train, 20% validation, and 20% test — or 80/10/10 for larger datasets. Keep all three splits fully separate throughout. Fit StandardScaler and RandomOverSampler on the training set only, and never let the test set influence any training or model-selection decision.
How do I choose the best K in KNN?
Test multiple K values (start with K=1, 3, and 5) and compare their performance on the validation set, not the test set. K=1 tends to overfit by memorizing individual points, while very large K smooths over real decision boundaries. Pick the K with the best validation F1 score, then evaluate that single choice once on the test set for your final reported performance.
How do I build a neural network if classical models plateau?
Build a network with an input layer (one neuron per feature), one or more hidden layers using nonlinear activation functions (ReLU is a strong default), and an output layer (Sigmoid for binary classification). Train it with gradient descent and backpropagation, and monitor both training loss and validation loss per epoch. Diverging validation loss signals overfitting — a cue to stop or regularize.
// Troubleshooting
Why is my model biased toward the majority class?
Your training set is likely imbalanced, so the model learns that predicting the majority class is usually correct. Fix it by applying RandomOverSampler to the training set only, duplicating minority-class samples until classes balance. Then report F1, precision, and recall instead of accuracy — because a model that always predicts the majority class can still score high accuracy while being useless.
Why does my model look accurate but perform poorly in production?
You likely have data leakage or misleading accuracy. Common causes: fitting StandardScaler on the full dataset instead of just training data, oversampling the validation or test sets, using the test set to tune hyperparameters, or reporting accuracy on imbalanced data. Fix by fitting preprocessing on training data only, keeping test data untouched until the final evaluation, and prioritizing F1 score.
Why did my SVM model perform worse than expected?
SVMs are not robust to outliers — a single extreme point can shift the support vectors and move the separating hyperplane significantly. Check your data for outliers before applying SVC and consider removing or clipping them. Also ensure features are scaled with StandardScaler, since SVM relies on distances that are distorted when feature ranges differ widely.
Why does my deep neural network act like a simple linear model?
You're probably missing activation functions in your hidden layers. Without nonlinear activations like ReLU, Sigmoid, or tanh, stacking multiple layers collapses mathematically into a single linear transformation, making depth pointless. Add a nonlinear activation function after each hidden layer so the network can learn complex, non-linear patterns.
// Comparisons
How does this framework compare to a generic 'just throw it at a model' approach?
A generic approach usually skips disciplined preprocessing and honest evaluation, producing models that look strong but fail on real data. This framework enforces shuffling before splitting, scaling fit only on training data, oversampling only the training set, testing multiple candidate models, and prioritizing F1 over accuracy. The result is a leak-free pipeline and performance numbers that actually reflect generalization.
How does F1 score compare to accuracy for evaluation?
Accuracy measures the overall fraction of correct predictions and is misleading on imbalanced data — a model predicting only the majority class can score high accuracy while missing every minority case. F1 score is the harmonic mean of precision and recall, balancing false positives and false negatives. Prefer F1 whenever classes are imbalanced; accuracy is only trustworthy when classes are roughly balanced.
How does KNN compare to SVM for classification?
KNN classifies a point by the majority label among its K nearest neighbors using Euclidean distance — simple and intuitive but sensitive to K and feature scaling. SVM finds the hyperplane that maximizes the margin between classes and often excels in high-dimensional feature spaces, but it's sensitive to outliers. Train both, compare F1 on the validation set, and pick the stronger performer.
How does Naive Bayes compare to Logistic Regression?
Naive Bayes is a probabilistic classifier that applies Bayes' rule and assumes features are conditionally independent given the class, predicting the class with the highest posterior (MAP). Logistic Regression fits data to the sigmoid function to output class probabilities without that independence assumption. Naive Bayes is fast and a good baseline; Logistic Regression often generalizes better when features are correlated.
// Advanced
What is the difference between the validation set and the test set?
The validation set is used during model development to tune hyperparameters and reality-check generalization — you can look at it repeatedly, but its loss never feeds back to update weights. The test set is used exactly once, after all model selection and tuning, to report final real-world performance. Using the test set to tune models defeats its purpose and inflates your reported numbers.
What is the kernel trick in SVM?
The kernel trick transforms features into a higher-dimensional space — for example adding an x² feature — so data that isn't linearly separable becomes separable by a hyperplane. This lets SVM draw complex, non-linear decision boundaries without explicitly computing the high-dimensional coordinates. Choosing the kernel is a hyperparameter you should tune on the validation set.
How does gradient descent update neural network weights?
Gradient descent computes the slope of the loss with respect to each weight — how much that weight contributed to the error — via backpropagation, then steps each weight in the direction that reduces loss. The learning rate controls the step size: too large overshoots the minimum, too small trains slowly. This repeats over many epochs until the loss stops decreasing meaningfully.
When should I use clustering instead of classification?
Use clustering when your data has no reliable labels and you want to discover natural groupings — for example, sensor readings from uncertain particle events. Skip the labels vector y, scale the features, and apply an algorithm like KMeans, then visualize the clusters to see if they align with domain knowledge. Use classification instead whenever you have trustworthy labels to learn from.
Can I use this framework for regression tasks too?
Yes — the pipeline is the same, but you swap classifiers for their regression equivalents in scikit-learn and change your metrics. Instead of precision, recall, and F1, you evaluate with loss functions like L1 (mean absolute error) or L2 (mean squared error, which penalizes outliers more). Encoding, scaling, and the train/validation/test split all apply identically.
How many models should I train before choosing one?
Train at least four baselines — KNN with a few K values, Gaussian Naive Bayes, Logistic Regression, and SVC — then compare their F1 scores. This gives you a spread across distance-based, probabilistic, and margin-based approaches so you're not betting on one algorithm. Only escalate to a neural network if these classical models plateau, since networks add complexity and training cost.