Frequently Asked Questions About Ko Romeo Deep Learning Neural Network Builder
21 answers covering everything from basics to advanced usage.
// Basics
What is a synapse in a neural network?
A synapse is the connection between two neurons, equivalent to the biological axon-to-dendrite junction. Each synapse carries a weight that determines the strength and relevance of the signal passing through it. Adjusting these weights across all synapses is literally what 'learning' means in a neural network — nothing else is being trained.
What is an epoch and how many do I need?
An epoch is one complete pass through your entire training dataset. During each epoch, every row is fed through the network, the cost function contribution is accumulated, and weights are updated via back propagation. You run multiple epochs to iteratively refine the weights until the cost function is minimised — the exact number depends on your data and how quickly the cost stops decreasing.
What are the two non-negotiable ingredients for deep learning?
Large volumes of data and sufficient processing power. This is why deep learning failed in the 1980s but succeeds now — the technology finally caught up. Before starting any project, audit both ingredients. Deploying deep learning without enough data or compute produces underperforming models, exactly as it did decades ago.
// How To
How do I know if my problem needs a continuous, binary, or categorical output layer?
Look at what you're predicting. A single value like a price is continuous — use one output neuron. A yes/no answer like 'will the customer leave?' is binary — use one neuron with sigmoid or threshold activation. A multi-class label is categorical — use multiple output neurons representing dummy variables. Determine this first, because it sets your output layer activation function before anything else.
How do I set up the cost function for my network?
Start with the standard cost function C = ½(ŷ - y)², the half-squared difference between the predicted value (ŷ) and actual value (y). This is the error signal your network minimises. Other cost functions exist for specific applications, but this is the default starting point. Remember: y is the real value, ŷ is the prediction.
How do I decide how many hidden layers to use?
Add one or more hidden layers between input and output — multiple hidden layers are what make it 'deep' learning. There's no single formula; start with one or two, evaluate performance, and iterate. Each hidden neuron autonomously discovers which input combinations matter. If performance is poor, one lever is adding depth, but also revisit activation functions, standardisation, and training epochs.
How do I interpret what a hidden neuron has learned?
After training, examine the weights each hidden neuron assigned to its inputs. Near-zero weights mean that input is irrelevant to that neuron; non-zero weights reveal the combination it detects. For example, a property model might reveal one neuron that fired only on properties over 100 years old using a rectifier threshold. Expect combinations you never anticipated.
// Troubleshooting
My network trains fine on small data but is impossibly slow on large data. What's wrong?
You're likely using batch gradient descent, which updates weights only after a full pass through all rows. Switch to Stochastic Gradient Descent, which updates weights after each individual row. This dramatically reduces computation per update cycle and prevents the curse of dimensionality from making training intractable at scale.
My network won't converge and the cost function stays high. What should I check?
Check four things in order: are your inputs standardised or normalised to comparable scales? Are you using appropriate activation functions (Rectifier in hidden layers, Sigmoid in output)? Do you have enough hidden layers to detect meaningful patterns? Have you run enough training epochs? Un-standardised inputs are the most common cause of a network that won't train.
Why is my whole network giving strange predictions after I changed one row's weights?
Because there is only one neural network — all rows share the same weights. There is no separate set of weights per row. Updating weights means updating them for the entire network, not for an individual observation. If you're treating each row as having its own weights, you've fundamentally misunderstood how a single network processes one row at a time using shared weights.
My deep learning code throws errors on import. Could it be my Python version?
Yes — check that you're running Python 3.x, not Python 2.7. Modern deep learning libraries require Python 3, and using 2.7 causes import and syntax failures before you even start building. Always verify your Python version before beginning any practical implementation.
// Comparisons
How does gradient descent compare to brute-force optimisation?
Brute force tests every possible weight combination and is computationally impossible — a 25-weight network with 1,000 values each needs 10^75 combinations. Gradient descent instead starts at an arbitrary point on the cost curve, computes the slope, and steps downhill iteratively toward the minimum. It's the only feasible approach at any meaningful scale, converging in a fraction of the operations.
How does the Rectifier function compare to the Sigmoid function?
The Rectifier outputs 0 for negative inputs and increases linearly for positive ones, making it the most popular choice for hidden layers. The Sigmoid produces a smooth S-curve between 0 and 1, ideal for output layers when predicting probabilities. In practice, most networks combine Rectifier in the hidden layers with Sigmoid in the output layer.
How does standardisation compare to normalisation?
Standardisation transforms inputs to mean 0 and variance 1, while normalisation scales values to a 0–1 range by subtracting the minimum and dividing by the range. Both put variables on comparable scales so weighted sums work correctly. Standardisation is the more common default; normalisation is used in specific scenarios. Refer to Yann LeCun's 'Efficient BackProp' (1998) for edge cases.
How does a perceptron compare to a deep neural network?
A perceptron is a single-layer feed-forward network — the simplest architecture, with input neurons connected directly to an output neuron, invented by Frank Rosenblatt in 1957. A deep neural network adds multiple hidden layers between input and output. Those hidden layers manufacture new derived attributes, giving deep networks the ability to detect complex patterns a perceptron never could.
// Advanced
Why do individual neurons have no predictive power on their own?
Like ants that can only build a colony collectively, a single neuron does almost nothing alone. The magic of a neural network emerges from billions of weighted connections working in concert across hidden layers. Predictive power is an emergent property of the connections and their weights, not of any individual node.
How does back propagation actually update the weights?
After a full epoch, the total cost function is calculated by summing ½(ŷ - y)² across all rows. Back propagation then feeds this error signal backward through the network, using the gradient to determine which direction to adjust each weight and by how much. All synapses are updated simultaneously, and the process repeats each epoch until the cost is minimised.
How do hidden layers create attributes a human analyst wouldn't think of?
Each hidden neuron combines input signals to detect a specific pattern, effectively manufacturing a brand-new derived attribute. In a property model, one neuron might combine area and distance-to-city to detect 'large homes far from downtown,' while another combines area, bedrooms, and age. The network discovers these automatically — this feature-engineering capability is deep learning's core advantage over shallow ML.
Why should I design systems assuming exponential progress?
Because storage and compute power grow exponentially, following Moore's Law. What is computationally impossible today may become trivially achievable within a decade. This is exactly why deep learning went from failure in the 1980s to dominance today. When architecting systems, factor in this trajectory rather than assuming today's constraints are permanent.
When does the Hyperbolic Tangent function make sense over Sigmoid?
Use the Hyperbolic Tangent function when negative output values are meaningful in your application. It's similar to Sigmoid but ranges from -1 to +1 instead of 0 to 1. This zero-centred range can be advantageous in certain hidden-layer contexts and specific applications where you want signals that can be both positive and negative.
Can I pre-specify what each hidden neuron should detect?
No — and doing so defeats the entire neural network paradigm. Hidden neurons discover their own feature combinations through training by adjusting weights. If you hard-code what a neuron should detect, you've abandoned the learning facility that makes the network powerful and reverted to hard-coded rules. Your job is to build the architecture and supply labelled data, then let training do the work.