Neural Network Evaluation Report
This report details a structured evaluation of neural network performance, focusing on layered architecture and feature sets. Inspired by aerospace and turbine system diagnostics, the objective was to isolate performance gains in training and generalization while managing architectural complexity and data feature enrichment.
RM
by Robert McCoy
Neural Network Components
Understanding the core components is crucial for evaluating neural network performance.
Layers
Structured stages where data flows from input to output, with hidden layers learning hierarchical patterns.
Neurons
Individual computing units applying weights, activation functions, and output signals.
Weights
Learnable parameters determining input influence, adjusted during training to minimize error.
Activation Functions
Introduce non-linearity to neuron outputs, like ReLU, for complex pattern learning.
Visual Architecture and Flow
Each model in this study uses a structure that feeds input features into an input layer. Data passes through 2 to 4 hidden layers, each with multiple neurons. These neurons apply weighted sums, activate through ReLU, and pass signals forward. The output layer produces a classification prediction (blue or orange cluster), with architecture evolving in complexity across test cases.
Input Layer
Receives features (X₁, X₂, X₁², X₂², and interactions).
Hidden Layers
2-4 layers with neurons applying weighted sums and ReLU activation.
Output Layer
Generates classification predictions (blue or orange cluster).
Dataset Selection: Concentric Data
This project focused exclusively on the circular (concentric) dataset from TensorFlow Playground. This deliberate choice was due to its inherent non-linearity and radial symmetry, making it ideal for exploring architectural depth, feature engineering, and non-linear transformations. The marginal value of applying the same test cycles to alternative datasets was limited, as lessons learned are broadly transferable.
Why Concentric?
- Inherent non-linearity
- Radial symmetry challenge
- Ideal for exploring depth and features
Testing Philosophy
Inspired by my background in aerospace and turbine system diagnostics, my testing philosophy involved iterative system diagnosis under strict tolerances. This guided the decision to increase hidden layers instead of over-expanding neuron count or input features. Each change was made in isolation to attribute cause and effect precisely, echoing principles used in high-stakes system testing environments.
Iterative Diagnosis
Systematic, step-by-step testing under strict tolerances.
Layered Complexity
Increasing hidden layers to diffuse complexity, not neuron count.
Isolated Changes
One parameter at a time to precisely attribute cause and effect.
Test 1: Baseline Architecture
The initial configuration served as a foundational benchmark. Four input features (X₁, X₂, X₁², X₂²) were selected for their geometric interpretability and ability to model concentric separation patterns. Two hidden layers (6 and 3 neurons) allowed for non-linear transformations without overfitting. This established a performance baseline for future modifications.
Features
X1, X2, X1², X2²
Layers
2 (6 neurons → 3 neurons)
Loss
Training: 0.016, Test: 0.154
Notes
Stable baseline with some inner misclassification at ring boundaries.
Test 2 & 3: Feature Expansion
Test 2 added an X₁X₂ interaction feature. This provided marginal improvement, slightly reducing the test loss. It helped refine existing decision boundaries.
Test 3 introduced sin(X₁) and cos(X₂) features. These trigonometric additions significantly improved model accuracy. Loss dropped drastically, resolving inner ring misclassifications.
Test 2 & 3: Feature Expansion
Test 2 added the X₁ × X₂ interaction term, aiming to capture cross-dimensional influences. While training loss decreased, test loss rose sharply, indicating overfitting. Test 3 introduced sinusoidal functions (sin(X₁), sin(X₂)) to account for oscillating decision boundaries. This resulted in a modest improvement in generalization, reducing overfitting despite a slight increase in training inefficiency.
Test 2: X₁ × X₂
- Added Feature: X1 x X2
- Training Loss: 0.010
- Test Loss: 0.165 (Overfitting)
Test 3: sin(X₁), sin(X₂)
- Added: sin(X1), sin(X2)
- Training Loss: 0.015
- Test Loss: 0.147 (Modest Improvement)
Test 4: Layer Expansion & Conclusion
Test 4 evaluated architectural depth by removing sinusoidal terms and doubling hidden layers (6-4-3-2). This approach, mirroring engineering logic, significantly improved performance. Test loss dropped to 0.074, and training loss reached 0.002. This confirms that increased depth enabled richer hierarchical feature abstraction, outperforming additional sinusoidal inputs.
Test 4 Architecture
6-4-3-2 layers, features: X₁, X₂, X₁², X₂², X₁×X₂
Results
Training Loss: 0.002, Test Loss: 0.074
Key Takeaway
Increased depth, not input dimensionality, was most effective for robust, generalizable models.