poimen-memory/knowledge/andrej-karpathy.md

Andrej Karpathy — Key Insights & Practices

Software 2.0

• Traditional software (1.0): explicit rules written by programmers.
• Software 2.0: behavior learned from data via neural networks. Code = weights.
• Implication: datasets are the new source code. Data curation > clever algorithms.
• Debug by inspecting data, not stepping through logic.

Training Neural Networks — A Recipe

1. Become one with the data — visualize, understand distributions, find patterns and anomalies before writing any model code.
2. Set up end-to-end training/eval skeleton — simplest possible model first. Get the pipeline working.
3. Overfit first — if model can't memorize a single batch, architecture is wrong.
4. Regularize — only add dropout, weight decay, augmentation after overfitting confirmed.
5. Tune — learning rate is the most important hyperparameter. Use LR finder.
6. Squeeze — ensembles, larger models, more data. Diminishing returns here.

Most Common Neural Net Mistakes

• Not looking at data first.
• Forgetting to set model to eval mode (BatchNorm, Dropout change behavior).
• Forgetting to zero gradients.
• Using softmax with cross-entropy (use logits directly).
• Not normalizing inputs.
• Applying augmentation to validation set.
• Silent shape broadcasting bugs — always assert tensor shapes.

LLM Insights (Post-GPT Era)

• LLMs are "operating systems" — CPU is the transformer, context window is RAM, training data is disk.
• Tokenization is a key bottleneck — BPE artifacts cause many failure modes.
• Temperature controls creativity vs precision. T=0 for factual, T>0 for creative.
• Chain-of-thought works because it gives the model "working memory" in the output tokens.
• Prompt engineering is programming in natural language. Be explicit, give examples.

Build Nanograd / Micrograd Philosophy

• Understand backpropagation by implementing it from scratch.
• A neural net is just: forward pass → compute loss → backward pass → update weights.
• Autograd: track operations, build computation graph, reverse-mode differentiation.
• Every complex framework (PyTorch, JAX) is built on these same primitives.

Practical ML Engineering

• Start simple: logistic regression baseline before deep learning.
• Measure everything: loss curves, gradient norms, weight distributions.
• Reproducibility: fix seeds, log hyperparameters, version datasets.
• Don't trust your code — trust your loss curve. If loss isn't going down, something is wrong.
• Data quality > model complexity. 10x data often beats 10x model size.

Scaling Laws

• Performance scales predictably with compute, data, and parameters (Chinchilla scaling).
• Compute-optimal training: balance model size and training tokens.
• Emergent abilities appear at scale — capabilities that don't exist in smaller models.

On AI Engineering

• The best AI engineers understand both ML and systems engineering.
• Inference optimization matters as much as training — quantization, batching, KV-cache.
• Eval is everything. If you can't measure it, you can't improve it.
• Build evaluation suites before building features.