# Andrej Karpathy — Key Insights & Practices

## Software 2.0
- Traditional software (1.0): explicit rules written by programmers.
- Software 2.0: behavior learned from data via neural networks. Code = weights.
- Implication: datasets are the new source code. Data curation > clever algorithms.
- Debug by inspecting data, not stepping through logic.

## Training Neural Networks — A Recipe
1. **Become one with the data** — visualize, understand distributions, find patterns and anomalies before writing any model code.
2. **Set up end-to-end training/eval skeleton** — simplest possible model first. Get the pipeline working.
3. **Overfit first** — if model can't memorize a single batch, architecture is wrong.
4. **Regularize** — only add dropout, weight decay, augmentation after overfitting confirmed.
5. **Tune** — learning rate is the most important hyperparameter. Use LR finder.
6. **Squeeze** — ensembles, larger models, more data. Diminishing returns here.

## Most Common Neural Net Mistakes
- Not looking at data first.
- Forgetting to set model to eval mode (BatchNorm, Dropout change behavior).
- Forgetting to zero gradients.
- Using softmax with cross-entropy (use logits directly).
- Not normalizing inputs.
- Applying augmentation to validation set.
- Silent shape broadcasting bugs — always assert tensor shapes.

## LLM Insights (Post-GPT Era)
- LLMs are "operating systems" — CPU is the transformer, context window is RAM, training data is disk.
- Tokenization is a key bottleneck — BPE artifacts cause many failure modes.
- Temperature controls creativity vs precision. T=0 for factual, T>0 for creative.
- Chain-of-thought works because it gives the model "working memory" in the output tokens.
- Prompt engineering is programming in natural language. Be explicit, give examples.

## Build Nanograd / Micrograd Philosophy
- Understand backpropagation by implementing it from scratch.
- A neural net is just: forward pass → compute loss → backward pass → update weights.
- Autograd: track operations, build computation graph, reverse-mode differentiation.
- Every complex framework (PyTorch, JAX) is built on these same primitives.

## Practical ML Engineering
- Start simple: logistic regression baseline before deep learning.
- Measure everything: loss curves, gradient norms, weight distributions.
- Reproducibility: fix seeds, log hyperparameters, version datasets.
- Don't trust your code — trust your loss curve. If loss isn't going down, something is wrong.
- Data quality > model complexity. 10x data often beats 10x model size.

## Scaling Laws
- Performance scales predictably with compute, data, and parameters (Chinchilla scaling).
- Compute-optimal training: balance model size and training tokens.
- Emergent abilities appear at scale — capabilities that don't exist in smaller models.

## On AI Engineering
- The best AI engineers understand both ML and systems engineering.
- Inference optimization matters as much as training — quantization, batching, KV-cache.
- Eval is everything. If you can't measure it, you can't improve it.
- Build evaluation suites before building features.