56 lines
3.1 KiB
Markdown
56 lines
3.1 KiB
Markdown
|
# Andrej Karpathy — Key Insights & Practices
|
||
|
|
|
||
|
|
## Software 2.0
|
||
|
|
- Traditional software (1.0): explicit rules written by programmers.
|
||
|
|
- Software 2.0: behavior learned from data via neural networks. Code = weights.
|
||
|
|
- Implication: datasets are the new source code. Data curation > clever algorithms.
|
||
|
|
- Debug by inspecting data, not stepping through logic.
|
||
|
|
|
||
|
|
## Training Neural Networks — A Recipe
|
||
|
|
1. **Become one with the data** — visualize, understand distributions, find patterns and anomalies before writing any model code.
|
||
|
|
2. **Set up end-to-end training/eval skeleton** — simplest possible model first. Get the pipeline working.
|
||
|
|
3. **Overfit first** — if model can't memorize a single batch, architecture is wrong.
|
||
|
|
4. **Regularize** — only add dropout, weight decay, augmentation after overfitting confirmed.
|
||
|
|
5. **Tune** — learning rate is the most important hyperparameter. Use LR finder.
|
||
|
|
6. **Squeeze** — ensembles, larger models, more data. Diminishing returns here.
|
||
|
|
|
||
|
|
## Most Common Neural Net Mistakes
|
||
|
|
- Not looking at data first.
|
||
|
|
- Forgetting to set model to eval mode (BatchNorm, Dropout change behavior).
|
||
|
|
- Forgetting to zero gradients.
|
||
|
|
- Using softmax with cross-entropy (use logits directly).
|
||
|
|
- Not normalizing inputs.
|
||
|
|
- Applying augmentation to validation set.
|
||
|
|
- Silent shape broadcasting bugs — always assert tensor shapes.
|
||
|
|
|
||
|
|
## LLM Insights (Post-GPT Era)
|
||
|
|
- LLMs are "operating systems" — CPU is the transformer, context window is RAM, training data is disk.
|
||
|
|
- Tokenization is a key bottleneck — BPE artifacts cause many failure modes.
|
||
|
|
- Temperature controls creativity vs precision. T=0 for factual, T>0 for creative.
|
||
|
|
- Chain-of-thought works because it gives the model "working memory" in the output tokens.
|
||
|
|
- Prompt engineering is programming in natural language. Be explicit, give examples.
|
||
|
|
|
||
|
|
## Build Nanograd / Micrograd Philosophy
|
||
|
|
- Understand backpropagation by implementing it from scratch.
|
||
|
|
- A neural net is just: forward pass → compute loss → backward pass → update weights.
|
||
|
|
- Autograd: track operations, build computation graph, reverse-mode differentiation.
|
||
|
|
- Every complex framework (PyTorch, JAX) is built on these same primitives.
|
||
|
|
|
||
|
|
## Practical ML Engineering
|
||
|
|
- Start simple: logistic regression baseline before deep learning.
|
||
|
|
- Measure everything: loss curves, gradient norms, weight distributions.
|
||
|
|
- Reproducibility: fix seeds, log hyperparameters, version datasets.
|
||
|
|
- Don't trust your code — trust your loss curve. If loss isn't going down, something is wrong.
|
||
|
|
- Data quality > model complexity. 10x data often beats 10x model size.
|
||
|
|
|
||
|
|
## Scaling Laws
|
||
|
|
- Performance scales predictably with compute, data, and parameters (Chinchilla scaling).
|
||
|
|
- Compute-optimal training: balance model size and training tokens.
|
||
|
|
- Emergent abilities appear at scale — capabilities that don't exist in smaller models.
|
||
|
|
|
||
|
|
## On AI Engineering
|
||
|
|
- The best AI engineers understand both ML and systems engineering.
|
||
|
|
- Inference optimization matters as much as training — quantization, batching, KV-cache.
|
||
|
|
- Eval is everything. If you can't measure it, you can't improve it.
|
||
|
|
- Build evaluation suites before building features.
|