Notes: Generative AI Landscape
The Compute (GPU) later
https://epochai.org/blog/predicting-gpu-performance
https://epochai.org/blog/trends-in-gpu-price-performance
The Data Layer
https://epochai.org/blog/will-we-run-out-of-ml-data-evidence-from-projecting-dataset
The Algorithmic Progress / LLM Layer
https://epochai.org/blog/projecting-compute-trends
- Bearish: slow compute cost-performance improvements and very little specialized hardware improvements. In this scenario, it takes 12 years for the cost of computation to fall by an OOM. The current 6-month doubling period can be maintained for another ~8 years.
- Middle of the road: Moderate compute cost-performance improvements and moderate improvements in specialized computing. In this scenario, it takes roughly 7 years for the cost of computation to fall by an OOM, and progress in specialized hardware helps sustain the trend ~3 additional years. The current 6-month doubling period can be maintained for another ~12 years.
- Bullish: Fast compute cost-performance improvements and substantial improvements in specialized computing. In this scenario, it takes 4 years for the cost of computation to fall by an OOM, and progress in specialized hardware helps sustain the trend ~6 additional years. The current 6-month doubling period can be maintained for another ~18 years.
https://colab.research.google.com/drive/1FasOOiA-oh7nCkd0cEtuqbA3DSCAeDI-?usp=sharing
https://arxiv.org/abs/2202.05924
https://epochai.org/blog/grokking-bioanchors
The API/ML System Layer
https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems
The Product Layer
https://www.lesswrong.com/posts/aqTAd7KzsYmHWYdei/why-copilot-accelerates-timelines Jared Kaplan | Scaling Laws and Their Implications for Coding AI