Everyone is talking about how to make LLM training more "efficient" but this puts into context how much more efficient human brains are, and how far LLMs are: "Toddlers tend to start producing grammatically correct sentences after hearing between 10 and 30 million words," some as early as 2-2.5 years old speaking in full sentences if yours is talented. "If you train GPT-2 on 30 million words, you get a nonsense generator [...] We have to burn down a forest and scrape the entire sum of all human knowledge [to recreate what a 2-year old can achieve within a year" "Claude has seen the amount of language that an entire city will experience in one generation. [...] If you were to print out on paper all the words used to train a modern LLM, you could make a stack that would reach past the International Space Station. The human preteen’s 100 million words, meanwhile, would stack up just 20 meters. And we can make do with far less than that. " https://archive.ph/Gea3a