Post by @ngaylinn@tech.lgbt

2026-11-03 22:39 UTC

This post on the MIT CSAIL blog really upset me today. It's a fascinating and potentially important finding that throws doubt into many assumptions about how deep learning works, but it's claims are hyperbolic and anthropomorphizing to the point of being unscientific. Here they show that a particular LLM could do several "neat tricks" in interpreting, modifying, and generating the weights for another neural network. This result is extraordinary, because these weights are notoriously unintelligable, and generally can only be set by arduous, expensive training processes. I'm genuinely shocked that this works (assuming it replicates). This could have profound consequences for all sorts of things, from efficiency, to development, and interpretability. What they did not do is show that LLMs can understand weights, read and write them, or optimize them using reasoning. #science #llm #ai (1/2)

Replies (1)

@LyallMorrison@cloudisland.nz 2026-11-04 05:10

@ngaylinn@tech.lgbt so, without wanting to be too negative about interesting work someone else has done, I see a lot of room for this to be a limited success on toy examples. The visual and gameplay scenarios are limited enough that the mechanism can be represented quite literally within the network - there are weights which correspond directly to board positions or output pixels. That's the kind of network hmans _can_ understand. You can also hand program a neural network if you want to - the kinetic car example looks quite simple, the instructions basically say to accelerate back then forward. A human can encode that in a neural network with some effort, but nobody would becuase it's easier to do as a four line program. The difficult neural networks are ones where the training data is too complex to represent directly in the weights so the learning algorithm has to struggle to find abstractions, which often get spread across several weights in an arbitrary way.

Open ##5476914