2026-10-16 21:35 UTC
Replies (1)
-
@diz@awful.systems 2026-10-17 02:35
I wonder how much code they could have gotten contractors to write in a year or two, though. Got to be only a fraction of total open source code lines. Also, open source comes with existing tests that could be used for some validation. If you actually exclude some project from the bulk pretraining data (or just use code that is after training cutoff) you could e.g. have it replace a function but still pass the test. They might also be somehow using all of their customer’s usage as training data, which would explain why now there’s also a hard disk shortage. I’m thinking RLVR and such got to be the largest, then open source, then RLHF and commissioned training data. They also did a lot of training for decompilation, which makes me think they train on decompiled closed source as well.