Elektrine lite

← Feed

@nfultz@awful.systems

2026-10-16 21:35 UTC

So, I was thinking about how modern LLMs got better at coding in the last 1 year, despite not having much good new training data. Half the job posts I see on linkedin are from snorkel, handshake and mercor etc to train llms for $50-90/hr. So I think there’s been quite a bit of new training data created as those companies scaled up. So everyone is paying openai to pay mercor to pay contractors to write code and then regurgitate it, rather than just paying for what they want directly. idk. I’m just skeptical that there was actually no human in the loop and that the gains were from unit tests / synthetic TDD.

Replies (1)

  • @diz@awful.systems 2026-10-17 02:35

    I wonder how much code they could have gotten contractors to write in a year or two, though. Got to be only a fraction of total open source code lines. Also, open source comes with existing tests that could be used for some validation. If you actually exclude some project from the bulk pretraining data (or just use code that is after training cutoff) you could e.g. have it replace a function but still pass the test. They might also be somehow using all of their customer’s usage as training data, which would explain why now there’s also a hard disk shortage. I’m thinking RLVR and such got to be the largest, then open source, then RLHF and commissioned training data. They also did a lot of training for decompilation, which makes me think they train on decompiled closed source as well.

    Open ##5073993