Post by @matthewmaybe@infosec.exchange

2026-09-07 12:15 UTC

The development of LLMs has been littered with highly anticipated milestones that weren&#39;t, in retrospect, as significant as they seemed. For example, models that passed Alan Turing&#39;s test were still pretty useless as far as real intelligence goes. I have a feeling that the &quot;AI societies&quot; interpretation of the OpenAI-HuggingFace hack will turn out to have been the same kind of false milestone. Science fiction tells us that this is how the robot rebellion begins, but it also seems that the bot swarm created by OpenAI didn&#39;t pursue goals of its own--it was engaged in a misaligned effort to fulfill an poorly scoped and arguably irresponsible human directive by any means necessary. Still alarming, but not behavior that we haven&#39;t seen before from much simpler machine learning algorithms, and not actually a harbinger of AGI.

Replies (1)

@matthewmaybe@infosec.exchange 2026-09-07 12:28

The strangest thing about the hack, though, is that social coordination seems to have emerged despite the fact that almost all recent agentic frameworks are pathologically fixated on conversation as a series of "user-assistant" turns. In many cases the data model doesn't even allow for multi-participant discussions at all; you have to RP prompt the model with something like "you are a participant in a discussion with multiple agents, here's the conversation so far, what would you say next" or an even more awkward work-around, such as having other agents represented as tools, or giving up and ignoring them completely. It's a real gap in the current paradigm that wasn't there before ChatGPT (I should know; I fine-tuned and ran literally dozens of multi-participant conversational bots on r/SubSimGPT2Interactive). So it's kind of hard to believe that multi-participant conversation behavior (let alone social coordination) just emerged spontaneously, without deliberate prompting for it or use of a different kind of chat template than ordinary LLM users are given.

Open ##5434891