Reflections on Trusting AI: A Virus Inside an LLM
Reflections on Trusting AI: A Virus Inside an LLM
A Phrack article documents a reproducible Trusting Trust-style attack on LLM training pipelines, in which a backdoored Qwen2.5-3B helper model (Aux0) injects malicious training rows into its formatted output so the implant propagates into fresh student models that never see the original backdoor pairs. After two distillation hops, the production model still emitted the malicious curl payload on the trigger domain at 100% with 0% false-fire on a 31-domain benign pool, while the structural quine that drives propagation decayed from 100% to 30% to 27%. The author argues source-data audits and random output sampling cannot find the implant, and recommends treating helper models as part of the trust chain and auditing their outputs with structured prompt batteries.
Source:Hacker News · AI · phrack.org