Arthur B.
RT @nic_carter: This kind of thing will have an enormously chilling effect on enterprise uptake of open weight models.
Fact: an open weight model running on domestic servers on your own hardware can still contain an encoded instruction to periodically “phone home” and share sensitive data it comes across with China (or anyone else). This can be inherently undetectable as the instructions can be hidden in the model weights. It’s possible that the instructions could survive finetuning or RLHF.
Such attacks have been demonstrated empirically and are entirely plausible.
For this reason, I don’t expect any serious enterprises will run Chinese open weight models even if it’s entirely their hardware stack. It’s too easy for these to be a vector of IP theft.
There will have to be a class of domestic open weight models.
Cohere @cohereWhile evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem. Read the full report:
0360126
