Science | AAAS
This new paper is worrying but not surprising. It describes a recent effort to (mis)use computational drug design software (especially as applied to avoid compounds with predicted toxicity), flipping the signs to have the model generate the most toxic molecules it could come up with. This was part of a biannual conference in Switzerland that looks at the implication of new technologies on chemical and biological weapons threats, and the authors (from Collaboration Pharmaceuticals) were invited to present on how AI/ML approaches could be weaponized in these areas. They were clearly shaken by the experience:
"The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting. Our work is rooted in building machine learning models for therapeutic and toxic targets to better assist in the design of new molecules for drug discovery. We have spent decades using computers and AI to improve human health—not to degrade it. We were naive in thinking about the potential misuse of our trade, as our aim had always been to avoid molecular features that could interfere with the many different classes of proteins essential to human life. Even our projects on Ebola and neurotoxins, which could have sparked thoughts about the potential negative implications of our machine learning models, had not set our alarm bells ringing."
The effort used known LD50 data to drive the modeling in the known "nerve gas" space, and it sppears to have worked very well. Relatively speaking, that is:
"In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules."
It would not surprise me if this modeling work uncovered a number of nondisclosed structures that have been quietly looked at in the past. It seems clear that there's a lot more structure-activity space in this area than has ever been declassified (the appearance of the "Novichok" compounds is one example of this). Now, keep in mind that we can't deliberately design our way to drugs so easily, so we won't be able to design horrible compounds in one shot, either. Just as there are considerations in drug discovery that narrow down these sorts of lead-generation efforts, there are such factors for chemical weapons: stability on storage, volatility (or lack of it), persistance in the environment, manufacturing concerns, etc. So it's not like all of these predicted highly toxic compounds would be militarily useful - but note that we're not just talking about state actors here. A big worry has always been homebrew types (as witness Aum Shinrikyo and their 1995 nerve-gas release in the Tokyo subway system). As it happened, Aum was not particularly competent, but that shouldn't make any of us feel better, because someone else might be. That goes for the various garage-ricin incidents over the years as well - ricin is (fortunately) not all that easy to turn into a weapon, and the people who try to do it are generally somewhat disconnected from reality (and also from technical proficiency).
But there's always that worrisome part of the Venn diagram where competence and evil intent overlap. It hit me in the aftermath of 9/11 (during that anthrax period) that I could personally think of a whole list of horrible possibilities for chemical and biological terrorism threats. I'd never thought of such things, but a bit of sustained thought brought all sorts of stuff to mind. That must have been a similar feeling to what the authors of this paper experienced. And while I'm fairly imaginative, I don't think I'm particularly diabolical. What, I wondered, about people who are a bit of both?
As mentioned, I'm not surprised at all that computational methods can point the way to such things, although I can see how the authors of this work found demonstrating this to be an unsettling experience. To be honest, I'm not all that worried about new nerve agents, at least not compared to my level of worry about the existing ones for which there is already a wide base of knowledge. That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us. One factor (mentioned in the paper) is that new compounds and new routes might be able to circumvent chemical controls and watch lists, though. The paper discusses ethics training, international agreements and guidelines, pledges of responsibility and so on. That's all fine, but history demonstrates that anyone truly interested in using such things will care nothing for these constraints. I feel about these the way that I felt about the "No First Use" pledge on nuclear weapons - that it guaranteed that the world could only be blown up by a liar. Thin comfort.
So we should be vigilant about potential misuse of these technologies, but at the same time we shouldn't imagine that this will be enough.
