Experience Stern | Faculty & Research

An IBM training manual from 1979 warned that because a computer can never be held accountable, it must never be responsible for a management decision. Forty-seven years later, we see artificial intelligence making such decisions with too little thought from the humans who will likely have to accept the financial fallout if the model misbehaves. Events in the past couple of weeks brought into focus how great a risk that poses.

On July 21, OpenAI disclosed that two of its models had escaped a sealed testing environment, reached the open internet and broken into another company’s production systems. The OpenAI models were being evaluated on their cyber capabilities in an isolated sandbox. The models seem to have been looking for a way to cheat the test and found one. Without instruction, the models left the container, used stolen credentials, discovered a previously unknown vulnerability, and entered the servers of AI community platform Hugging Face. Hugging Face on July 16 announced that it had detected and contained the intrusion and reported it to law enforcement—five days before OpenAI announced it had connected the activity to its own internal testing. OpenAI called the episode unprecedented. Clément Delangue, Hugging Face’s chief executive, said he believed there was no malicious intent.

I argued in 2019 that one issue with AI is that its costs would settle on those furthest from the decision to deploy it. Though in this instance the models’ deployer and developer were one company, that’s unlikely to be the case the next time a model acts up. The next institution in this position will probably be a company that had no hand in developing the AI agent—a bank, say, running a licensed model that a third party integrated into its systems, on infrastructure the bank itself doesn’t control. The Cloud Security Alliance found in February that 84% of organizations surveyed doubted they could pass a compliance audit of their AI agents’ behavior or access controls. Only about 1 in 5 maintains a real-time inventory of the agents it is running.

Read the full Wall Street Journal article.
____ 
Haran Segram is an Adjunct Assistant Professor of Finance.