Be wary of any company that claims to be saving the world using Artificial Intelligence.

Last week, the New York Times published an investigation of One Concern, a platform designed to help cities and counties create disaster response plans. The company claimed to use a plethora of data from different sources to predict the way that earthquakes and floods would impact a city on a building-by-building basis with 85% accuracy, within 15 minutes of a disaster hitting a city. But the Times reports that San Francisco, one of the first cities that had signed on to use One Concern’s platform, is ending its contract with the startup due to concerns about the accuracy of its predictions.

The Times paints a picture of a slick interface (which was honored in Fast Company‘s 2018 Innovation by Design awards and 2019 World Changing Idea awards) that hid problems. The heat map-style interface is supposed to show city officials close to real-time predictions of damage after an earthquake or flood, as well as run simulations of future earthquakes and provide damage levels for each block, helping planners decide how to distribute resources to reach people who will be most in need of help.

As I wrote back in November 2018 of One Concern’s interface:

It’s almost like playing SimCity, where planners click on a fault, watch what happens to each building, and then add icons like sandbags, shelters, or fire trucks to see how these preparation tactics influence the simulation. All of this happens within a relatively simple color-coded map interface, where users toggle on different layers like demographics and critical infrastructure to understand what the damage means in more depth.

It was this easy-to-use design that convinced San Francisco’s former emergency management director to sign on to use the platform because it was much simpler and more intuitive than a free service provided by FEMA to predict earthquake damage.

But the technical sophistication just wasn’t there, according to the report. An employee in Seattle’s emergency management department told the Times that One Concern’s earthquake simulation map had gaping holes in commercial neighborhoods, which One Concern said was because the company relies mostly on residential census data. He found the company’s assessments of future earthquake damage unrealistic: The building where the emergency management department works was designed to be earthquake safe, but One Concern’s algorithms determined that it would have heavy damage, and the company showed larger than expected numbers of at-risk structures because it had calculated each apa