It is also an illustration of a problem with AI that trusts it environment and how such trust could have really bad consequences if said AI is allowed to anything important.
People try to social engineer people all the time, and it works. The consequences are somewhat limited because you are limited to breadth (mass media, phishing etc.) or depth (one on one interactions), but can still be scary.
If an AI isn't more resistant, we face the risk that any re-purposing of data from one AI in other copies, or allowing the AI to massively multitask makes social-engineering of AI far more wide-reaching, and hence far more worthwhile for attackers.
It'll be interesting to see what kinds of attacks will get directed at e.g. customer service bots in the future, and what brands ends up damaged as a result.
People try to social engineer people all the time, and it works. The consequences are somewhat limited because you are limited to breadth (mass media, phishing etc.) or depth (one on one interactions), but can still be scary.
If an AI isn't more resistant, we face the risk that any re-purposing of data from one AI in other copies, or allowing the AI to massively multitask makes social-engineering of AI far more wide-reaching, and hence far more worthwhile for attackers.
It'll be interesting to see what kinds of attacks will get directed at e.g. customer service bots in the future, and what brands ends up damaged as a result.