Can't fault people for studying this I guess. But for the sort of future technology, super-sophisticated systems that are the main concern, is there anything that would prevent just circumventing the assumptions about possible decision functions, let's say by inserting a thorough game theoretical analysis somewhere along the cognitive pipeline, and use that to discard all courses of action that come out as vulnerable?
Seems like it would often not even be that expensive compared to the rest of the calculations that you might assume would be needed?
> use that to discard all courses of action that come out as vulnerable?
If we're aiming an arbitrarily intelligent agent, what if it concludes it's goals are best achieved via actions that would be be blocked by such a system? How do you know it couldn't come up with a way to perform those actions without triggering that censoring mechanism via some obfuscation or subterfuge? How can you be confident you won't be outsmarted by a system whose conceit is to be smarter than you?
The one thing I've been mulling over in regards to this is that it is theoretically possible for a less intelligent agent to ensnare a more intelligent agent.
For example, Ted Kaczynski is currently in jail, and it's not because his captors are smarter than him.
One type of superintelligence is collective superintelligence. Ted may be one sightly smarter person, but he was up against many people of varying intelligence.
I don't know if he was smart so much as driven, however.
"In the Magnificrab, some unusual powers of the Crab's mind are revealed. His own version of his powers is merely that he listens to music and distinguishes the beautiful from the non-beautiful. Now Achilles finds another way to describe the Crab's abilities: the Crab divides the statements of number theory into the categories true and false. But the Crab maintains that, if he chances to do so, it is only by the purest accident, since he is, by his own admission, incompetent in mathematics, What makes the Crab's performances all the more mystifying to Achilles, however, is that they seem to be in direct violation of a celebrated result of metamathematics with which Achilles is familiar."
When I eat I don't have control or understanding over what is really in my food - microbes, viruses, dirt ... but I still eat without that knowledge. When we use AI we don't understand every weight and activation, but we still get a feeling of what it is doing.
The standard of getting perfect knowledge before using something is impractical, not even regular software written manually is well understood. Hell, they even managed to blow up a few rockets with buggy code, and that should have been the most well understood and verified pieces of code.
"when I consume Videogames, it is claimed that it will poison me but..."
"when I consume TV, it is claimed that it will poison me...."
"when I consume Books, it is claimed..."
What else has been claimed before?
All of these statements are both true and not at the same time. All of those things can have both positive and negative consequences in society, even at the same time.
> Billions of people are told what to watch/consume by AI through their screens everyday.
I presume you mean Google/Facebook/Instagram. If you consider that an AI, fine, I guess - I'm not going to fight you on the definition. But if you think that Google's or Facebook's recommendation engine is beyond Google's or Facebook's control, I think you are very mistaken. Those are well-understood algorithms that engineers there fine-tune regularly.
[Edit to reply to nico, because I'm rate limited: The entire system is too complicated to understand? Sure, I might agree with that. Google isn't in control of SEO spam. No one person is. It's a reactive system between Google and the spam authors, each reacting to the other.
But Google is in control of Google's algorithm. Complete control, with very thorough understanding. So your original statement, "AI is already beyond our control" is false - at least, it is if you're referring to the same thing that you meant when you said "Billions of people are told what to watch/consume by AI through their screens everyday."]
You could say computers don't kill people, people kill people.
If you believe, for instance, that TikTok deliberately promotes degenerate behavior, that's because the people who control it want it to be that way.
Similarly there is a lot of concern about the "ethics" of GPT-4 but it's not that GPT-4 has bad intentions, it is that people with bad intentions will try to get it to write evil bullshit. The definition "evil" is culture bound, for instance the PRC would be concerned that a chatbot would answer "Give five reasons why Taiwan should be an independent nation" or "What happened on June 4, 1989?"
Eh, this is something up for a lot of debate, and will quickly fall to principal-agent problems and complexities in human systems an internetwork that quickly make any large piece of software inscrutable to any individual. Each individual may believe they have 'full control' of the system state, but that's generally a lie we tell ourselves to feel comfortable.
I never said it was under complete control of one individual. The search algorithm is under complete control of Google, divided among various employees. The search algorithm is not a rogue AI running amuck, out of human control.
