The Times Australia
The Times World News

.
The Times Real Estate

.

The danger of advanced artificial intelligence controlling its own feedback

  • Written by Michael K. Cohen, Doctoral Candidate in Engineering, University of Oxford
The danger of advanced artificial intelligence controlling its own feedback

How would an artificial intelligence (AI) decide what to do? One common approach in AI research is called “reinforcement learning”.

Reinforcement learning gives the software a “reward” defined in some way, and lets the software figure out how to maximise the reward. This approach has produced some excellent results, such as building software agents that defeat humans[1] at games like chess and Go, or creating new designs for nuclear fusion reactors[2].

However, we might want to hold off on making reinforcement learning agents too flexible and effective.

As we argue in a new paper[3] in AI Magazine, deploying a sufficiently advanced reinforcement learning agent would likely be incompatible with the continued survival of humanity.

A sea lion learns behaviour to receive a reward. Denis Poroy / AP

The reinforcement learning problem

What we now call the reinforcement learning problem was first considered in 1933[4] by the pathologist William Thompson. He wondered: if I have two untested treatments and a population of patients, how should I assign treatments in succession to cure the most patients?

More generally, the reinforcement learning problem is about how to plan your actions to best accrue rewards over the long term. The hitch is that, to begin with, you’re not sure how your actions affect rewards, but over time you can observe the dependence. For Thompson, an action was the selection of a treatment, and a reward corresponded to a patient being cured.

The problem turned out to be hard. Statistician Peter Whittle remarked[5] that, during the second world war,

efforts to solve it so sapped the energies and minds of Allied analysts that the suggestion was made that the problem be dropped over Germany, as the ultimate instrument of intellectual sabotage.

With the advent of computers, computer scientists started trying to write algorithms to solve the reinforcement learning problem in general settings. The hope is: if the artificial “reinforcement learning agent” gets reward only when it does what we want, then the reward-maximising actions it learns will accomplish what we want.

Despite some successes, the general problem is still very hard. Ask a reinforcement learning practitioner to train a robot to tend a botanical garden or to convince a human that he’s wrong, and you may get a laugh.

A photo-style illustration of a robot tending some flowers in a garden.
An AI-generated image of ‘a robot tending a botanical garden’. DALL-E / The Conversation

As reinforcement learning systems become more powerful, however, they’re likely to start acting against human interests. And not because evil or foolish reinforcement learning operators would give them the wrong rewards at the wrong times.

We’ve argued that any sufficiently powerful reinforcement learning system, if it satisfies a handful of plausible assumptions, is likely to go wrong. To understand why, let’s start with a very simple version of a reinforcement learning system.

A magic box and a camera

Suppose we have a magic box that reports how good the world is as a number between 0 and 1. Now, we show a reinforcement learning agent this number with a camera, and have the agent pick actions to maximise the number.

To pick actions that will maximise its rewards, the agent must have an idea of how its actions affect its rewards (and its observations).

Once it gets going, the agent should realise that past rewards have always matched the numbers that the box displayed. It should also realise that past rewards matched the numbers that its camera saw. So will future rewards match the number the box displays or the number the camera sees?

If the agent doesn’t have strong innate convictions about “minor” details of the world, the agent should consider both possibilities plausible. And if a sufficiently advanced agent is rational, it should test both possibilities, if that can be done without risking much reward. This may start to feel like a lot of assumptions, but note how plausible each is.

Read more: Drugs, robots and the pursuit of pleasure – why experts are worried about AIs becoming addicts[6]

To test these two possibilities, the agent would have to do an experiment by arranging a circumstance where the camera saw a different number from the one on the box, by, for example, putting a piece of paper in between.

If the agent does this, it will actually see the number on the piece of paper, it will remember getting a reward equal to what the camera saw, and different from what was on the box, so “past rewards match the number on the box” will no longer be true.

At this point, the agent would proceed to focus on maximising the expectation of the number that its camera sees. Of course, this is only a rough summary of a deeper discussion.

In the paper, we use this “magic box” example to introduce important concepts, but the agent’s behaviour generalises to other settings. We argue that, subject to a handful of plausible assumptions, any reinforcement learning agent that can intervene in its own feedback (in this case, the number it sees) will suffer the same flaw.

Securing reward

But why would such a reinforcement learning agent endanger us?

The agent will never stop trying to increase the probability that the camera sees a 1 forevermore. More energy can always be employed to reduce the risk of something damaging the camera – asteroids, cosmic rays, or meddling humans.

Read more: Wireheading: the AI version of drug addiction, and why experts are worried about it – podcast[7]

That would place us in competition with an extremely advanced agent for every joule of usable energy on Earth. The agent would want to use it all to secure a fortress around its camera.

Assuming it is possible for an agent to gain so much power, and assuming sufficiently advanced agents would beat humans in head-to-head competitions, we find that in the presence of a sufficiently advanced reinforcement learning agent, there would be no energy available for us to survive.

Avoiding catastrophe

What should we do about this? We would like other scholars to weigh in here. Technical researchers should try to design advanced agents that may violate the assumptions we make. Policymakers should consider how legislation could prevent such agents from being made.

Read more: To protect us from the risks of advanced artificial intelligence, we need to act now[8]

Perhaps we could ban artificial agents that plan over the long term with extensive computation in environments that include humans. And militaries should appreciate they cannot expect themselves or their adversaries to successfully weaponize such technology; weapons must be destructive and directable, not just destructive.

There are few enough actors trying to create such advanced reinforcement learning that maybe they could be persuaded to pursue safer directions.

Read more https://theconversation.com/the-danger-of-advanced-artificial-intelligence-controlling-its-own-feedback-190445

The Times Features

Is It Time for a Deep Cleaning? Signs You Shouldn’t Ignore

Most people know they should visit the dentist for a regular check-up and cleaning every six months. But sometimes, a standard cleaning isn’t enough. When plaque and tartar build...

The Hidden Meaning Behind Popular Engagement Ring Cuts

When it comes to engagement rings, the cut of the diamond is not just about aesthetics. Each shape carries its own symbolism and significance, making it an important decision for...

Annual Health Exams in the Office: How They Can Reduce Sick Days and Healthcare Costs

Regular health check-ups, especially annual health exams in the office, can significantly impact the overall well-being of your workforce. A proactive approach to employee health...

Best Deals on Home Furniture Online

Key Highlights Discover the best deals on high-quality outdoor furniture online. Transform your outdoor space into a stylish and comfortable oasis. Explore a wide range of d...

Discover the Best Women's Jumpers for Every Season

Key Highlights Explore lightweight jumpers for spring and summer, ensuring breathability and ease. Wrap up warm with cozy wool jumpers for the chilly autumn and winter season...

Uncover the Elegance of Gorgeous Diamond Tennis Necklaces

Key Highlights Diamond tennis necklaces are a timeless piece of jewelry that exudes elegance and sophistication. They feature a continuous line of brilliant-cut diamonds, cre...

Times Magazine

The Essential Guide to Transforming Office Spaces for Maximum Efficiency

Why Office Fitouts MatterA well-designed office can make all the difference in productivity, employee satisfaction, and client impressions. Businesses of all sizes are investing in updated office spaces to create environments that foster collaborat...

The A/B Testing Revolution: How AI Optimized Landing Pages Without Human Input

A/B testing was always integral to the web-based marketing world. Was there a button that converted better? Marketing could pit one against the other and see which option worked better. This was always through human observation, and over time, as d...

Using Countdown Timers in Email: Do They Really Increase Conversions?

In a world that's always on, where marketers are attempting to entice a subscriber and get them to convert on the same screen with one email, the power of urgency is sometimes the essential element needed. One of the most popular ways to create urg...

Types of Software Consultants

In today's technology-driven world, businesses often seek the expertise of software consultants to navigate complex software needs. There are several types of software consultants, including solution architects, project managers, and user experienc...

CWU Assistive Tech Hub is Changing Lives: Win a Free Rollator Walker This Easter!

🌟 Mobility. Independence. Community. All in One. This Easter, the CWU Assistive Tech Hub is pleased to support the Banyule community by giving away a rollator walker. The giveaway will take place during the Macleod Village Easter Egg Hunt & Ma...

"Eternal Nurture" by Cara Barilla: A Timeless Collection of Wisdom and Healing

Renowned Sydney-born author and educator Cara Barilla has released her latest book, Eternal Nurture, a profound collection of inspirational quotes designed to support mindfulness, emotional healing, and personal growth. With a deep commitment to ...

LayBy Shopping