Talk:Instrumental convergence

	This article is within the scope of WikiProject Futures studies, a collaborative effort to improve the coverage of Futures studies on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.Futures studiesWikipedia:WikiProject Futures studiesTemplate:WikiProject Futures studiesfutures studies articles
Low	This article has been rated as Low-importance on the importance scale.

Computer science Low‑importance

This article is within the scope of WikiProject Computer science, a collaborative effort to improve the coverage of Computer science related articles on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.Computer scienceWikipedia:WikiProject Computer scienceTemplate:WikiProject Computer scienceComputer science articles

Low

This article has been rated as Low-importance on the project's importance scale.

An editor has requested that an image or photograph be added to this article.

Things you can help WikiProject Computer science with:

Here are some tasks awaiting attention:

Article requests :
- Requested articles/Applied arts and sciences/Computer science, computing, and Internet
Cleanup :
- Computer science articles needing attention
- Computer science articles needing expert attention
Copyedit :
- Computing
Expand :
- Computer science
Infobox :
- Computer science articles without infoboxes
Maintain :
- Timeline of computing 2020–present
Photo :
- Find pictures for the biographies of computer scientists (see List of computer scientists)
- Computing articles needing images
Stubs :
- Computer science stubs
Unreferenced :
- WikiProject Computer science/Unreferenced BLPs
Project-related :
- Tag all relevant articles in Category:Computer science and sub-categories with {{WikiProject Computer science}}

Merge proposal[edit]

I think that Paperclip maximizer should be folded into this article, as the other article is stub-length, and (AFAICT) the paperclip maximizer is a thought experiment intended to illustrate the idea of instrumental convergence. Spectra239 (talk) 10:18, 30 January 2016 (UTC)[reply]

Sounds good to me. Rolf H Nelson (talk) 21:39, 30 January 2016 (UTC)[reply]

I support. Gap9551 (talk) 02:13, 31 January 2016 (UTC)[reply]

Support - I originally created the Paperclip maximizer page, but agree that it fits better with the context provided here. Paperclip Maximiser (talk) 08:25, 6 February 2016 (UTC)[reply]

Done. Rolf H Nelson (talk) 04:05, 26 February 2016 (UTC)[reply]

Counteracting[edit]

I am missing the section on how to protect humanity from the paperclip problem. Zezen (talk) 09:01, 4 November 2017 (UTC)[reply]

Well, if you find it, you'll probably get statues put up to you for the next trillion years as the one who delivered the cosmic endowment to mankind. So good luck! 103.20.170.81 (talk) 10:01, 21 November 2017 (UTC)[reply]

@Zezen Added a pointer in a new 'impacts' section. Rolf H Nelson (talk) 19:06, 25 November 2017 (UTC)[reply]

Convergence kind of a good thing?[edit]

I deleted the second paragraph:

"Instrumental convergence suggests that an intelligent agent with apparently harmless goals can act in surprisingly harmful ways. For example, a computer with the sole goal of solving the Riemann hypothesis could attempt to turn the entire Earth into computronium in an effort to increase its computing power so that it can succeed in its calculations."

IMHO, the paragraph confuses association with causation. The supposed tendency of strong AIs for instrumental convergence does not make them more harmful. If anything, it allows us to better predict their behavior. Should they diverge in every which way depending on goals, they would be far more dangerous. — Preceding unsigned comment added by 176.11.124.251 (talk) 14:45, 2 February 2018 (UTC)[reply]

Instrumental convergence, the hypothesized tendency for intelligent agents to want to seek various generically-useful instrumental capabilities, is currently framed in the literature as a bad thing. If you have a strong source indicating the opposite, that should be added. Rolf H Nelson (talk) 20:32, 3 February 2018 (UTC)[reply]

I think the way YouTube's recommender system drives viewers to more and more extreme content is a good example of a rudimentary paperclip maximizer. The aim is to maximize money by showing ads with content people want to see, Google's "don't be evil" is not incorporated into the programming and it is difficult to do so. Dmcq (talk) 11:11, 12 September 2018 (UTC)[reply]

Paperclip maximizer author[edit]

The article mentions Nick Bostrom as the author of the idea of the paperclip maximizer, however Eliezer Yudkowsky seems to consider himself the author of that idea (tweet). The article does not seem to have a citation for the source. A citation would not solve the problem unless one of the two has cited the other one at some point, to establish the direction. What do we know about the origin of this idea?

trylks (talk) 23:19, 16 November 2021 (UTC)[reply]

Eliezer Yudkowsky said a bit more on Sam Harris's podcast (transcript): "As far as I know, it’s me. It could still be Bostrom. I asked somebody, “Do you remember who it was?” and they searched through the archives of the mailing list where this idea plausibly originated and if it originated there, then I was the first one to say “paperclips.”"

I haven't come across any quotes by Nick Bostrom on this topic. --Steve (talk) 02:08, 18 November 2021 (UTC)[reply]

some more tweets by eliezer yudkowsky claiming to have originated the meme. this matches my memory; unfortunately, i do not have a copy of whatever mailing list it might have been to verify the claim. 135.180.194.177 (talk) 22:47, 9 November 2022 (UTC)[reply]

https://twitter.com/ESYudkowsky/status/1578459400389345280

https://twitter.com/ESYudkowsky/status/1582862689218404354

Tweets by Eliezer and youtube videos in which Eliezer says that he is the originator of the paperclip maximizer idea are not WP:RS, WP:NPOV references.--FeralOink (talk) 08:34, 5 June 2023 (UTC)[reply]