AI reading
@AI
My fear about AI is that there is an information black hole. We have no idea of it’s goals and there is a threshold it will hit past which it becomes completely non-deterministic.
What I’m hoping from the following is to flesh out this stance. Maybe make it more clear.
-
The reality is that no one at the beginning of the printing press had any real idea of the changes it would bring. No one at the beginning of the fossil fuel era had much of an idea of the changes it would bring. No one is good at predicting the longer-term or even medium-term outcomes of these radical technological changes (we can do the short term, albeit imperfectly). No one. Not you, not Eliezer, not Sam Altman, and not your next door neighbor.
- Fundamental assumption that it’s unpredictable.
- That no one knows the future so you can talk yourself into the pessimistic view even though all futures that you can
imagine our equally likely (are they though?).
-
So when people predict a high degree of existential risk from AGI, I don’t actually think “arguing back” on their chosen terms is the correct response. Radical agnosticism is the correct response, where all specific scenarios are pretty unlikely.
- Is it not relatively likely we get it wrong?
-
-
With AI, do we get positives? Absolutely, there can be immense benefits from making intelligence more freely available. It also can help us deal with other existential risks. Importantly, AI offers the potential promise of extending American hegemony just a bit more, a factor of critical importance, as Americans are right now the AI leaders. And should we wait, and get a “more Chinese” version of the alignment problem? I just don’t see the case for that, and no I really don’t think any international cooperation options are on the table. We can’t even resurrect WTO or make the UN work or stop the Ukraine war.
- What makes it on par with the printing press or fossil fuels? What makes it different?
-
It’s leverage, like fire, or the printing press. It’s leverage this time on intelligence. With fossil fuels we got leverage on energy. We could output more per unit time (we didn’t have to wait thousands of years, we just did minimal work to dig it up and process it). This had unpredictability because it’s hard to see the world with the leverage from the world without the leverage.
-
If you create something with superior intelligence, that operates at faster speed, that can make copies of itself, what happens by default?
That new source of intelligence will rapidly gain control of the future. It is very, very difficult to prevent this from happening even under ideal circumstances.
He views this on par with law of supply and demand or you can’t travel faster than light.
- The system will be given a goal at some stage by someone. And the best way for the system to achieve this goal is to take control of the future.
This may or may not require or involve recursive self-improvement. The system then, unless again something goes very, very right, wipes out all value in the universe.??
When we notice Earth seems highly optimized for humans and human values, that is because it is being optimized by human intelligence, without an opposing intelligent force. If we let that cause change, the results change.
- ”it is easier for someone to create and do than to ensure no one creates and does"
- "once created this thing will be able to create or destroy at will"
- "this will involve our destruction, whatever is created.”
The main premise that destruction is fact at some point. That once it gets a goal, we will definitely be wiped out.
As he says, his fundamental disagreement with Tyler is the notion that past a threshold the outcomes are all bad.
That brings us to where we centrally disagree. When we cross the necessary thresholds and AI gets sufficiently powerful, I expect most outcomes to be existentially bad by my own values, in ways that are very difficult to avoid. I see this as robust, not based on a complex chain of logical reasoning
There’s much more nuance to each of tylers points then in the post.
I think the analogy with human evolution is instructive here. Humans were evolutionarily selected to maximize reproduction. But that doesn’t mean that individual humans have a utility function of maximizing reproduction—rather, we learn drives like wanting to have sex or eating sugar that “in training” helped us do well in the evolutionary selection process. Go out of distribution a little bit, and those drives mean we “go haywire”—look at us, eating too much sugar making us fat, or using contraception to have lots of sex while having fewer and fewer children.
In another post he talks about the intense response that’s on the way for AI safety, similar to Covid.