连环推 ×2第1部分：感谢 Gavin 进行了一场格外深思熟虑的交流。我平时不太花时间在社交媒体上，但这次想参与进来，因为它真正触及了一场重要对话的核心。
首先，关于监管，我认为"要么通过监管把权力集中在少数选定的公司和政客手中，要么广泛分散"是一个虚假的两难选择。我知道硅谷有一种简化的说法，认为监管=监管捕获=权力集中，但我一直觉得这是对世界过于简化的描绘。这个圈子之外的很多人把监管视为限制企业权力、惠及普通人的工具。我并不完全同意那种观点，而是认为事情很复杂，真的取决于"监管"具体包含什么。但尤其值得注意的是，我认为持"监管=监管捕获=权力集中"框架的人往往低估了客观、公正的制度程序所具有的去中心化力量。举个粗糙的类比：正式法院系统有时会显得刻板和精英化，但它在捍卫弱势个体权利方面远比替代方案——暴民正义——做得好。在最佳状态下，制度可以把权力赋予想法而非个人，从而实现权力分散。
这正是 Anthropic 一直非常谨慎制定政策提案的原因。我们非常努力地提出对前沿 AI 公司不利（减缓其发展）但对小型竞争者有利的提案。加州 SB53（我们支持的）甚至备受诟病的 SB 1047（我们对其态度矛盾）都完全豁免了收入或模型训练成本低于一定门槛的公司（SB 53 是 5 亿美元，1047 更低但我们对这一点提出了异议）。最近，我们在 CAISI 和白宫倡导的测试流程对前沿模型的测试要求比非前沿模型更严格——这对挑战者更有利。同样，"Pacing the Frontier"公开信设想（或至少 Anthropic 偏好的实施方案设想）调节最顶尖模型的推进速度，同时不限制追赶中的参与者。这损害了前沿实验室的商业利益，却帮助了挑战者，包括开放权重模型！
总的来说，我的观点是，AI *在结构上*就是一种倾向于集中权力的技术，原因与监管无关（更多与 scaling laws 的极端含义有关）。开放权重确实在这方面有些帮助，但远非充分的解决方案，因为它们只是将集中程度部分转移到拥有最多算力和芯片的人手中（这些人大致就是前沿实验室，加上可能还有硬件供应商）。相比之下，我认为正确的"游戏规则"可以同时做到：（a）应对 AI 的网络、生物和 alignment 风险；（b）在制度上约束前沿 AI 公司的权力；（c）为开放权重模型留出空间，同时解决它们带来的特定风险。
顺便说一下，我并不认为过去几个月的事件"未能促成[我]倾向的监管路径"。据报道，特朗普政府采取的做法——对前沿模型进行部署前测试，以及在开放权重模型接近前沿水平时也对其进行测试——我非常支持，当然还得看到细节才能确认。我也支持 Demis Hassabis 关于类似 FINRA 机构的设想。这与六个月前形成鲜明对比，当时业界大多数人还在推动全盘否决各州监管，而联邦层面似乎也没有任何明确方案。
2/2 其次，关于AI的舆论传播。我认为我的言论并没有被不公正地负面化。实际上，我的言论在风险与益处之间基本平衡：我分别就这两方面各写过一篇重要文章，即便在讨论风险的采访中，我也会确保频繁提及惊人好处并提出应对风险的可行方案（我采访中的短片段往往被裁得比较负面，因为那样更有流量）。事实上，我写《博爱机器》是因为我觉得AI行业没能在描绘科技如何彻底改变世界方面足够鼓舞人心。这篇文章的大部分内容都在反驳对AI在健康与生物领域潜力的怀疑，并阐明为什么我认为未来5到10年内治愈大多数人类疾病是切实可行的，即便这在普通人乃至生物学家（我也曾是其中之一！）听来疯狂。还有，如果你阅读我最新的文章（《人工智能指数的政策》），我提出了具体建议来简化和加快FDA流程，确保源源不断由AI加速研发的药物不会被监管机构拖延。我深知这种紧迫感：我父亲在直接作用抗病毒药物（索非布韦）问世前几年死于丙型肝炎，这种药物能治愈95%的患者，也很可能治好他。
我同意公众对AI持负面看法（且这是个大问题），但不认为这主要由我或其他AI领导人发表的AI风险警告所致。我认为这根本上是信任危机。普通人既不信任企业、政府，也不信任科技产业，总觉得我们在耍什么新花招来坑他们。这根源可追溯到几十年前，AI不过是它最新的化身。我觉得搞一场花哨积极的营销活动（有建议Anthropic这么做）并非赢回信任之道——今时今日，说“AI能治愈癌症”听起来更像陈旧套话而非激励，大多数人觉得这是一种欺骗。真正管用的是*真的把癌症治好*。我认为目前对AI公司（包括Anthropic）来说最精准的批评是，我们都还未兑现造福世界的宏大诺言。这完全怪我们自己，所以我反倒希望你能这样批评，而不是琢磨那些传播技巧和市场策略的话题。
不过我们正尽力改进：Anthropic正快速推进其在生物和医学领域的努力，我们希望在未来的几年里能取得惊人成果，并在接下来的几个月内窥得一缕曙光。一旦我们真正实现了一些实质成就，全世界都会以最大音量听到，打包票。但在那一天到来前，我不想说空话；同时，我忍不住要对AI带来的实际风险及其应对方案坦诚相告。坦诚是基于良心最该做的事，且就公众信誉和信任度而言，它绝不输给甚至有可能优于那种忽视或掩饰人们心里本能感受到的真实威胁的态度。
引用 Gavin Baker @GavinSBakerSholto, thank you for setting the record straight. Larger issue is that multiple very serious people in Silicon Valley have heard some variation of this and believe it to be true. And the reason it is believable to so many is that it is consistent with Dario’s public messaging and what he outlined in the essay you shared: this technology *might* be dangerous for humans in multiple ways, could lead to extreme concentration of economic power (as outlined in the essay) and therefore needs to be regulated thoughtfully. I agree with the potential risks and I believe Dario makes all of these arguments in good faith. As discussed on the pod, if one agrees that AI *might* be dangerous, there are two ways to address this potential risk. Either concentrate it in the hands of a chosen few companies and politicians via regulation or distribute it widely. Essentially boils down to whether one believes AI is too dangerous to concentrate or too dangerous to distribute. There are reasonable arguments on both sides, but I profoundly agree with Zuckerberg’s statement that: “The notion that AI is so dangerous that the only safe path is an extreme concentration of power seems inherently problematic. Historically, hoping that an absolute power will benevolently provide for humanity if sufficiently enlightened has not led to safe or positive outcomes.” And as Dario says in the aforementioned essay, “some may object that we can simply keep AIs in check with a balance of power between many AI systems, as we do with humans.” I believe this is the best path forward: I want as many AIs as possible to maximize the odds that one shares my own particular values. And as Dario notes, no human has ever been able to take over the world. At this point, I think safe to say that Dario has lost the argument. His messaging has failed to result in his preferred regulatory path. The fact that the only solution to the recent incident where an unreleased advanced OpenAI model hacked Hugging Face was an 查看被引原帖 ↗
查看英文原文
1/2 Thanks Gavin for an especially thoughtful exchange. I don't usually spend much time on social media but I wanted to engage here because it really brings out the heart of an important conversation.
First, on regulation, I think that “either concentrate it in the hands of a chosen few companies and politicians via regulation or distribute it widely” is a false choice. I know that there’s a sort of Silicon Valley shorthand where regulation = regulatory capture = concentration of power, but I’ve always found this to be an overly simplified picture of the world. Many people outside this bubble think of regulation as something that constrains corporate power and benefits ordinary people. I don’t necessarily agree with that perspective either, rather I think it’s complicated and really depends on what the “regulation” consists of. But in particular I think that those in the “regulation = regulatory capture = concentration of power” frame often underrate the decentralizing power of objective and fair institutional processes. A crude analogy is that the formal court system can sometimes feel stuffy and elitist, but it does a much better job of defending the rights of vulnerable individuals than the alternative, mob justice. At their best, institutions can vest power in ideas rather than people, and thereby decentralize that power.
This is why Anthropic has always made its policy proposals very carefully. We try very hard to make proposals that disadvantage (slow down) frontier AI companies while *advantaging* smaller competitors. California’s SB53 (which we supported), and even the much-maligned SB 1047 (which we were ambivalent on), completely exempt any company below a certain amount of revenue or model training costs from being covered at all (it was $500M for SB 53, lower for 1047 but we objected to that). More recently the testing process we’ve advocated for at CAISI and the White House involves more rigorous tests for frontier models than off-frontier models — something that differentially advantages challengers. Similarly, the “Pacing the Frontier” letter envisions (or at least Anthropic’s preferred implementation of it envisions) modulating the pace of the very best models while not constraining those who are catching up. This hurts the business interests of the frontier labs and helps challengers, including open-weights!
Overall my view is that AI is *structurally* a technology that tends to concentrate power, for reasons that have nothing to do with regulation (more to do with the extreme implications of the scaling laws). Open-weights do help some with this but are nowhere near a sufficient solution because they simply shift the concentration somewhat to those with the most compute and chips (which are roughly the frontier labs plus maybe hardware providers). By contrast I think the right “rules of the road” can simultaneously (a) address AI’s cyber/bio/alignment risks, (b) institutionally constrain the power of the frontier AI companies, and (c) leave room for open-weights models while also addressing the specific risks that they bring.
BTW I do not think that the events of the last few months have “failed to result in [my] preferred regulatory path”. The approach that the Trump administration is reported to be taking — pre-deployment testing for frontier models, and also testing of open-weights models when they get closer to the frontier — is one that I am very supportive of, though of course I have to see the details to be sure. I am also supportive of Demis Hassabis’ ideas around a FINRA-like entity. This contrasts with six months ago when most of the industry was still pushing for preemption of all state regulation and no apparent federal approach either.
2/2 Second, on the messaging around AI. I do not agree that my messaging has been disproportionately negative. In fact it has been about equally balanced between risks and benefits: I’ve written one major essay about each, and even in interviews where I discuss the risks, I make sure to frequently mention the incredible benefits as well as proposing possible solutions to the risks (short clips from my interviews that end up on social media tend to be disproportionately negative, as that gets clicks). In fact, I wrote Machines of Loving Grace because I didn’t feel the AI industry was painting an inspiring enough picture of how the technology could radically transform the world for the better. The bulk of the essay is devoted to refuting skepticism of AI’s potential in health and biology, and showing why I think it will actually be possible to cure most human disease in ~5-10 years, as crazy as it may sound to ordinary people and frankly to biologists as well (I used to be one!). And, if you read my most recent essay (Policy on the AI Exponential), I discuss concrete proposals for how to streamline the FDA process to make sure the deluge of AI-accelerated drugs isn’t slowed down by the regulatory process. I feel the urgency here: I lost my father to Hepatitis C only a few years before the development of direct-acting antivirals (sofosbuvir), which cure 95% of patients and probably would have cured him.
I do agree that the public has a negative view of AI (and that this is a big problem), but I don’t think it is primarily caused by me or any other AI leader warning about AI’s risks. I think it is fundamentally a crisis of trust. I think that ordinary people don’t trust companies, governments, or the tech industry and always suspect that we are cooking up some new way to screw them over. The causes of this go back decades and AI is just the latest iteration of it. I don’t think that a glitzy marketing campaign with a positive spin (which some have advocated that Anthropic do) is the way to win back that trust — at this point, saying that AI will cure cancer is more a cliche than it is inspiring, and most people think it is deceptive. The thing that will work is *actually curing cancer*. I think by far the most accurate criticism of AI companies including Anthropic is that we haven’t yet delivered on our big promises to benefit the world. That is totally on us, and I think it’s the criticism you should be making, instead of all this stuff about messaging and marketing.
We are however doing our best to fix this: Anthropic is ramping up its efforts very quickly in biology and medicine, and we hope to have incredible results in the coming years and some early glimmers in the coming months. When we’ve actually accomplished something real, the whole world will hear about it, as loudly as possible, you have my word on that. But until then I don’t want to make empty promises, and in the meantime I feel compelled to speak honestly about the very real risks of AI and how to address them. Honesty is the right thing on the merits, and in terms of public credibility and trust it is no worse than, and may in fact be better than, an approach that ignores or distracts from risks which people instinctively understand are real.
First, on regulation, I think that “either concentrate it in the hands of a chosen few companies and politicians via regulation or distribute it widely” is a false choice. I know that there’s a sort of Silicon Valley shorthand where regulation = regulatory capture = concentration of power, but I’ve always found this to be an overly simplified picture of the world. Many people outside this bubble think of regulation as something that constrains corporate power and benefits ordinary people. I don’t necessarily agree with that perspective either, rather I think it’s complicated and really depends on what the “regulation” consists of. But in particular I think that those in the “regulation = regulatory capture = concentration of power” frame often underrate the decentralizing power of objective and fair institutional processes. A crude analogy is that the formal court system can sometimes feel stuffy and elitist, but it does a much better job of defending the rights of vulnerable individuals than the alternative, mob justice. At their best, institutions can vest power in ideas rather than people, and thereby decentralize that power.
This is why Anthropic has always made its policy proposals very carefully. We try very hard to make proposals that disadvantage (slow down) frontier AI companies while *advantaging* smaller competitors. California’s SB53 (which we supported), and even the much-maligned SB 1047 (which we were ambivalent on), completely exempt any company below a certain amount of revenue or model training costs from being covered at all (it was $500M for SB 53, lower for 1047 but we objected to that). More recently the testing process we’ve advocated for at CAISI and the White House involves more rigorous tests for frontier models than off-frontier models — something that differentially advantages challengers. Similarly, the “Pacing the Frontier” letter envisions (or at least Anthropic’s preferred implementation of it envisions) modulating the pace of the very best models while not constraining those who are catching up. This hurts the business interests of the frontier labs and helps challengers, including open-weights!
Overall my view is that AI is *structurally* a technology that tends to concentrate power, for reasons that have nothing to do with regulation (more to do with the extreme implications of the scaling laws). Open-weights do help some with this but are nowhere near a sufficient solution because they simply shift the concentration somewhat to those with the most compute and chips (which are roughly the frontier labs plus maybe hardware providers). By contrast I think the right “rules of the road” can simultaneously (a) address AI’s cyber/bio/alignment risks, (b) institutionally constrain the power of the frontier AI companies, and (c) leave room for open-weights models while also addressing the specific risks that they bring.
BTW I do not think that the events of the last few months have “failed to result in [my] preferred regulatory path”. The approach that the Trump administration is reported to be taking — pre-deployment testing for frontier models, and also testing of open-weights models when they get closer to the frontier — is one that I am very supportive of, though of course I have to see the details to be sure. I am also supportive of Demis Hassabis’ ideas around a FINRA-like entity. This contrasts with six months ago when most of the industry was still pushing for preemption of all state regulation and no apparent federal approach either.
2/2 Second, on the messaging around AI. I do not agree that my messaging has been disproportionately negative. In fact it has been about equally balanced between risks and benefits: I’ve written one major essay about each, and even in interviews where I discuss the risks, I make sure to frequently mention the incredible benefits as well as proposing possible solutions to the risks (short clips from my interviews that end up on social media tend to be disproportionately negative, as that gets clicks). In fact, I wrote Machines of Loving Grace because I didn’t feel the AI industry was painting an inspiring enough picture of how the technology could radically transform the world for the better. The bulk of the essay is devoted to refuting skepticism of AI’s potential in health and biology, and showing why I think it will actually be possible to cure most human disease in ~5-10 years, as crazy as it may sound to ordinary people and frankly to biologists as well (I used to be one!). And, if you read my most recent essay (Policy on the AI Exponential), I discuss concrete proposals for how to streamline the FDA process to make sure the deluge of AI-accelerated drugs isn’t slowed down by the regulatory process. I feel the urgency here: I lost my father to Hepatitis C only a few years before the development of direct-acting antivirals (sofosbuvir), which cure 95% of patients and probably would have cured him.
I do agree that the public has a negative view of AI (and that this is a big problem), but I don’t think it is primarily caused by me or any other AI leader warning about AI’s risks. I think it is fundamentally a crisis of trust. I think that ordinary people don’t trust companies, governments, or the tech industry and always suspect that we are cooking up some new way to screw them over. The causes of this go back decades and AI is just the latest iteration of it. I don’t think that a glitzy marketing campaign with a positive spin (which some have advocated that Anthropic do) is the way to win back that trust — at this point, saying that AI will cure cancer is more a cliche than it is inspiring, and most people think it is deceptive. The thing that will work is *actually curing cancer*. I think by far the most accurate criticism of AI companies including Anthropic is that we haven’t yet delivered on our big promises to benefit the world. That is totally on us, and I think it’s the criticism you should be making, instead of all this stuff about messaging and marketing.
We are however doing our best to fix this: Anthropic is ramping up its efforts very quickly in biology and medicine, and we hope to have incredible results in the coming years and some early glimmers in the coming months. When we’ve actually accomplished something real, the whole world will hear about it, as loudly as possible, you have my word on that. But until then I don’t want to make empty promises, and in the meantime I feel compelled to speak honestly about the very real risks of AI and how to address them. Honesty is the right thing on the merits, and in terms of public credibility and trust it is no worse than, and may in fact be better than, an approach that ignores or distracts from risks which people instinctively understand are real.
