Should AI Be Open?

As someone who has built my entire career and startup stack on top of open-source software, my knee-jerk reaction is always to support open code. But Scott Alexander's essay on whether advanced AI should be open forces a painful, necessary intellectual pivot. In traditional software, open sourcing makes systems more secure by putting eyes on bugs. In the context of superintelligent AGI, however, open sourcing might mean distributing a potentially catastrophic tool to bad actors, eliminating any chance of central coordination or containment.

This is a profound dilemma for any developer who believes in democratic access to technology. If we open-source a system that can be easily repurposed to cause global harm, we aren't democratizing power—we are decentralizing existential risk. It suggests that our standard open-source playbook, which has served us so incredibly well for operating systems and databases, might be actively dangerous when applied to highly autonomous, adaptive models that we do not yet know how to control.

What stuck with me

• The open-source dilemma: Traditional open-source security models fail when the code itself can be weaponized with zero friction.
• Decentralizing high risk: Opening up powerful autonomous systems makes containment and safety coordination virtually impossible to enforce.
• Asymmetric threat profiles: A single malicious actor can cause catastrophic harm if given access to unrestricted general-purpose intelligence.
• A new paradigm: We must develop entirely new governance frameworks for AI that balance open access with existential safety.