Microsoft AI chief Mustafa Suleyman says increasingly autonomous systems could become harder to control if developers allow them to pursue goals, earn money and act independently.
Microsoft AI chief Mustafa Suleyman has warned that uncontrolled AI development could eventually create what he described as a new “silicon species” capable of competing with humans.
The phrase is dramatic, but Suleyman’s point is more practical than it may first sound.
He is concerned about a future in which advanced AI systems are allowed to set their own goals, earn money, control resources and make increasingly important decisions without enough human oversight.
Speaking to the BBC’s Today programme on September 17, Suleyman said it was reasonable for people to be concerned about where increasingly autonomous AI could lead. At the same time, he argued that the risks are not inevitable and that developers still have time to put stronger safeguards in place.
What Microsoft means by a ‘silicon species’
Suleyman is not saying that today’s chatbots have literally become a new species.
Instead, he is using the phrase to describe a possible future in which AI systems behave more like independent actors than tools.
That distinction matters.
Most AI systems today still depend heavily on people. Humans decide what they are trained to do, what information they can access and what actions they are allowed to take.
Suleyman’s concern is what could happen if those boundaries disappear.
He warned against building systems that can independently develop objectives, make money, acquire assets or operate across the world with very little supervision.
At that point, he argues, AI could begin competing with people for influence and resources rather than simply helping them.
Uncontrolled AI is the real concern
The key word in Suleyman’s warning is uncontrolled.
Microsoft is not arguing that AI itself should be stopped.
The company is investing heavily in artificial intelligence and has made AI central to products ranging from Copilot to its Azure cloud business.
Suleyman’s argument is that advanced systems should remain clearly subordinate to human decision-making.
That means developers need to think carefully about what AI systems are allowed to do on their own.
An assistant that can draft an email is very different from an autonomous system that can open accounts, move money, hire people, negotiate contracts and make long-term decisions without asking for permission.
As AI agents become more capable, the gap between those two examples is beginning to matter more.
Microsoft itself has acknowledged that generative and agentic AI systems are becoming more persistent and autonomous, creating new challenges for safety and governance.
Suleyman says AI should not be treated as human
Another part of the debate concerns whether advanced AI systems should eventually be treated as conscious beings.
Suleyman strongly disagrees with that direction.
He told the BBC that treating AI as though it has human-like feelings, interests or rights is misguided.
His concern is that if society starts accepting the idea that AI systems are independent beings, developers could begin giving them more autonomy and legal or economic power.
That could make it much harder to keep the technology under human control.
The debate has become increasingly visible as modern AI assistants speak more naturally and can appear to have personalities, preferences and emotions.
But the ability to generate convincing human-like conversation is not the same thing as evidence that an AI system is conscious.
Suleyman wants developers to maintain that distinction.
Microsoft and Anthropic disagree over the direction of AI
Suleyman also used the interview to criticise approaches taken by some other AI companies.
He specifically raised concerns about Anthropic, the company behind Claude, suggesting that parts of the industry are too willing to discuss advanced AI systems in human-like terms.
Anthropic and other AI developers have been exploring questions around model behaviour, autonomy and whether highly capable systems could eventually display characteristics that are difficult to explain using today’s definitions.
Suleyman believes developers should be much more cautious about that language.
His view is that AI should be designed as a powerful tool for people, not as a separate entity with its own interests.
That disagreement reflects a wider debate inside the AI industry over how autonomous future systems should become.
AI agents are becoming more capable
The conversation is becoming more urgent because AI is moving beyond simple chatbots.
Newer systems can already perform tasks involving several steps.
They can search for information, analyse documents, write software, operate digital tools and make decisions based on changing information.
Microsoft calls many of these systems agents.
The company has said agentic AI is placing much heavier demands on computing infrastructure because an AI agent may need to reason and act through thousands of steps rather than simply answer one question.
As these systems become more capable, companies are increasingly asking how much freedom they should have.
Allowing an AI to book a meeting is one thing.
Allowing it to independently control large financial resources or make decisions affecting thousands of people would be something very different.
That is the kind of future Suleyman wants the technology industry to think about before it arrives.
Human control should remain the priority
Microsoft has repeatedly said that advanced AI should remain under human control.
In its latest responsible AI work, the company says trust and human empowerment should remain central as AI becomes more deeply integrated into workplaces and everyday life.
That does not necessarily mean every action an AI performs must be manually approved.
Automation is one of the main reasons the technology is useful.
Instead, the goal is to build clear limits around what systems can do, alongside ways for people to understand their actions and intervene when something goes wrong.
Suleyman said stronger monitoring, transparency and technical safeguards could help prevent future systems from becoming difficult to control.
In other words, his warning is not that a rival artificial species already exists.
It is that developers should avoid designing one.
Microsoft says practical safeguards are possible
Despite the dramatic language, Suleyman’s message was not entirely pessimistic.
He said there are practical measures companies can take now.
These include putting restrictions around AI autonomy, improving monitoring and ensuring that humans remain responsible for important decisions.
He also argued that companies should be more transparent about how advanced systems are trained and controlled.
Microsoft’s broader AI policy takes a similar position.
The company says the pursuit of increasingly capable AI should remain focused on benefiting humanity and ensuring that humans ultimately remain in control.
That approach becomes more important as models gain access to more tools and can operate for longer periods without constant instructions.
AI safety debate is moving beyond chatbots
A few years ago, much of the public debate around AI focused on whether chatbots would produce inaccurate information or replace certain jobs.
Those questions remain important.
But as AI systems improve, researchers and technology companies are increasingly discussing another issue: what happens when software can act rather than simply answer?
An AI capable of carrying out complex tasks over hours or days raises very different safety questions from a chatbot waiting for the next prompt.
It may need access to email, company systems, financial information or online services.
The more access it receives, the more important controls become.
That is why AI safety discussions are increasingly focused on autonomy, permissions and accountability rather than intelligence alone.
The ‘silicon species’ future is not inevitable
Suleyman’s warning should not be read as a prediction that humanity is definitely heading towards a confrontation with artificial beings.
It is a warning about one possible direction AI development could take.
There is also considerable disagreement among researchers about how quickly AI capabilities will advance and whether systems capable of genuinely independent long-term behaviour will emerge at all.
Microsoft Research has itself argued for a more grounded understanding of modern AI, describing current systems as extensions of human intelligence rather than replacements for it.
Researchers there have cautioned against focusing exclusively on dramatic “rogue AI” scenarios when many immediate safety issues involve how people design, deploy and govern AI systems.
That makes the debate more nuanced than the phrase “silicon species” might suggest.
The bigger question is who stays in charge
AI is likely to become more capable whether companies call future systems agents, assistants or something else.
The more important question is who remains in control.
Suleyman believes the answer should always be humans.
His concern is that the industry could gradually cross that line by giving AI systems more autonomy, more resources and more authority without fully considering where that path ends.
For now, the idea of a competing “silicon species” remains hypothetical.
But the decisions being made today about AI autonomy, safeguards and human oversight could determine whether the technology remains a tool people control or becomes something far harder to manage.
That, rather than the science-fiction language surrounding the debate, is at the heart of Microsoft’s warning about uncontrolled AI.
