In the past, when we talked about "killer robots,"
it was easy to think it was something out of a sci-fi movie.
A machine that sees a human by itself.
Decides on its own who is the enemy.
Then decides to fire on its own.
But what is really worrying the United Nations and the International Committee of the Red Cross now isn’t:
"Will this kind of thing ever appear?"
It’s:
"Should we set boundaries now before it’s too late?"
On August 25,
UN Secretary-General António Guterres and ICRC President Mirjana Spoljaric again jointly called on countries to
start discussing legally binding rules on autonomous weapons.
They used a very serious phrase:
The world is dangerously approaching a moral red line.
That is:
Allowing machines autonomy to select humans as attack targets.
What Exactly Are "Autonomous Weapons"?
First, let's clarify the most easily confused point.
Autonomous weapons are NOT:
A person remotely controlling a drone,
spotting a target,
and then pressing the attack button themselves.
That is still a human choosing a specific target.
The ICRC’s understanding of autonomous weapons is:
Once activated by a person,
the system can, based on:
sensors,
software,
and environmental signals,
match certain "target characteristics" on its own,
then select and apply force to the target.
Put simply:
Humans decide to activate the weapon, but the system itself chooses which specific target gets hit, not necessarily the human operator in real time.
Autonomous Weapons Are Not the Same as AI
This is an important point.
We commonly hear "autonomous weapons" and equate them directly with AI weapons.
But that’s not entirely accurate.
Some autonomous weapons are simply:
sensors + fixed rules + software.
They don’t necessarily use the machine learning or generative AI we know today.
For example, certain air defense systems
can automatically detect fast incoming missiles or rockets,
judge whether they meet specific criteria,
and then automatically intercept.
These systems may rely mostly on fixed rules,
not on an AI that "thinks."
What truly complicates things is:
Increasingly powerful AI is being incorporated into more military sensing, recognition, decision support, and autonomy systems.
So the issue is not just:
Can weapons move and act on their own?
But:
Will they eventually understand complex environments by themselves,
and independently determine who might be a target?
Weapons That Can "Shoot Themselves" Have Long Existed
This technology isn't completely new.
For example, some naval defense systems
must detect missiles,
judge threats,
track,
and intercept within extremely short timeframes.
If each step waits for a human confirmation,
the missile might have already hit the ship.
Therefore,
some air defense, anti-missile, or active protection systems on vehicles
already have a degree of autonomous target selection and attack ability.
But these systems usually have an important limitation.
They operate within:
limited areas,
limited durations,
and under clear conditions,
targeting very specific things.
Such as:
incoming missiles,
rockets,
artillery shells,
or military vehicles.
This is vastly different from sending a system into a city and having it
independently determine:
"Is this person the enemy?"
Not even comparable in difficulty.
The Real Red Line: Targets Changing from "Objects" to "Humans"
Imagine a defense system seeing:
A missile flying at high speed toward a naval ship.
It needs to identify:
speed,
direction,
radar signature,
flight trajectory.
These signals are relatively clear.
But if it sees a person?
The situation is completely different.
This person might be:
carrying a weapon.
Or just holding something that looks like a weapon.
They could be a combatant,
or a civilian.
They might have just been fighting,
but now surrendered.
They might be wounded,
unable to continue fighting.
They could be standing among other civilians.
At this point,
what’s needed is no longer just:
"Is the identification accurate?"
But:
"Legally and based on the situation, can we use force against this person right now?"
This requires,
not just image classification,
but also:
context,
proportionality,
distinction,
military necessity,
and human judgment.
Even 99% AI Accuracy Doesn’t Mean the Final Decision Can Be Fully Automated
Suppose there’s an AI
that recognizes combatants with very high accuracy.
The problem is not over.
Because:
Recognizing who a person is, and deciding if it’s lawful to kill them right now, are two different issues.
It’s like medical AI.
AI may accurately identify a likely tumor in an image.
But that doesn’t mean it should decide on high-risk treatment next.
Or the justice system.
AI can organize:
cases,
precedents,
and risk data.
But "how much freedom a person should lose"
is a different kind of authority.
Weapons are even more extreme.
Because the last mistake,
may leave no room for correction.
Humans Just Watching Isn’t Necessarily "Human Control"
There’s another pitfall here.
We easily say:
"It’s okay, as long as a person is watching at the end."
But if a system:
generates 20 targets per second,
each with a green box around it,
and gives the operator just two seconds to press:
"Approve,"
did that person truly reassess the situation?
Or is that just:
rubber-stamping the machine’s decision?
Meaningful human control cannot just be having a "confirm" button visible on a screen.
Humans must have:
enough information,
enough time,
real understanding,
and the power to
stop,
modify,
or cancel the attack.
If humans don’t have time to really think,
then "human-in-the-loop" may be merely symbolic.
If Machines Kill the Wrong Person, Who is Responsible?
This is the toughest question about autonomous weapons.
Suppose a system makes a mistake.
Causing an attack that should never have happened.
The first question is:
Who is responsible?
Is it:
the algorithm programmer?
the sensor manufacturer?
the weapons company?
the commander who ordered deployment?
the operator who activated the system?
or the state that approved procurement?
You cannot bring a machine to a military court
and ask,
"Why did you make that decision?"
A machine has no legal responsibility.
Ultimately, responsibility must lie with humans.
But as systems get more complex,
humans understand less about how a specific result happened,
and responsibility becomes harder to determine.
This is a key reason why the UN and ICRC want clear rules before the technology spreads widely.
Don’t We Already Have International Laws?
Yes, but not specifically for autonomous weapons.
War is not lawless.
Existing international humanitarian law still applies to new weapons.
Armies must still abide by:
distinguishing military targets from civilians,
proportionality,
precautions during attack,
and other laws of war.
The issue is:
There is no global legal agreement specifically setting clear rules for autonomous weapons,
defining which types must be banned,
which can be used,
and what level of human control must be retained.
The ICRC argues at least two categories should be banned:
Autonomous weapons whose effects cannot be adequately understood, predicted, or explained.
And:
Autonomous weapons designed or used to autonomously attack humans.
Other autonomous systems not banned should also be restricted in terms of:
what they can attack,
where they can be used,
how long they can operate,
and how humans supervise and can promptly stop them.
Why the Urgency Now?
Because technological advancement is outpacing international negotiations.
Countries started formally discussing autonomous weapons around 2014.
More than a decade later,
the technical capabilities are completely different.
Better sensors.
Faster chips.
Cheaper drones.
Stronger AI image recognition.
Military systems can process much more data simultaneously.
So what could once be delayed with "let’s wait and see,"
now means:
After large-scale deployment,
it will be far harder to establish shared regulations.
The joint UN statement on August 25 also conveys this message:
Set the boundaries first, then let technology advance.
Not let technology run unchecked and after an incident ask,
"Should we have put in brakes?"
This Issue Extends Beyond Weapons
Autonomous weapons are just a very extreme example of an AI-era dilemma.
As AI grows stronger,
we encounter the same question daily:
Which tasks can we fully entrust to AI?
Which should AI only suggest?
Which require a final human decision?
For example:
AI can help doctors find medical lesions.
But should it decide high-risk treatments on its own?
AI can analyze resumes.
But should it decide who never gets called for an interview without human knowledge?
AI can analyze suspect images.
But should it directly decide who law enforcement acts on?
AI can predict financial risk.
But can it cut off critical financial services without confirmation?
Once technology matures,
the hardest question is often not:
"Can AI do it?"
But:
"Even if it can, are we willing to hand over that power?"
Why Is the Final Button So Important?
Because some decisions
aren't just about who calculates faster.
They involve:
responsibility,
value judgments,
laws,
circumstances,
and whether humans are willing to be accountable for the outcome.
So what really matters about autonomous weapons is not "Will AI one day become killer robots?"
It's:
When machines become capable of completing more steps independently,
are humans still willing to clearly stipulate a point where they must stop?
If life-and-death decisions become just algorithmic outcomes,
what truly disappears is not just manual control,
but:
who ultimately bears responsibility for a person’s life.
Today, make some progress with AI.
Learn a new AI skill every day.
Save a bit of time daily.
Improve your abilities gradually.
SasaDaily, growing with you.
