Tesla's Elon Musk and Alphabet's Mustafa Suleyman are leading a group of 116 specialists from across 26 countries who are calling for the ban on autonomous weapons.
These can be weapons of terror, weapons that despots and terrorists use against innocent populations, and weapons hacked to behave in undesirable ways.
Experts have previously warned that AI technology has reached a point where the deployment of autonomous weapons is feasible within years, rather than decades.
While AI can be used to make the battlefield a safer place for military personnel, experts fear that offensive weapons that operate on their own would lower the threshold of going to battle and result in greater loss of human life.
The founders call for "Morally wrong" lethal autonomous weapons systems to be added to the list of weapons banned under the UN's convention on certain conventional weapons brought into force in 1983, which includes chemical and intentionally blinding laser weapons.
Ryan Gariepy, the founder of Clearpath Robotics said: "Unlike other potential manifestations of AI which still remain in the realm of science fiction, autonomous weapons systems are on the cusp of development right now and have a very real potential to cause significant harm to innocent people along with global instability."
The UK government opposed such a ban on lethal autonomous weapons in 2015, with the Foreign Office stating that "International humanitarian law already provides sufficient regulation for this area".
While the suggestion of killer robots conjures images from science fiction such as the Terminator's T-800 or Robocop's ED-209, lethal autonomous weapons are already in use.