With Driverless Cars, a Safety Dilemma Arises: Would you buy a car that might decide to kill you?

[mahatmakanejeeves]

With Driverless Cars, a Safety Dilemma Arises

Survey respondents say cars should try to protect as many pedestrians as possible in an accident scenario

By Amy Dockser Marcus
[e-mail]

June 23, 2016 2:00 p.m. ET

Would you buy a car that might decide to kill you?

It is a question social-science researchers are exploring amid the development of driverless cars. While commercial applications may be years away, any fully autonomous vehicle that eventually takes to the road will need to make decisions—like whether to swerve to miss one pedestrian at the risk of hitting another. ... Many ethicists argue that a public conversation should be part of the development process.

In a study published in Science, researchers found people want the cars to be programmed to minimize casualties while on the road. But when asked about what kind of vehicle they might actually purchase, they chose a car that would protect the car occupants first. ... The paper describes a series of online surveys that posited various scenarios.

In one, participants were asked to imagine that they are in a self-driving vehicle traveling at the speed limit. Out of nowhere, 10 pedestrians appear in the direct path of the car. Should engineers program the car to swerve off the road in such instances, killing the car occupant but leaving the 10 pedestrians unharmed, or keep going, killing the 10 people?

[whatthehey]

1. A rather contrived dilemma

Are these ten suddenly-appearing pedestrians on a barrierless mountain pass? Clustered around both sides of a car with a convoy of 18 wheelers coming in the opposite lane at 70mph? What's wrong with braking and steering into a kerb or verge where a passenger fatality is pretty unlikely? What's wrong with simply minimizing impact in both force and quantiity by slowing as much as possible and steering towards the clearest available direction?

[Kentonio]

4. There are always going to be edge cases where it becomes an issue.

And unlike with people were you can just leave it to their determination and judge them later, the cars have to be programmed to make decisions in these cases. Otherwise you WILL end up with occasional cases where a group of pedestrian children get mowed down because the cars programming tells it that protecting the occupant is paramount.

[whatthehey]

7. But programming doesn't work like that

It's not going to have a magic number or type of victim where it is directed to kill the occupant. You can't list all possible scenarios and say if two people under 15 years of age do this, if 1 person of 80 do that. The brakes and steering sill simply be programmed to reduce impact severity and quantity as much as possible, essentially to do what humans do but much much faster and with far better precision. Slow as quickly as possible and avoid the obstacle(s) as much as possible. What do human drivers do when a group of kids suddenly run into the road? Slam on the brakes and try and steer away from them. Autonomous cars will do the same. If either will not avoid the collision, hit as few of the ten teleporting toddlers as you can.

[Kentonio]

9. Programming works exactly like that.

There will certainly come a point not too far away when the built in cameras will be able to detect vital signs and then questions like whether the life of the occupant or the life of pedestrians have the higher priority will become important. If it isn't factored in then you'll just get cases where the car tried to avoid others while keeping the occupant safe, which could result in a driverless car killing multiple pedestrians when it could have simply killed the occupant. Better to have the discussion today rather than end up with it litigated in court by the families of dead people.

I actually suspect the occupants life will be the one determined to highest priority. The famous moral test shows that people are deeply uncomfortable about any decision to deliberately kill an innocent person to save other innocent people, plus if the occupant isn't highest priority it could have a deeply damaging effect PR wise on the industry.

[whatthehey]

12. No - it works with input X=output Y not with every possible scenario linked to a specific response

You're also falling for the false dilemma. There is neither the need nor the basis on which to triage life a against life b. The only relevant programming will be to minimize impact if it cannot be avoided. A car that decides to fling itself off an Alpine pass rather than hit a pair of pedestrians (assuming it has only one occupant) lacks the sensors to know whether it will crash into a cottage over the cliff with three innocent bystanders, thereby increasing body count. If it decides to hit a brick wall and kill the driver (is that 150lb in the back seat luggage or another passenger who needs saving by the way?) it cannot know that the brick wall will collapse a building with ten people inside. See, contrived scenarios can work both ways and have too many unknowable parameters to program this kind of utilitarian woolgathering when simply avoid/minimize impact does the trick just like humans do, but better.

[Kentonio]

14. It doesn't need to have every possible scenario factored in.

It can be as easy as 'if exterior potential victims > potential interior victims then take course of action that prioritizes the safety of exterior victims'. Obviously this assumes a level of camera technology that we're not at yet (but probably will be fairly soon). How it then prioritizes that safety is of course then down to its other programmed routines.

You're quite correct about the potentially unknowable outcomes of any decision of course, but if there's a public consensus on what the car should attempt to prioritize (and then that's written into legislation), then the manufacturers probably wouldn't be held liable for scenarios totally outside their reasonable expectation of control.

[Jerry442]

29. If a device is intelligent enough to exist in a complex human environment...

...then it will have to have something resembling values.

Already our brainless cars prioritize the well-being of an adult in the front passenger seat over that of a small child who might be seated there.

[Kentonio]

33. Values aren't necessary, just priority commands. Value driven AI is a fair way off yet.

[Jerry442]

34. Like it or not, our values often show up in our technology.

The Mitsubishi A6M Zero was designed with essentially no protection for the pilot. That might have been the best military option at the time, but someone had to make that decision.

[stevenleser]

46. That is a very interesting point.

And true.

[stevenleser]

45. I agree for two reasons

First of all, how many times is this scenario encountered nowadays? I dont think it is common. When it does happen the driver is at fault for the scenario happening in the first place a fair percentage of the time. That part would be eliminated.

Second of all, for the remaining cases, the decision making is simple. Since the car has been following the rules of the road, the decision should be to protect the occupants of the car while doing what you can to minimize the casualties of the folks who are not following the rules of the road/pedestrians.

This is simple stuff.

[Todays_Illusion]

59. I agree it is very contrived and provides cause to wonder abouit the result will be used.

Will this be part of a movement to influence opinion about driverless automobiles.

[leeroysphitz]

2. Why do so many will smith movies turn into real life problems? n/t

[giftedgirl77]

11. ....

[anigbrowl]

19. Because most Will Smith movies are based on real life problems

The Trolley PRoblem (should you act in a way that will kill someone in order to save a greater number of lives at risk) is a well-known ethical problem that has had entire books written about it. While it appears to promote stark utilitarianism, the deeper insight is that you rarely, if ever, enjoy so much certitude about the world that you can make pure utilitarian decisions, and so one is better off as a Kantian deontologist for practical purposes.

[Journeyman]

3. Keen way for a group to plot to kill someone -- simply step in front of their car and . . .

let the vehicle do it for you.

Seriously, however, this is a dilemma we as a society will need to address before such vehicles become commonplace. Yet another incident, sad to say, where our technologies outstrip our ethical considerations.

[tblue37]

65. One clever guy could commit that murder all by himself:

[ManiacJoe]

5. From an engineering and decision making process,

this is actually much easier than the article suggests.

For example:
Avoid the head-on collisions.
To prevent pedestrian injuries, keep the car on the road if evading off-road is risky to others.
To prevent chain reactions, keep the car in its lane as needed.
Evade off road if not overly risky to the occupants.

[DawgHouse]

6. I haven't trusted a car since I read Christine.

[Glassunion]

8. I agree with the survey response.

The car should try to protect as many pedestrians as possible.

A car striking a pedestrian has a higher probability of killing someone based on the simple physics involved. The occupants of the vehicle are protected by many layers of safety devices (crumple zones, seat belts, air bags, etc...).

[Shandris]

10. Well, I certainly can't see any reason why we should be slightly suspicious of...

...a company with Alphabet links and ties to every major national government - almost all of which are completely indifferent to the suffering of their own people - as well as a hand in AI development when it seeks to discover exactly what limits people have on allowing software and AI's to decide when and where people will be allowed to die for 'the greater good'.

No reason at all.

[LWolf]

13. This is

another reiteration of a very commonly posed ethical dilemma that has been debated for, well for longer than I've looked back to find.

The needs of the many vs the needs of the few, or one. I don't see that it is any more applicable to driverless cars than any other situation. I guess one could argue that the decision in how to handle that dilemma is taken out of human hands, but it's humans who program the computer. It doesn't "decide." It obeys its programming. The decision is out of the drivers' hands, though. We don't like the loss of control or choice.

[tblue37]

66. Yep. Statistically flying is *much* safer than traveling by car, yet people fear flying much more--

mainly because they are powerless to affect the situation.

Oh, and they are way high up there. As one comic put it, "After a plane crash, the pilots aren't getting out of the plane to exchange insurance information.

[jberryhill]

15. "Would you buy a car that is less likely to kill you" is a better question

Would you quit smoking if you knew that next month you'd be hit by a meteor?

[DeadLetterOffice]

38. Ah, the Sweet Meteor of Death...

[Lee-Lee]

16. I expect liability issues and questions to be a huge hurdle

Say you buy a self driving car. The car makes an error and is in an accident where a person is killed or injured.

Are you, the car owner responsible? Or is the software team and company that wrote the code?

How many lawsuits can a software team defend against if they get named as a defendant every time a self driving car is in a collision?

[Daemonaquila]

17. Simple answer - I will never buy a driverless car.

It's not just about the safety decisions. It's about not wanting expensive breakdowns I can't fix, or being out of control of the speed and route *I* want, or generally hating technology that thinks it's smarter than me but isn't. It's also about dumbing down our society even further, taking away another important right of passage (along with responsibility, honing judgment, etc.) from kids, and letting everyone's skills and judgment atrophy so that if anything happened to their car, the communications systems that would keep it all working, etc., we have a large portion of society sitting around asking each other what to do.

[mahatmakanejeeves]

18. Get back to me when you're 90 years old.

You have the car to take you to the grocery store and doctors' appointments.

Because of your deteriorating eyesight and reaction time, you no longer feel safe behind the wheel. Is someone going to take you where you want to go, day after day? I doubt it.

Your car does the job for you, and it never complains.

I wouldn't worry about "technology that thinks it's smarter than me but isn't." I would worry about having your - or my - having a bad day, or not having received enough sleep last night, or trying to get home before too long, even though I really should pull over.

[HereSince1628]

20. Will 90 year olds be able to afford a robot driver?

I suppose that depends a bit on when people turn 90 and how many old used robot drivers are on the market.

Personally I can't see someone on modest SS being able to afford one though I can imagine a robot driving eldercare bus.

With no drivers to put on salary small towns and rural areas that have no masses to justify mass transit might even be able to afford such things.

[Codeine]

23. Self-driving technology isn't likely to be terribly expensive

and will have greater reaction time and better judgment than anyone short of a Formula 1 driver.

[csziggy]

25. Perhaps they would not have to own one, just hire it like a taxi

My 95 year old mother lives in a small town with no mass transit and no taxi service. Her home caretakers are supposed to have cars so they can take her places, but their cars are not reliable or safe. My sister who lived 50 miles away visits once or twice a week to take Mom to the doctor and grocery shopping. If it weren't for her, Mom would have to move into a nursing home which would accelerate her dementia.

I could see a company - or even the town - having one or two driverless cars on call, just like city people can call for a taxi or Uber. It wouldn't work for Mom at this point - her dementia has worsened in the last two years - but for many of the elderly in that town that should not be driving, it would be wonderful.

By the time I get so I can't drive I might be able to adjust to the idea of a driverless car to either own or that I could call to take me places. If I take after my Mom, I have a good 25 + years before I have to worry about it, though.

[HereSince1628]

27. Yes, I can sort of see robot taxis/vans as something that could work

because there's no salary and other overhead for a driver.

I guess you'd have to have an onboard communications service to deal with mechanical failures or other emergencies. But I think an investment under current dollars $40K and whatever insurance would be is something a co-op or eldercare program could make. And recoup at modest fairs elderly and handicapped could afford.

[csziggy]

28. For elders it would be good to have some protections

Like my Mom at this point. She forgets where she's going so having precautions to keep her from getting out of the car at stoplights would be good. Maybe set it up so when she arrives at her destination, someone would let her out and accompany her inside. With all us Boomers getting old, services like this would be so helpful in the near future!

Right now if my sister could arrange for a scheduled service to pick Mom up, carry Mom and her caretaker to appointments and shopping, it would reduce my sister's stress a lot. But there are no services withing a reasonable distance or at a reasonable cost.

Or even for some of our friends that live here in town. One couple we know are both disabled and on SSD. They can't afford a car, the bus is a 1.5 mile walk which neither of them can manage. At least once a week my husband takes them places they need to go - medical appointments, shopping, monthly to pay all their bills (in cash since they can't afford a checking account), etc.

Tomorrow he has to take the husband to the court house for jury duty - the court will not excuse him in advance. In addition, my husband will take the wife to a medical appointment and spend the day with her - she had a pacemaker put in a few weeks ago and cannot be left alone. That couple would love to have an alternative that was affordable so they wouldn't have to impose on my husband!

[tblue37]

70. This reminds me of one 5-year period when I kept getting called for jury duty. Over and over again,

nine times!

I kept explaining that I am severely hearing impaired, even with high quality hearing aids, and don't know sign. I would never be able to hear the lawyers' questions, the witness testimony, or the judge's instructions.

Finally I got them to allow me an "extended" exemption, though my exemption notice said the exemption must not be presumed to be permanent and could be withdrawn at any time. As deaf as I already am, my condition (Meniere's disease) is progressive, so it gets worse, which I also explained, but bureaucracy isn't rational.

[csziggy]

74. Yeah, my Dad kept getting called the same way

And got excused for the same reason. By the third time they called him in, he was fed up and turned off his hearing aids when he got to the courthouse. Without the aids he was profoundly deaf and couldn't hear anything at all. The court clerk got so frustrated with him she made sure he got put on the exempt list permanently. By the time Dad died he was so deaf the hearing aids didn't help anymore.

I got called in 2008 the year after they removed the meniscus from my other knee. When I called in I asked about handicapped parking since I had a permit. The clerk told me "No problem, there are spaces right next to the entrance and in the underground garage next to the elevator. When I got there all the spaces next to the entrance were filled with dumpsters for the remodeling they were doing to the courthouse. The pay parking garage next to the entrance was filled to capacity and there was no street parking at all.

The underground garage was restricted to employees. I finally convinced the guard to let me in there, showing him my jury summons and that I was already late. Yeah, the handicap spaces were next to the elevators - which were also restricted to employees. I had to walk the length of the block to get to the security check point, then back the same distance to where the employee elevators were located to check in with the clerk, then halfway back to wait in the courtroom for jury selection. By the time they started I was in so much pain I couldn't pay attention. I was selected for the jury but that day I got to the pay parking lot early enough to get in. Then after we sat around for three hours the defendant plead out.

I can't wait until I am too old for jury duty.

[tblue37]

77. Without aids I am profoundly deaf. With them, I still can't understand most speech unless

am about 3 feet or less away from the speaker and directly in front of him or her. I also have to be able to see the speaker's lips, because I rely heavily on lipreading.

And even under ideal conditions like that, if the speaker mumbles or swallows his/her words (the way so many do), I still can't understand what is being said.

I would be useless in a courtroom.

[Uben]

36. The rich get old, too.

In fact, they are more likely to reach ninety than the poor.

[tblue37]

69. We need individual transport pods--like in "Minority Report," or "The Jetsons":

https://m.

#


https://m.

#

[tblue37]

67. I don't even use cruise control when driving. nt

[tblue37]

68. When I read this part of your reply I thought about velcro sneakers:

"It's also about dumbing down our society even further, taking away another important right of passage (along with responsibility, honing judgment, etc.) from kids, and letting everyone's skills and judgment atrophy. . . ."

[EllieBC]

73. My daughter's physio therapist hates Velcro shoes.

Because they've led to a delay in some fine motor development.

[Blue_Tires]

21. I'm not ever buying one, so no dilemma for me...

[Major Nikon]

37. Many of the technologies involved are already available today

Very soon many of them will be standard equipment on all cars.

[Blue_Tires]

52. As long as I'm the one in total control...

[Major Nikon]

54. You probably aren't already

If your car has cruise control, anti-lock brakes, stability control, and/or traction control, you aren't the one in total control. By 2022, all new passenger vehicles will have automatic emergency braking as standard equipment. My latest vehicle already has it along with adaptive cruise control and a few other automation functions.

[Blue_Tires]

55. You're conflating driver aids with full automation...

FWIW, I've used cruise control a grand total of two times on my current car, and the first time was just to see how it worked...

And I can turn off traction/stability control whenever I want.

[Major Nikon]

57. Please read post #37 again

"Full automation" even if and when it appears, will most certainly have the capability of being overridden. So the capability of overriding parts of the functionality isn't what differentiates it in the first place.

The point is "full automation" isn't going to just suddenly appear. There's going to be a series of technologies employed which will lead up to it. And while you will be able to override some of them, you probably aren't going to want to because like traction control, stability control, anti-lock brakes, and many other forms of automation they are going to make driving safer.

[JustABozoOnThisBus]

22. What would a real-life human driver do?

a) slam on the brakes and yell "oh, shit!"
b) honk the horn and aim for the least pedestrian-dense part of the road and yell "oh, shit!"
c) hit the pedestrians, then look up from the phone/coffee/burger and yell "oh, shit!"
d) decide for the good of the many, to drive over the cliff. On the way down, yell "oh, shit!"

Good luck programming a driverless car to make ethical decisions. Maybe self-sacrifice can be a "setting" in the system.

[Codeine]

24. Panic-lock the brakes while turning the car,

thus assuring there's no control whatsoever.

[PoliticAverse]

26. I'm guessing the 1% will be able to pay for the "personal protection priority" option. n/t

[Jerry442]

31. I was going to go there. You beat me to it. NT

[flvegan]

30. Was there a follow up "What if the 10 pedestrians are Kardashians?" question?

Cuz then it gets to be a REALLY easy answer.

Yes, I'm just kidding.

[trof]

32. You'll have an option.

For "Protect the vehicle occupant(s), press 1."
For "Protect the most number of potential victims, press 2."
Sorry, I'm pressing "1".

[mahatmakanejeeves]

41. "I'm sorry. I didn't get that answer. Would you say that again?"

"Did you say 'shipping department'?"

[Uben]

35. Hell to the no!

I was a career machinist. I've had software problems nearly kill me more than once.

[JustABozoOnThisBus]

40. As a career machinist, I'd bet you've had close calls with human error.

Just the sleepy/drunk/distracted fork lift driver can make the plant floor interesting.

[Uben]

75. Oh yeah!

I've had parts thrown at me more than once. I would always dry run a program, but sometimes missed the programming errors...it's easy to do. Never was really injured by that other than a few bruises, but if the part had been a foot in the other direction it would have been lights out! I know guys who have been injured severely. I made it through my career with a lot of luck and a strict detail to safety.

[tblue37]

71. Considering how often your cell phone drops calls or your tablet freezes up or gets

disconnected from your home wi-fi, would you really trust your car's automated driving system to never have a glitch?

Heck, even the cable goes out on your TV from time to time when you are watching a show on HBO.

And what about the possibility of interference from other devices in the vicinity?

[One_Life_To_Give]

39. When given the choice while driving what wou9ld You do?

Imagine deciding between killing ten kindergartners vs driving off a cliff. How would each of us choose? Will a community programmed computer choose differently? And if so what does it say about us?

[sendero]

42. I will never be interested...

.... in a driverless car, and I suspect I am not remotely alone.

I code in big complex systems for a living and I'm really not interested in finding the fatal bug in their system.

[mahatmakanejeeves]

44. Driverless Cars Should Kill Passengers To Save Lives - But Then People Won't Buy Them

Jun 24, 2016 @ 06:50 AM

Driverless Cars Should Kill Passengers To Save Lives - But Then People Won't Buy Them

People want driverless cars to act for the greater good in traffic collisions – but they don’t want to be in the cars when they do so.

In a series of surveys, researchers asked people whether autonomous vehicles (AVs) should swerve to avoid hitting a group of pedestrians, even if that meant killing the occupant of the car. Most people gave the greater good answer, that saving many lives was better than saving one, but those people don’t want to be in a car that would make that choice.

Even when the researchers introduced variations to the pedestrian/swerving example to make it a more difficult choice, such as suggesting a family member or a child was in the car, participants still objectively want the car to save the most people possible. Just not when it’s them or their own children.

“Although people tend to agree that everyone would be better off if AVs were utilitarian (in the sense of minimizing the number of casualties on the road), these same people have a personal incentive to ride in AVs that will protect them at all costs,” the researchers said in a paper for Science magazine.

[Initech]

47. Cars are already becoming self aware.

Shit, look at what happened to Anton Yelchin - dude was crushed by his own car after the electronic gear shifter put itself in reverse when he had it in neutral. With driverless cars, if it could happen to him, it can happen to anyone.

[mahatmakanejeeves]

48. Is that an accurate description of what happened?

Is it not the case that he thought he had the car in park, but the transmission was really in neutral?

Anton Yelchin’s Death Highlights a Known Issue With Jeeps

By CHRISTOPHER JENSEN JUNE 21, 2016

....
The problem involves an electronic gearshift, whose operation is similar to that of a video-game joystick. It has confused many drivers, who thought they had left their cars in park, only to find they were in neutral, and started rolling away after the drivers stepped out.
....

Last August officials at the highway safety administration told Fiat Chrysler that it was investigating complaints from 14 owners of some of Fiat Chrysler’s most popular models. The owners said their vehicles had rolled away — in some cases causing accidents and injuries — when they left the vehicles, thinking they were safely in the parking gear.

“As I stepped out, the vehicle started moving backwards and the driver door struck me, knocking me to the ground on my back,” a Jeep Grand Cherokee owner from Morganton, N.C., wrote to safety officials last summer. “The left front tire rolled over my pelvic area, causing serious injury.”

[Initech]

49. That's what I heard happened.

But it's still scary to think that cars aren't safe like that. After hearing about that, I wouldn't buy a current model Jeep if you paid me.

[TacoD]

50. If an airline pilot ever had to choose whether to kill everyone on the plane in order to save a much

larger number of people on the ground, I would have no choice but to trust him or her to make the right choice. Isn't this the same thing?

[anoNY42]

51. A PSA is needed

educating the public on how they are already "programmed" to kill themselves and others when they get behind the wheel. We really do make terrible decisions all the time (source: my experience on the mean streets of Florida).

[RAFisher]

56. With my last job I was driving all time on Florida streets. I'm all for driverless cars.

I was on the fence before but now I can't wait for driverless cars.

[anoNY42]

63. Ouch

I feel for ya.

[Vinca]

53. I wouldn't buy a driverless car.

For starters, I like driving. Secondly, computers aren't perfect. GPS keeps sending tractor trailer trucks up a nearby narrow and winding road that borders a stream and they keep sliding into the water.

[Todays_Illusion]

58. Right now we are all at the mercey of other drivers and have managed to reduce hiway deaths/per

mile traveled. That knowledge and our collective experience and memories of people we knew who were killed in an unnecessary automobile crash has not stopped many of us from driving.

I agree with whatthehey, this is a contrived question, is the research intended to carry an anti-driverless car message?

[Little Tich]

60. I would trust the driverless car over a normal car, unless the driver is a woman.

Women are safer drivers.

The Telegraph, 7 JANUARY 2016:Women are better drivers than men, says new survey
http://www.telegraph.co.uk/cars/news/women-are-better-drivers-than-men-says-new-survery/

[alphafemale]

61. So you are saying pedestrians would be walking where they obviously aren't supposed to be

They are walking on a vehicle roadway in an area with blind curves or with low visibility.

They are like people walking on train tracks.

Yeah, stupidity is fatal sometimes.

[mahatmakanejeeves]

62. Self-Driving {sic} Tesla Was Involved in Fatal Crash, U.S. Says

Self-Driving Tesla Was Involved in Fatal Crash, U.S. Says

I do not own a Tesla, and I am not familiar with its various features. I see that whatthehey has pointed out that the headline in the New York Times is in error, and that this Tesla was in its autopilot mode and not self-driving.

Except this wasn't one, ...

Tesla is very clear about what autopilot is, and a self-driving car it is not.


Note the bits about assistance, reducing driver workload, being alert and keeping your hands on the wheel at all times?

A self-driving car would make all those things irrelevant and in fact exclusionary.

https://www.teslamotors.com/presskit/autopilot

[Darb]

64. Driverless will be the only choice in 10 years.

Won't be able to get insurance otherwise.

[killbotfactory]

72. People are terrible drivers and scare the shit out of me

Please bring on the self-driving cars, ASAP.

[Agnosticsherbet]

76. A car is not a conscious being.

It can not decide to kill.