NOT WITHOUT US

A Challenge to Computer Professionals to Help Bring the Present

Insanity to a Halt

By Joseph Weizenbaum, professor of computer science at MIT

(This is an English translation of a talk given in German to the

Association of Computer Professionals in West Germany in July 1986.

You are welcome to reproduce and distribute it.)

 

Whenever I come to Europe, especially to West Germany, I am

amazed by the normality of everyday life: superhighways, "music" that

assaults one in restaurants, the many parks, the forests of television

antennas on the roofs of houses and so on. I am amazed because of

Europe's geographic position and all that follows from it. In West

Germany, for example, there is the border with the other Germany,

dense with military installations of all sorts. There are holes in

the street that are intended to be filled with nuclear land mines if

Russian tanks should come. These are signs of Europe's physical and

psychological proximity to the final catastrophe.

We in America are, in a certain sense, no more distant from the

catastrophe than the Europeans are. Not only Chernobyl, but also the

threat of war is everywhere. And war is everyone's enemy. In case of

war, regardless of whether unintentionally initiated by technology

allegedly designed to avert war, or by so-called statesmen or women

who thought it their duty to push the button, you may die ten minutes

earlier than we in fortress America, but we shall all die.

But we have no holes in our streets for atomic land mines that

are intended to delay Soviet tank regiments. We see our missile silos

only now and then -- that is, only whenever it pleases someone to show

them to us on television. No matter how passionately our government

tries to convince us that the nasty Soviets are effectively as near to

us as to Europeans, that they threaten us from Cuba and Nicaragua,

Americans are, on the whole, quite unconvinced and untroubled by such

efforts. The American experience of war has allowed us to develop an

"it can't happen here" attitude, rather than a concrete fear of what

appears to be far removed from the immediate concerns of daily life.

We know that it is emotionally impossible for anyone to live for

very long in the face of immediate threats to existence without

bringing to bear psychological mechanisms that will exclude these

dangers from consciousness, permitting them to surface only rarely.

But when repression necessitates systematically misdirected efforts,

or excludes potentially life-saving behavior, then it is time to

replace it with a conscious effort to find the prod to correct action.

That time has come for computer professionals. We now have the

power radically to turn the state of the world in directions conducive

to life.

In order to gain the necessary courage (not all of us are saints

or heroes) we have to understand that for us as individuals, as well

as for those we love, our present behavior is far more dangerous, even

life threatening, than what healthy common sense now demands of us.

None of the weapons that today threaten every human being with murder,

and whose design, manufacture and sale condemns countless people to

starvation, could be developed without the earnest cooperation of

computer professionals. Without us, the arms race, especially the

qualitative arms race, cannot march another step.

What does this say to us?

First, that we computer experts -- as well as specialists in many

other technical domains -- share in the guilt of having brought about

the present dangerous state of the world. Those among us who, perhaps

without being aware of it, devote our talents and strengths to death

rather than to life have little right to curse politicians, statesmen

and women for not bringing us peace. It isn't enough to make pretty

posters that can be carried in demonstrations. Those who carry them

must care whether their daily work helps to make possible the very

devices the use of which they are protesting.

At this point, the domain called Artificial Intelligence (AI)

comes especially to mind. Many of the technical tasks and problems in

this subdiscipline of computer science stimulate the imagination and

creativity of technically oriented workers particularly strongly.

Goals like making a thinking being out of the computer, giving the

computer the ability to understand spoken language, making it possible

for the computer to see, offer nearly irresistible temptations to

those among us who have not fully sublimated our playful sandbox

fantasies, or who mean to satisfy our delusions of omnipotence on the

computer stage. Such tasks are extraordinarily demanding and

interesting. Robert Oppenheimer called them sweet. Besides, research

projects in these areas are generously funded. The required moneys

usually come out of the coffers of the military, at least in America.

It is enormously tempting and, in Artificial Intelligence work,

seductively simple to lose or hide oneself in details, in subproblems

and their subproblems and so on. The actual problems on which one

works -- and which are so generously supported -- are disguised and

transformed until their representations are mere fables: harmless,

innocent, lovely fairy tales.

Here is an example. A doctoral student characterized his

projected dissertation task as follows. A child, six or seven years

old, sits in front of a computer display that shows a kitten and a

bear, in full color. The kitten is playing with a ball. The child

speaks to the computer system: "The bear should say 'thank you' when

someone gives him something." The system responds in a synthetic, but

nevertheless pleasing voice: "Thank you, I understand." Then the

child again: "Kitty, give your ball to your friend." Immediately we

see the kitten on the computer display throw the ball to the bear.

Then we hear the bear say: "Thank you, my dear kitten."

This is the kernel of what the system, development of which is to

constitute the student's doctoral work, is to accomplish. Seen from a

technical point of view, the system is to understand spoken

instructions -- that alone is not simple -- and translate them into a

computer program which it is then to integrate seamlessly into its own

computational structure. Not at all trivial, and beyond that, quite

touching.

Now a translation to reality. A fighter pilot is addressed by

his pilot's assistant system: "Sir, I see an enemy tank column below.

Your orders, please." The pilot: "When you see something like that,

don't bother me, destroy the bastards and record the action. That's

all." The system answers: "Yes, sir!" and the plane's rockets fly

earthward.

This pilot's assistant system is one of three weapons systems

that are expressly described, mainly as a problem for artificial

intelligence, in the Strategic Computing Initiative, a new major

research and development program of the American military. Over

$600,000,000 are to be spent on this program in the next four or five

years.

It isn't my intention to assail or revile military systems at

this point. I intend this example from the actual practice of

academic artificial intelligence research in America to illustrate the

euphemistic linguistic dissimulation whose effect it is to hinder

thought and, ultimately, to still conscience.

I don't know whether it is especially computer science or its

subdiscipline Artificial Intelligence that has such an enormous

affection for euphemism. We speak so readily of computer systems that

understand, that see, decide, make judgments, and so on, without

ourselves recognizing our own superficiality and immeasurable naivete

with respect to these concepts. We anesthetize our ability to

evaluate the quality of our work and, what is more important, to

identify and become conscious of its end use.

The student mentioned above imagines his work to be about

computer games for children, involving perhaps toy kittens, bears and

balls. Its actual and intended end use will probably mean that some

day a young man, quite like the student himself -- someone with

parents and possibly a girl friend -- will be set afire by an

exploding missile sent his way by a system shaped by his own research.

The psychological distance between the student's conception of his

work and its actual implications is astronomic. It is precisely that

enormous distance that makes it possible not to know and not to ask if

one is doing sensible work or contributing to the greater efficiency

of murderous devices.

One can't escape this state without asking, again and again:

"What do I actually do? What is the final application and use of my

work? Am I content or ashamed to have contributed to this use?"

I am reminded in this context of a well known American journalist

who, during a Middle East highjacking, suggested that under certain

circumstances the Israelis shoot ten Arab prisoners and, should the

circumstances not change, shoot ten more the next day, and so on. He

should not have made this suggestion unless he was prepared to go

personally among the prisoners and look into the eyes of the men, some

of whom would hear him say, "You, you will die today." He should have

been prepared as well to hold the pistol to the heads of those he

selected and command his own finger to pull the trigger.

Just so should we ask ourselves about our own work. Once we have

abandoned the prettifying of our language, we can begin to speak among

ourselves realistically and in earnest about our work as computer

professionals.

"You, colleague of many years, you are working on a machine

consisting of two to the fifteenth and more microprocessors running

simultaneously. With the help of such a machine one can first

simulate then construct much more efficient, smaller and lighter

hydrogen bombs. Imagine, for a moment, you were an eyewitness at

Hiroshima in 1945; you saw people stripped of their skin die. Would

you want to make this happen thousands of times more? Would you so

torture a single human being with your own hands? If you would not,

regardless of what end would be served, then you must stop your work."

One should ask similar questions with respect to other branches

of computer science, for example, with respect to attempts to make it

possible for computer systems to see. Progress in this domain will be

used to steer missiles like the Cruise and Pershing ever more

precisely to their targets, where murder will be committed.

Many will argue that the computer is merely a tool. As such it

can be used for good or evil. In and of itself, it is value free.

Scientists and technicians cannot know how the products of their work

will be applied, whether they will find a good or an evil use. Hence

scientists and technicians cannot be held responsible for the final

application of their work.

That point of view is manifested in the world famous Draper

Laboratory, next door to the MIT building where I work. Draper is

devoted almost entirely to missile guidance and submarine navigation.

Many of the scientists employed there argue that the systems they work

on can take men to the moon and bring them back, as well as guarantee

that missiles aimed at Moscow will actually hit Moscow, their target.

They cannot know in advance, they say, which of these two or still

other goals their work will serve in the end. How then can they be

held responsible for all the possible consequences of their work?

So it is, on the whole, with computer professionals. The

doctoral student I mentioned, who wishes to be able to converse with

his computer display, does in fact believe that future applications of

his work will be exclusively in innocent applications like children's

games. Perhaps his research is not sponsored by the Pentagon's

Strategic Computing Initiative; perhaps he never even heard of SCI.

How then can he be held responsible if his work is put to anti-human

use?

Here is where we come to the essence of the matter. Today we

know with virtual certainty that every scientific and technical result

will, if at all possible, be put to use in military systems.

The computer, together with the history of its development, is

perhaps the key example. But we should also think in this connection

of everything that has to do with flight, or of things atomic, of

communications systems, satellites, space ships, and most of the

scientific achievements of the human genius. We may then convince

ourselves that in the concrete world in which we live, the burden of

proof rests with those who assert that a specific new development is

immune from the greed of the military.

In these circumstances, scientific and technical workers cannot

escape their responsibility to inquire about the end use of their

work. They must then decide, once they know to what end it will be

used, whether or not they would serve these ends with their own hands.

I don't believe the military, in and of itself, to be an evil.

Nor would I assert that the fact that a specific technology that has

been adopted by the military is, on that ground alone, an evil. In

the present state of the evolution of the sovereign nation-state -- in

other words, in the insane asylum in which we live -- each state needs

a military just as every city needs a fire department. But no one

pleads for a fire station on every corner, and no one wishes for a

city fire department that makes a side business of committing arson in

the villages adjacent to the city.

But we see our entire world, particularly its universities and

science and engineering facilities, being more profoundly militarized

every day. "Little" wars burn in almost every part of the earth.

(They serve, in part, to test the high tech weapons of the "more

advanced nations.") More than half of all the earth's scientists and

engineers work more or less directly in military institutions, or in

institutions supported by the military. That is an evil that must be

resisted.

We must also recognize that it is only our already internalized

habit of prettifying our language, in order not to arouse our

conscience, that permits us to speak in terms of weapons and weapons

delivery systems at all, when we are, in fact, discussing atomic

explosives and hydrogen bombs. Those aren't weapons, they are mass

murder machines and mass murder machine delivery systems. That is how

we should speak of them: clearly, distinctly, and without evasion.

Once we recognize that a nuclear mass murder machine is nothing other

than an instant Auschwitz -- without railroads or Eichmanns or Dr.

Mengele, but an Auschwitz just the same -- can we continue then to

work on systems that steer these devices to living cities?

That is the question I ask. Each of us must earnestly ask

ourselves such questions and deeply consider the responses we find in

ourselves. Our answers must finally manifest themselves in our

actions -- concretely, in what we do every day.

Probably the most pandemic mental illness of our time is the

almost universally held belief that the individual is powerless. This

self-fulfilling delusion will surely be offered as a counter argument

to my theses. I demand, do I not, that a whole profession refuse to

participate in the murderous insanity of our time. "That cannot be

effective," I can already hear it said," That is plainly impossible.

After all, if I don't do it, someone else will." First, and on the

most elementary level, "If I don't do it, someone else will" cannot

serve as a basis of moral behavior. Every crime imaginable can be

justified with those words. For example: If I don't steal the

sleeping drunk's money, someone else will. But it is not at all

trivial to ask after the meaning of effectiveness in the present

context. Surely, effectiveness is not a binary matter, an either/or

matter. To be sure, if what I say here were to induce a strike on the

part of all scientists with respect to weapons work, that would have

to be counted as effective. But there are many much more modest

measures of effectiveness.

I think it was George Orwell who once wrote, "The highest duty of

intellectuals in these times is to speak the simplest truths in the

simplest possible words." For me that means, first of all, to

articulate the absurdity of our work in my actions, my writings and

with my voice. I hope thereby to stir my students, my colleagues,

everyone to whom I can speak directly. I hope to encourage those who

have already begun to think similarly, and to be encouraged by them,

and possibly rouse others out of their slumber. Courage, like fear is

catching.

Even the most modest success in such attempts has to be counted

as effective. Beyond that, in speaking as I do, I put what I discuss

here on the public agenda and contribute to its legitimation. These

are modest goals that can surely be reached.

But, finally, I want to address such larger goals as, for

example:

Ridding the world of nuclear mass murder devices and perhaps also

of nuclear power generators.

So reordering the world that it becomes impossible ever again to

convince workers of one country that it is a necessity of life that

they feed their families on the flesh and the blood and the tears of

people of other countries. (That is, unfortunately, the fate of many

workers today, and not only those who earn their daily bread in

armaments factories, but equally those of us whose daily work is to

sharpen high tech weapons.)

So reordering the world that every human being has available to

him or herself all material goods necessary for living in dignity. (I

have often heard well-meaning people say that, if we apply technology,

especially computer and communications technology wisely, we may reach

this goal in perhaps 50 to 100 years. But we can reach it sooner, and

without waiting for technological advances. For the obstacle is not

the absence of technology, it is the absence of political will.)

I once heard Elie Wiesel say: "We must believe the impossible is

possible." I understood that in two different ways. First, had we

been able to believe that "the land of the poets and the thinkers"

could give birth to human extermination factories, we might not have

had to experience Bergen Belsen. The impossible horror proved

possible and became reality.

But there is a more hopeful interpretation. It seemed impossible

in the America of only 150 years ago ever to abolish the slavery of

the black people. The entire economy of America's south was built on

cotton. Cotton could neither be planted nor harvested, it was

believed, without the unpaid toil of thousands of human beings out of

whose wretchedness the plantation master could squeeze his profit.

Nevertheless, at first only a few farseeing men and women, dreamers

all, in Massachusetts, later many more citizens, came to believe the

impossible was possible, that the slaves could be freed and slavery

ended.

The impossible goals I mention here are possible, just as it is

possible that we will destroy the human race. I alone can neither

achieve the one nor prevent the other. But neither can it be done

without me, without us.

I have no right to demand anything from my colleagues. But they

must know that we have the power either to increase the efficiency of

the mass murder instruments we have and thereby make the murder of our

children more likely, or to bring the present insanity to a halt, so

that we and our children have a chance to live in human dignity.

Let us think about what we actually accomplish in our work, about

how it will be used, and whether we are in the service of life or

death.

Computers for Christ - Chicago

---

Index of Articles of Interest    Home

  1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 317 | 318 | 319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333