Go to the previous, next section.

Crashing The System

A Brief History of Telephony / Bell's Golden Vaporware / Universal Service / Wild Boys and Wire Women / The Electronic Communities / The Ungentle Giant / The Breakup / In Defense of the System / The Crash PostMortem / Landslides in Cyberspace

On January 15, 1990, AT&T's long-distance telephone switching system crashed.

This was a strange, dire, huge event. Sixty thousand people lost their telephone service completely. During the nine long hours of frantic effort that it took to restore service, some seventy million telephone calls went uncompleted.

Losses of service, known as "outages" in the telco trade, are a known and accepted hazard of the telephone business. Hurricanes hit, and phone cables get snapped by the thousands. Earthquakes wrench through buried fiber-optic lines. Switching stations catch fire and burn to the ground. These things do happen. There are contingency plans for them, and decades of experience in dealing with them. But the Crash of January 15 was unprecedented. It was unbelievably huge, and it occurred for no apparent physical reason.

The crash started on a Monday afternoon in a single switching-station in Manhattan. But, unlike any merely physical damage, it spread and spread. Station after station across America collapsed in a chain reaction, until fully half of AT&T's network had gone haywire and the remaining half was hard-put to handle the overflow.

Within nine hours, AT&T software engineers more or less understood what had caused the crash. Replicating the problem exactly, poring over software line by line, took them a couple of weeks. But because it was hard to understand technically, the full truth of the matter and its implications were not widely and thoroughly aired and explained. The root cause of the crash remained obscure, surrounded by rumor and fear. The crash was a grave corporate embarrassment. The "culprit" was a bug in AT&T's own software -- not the sort of admission the telecommunications giant wanted to make, especially in the face of increasing competition. Still, the truth was told, in the baffling technical terms necessary to explain it.

Somehow the explanation failed to persuade American law enforcement officials and even telephone corporate security personnel. These people were not technical experts or software wizards, and they had their own suspicions about the cause of this disaster.

The police and telco security had important sources of information denied to mere software engineers. They had informants in the computer underground and years of experience in dealing with high-tech rascality that seemed to grow ever more sophisticated. For years they had been expecting a direct and savage attack against the American national telephone system. And with the Crash of January 15 -- the first month of a new, high-tech decade -- their predictions, fears, and suspicions seemed at last to have entered the real world. A world where the telephone system had not merely crashed, but, quite likely, been crashed -- by "hackers."

The crash created a large dark cloud of suspicion that would color certain people's assumptions and actions for months. The fact that it took place in the realm of software was suspicious on its face. The fact that it occurred on Martin Luther King Day, still the most politically touchy of American holidays, made it more suspicious yet.

The Crash of January 15 gave the Hacker Crackdown its sense of edge and its sweaty urgency. It made people, powerful people in positions of public authority, willing to believe the worst. And, most fatally, it helped to give investigators a willingness to take extreme measures and the determination to preserve almost total secrecy. An obscure software fault in an aging switching system in New York was to lead to a chain reaction of legal and constitutional trouble all across the country.

#

Like the crash in the telephone system, this chain reaction was ready and waiting to happen. During the 1980s, the American legal system was extensively patched to deal with the novel issues of computer crime. There was, for instance, the Electronic Communications Privacy Act of 1986 (eloquently described as "a stinking mess" by a prominent law enforcement official). And there was the draconian Computer Fraud and Abuse Act of 1986, passed unanimously by the United States Senate, which later would reveal a large number of flaws. Extensive, wellmeant efforts had been made to keep the legal system up to date. But in the day-to-day grind of the real world, even the most elegant software tends to crumble and suddenly reveal its hidden bugs.

Like the advancing telephone system, the American legal system was certainly not ruined by its temporary crash; but for those caught under the weight of the collapsing system, life became a series of blackouts and anomalies.

In order to understand why these weird events occurred, both in the world of technology and in the world of law, it's not enough to understand the merely technical problems. We will get to those; but first and foremost, we must try to understand the telephone, and the business of telephones, and the community of human beings that telephones have created.

#

Technologies have life cycles, like cities do, like institutions do, like laws and governments do.

The first stage of any technology is the Question Mark, often known as the "Golden Vaporware" stage. At this early point, the technology is only a phantom, a mere gleam in the inventor's eye. One such inventor was a speech teacher and electrical tinkerer named Alexander Graham Bell.

Bell's early inventions, while ingenious, failed to move the world. In 1863, the teenage Bell and his brother Melville made an artificial talking mechanism out of wood, rubber, gutta-percha, and tin. This weird device had a rubber-covered "tongue" made of movable wooden segments, with vibrating rubber "vocal cords," and rubber "lips" and "cheeks." While Melville puffed a bellows into a tin tube, imitating the lungs, young Alec Bell would manipulate the "lips," "teeth," and "tongue," causing the thing to emit high-pitched falsetto gibberish.

Another would-be technical breakthrough was the Bell "phonautograph" of 1874, actually made out of a human cadaver's ear. Clamped into place on a tripod, this grisly gadget drew sound-wave images on smoked glass through a thin straw glued to its vibrating earbones.

By 1875, Bell had learned to produce audible sounds -- ugly shrieks and squawks -- by using magnets, diaphragms, and electrical current. Most "Golden Vaporware" technologies go nowhere.

But the second stage of technology is the Rising Star, or, the "Goofy Prototype," stage. The telephone, Bell's most ambitious gadget yet, reached this stage on March 10, 1876. On that great day, Alexander Graham Bell became the first person to transmit intelligible human speech electrically. As it happened, young Professor Bell, industriously tinkering in his Boston lab, had spattered his trousers with acid. His assistant, Mr. Watson, heard his cry for help -- over Bell's experimental audiotelegraph. This was an event without precedent.

Technologies in their "Goofy Prototype" stage rarely work very well. They're experimental, and therefore halfbaked and rather frazzled. The prototype may be attractive and novel, and it does look as if it ought to be good for something-or-other. But nobody, including the inventor, is quite sure what. Inventors, and speculators, and pundits may have very firm ideas about its potential use, but those ideas are often very wrong.

The natural habitat of the Goofy Prototype is in trade shows and in the popular press. Infant technologies need publicity and investment money like a tottering calf need milk. This was very true of Bell's machine. To raise research and development money, Bell toured with his device as a stage attraction.

Contemporary press reports of the stage debut of the telephone showed pleased astonishment mixed with considerable dread. Bell's stage telephone was a large wooden box with a crude speaker-nozzle, the whole contraption about the size and shape of an overgrown Brownie camera. Its buzzing steel soundplate, pumped up by powerful electromagnets, was loud enough to fill an auditorium. Bell's assistant Mr. Watson, who could manage on the keyboards fairly well, kicked in by playing the organ from distant rooms, and, later, distant cities. This feat was considered marvellous, but very eerie indeed.

Bell's original notion for the telephone, an idea promoted for a couple of years, was that it would become a mass medium. We might recognize Bell's idea today as something close to modern "cable radio." Telephones at a central source would transmit music, Sunday sermons, and important public speeches to a paying network of wired-up subscribers.

At the time, most people thought this notion made good sense. In fact, Bell's idea was workable. In Hungary, this philosophy of the telephone was successfully put into everyday practice. In Budapest, for decades, from 1893 until after World War I, there was a government-run information service called "Telefon Hirmondo(C)." Hirmondo(C) was a centralized source of news and entertainment and culture, including stock reports, plays, concerts, and novels read aloud. At certain hours of the day, the phone would ring, you would plug in a loudspeaker for the use of the family, and Telefon Hirmondo(C) would be on the air -- or rather, on the phone.

Hirmondo(C) is dead tech today, but Hirmondo(C) might be considered a spiritual ancestor of the modern telephone-accessed computer data services, such as CompuServe, GEnie or Prodigy. The principle behind Hirmondo(C) is also not too far from computer "bulletin board systems" or BBS's, which arrived in the late 1970s, spread rapidly across America, and will figure largely in this book.

We are used to using telephones for individual person-to-person speech, because we are used to the Bell system. But this was just one possibility among many. Communication networks are very flexible and protean, especially when their hardware becomes sufficiently advanced. They can be put to all kinds of uses. And they have been -- and they will be.

Bell's telephone was bound for glory, but this was a combination of political decisions, canny infighting in court, inspired industrial leadership, receptive local conditions and outright good luck. Much the same is true of communications systems today.

As Bell and his backers struggled to install their newfangled system in the real world of nineteenth-century New England, they had to fight against skepticism and industrial rivalry. There was already a strong electrical communications network present in America: the telegraph. The head of the Western Union telegraph system dismissed Bell's prototype as "an electrical toy" and refused to buy the rights to Bell's patent. The telephone, it seemed, might be all right as a parlor entertainment -- but not for serious business.

Telegrams, unlike mere telephones, left a permanent physical record of their messages. Telegrams, unlike telephones, could be answered whenever the recipient had time and convenience. And the telegram had a much longer distance-range than Bell's early telephone. These factors made telegraphy seem a much more sound and businesslike technology -- at least to some.

The telegraph system was huge, and well-entrenched. In 1876, the United States had 214,000 miles of telegraph wire, and 8500 telegraph offices. There were specialized telegraphs for businesses and stock traders, government, police and fire departments. And Bell's "toy" was best known as a stage-magic musical device.

The third stage of technology is known as the "Cash Cow" stage. In the "cash cow" stage, a technology finds its place in the world, and matures, and becomes settled and productive. After a year or so, Alexander Graham Bell and his capitalist backers concluded that eerie music piped from nineteenth-century cyberspace was not the real selling-point of his invention. Instead, the telephone was about speech -- individual, personal speech, the human voice, human conversation and human interaction. The telephone was not to be managed from any centralized broadcast center. It was to be a personal, intimate technology.

When you picked up a telephone, you were not absorbing the cold output of a machine -- you were speaking to another human being. Once people realized this, their instinctive dread of the telephone as an eerie, unnatural device, swiftly vanished. A "telephone call" was not a "call" from a "telephone" itself, but a call from another human being, someone you would generally know and recognize. The real point was not what the machine could do for you (or to you), but what you yourself, a person and citizen, could do through the machine. This decision on the part of the young Bell Company was absolutely vital.

The first telephone networks went up around Boston -- mostly among the technically curious and the well-to-do (much the same segment of the American populace that, a hundred years later, would be buying personal computers). Entrenched backers of the telegraph continued to scoff.

But in January 1878, a disaster made the telephone famous. A train crashed in Tarriffville, Connecticut. Forward-looking doctors in the nearby city of Hartford had had Bell's "speaking telephone" installed. An alert local druggist was able to telephone an entire community of local doctors, who rushed to the site to give aid. The disaster, as disasters do, aroused intense press coverage. The phone had proven its usefulness in the real world.

After Tarriffville, the telephone network spread like crabgrass. By 1890 it was all over New England. By '93, out to Chicago. By '97, into Minnesota, Nebraska and Texas. By 1904 it was all over the continent.

The telephone had become a mature technology. Professor Bell (now generally known as "Dr. Bell" despite his lack of a formal degree) became quite wealthy. He lost interest in the tedious day-to-day business muddle of the booming telephone network, and gratefully returned his attention to creatively hacking-around in his various laboratories, which were now much larger, better ventilated, and gratifyingly better-equipped. Bell was never to have another great inventive success, though his speculations and prototypes anticipated fiber-optic transmission, manned flight, sonar, hydrofoil ships, tetrahedral construction, and Montessori education. The "decibel," the standard scientific measure of sound intensity, was named after Bell.

Not all Bell's vaporware notions were inspired. He was fascinated by human eugenics. He also spent many years developing a weird personal system of astrophysics in which gravity did not exist.

Bell was a definite eccentric. He was something of a hypochondriac, and throughout his life he habitually stayed up until four A.M., refusing to rise before noon. But Bell had accomplished a great feat; he was an idol of millions and his influence, wealth, and great personal charm, combined with his eccentricity, made him something of a loose cannon on deck. Bell maintained a thriving scientific salon in his winter mansion in Washington, D.C., which gave him considerable backstage influence in governmental and scientific circles. He was a major financial backer of the the magazines Science and National Geographic, both still flourishing today as important organs of the American scientific establishment. Bell's companion Thomas Watson, similarly wealthy and similarly odd, became the ardent political disciple of a 19th-century science-fiction writer and would-be social reformer, Edward Bellamy. Watson also trod the boards briefly as a Shakespearian actor.

There would never be another Alexander Graham Bell, but in years to come there would be surprising numbers of people like him. Bell was a prototype of the high-tech entrepreneur. High-tech entrepreneurs will play a very prominent role in this book: not merely as technicians and businessmen, but as pioneers of the technical frontier, who can carry the power and prestige they derive from high-technology into the political and social arena.

Like later entrepreneurs, Bell was fierce in defense of his own technological territory. As the telephone began to flourish, Bell was soon involved in violent lawsuits in the defense of his patents. Bell's Boston lawyers were excellent, however, and Bell himself, as an elecution teacher and gifted public speaker, was a devastatingly effective legal witness. In the eighteen years of Bell's patents, the Bell company was involved in six hundred separate lawsuits. The legal records printed filled 149 volumes. The Bell Company won every single suit.

After Bell's exclusive patents expired, rival telephone companies sprang up all over America. Bell's company, American Bell Telephone, was soon in deep trouble. In 1907, American Bell Telephone fell into the hands of the rather sinister J.P. Morgan financial cartel, robber-baron speculators who dominated Wall Street.

At this point, history might have taken a different turn. American might well have been served forever by a patchwork of locally owned telephone companies. Many state politicians and local businessmen considered this an excellent solution.

But the new Bell holding company, American Telephone and Telegraph or AT&T, put in a new man at the helm, a visionary industrialist named Theodore Vail. Vail, a former Post Office manager, understood large organizations and had an innate feeling for the nature of large-scale communications. Vail quickly saw to it that AT&T seized the technological edge once again. The Pupin and Campbell "loading coil," and the deForest "audion," are both extinct technology today, but in 1913 they gave Vail's company the best long-distance lines ever built. By controlling long-distance -- the links between, and over, and above the smaller local phone companies -- AT&T swiftly gained the whip-hand over them, and was soon devouring them right and left.

Vail plowed the profits back into research and development, starting the Bell tradition of huge-scale and brilliant industrial research.

Technically and financially, AT&T gradually steamrollered the opposition. Independent telephone companies never became entirely extinct, and hundreds of them flourish today. But Vail's AT&T became the supreme communications company. At one point, Vail's AT&T bought Western Union itself, the very company that had derided Bell's telephone as a "toy." Vail thoroughly reformed Western Union's hidebound business along his modern principles; but when the federal government grew anxious at this centralization of power, Vail politely gave Western Union back.

This centralizing process was not unique. Very similar events had happened in American steel, oil, and railroads. But AT&T, unlike the other companies, was to remain supreme. The monopoly robber-barons of those other industries were humbled and shattered by government trust-busting. Vail, the former Post Office official, was quite willing to accommodate the US government; in fact he would forge an active alliance with it. AT&T would become almost a wing of the American government, almost another Post Office -- though not quite. AT&T would willingly submit to federal regulation, but in return, it would use the government's regulators as its own police, who would keep out competitors and assure the Bell system's profits and preeminence.

This was the second birth -- the political birth -- of the American telephone system. Vail's arrangement was to persist, with vast success, for many decades, until 1982. His system was an odd kind of American industrial socialism. It was born at about the same time as Leninist Communism, and it lasted almost as long -- and, it must be admitted, to considerably better effect.

Vail's system worked. Except perhaps for aerospace, there has been no technology more thoroughly dominated by Americans than the telephone. The telephone was seen from the beginning as a quintessentially American technology. Bell's policy, and the policy of Theodore Vail, was a profoundly democratic policy of universal access. Vail's famous corporate slogan, "One Policy, One System, Universal Service," was a political slogan, with a very American ring to it. The American telephone was not to become the specialized tool of government or business, but a general public utility. At first, it was true, only the wealthy could afford private telephones, and Bell's company pursued the business markets primarily. The American phone system was a capitalist effort, meant to make money; it was not a charity. But from the first, almost all communities with telephone service had public telephones. And many stores -- especially drugstores -- offered public use of their phones. You might not own a telephone -- but you could always get into the system, if you really needed to.

There was nothing inevitable about this decision to make telephones "public" and "universal." Vail's system involved a profound act of trust in the public. This decision was a political one, informed by the basic values of the American republic. The situation might have been very different; and in other countries, under other systems, it certainly was.

Joseph Stalin, for instance, vetoed plans for a Soviet phone system soon after the Bolshevik revolution. Stalin was certain that publicly accessible telephones would become instruments of anti-Soviet counterrevolution and conspiracy. (He was probably right.) When telephones did arrive in the Soviet Union, they would be instruments of Party authority, and always heavily tapped. (Alexander Solzhenitsyn's prison-camp novel The First Circle describes efforts to develop a phone system more suited to Stalinist purposes.)

France, with its tradition of rational centralized government, had fought bitterly even against the electric telegraph, which seemed to the French entirely too anarchical and frivolous. For decades, nineteenth-century France communicated via the "visual telegraph," a nation-spanning, government-owned semaphore system of huge stone towers that signalled from hilltops, across vast distances, with big windmill-like arms. In 1846, one Dr. Barbay, a semaphore enthusiast, memorably uttered an early version of what might be called "the security expert's argument" against the open media.

"No, the electric telegraph is not a sound invention. It will always be at the mercy of the slightest disruption, wild youths, drunkards, bums, etc... The electric telegraph meets those destructive elements with only a few meters of wire over which supervision is impossible. A single man could, without being seen, cut the telegraph wires leading to Paris, and in twenty-four hours cut in ten different places the wires of the same line, without being arrested. The visual telegraph, on the contrary, has its towers, its high walls, its gates well-guarded from inside by strong armed men. Yes, I declare, substitution of the electric telegraph for the visual one is a dreadful measure, a truly idiotic act."

Dr. Barbay and his high-security stone machines were eventually unsuccessful, but his argument -- that communication exists for the safety and convenience of the state, and must be carefully protected from the wild boys and the gutter rabble who might want to crash the system -- would be heard again and again.

When the French telephone system finally did arrive, its snarled inadequacy was to be notorious. Devotees of the American Bell System often recommended a trip to France, for skeptics.

In Edwardian Britain, issues of class and privacy were a ball-and-chain for telephonic progress. It was considered outrageous that anyone -- any wild fool off the street -- could simply barge bellowing into one's office or home, preceded only by the ringing of a telephone bell. In Britain, phones were tolerated for the use of business, but private phones tended be stuffed away into closets, smoking rooms, or servants' quarters. Telephone operators were resented in Britain because they did not seem to "know their place." And no one of breeding would print a telephone number on a business card; this seemed a crass attempt to make the acquaintance of strangers.

But phone access in America was to become a popular right; something like universal suffrage, only more so. American women could not yet vote when the phone system came through; yet from the beginning American women doted on the telephone. This "feminization" of the American telephone was often commented on by foreigners. Phones in America were not censored or stiff or formalized; they were social, private, intimate, and domestic. In America, Mother's Day is by far the busiest day of the year for the phone network.

The early telephone companies, and especially AT&T, were among the foremost employers of American women. They employed the daughters of the American middle-class in great armies: in 1891, eight thousand women; by 1946, almost a quarter of a million. Women seemed to enjoy telephone work; it was respectable, it was steady, it paid fairly well as women's work went, and -- not least -- it seemed a genuine contribution to the social good of the community. Women found Vail's ideal of public service attractive. This was especially true in rural areas, where women operators, running extensive rural partylines, enjoyed considerable social power. The operator knew everyone on the party-line, and everyone knew her.

Although Bell himself was an ardent suffragist, the telephone company did not employ women for the sake of advancing female liberation. AT&T did this for sound commercial reasons. The first telephone operators of the Bell system were not women, but teenage American boys. They were telegraphic messenger boys (a group about to be rendered technically obsolescent), who swept up around the phone office, dunned customers for bills, and made phone connections on the switchboard, all on the cheap.

Within the very first year of operation, 1878, Bell's company learned a sharp lesson about combining teenage boys and telephone switchboards. Putting teenage boys in charge of the phone system brought swift and consistent disaster. Bell's chief engineer described them as "Wild Indians." The boys were openly rude to customers. They talked back to subscribers, saucing off, uttering facetious remarks, and generally giving lip. The rascals took Saint Patrick's Day off without permission. And worst of all they played clever tricks with the switchboard plugs: disconnecting calls, crossing lines so that customers found themselves talking to strangers, and so forth.

This combination of power, technical mastery, and effective anonymity seemed to act like catnip on teenage boys.

This wild-kid-on-the-wires phenomenon was not confined to the USA; from the beginning, the same was true of the British phone system. An early British commentator kindly remarked: "No doubt boys in their teens found the work not a little irksome, and it is also highly probable that under the early conditions of employment the adventurous and inquisitive spirits of which the average healthy boy of that age is possessed, were not always conducive to the best attention being given to the wants of the telephone subscribers."

So the boys were flung off the system -- or at least, deprived of control of the switchboard. But the "adventurous and inquisitive spirits" of the teenage boys would be heard from in the world of telephony, again and again.

The fourth stage in the technological life-cycle is death: "the Dog," dead tech. The telephone has so far avoided this fate. On the contrary, it is thriving, still spreading, still evolving, and at increasing speed.

The telephone has achieved a rare and exalted state for a technological artifact: it has become a household object. The telephone, like the clock, like pen and paper, like kitchen utensils and running water, has become a technology that is visible only by its absence. The telephone is technologically transparent. The global telephone system is the largest and most complex machine in the world, yet it is easy to use. More remarkable yet, the telephone is almost entirely physically safe for the user.

For the average citizen in the 1870s, the telephone was weirder, more shocking, more "high-tech" and harder to comprehend, than the most outrageous stunts of advanced computing for us Americans in the 1990s. In trying to understand what is happening to us today, with our bulletin board systems, direct overseas dialling, fiberoptic transmissions, computer viruses, hacking stunts, and a vivid tangle of new laws and new crimes, it is important to realize that our society has been through a similar challenge before -- and that, all in all, we did rather well by it.

Bell's stage telephone seemed bizarre at first. But the sensations of weirdness vanished quickly, once people began to hear the familiar voices of relatives and friends, in their own homes on their own telephones. The telephone changed from a fearsome high-tech totem to an everyday pillar of human community.

This has also happened, and is still happening, to computer networks. Computer networks such as NSFnet, BITnet, USENET, JANET, are technically advanced, intimidating, and much harder to use than telephones. Even the popular, commercial computer networks, such as GEnie, Prodigy, and CompuServe, cause much head-scratching and have been described as "user-hateful." Nevertheless they too are changing from fancy high-tech items into everyday sources of human community.

The words "community" and "communication" have the same root. Wherever you put a communications network, you put a community as well. And whenever you take away that network -- confiscate it, outlaw it, crash it, raise its price beyond affordability -- then you hurt that community.

Communities will fight to defend themselves. People will fight harder and more bitterly to defend their communities, than they will fight to defend their own individual selves. And this is very true of the "electronic community" that arose around computer networks in the 1980s -- or rather, the various electronic communities, in telephony, law enforcement, computing, and the digital underground that, by the year 1990, were raiding, rallying, arresting, suing, jailing, fining and issuing angry manifestos.

None of the events of 1990 were entirely new. Nothing happened in 1990 that did not have some kind of earlier and more understandable precedent. What gave the Hacker Crackdown its new sense of gravity and importance was the feeling -- the community feeling - that the political stakes had been raised; that trouble in cyberspace was no longer mere mischief or inconclusive skirmishing, but a genuine fight over genuine issues, a fight for community survival and the shape of the future. These electronic communities, having flourished throughout the 1980s, were becoming aware of themselves, and increasingly, becoming aware of other, rival communities. Worries were sprouting up right and left, with complaints, rumors, uneasy speculations. But it would take a catalyst, a shock, to make the new world evident. Like Bell's great publicity break, the Tarriffville Rail Disaster of January 1878, it would take a cause celebre.

That cause was the AT&T Crash of January 15, 1990. After the Crash, the wounded and anxious telephone community would come out fighting hard.

#

The community of telephone technicians, engineers, operators and researchers is the oldest community in cyberspace. These are the veterans, the most developed group, the richest, the most respectable, in most ways the most powerful. Whole generations have come and gone since Alexander Graham Bell's day, but the community he founded survives; people work for the phone system today whose great-grandparents worked for the phone system. Its specialty magazines, such as Telephony, AT&T Technical Journal, Telephone Engineer and Management, are decades old; they make computer publications like Macworld and PC Week look like amateur johnny-come-latelies.

And the phone companies take no back seat in hightechnology, either. Other companies' industrial researchers may have won new markets; but the researchers of Bell Labs have won seven Nobel Prizes. One potent device that Bell Labs originated, the transistor, has created entire groups of industries. Bell Labs are world-famous for generating "a patent a day," and have even made vital discoveries in astronomy, physics and cosmology.

Throughout its seventy-year history, "Ma Bell" was not so much a company as a way of life. Until the cataclysmic divestiture of the 1980s, Ma Bell was perhaps the ultimate maternalist mega-employer. The AT&T corporate image was the "gentle giant," "the voice with a smile," a vaguely socialist-realist world of cleanshaven linemen in shiny helmets and blandly pretty phone-girls in headsets and nylons. Bell System employees were famous as rock-ribbed Kiwanis and Rotary members, Little-League enthusiasts, school-board people.

During the long heyday of Ma Bell, the Bell employee corps were nurtured top-to-bottom on a corporate ethos of public service. There was good money in Bell, but Bell was not about money; Bell used public relations, but never mere marketeering. People went into the Bell System for a good life, and they had a good life. But it was not mere money that led Bell people out in the midst of storms and earthquakes to fight with toppled phone-poles, to wade in flooded manholes, to pull the redeyed graveyard-shift over collapsing switching-systems. The Bell ethic was the electrical equivalent of the postman's: neither rain, nor snow, nor gloom of night would stop these couriers.

It is easy to be cynical about this, as it is easy to be cynical about any political or social system; but cynicism does not change the fact that thousands of people took these ideals very seriously. And some still do.

The Bell ethos was about public service; and that was gratifying; but it was also about private power, and that was gratifying too. As a corporation, Bell was very special. Bell was privileged. Bell had snuggled up close to the state. In fact, Bell was as close to government as you could get in America and still make a whole lot of legitimate money.

But unlike other companies, Bell was above and beyond the vulgar commercial fray. Through its regional operating companies, Bell was omnipresent, local, and intimate, all over America; but the central ivory towers at its corporate heart were the tallest and the ivoriest around.

There were other phone companies in America, to be sure; the so-called independents. Rural cooperatives, mostly; small fry, mostly tolerated, sometimes warred upon.

For many decades, "independent" American phone companies lived in fear and loathing of the official Bell monopoly (or the "Bell Octopus," as Ma Bell's nineteenth-century enemies described her in many angry newspaper manifestos). Some few of these independent entrepreneurs, while legally in the wrong, fought so bitterly against the Octopus that their illegal phone networks were cast into the street by Bell agents and publicly burned.

The pure technical sweetness of the Bell System gave its operators, inventors and engineers a deeply satisfying sense of power and mastery. They had devoted their lives to improving this vast nation-spanning machine; over years, whole human lives, they had watched it improve and grow. It was like a great technological temple. They were an elite, and they knew it -- even if others did not; in fact, they felt even more powerful because others did not understand. The deep attraction of this sensation of elite technical power should never be underestimated. "Technical power" is not for everybody; for many people it simply has no charm at all. But for some people, it becomes the core of their lives. For a few, it is overwhelming, obsessive; it becomes something close to an addiction. People -- especially clever teenage boys whose lives are otherwise mostly powerless and put-upon -- love this sensation of secret power, and are willing to do all sorts of amazing things to achieve it. The technical power of electronics has motivated many strange acts detailed in this book, which would otherwise be inexplicable.

So Bell had power beyond mere capitalism. The Bell service ethos worked, and was often propagandized, in a rather saccharine fashion. Over the decades, people slowly grew tired of this. And then, openly impatient with it. By the early 1980s, Ma Bell was to find herself with scarcely a real friend in the world. Vail's industrial socialism had become hopelessly out-of-fashion politically. Bell would be punished for that. And that punishment would fall harshly upon the people of the telephone community.

#

In 1983, Ma Bell was dismantled by federal court action. The pieces of Bell are now separate corporate entities. The core of the company became AT&T Communications, and also AT&T Industries (formerly Western Electric, Bell's manufacturing arm). AT&T Bell Labs became Bell Communications Research, Bellcore. Then there are the Regional Bell Operating Companies, or RBOCs, pronounced "arbocks."

Bell was a titan and even these regional chunks are gigantic enterprises: Fortune 50 companies with plenty of wealth and power behind them. But the clean lines of "One Policy, One System, Universal Service" have been shattered, apparently forever.

The "One Policy" of the early Reagan Administration was to shatter a system that smacked of noncompetitive socialism. Since that time, there has been no real telephone "policy" on the federal level. Despite the breakup, the remnants of Bell have never been set free to compete in the open marketplace.

The RBOCs are still very heavily regulated, but not from the top. Instead, they struggle politically, economically and legally, in what seems an endless turmoil, in a patchwork of overlapping federal and state jurisdictions. Increasingly, like other major American corporations, the RBOCs are becoming multinational, acquiring important commercial interests in Europe, Latin America, and the Pacific Rim. But this, too, adds to their legal and political predicament.

The people of what used to be Ma Bell are not happy about their fate. They feel ill-used. They might have been grudgingly willing to make a full transition to the free market; to become just companies amid other companies. But this never happened. Instead, AT&T and the RBOCS ("the Baby Bells") feel themselves wrenched from side to side by state regulators, by Congress, by the FCC, and especially by the federal court of Judge Harold Greene, the magistrate who ordered the Bell breakup and who has been the de facto czar of American telecommunications ever since 1983.

Bell people feel that they exist in a kind of paralegal limbo today. They don't understand what's demanded of them. If it's "service," why aren't they treated like a public service? And if it's money, then why aren't they free to compete for it? No one seems to know, really. Those who claim to know keep changing their minds. Nobody in authority seems willing to grasp the nettle for once and all.

Telephone people from other countries are amazed by the American telephone system today. Not that it works so well; for nowadays even the French telephone system works, more or less. They are amazed that the American telephone system still works at all, under these strange conditions.

Bell's "One System" of long-distance service is now only about eighty percent of a system, with the remainder held by Sprint, MCI, and the midget long-distance companies. Ugly wars over dubious corporate practices such as "slamming" (an underhanded method of snitching clients from rivals) break out with some regularity in the realm of long-distance service. The battle to break Bell's long-distance monopoly was long and ugly, and since the breakup the battlefield has not become much prettier. AT&T's famous shame-and-blame advertisements, which emphasized the shoddy work and purported ethical shadiness of their competitors, were much remarked on for their studied psychological cruelty. There is much bad blood in this industry, and much long-treasured resentment. AT&T's post-breakup corporate logo, a striped sphere, is known in the industry as the "Death Star" (a reference from the movie Star Wars, in which the "Death Star" was the spherical hightech fortress of the harsh-breathing imperial ultra-baddie, Darth Vader.) Even AT&T employees are less than thrilled by the Death Star. A popular (though banned) T-shirt among AT&T employees bears the old-fashioned Bell logo of the Bell System, plus the newfangled striped sphere, with the before-and-after comments: "This is your brain -- This is your brain on drugs!" AT&T made a very well-financed and determined effort to break into the personal computer market; it was disastrous, and telco computer experts are derisively known by their competitors as "the pole-climbers." AT&T and the Baby Bell arbocks still seem to have few friends. Under conditions of sharp commercial competition, a crash like that of January 15, 1990 was a major embarrassment to AT&T. It was a direct blow against their much-treasured reputation for reliability. Within days of the crash AT&T's Chief Executive Officer, Bob Allen, officially apologized, in terms of deeply pained humility: "AT&T had a major service disruption last Monday. We didn't live up to our own standards of quality, and we didn't live up to yours. It's as simple as that. And that's not acceptable to us. Or to you... We understand how much people have come to depend upon AT&T service, so our AT&T Bell Laboratories scientists and our network engineers are doing everything possible to guard against a recurrence... We know there's no way to make up for the inconvenience this problem may have caused you."

Mr Allen's "open letter to customers" was printed in lavish ads all over the country: in the Wall Street Journal, USA Today, New York Times, Los Angeles Times, Chicago Tribune, Philadelphia Inquirer, San Francisco Chronicle Examiner, Boston Globe, Dallas Morning News, Detroit Free Press, Washington Post, Houston Chronicle, Cleveland Plain Dealer, Atlanta Journal Constitution, Minneapolis Star Tribune, St. Paul Pioneer Press Dispatch, Seattle Times/Post Intelligencer, Tacoma News Tribune, Miami Herald, Pittsburgh Press, St. Louis Post Dispatch, Denver Post, Phoenix Republic Gazette and Tampa Tribune.

In another press release, AT&T went to some pains to suggest that this "software glitch" might have happened just as easily to MCI, although, in fact, it hadn't. (MCI's switching software was quite different from AT&T's -- though not necessarily any safer.) AT&T also announced their plans to offer a rebate of service on Valentine's Day to make up for the loss during the Crash.

"Every technical resource available, including Bell Labs scientists and engineers, has been devoted to assuring it will not occur again," the public was told. They were further assured that "The chances of a recurrence are small -- a problem of this magnitude never occurred before."

In the meantime, however, police and corporate security maintained their own suspicions about "the chances of recurrence" and the real reason why a "problem of this magnitude" had appeared, seemingly out of nowhere. Police and security knew for a fact that hackers of unprecedented sophistication were illegally entering, and reprogramming, certain digital switching stations. Rumors of hidden "viruses" and secret "logic bombs" in the switches ran rampant in the underground, with much chortling over AT&T's predicament, and idle speculation over what unsung hacker genius was responsible for it. Some hackers, including police informants, were trying hard to finger one another as the true culprits of the Crash.

Telco people found little comfort in objectivity when they contemplated these possibilities. It was just too close to the bone for them; it was embarrassing; it hurt so much; it was hard even to talk about.

There has always been thieving and misbehavior in the phone system. There has always been trouble with the rival independents, and in the local loops. But to have such trouble in the core of the system, the long-distance switching stations, is a horrifying affair. To telco people, this is all the difference between finding roaches in your kitchen and big horrid sewer-rats in your bedroom.

From the outside, to the average citizen, the telcos still seem gigantic and impersonal. The American public seems to regard them as something akin to Soviet apparats. Even when the telcos do their best corporatecitizen routine, subsidizing magnet high-schools and sponsoring news-shows on public television, they seem to win little except public suspicion.

But from the inside, all this looks very different. There's harsh competition. A legal and political system that seems baffled and bored, when not actively hostile to telco interests. There's a loss of morale, a deep sensation of having somehow lost the upper hand. Technological change has caused a loss of data and revenue to other, newer forms of transmission. There's theft, and new forms of theft, of growing scale and boldness and sophistication. With all these factors, it was no surprise to see the telcos, large and small, break out in a litany of bitter complaint.

In late '88 and throughout 1989, telco representatives grew shrill in their complaints to those few American law enforcement officials who make it their business to try to understand what telephone people are talking about. Telco security officials had discovered the computerhacker underground, infiltrated it thoroughly, and become deeply alarmed at its growing expertise. Here they had found a target that was not only loathsome on its face, but clearly ripe for counterattack.

Those bitter rivals: AT&T, MCI and Sprint -- and a crowd of Baby Bells: PacBell, Bell South, Southwestern Bell, NYNEX, USWest, as well as the Bell research consortium Bellcore, and the independent long-distance carrier Mid-American -- all were to have their role in the great hacker dragnet of 1990. After years of being battered and pushed around, the telcos had, at least in a small way, seized the initiative again. After years of turmoil, telcos and government officials were once again to work smoothly in concert in defense of the System. Optimism blossomed; enthusiasm grew on all sides; the prospective taste of vengeance was sweet.

#

From the beginning -- even before the crackdown had a name -- secrecy was a big problem. There were many good reasons for secrecy in the hacker crackdown. Hackers and code-thieves were wily prey, slinking back to their bedrooms and basements and destroying vital incriminating evidence at the first hint of trouble. Furthermore, the crimes themselves were heavily technical and difficult to describe, even to police -- much less to the general public.

When such crimes had been described intelligibly to the public, in the past, that very publicity had tended to increase the crimes enormously. Telco officials, while painfully aware of the vulnerabilities of their systems, were anxious not to publicize those weaknesses. Experience showed them that those weaknesses, once discovered, would be pitilessly exploited by tens of thousands of people -- not only by professional grifters and by underground hackers and phone phreaks, but by many otherwise more-or-less honest everyday folks, who regarded stealing service from the faceless, soulless "Phone Company" as a kind of harmless indoor sport. When it came to protecting their interests, telcos had long since given up on general public sympathy for "the Voice with a Smile." Nowadays the telco's "Voice" was very likely to be a computer's; and the American public showed much less of the proper respect and gratitude due the fine public service bequeathed them by Dr. Bell and Mr. Vail. The more efficient, high-tech, computerized, and impersonal the telcos became, it seemed, the more they were met by sullen public resentment and amoral greed.

Telco officials wanted to punish the phone-phreak underground, in as public and exemplary a manner as possible. They wanted to make dire examples of the worst offenders, to seize the ringleaders and intimidate the small fry, to discourage and frighten the wacky hobbyists, and send the professional grifters to jail. To do all this, publicity was vital.

Yet operational secrecy was even more so. If word got out that a nationwide crackdown was coming, the hackers might simply vanish; destroy the evidence, hide their computers, go to earth, and wait for the campaign to blow over. Even the young hackers were crafty and suspicious, and as for the professional grifters, they tended to split for the nearest state-line at the first sign of trouble. For the crackdown to work well, they would all have to be caught red-handed, swept upon suddenly, out of the blue, from every corner of the compass. And there was another strong motive for secrecy. In the worst-case scenario, a blown campaign might leave the telcos open to a devastating hacker counter-attack. If there were indeed hackers loose in America who had caused the January 15 Crash -- if there were truly gifted hackers, loose in the nation's long-distance switching systems, and enraged or frightened by the crackdown -- then they might react unpredictably to an attempt to collar them. Even if caught, they might have talented and vengeful friends still running around loose. Conceivably, it could turn ugly. Very ugly. In fact, it was hard to imagine just how ugly things might turn, given that possibility. Counter-attack from hackers was a genuine concern for the telcos. In point of fact, they would never suffer any such counter-attack. But in months to come, they would be at some pains to publicize this notion and to utter grim warnings about it.

Still, that risk seemed well worth running. Better to run the risk of vengeful attacks, than to live at the mercy of potential crashers. Any cop would tell you that a protection racket had no real future.

And publicity was such a useful thing. Corporate security officers, including telco security, generally work under conditions of great discretion. And corporate security officials do not make money for their companies. Their job is to prevent the loss of money, which is much less glamorous than actually winning profits. If you are a corporate security official, and you do your job brilliantly, then nothing bad happens to your company at all. Because of this, you appear completely superfluous. This is one of the many unattractive aspects of security work. It's rare that these folks have the chance to draw some healthy attention to their own efforts.

Publicity also served the interest of their friends in law enforcement. Public officials, including law enforcement officials, thrive by attracting favorable public interest. A brilliant prosecution in a matter of vital public interest can make the career of a prosecuting attorney. And for a police officer, good publicity opens the purses of the legislature; it may bring a citation, or a promotion, or at least a rise in status and the respect of one's peers.

But to have both publicity and secrecy is to have one's cake and eat it too. In months to come, as we will show, this impossible act was to cause great pain to the agents of the crackdown. But early on, it seemed possible -- maybe even likely -- that the crackdown could successfully combine the best of both worlds. The arrest of hackers would be heavily publicized. The actual deeds of the hackers, which were technically hard to explain and also a security risk, would be left decently obscured. The threat hackers posed would be heavily trumpeted; the likelihood of their actually committing such fearsome crimes would be left to the public's imagination. The spread of the computer underground, and its growing technical sophistication, would be heavily promoted; the actual hackers themselves, mostly bespectacled middle-class white suburban teenagers, would be denied any personal publicity.

It does not seem to have occurred to any telco official that the hackers accused would demand a day in court; that journalists would smile upon the hackers as "good copy;" that wealthy high-tech entrepreneurs would offer moral and financial support to crackdown victims; that constitutional lawyers would show up with briefcases, frowning mightily. This possibility does not seem to have ever entered the game-plan.

And even if it had, it probably would not have slowed the ferocious pursuit of a stolen phone-company document, mellifluously known as "Control Office Administration of Enhanced 911 Services for Special Services and Major Account Centers."

In the chapters to follow, we will explore the worlds of police and the computer underground, and the large shadowy area where they overlap. But first, we must explore the battleground. Before we leave the world of the telcos, we must understand what a switching system actually is and how your telephone actually works.

#

To the average citizen, the idea of the telephone is represented by, well, a telephone: a device that you talk into.

To a telco professional, however, the telephone itself is known, in lordly fashion, as a "subset." The "subset" in your house is a mere adjunct, a distant nerve ending, of the central switching stations, which are ranked in levels of hierarchy, up to the long-distance electronic switching stations, which are some of the largest computers on earth. Let us imagine that it is, say, 1925, before the introduction of computers, when the phone system was simpler and somewhat easier to grasp. Let's further imagine that you are Miss Leticia Luthor, a fictional operator for Ma Bell in New York City of the 20s. Basically, you, Miss Luthor, are the "switching system." You are sitting in front of a large vertical switchboard, known as a "cordboard," made of shiny wooden panels, with ten thousand metal-rimmed holes punched in them, known as jacks. The engineers would have put more holes into your switchboard, but ten thousand is as many as you can reach without actually having to get up out of your chair.

Each of these ten thousand holes has its own little electric lightbulb, known as a "lamp," and its own neatly printed number code.

With the ease of long habit, you are scanning your board for lit-up bulbs. This is what you do most of the time, so you are used to it.

A lamp lights up. This means that the phone at the end of that line has been taken off the hook. Whenever a handset is taken off the hook, that closes a circuit inside the phone which then signals the local office, i.e. you, automatically. There might be somebody calling, or then again the phone might be simply off the hook, but this does not matter to you yet. The first thing you do, is record that number in your logbook, in your fine American public-school handwriting. This comes first, naturally, since it is done for billing purposes.

You now take the plug of your answering cord, which goes directly to your headset, and plug it into the lit-up hole. "Operator," you announce.

In operator's classes, before taking this job, you have been issued a large pamphlet full of canned operator's responses for all kinds of contingencies, which you had to memorize. You have also been trained in a proper nonregional, non-ethnic pronunciation and tone of voice. You rarely have the occasion to make any spontaneous remark to a customer, and in fact this is frowned upon (except out on the rural lines where people have time on their hands and get up to all kinds of mischief).

A tough-sounding user's voice at the end of the line gives you a number. Immediately, you write that number down in your logbook, next to the caller's number, which you just wrote earlier. You then look and see if the number this guy wants is in fact on your switchboard, which it generally is, since it's generally a local call. Long distance costs so much that people use it sparingly.

Only then do you pick up a calling-cord from a shelf at the base of the switchboard. This is a long elastic cord mounted on a kind of reel so that it will zip back in when you unplug it. There are a lot of cords down there, and when a bunch of them are out at once they look like a nest of snakes. Some of the girls think there are bugs living in those cable-holes. They're called "cable mites" and are supposed to bite your hands and give you rashes. You don't believe this, yourself.

Gripping the head of your calling-cord, you slip the tip of it deftly into the sleeve of the jack for the called person. Not all the way in, though. You just touch it. If you hear a clicking sound, that means the line is busy and you can't put the call through. If the line is busy, you have to stick the calling-cord into a "busy-tone jack," which will give the guy a busy-tone. This way you don't have to talk to him yourself and absorb his natural human frustration.

But the line isn't busy. So you pop the cord all the way in. Relay circuits in your board make the distant phone ring, and if somebody picks it up off the hook, then a phone conversation starts. You can hear this conversation on your answering cord, until you unplug it. In fact you could listen to the whole conversation if you wanted, but this is sternly frowned upon by management, and frankly, when you've overheard one, you've pretty much heard 'em all.

You can tell how long the conversation lasts by the glow of the calling-cord's lamp, down on the calling-cord's shelf. When it's over, you unplug and the calling-cord zips back into place.

Having done this stuff a few hundred thousand times, you become quite good at it. In fact you're plugging, and connecting, and disconnecting, ten, twenty, forty cords at a time. It's a manual handicraft, really, quite satisfying in a way, rather like weaving on an upright loom.

Should a long-distance call come up, it would be different, but not all that different. Instead of connecting the call through your own local switchboard, you have to go up the hierarchy, onto the long-distance lines, known as "trunklines." Depending on how far the call goes, it may have to work its way through a whole series of operators, which can take quite a while. The caller doesn't wait on the line while this complex process is negotiated across the country by the gaggle of operators. Instead, the caller hangs up, and you call him back yourself when the call has finally worked its way through.

After four or five years of this work, you get married, and you have to quit your job, this being the natural order of womanhood in the American 1920s. The phone company has to train somebody else -- maybe two people, since the phone system has grown somewhat in the meantime. And this costs money.

In fact, to use any kind of human being as a switching system is a very expensive proposition. Eight thousand Leticia Luthors would be bad enough, but a quarter of a million of them is a military-scale proposition and makes drastic measures in automation financially worthwhile.

Although the phone system continues to grow today, the number of human beings employed by telcos has been dropping steadily for years. Phone "operators" now deal with nothing but unusual contingencies, all routine operations having been shrugged off onto machines. Consequently, telephone operators are considerably less machine-like nowadays, and have been known to have accents and actual character in their voices. When you reach a human operator today, the operators are rather more "human" than they were in Leticia's day -- but on the other hand, human beings in the phone system are much harder to reach in the first place.

Over the first half of the twentieth century, "electromechanical" switching systems of growing complexity were cautiously introduced into the phone system. In certain backwaters, some of these hybrid systems are still in use. But after 1965, the phone system began to go completely electronic, and this is by far the dominant mode today. Electromechanical systems have "crossbars," and "brushes," and other large moving mechanical parts, which, while faster and cheaper than Leticia, are still slow, and tend to wear out fairly quickly.

But fully electronic systems are inscribed on silicon chips, and are lightning-fast, very cheap, and quite durable. They are much cheaper to maintain than even the best electromechanical systems, and they fit into half the space. And with every year, the silicon chip grows smaller, faster, and cheaper yet. Best of all, automated electronics work around the clock and don't have salaries or health insurance.

There are, however, quite serious drawbacks to the use of computer-chips. When they do break down, it is a daunting challenge to figure out what the heck has gone wrong with them. A broken cordboard generally had a problem in it big enough to see. A broken chip has invisible, microscopic faults. And the faults in bad software can be so subtle as to be practically theological. If you want a mechanical system to do something new, then you must travel to where it is, and pull pieces out of it, and wire in new pieces. This costs money. However, if you want a chip to do something new, all you have to do is change its software, which is easy, fast and dirt-cheap. You don't even have to see the chip to change its program. Even if you did see the chip, it wouldn't look like much. A chip with program X doesn't look one whit different from a chip with program Y. With the proper codes and sequences, and access to specialized phone-lines, you can change electronic switching systems all over America from anywhere you please.

And so can other people. If they know how, and if they want to, they can sneak into a microchip via the special phonelines and diddle with it, leaving no physical trace at all. If they broke into the operator's station and held Leticia at gunpoint, that would be very obvious. If they broke into a telco building and went after an electromechanical switch with a toolbelt, that would at least leave many traces. But people can do all manner of amazing things to computer switches just by typing on a keyboard, and keyboards are everywhere today. The extent of this vulnerability is deep, dark, broad, almost mind-boggling, and yet this is a basic, primal fact of life about any computer on a network.

Security experts over the past twenty years have insisted, with growing urgency, that this basic vulnerability of computers represents an entirely new level of risk, of unknown but obviously dire potential to society. And they are right.

An electronic switching station does pretty much everything Letitia did, except in nanoseconds and on a much larger scale. Compared to Miss Luthor's ten thousand jacks, even a primitive 1ESS switching computer, 60s vintage, has a 128,000 lines. And the current AT&T system of choice is the monstrous fifth-generation 5ESS.

An Electronic Switching Station can scan every line on its "board" in a tenth of a second, and it does this over and over, tirelessly, around the clock. Instead of eyes, it uses "ferrod scanners" to check the condition of local lines and trunks. Instead of hands, it has "signal distributors," "central pulse distributors," "magnetic latching relays," and "reed switches," which complete and break the calls. Instead of a brain, it has a "central processor." Instead of an instruction manual, it has a program. Instead of a handwritten logbook for recording and billing calls, it has magnetic tapes. And it never has to talk to anybody. Everything a customer might say to it is done by punching the direct-dial tone buttons on your subset.

Although an Electronic Switching Station can't talk, it does need an interface, some way to relate to its, er, employers. This interface is known as the "master control center." (This interface might be better known simply as "the interface," since it doesn't actually "control" phone calls directly. However, a term like "Master Control Center" is just the kind of rhetoric that telco maintenance engineers -- and hackers -- find particularly satisfying.) Using the master control center, a phone engineer can test local and trunk lines for malfunctions. He (rarely she) can check various alarm displays, measure traffic on the lines, examine the records of telephone usage and the charges for those calls, and change the programming.

And, of course, anybody else who gets into the master control center by remote control can also do these things, if he (rarely she) has managed to figure them out, or, more likely, has somehow swiped the knowledge from people who already know.

In 1989 and 1990, one particular RBOC, BellSouth, which felt particularly troubled, spent a purported $1.2 million on computer security. Some think it spent as much as two million, if you count all the associated costs. Two million dollars is still very little compared to the great cost-saving utility of telephonic computer systems.

Unfortunately, computers are also stupid. Unlike human beings, computers possess the truly profound stupidity of the inanimate.

In the 1960s, in the first shocks of spreading computerization, there was much easy talk about the stupidity of computers -- how they could "only follow the program" and were rigidly required to do "only what they were told." There has been rather less talk about the stupidity of computers since they began to achieve grandmaster status in chess tournaments, and to manifest many other impressive forms of apparent cleverness.

Nevertheless, computers still are profoundly brittle and stupid; they are simply vastly more subtle in their stupidity and brittleness. The computers of the 1990s are much more reliable in their components than earlier computer systems, but they are also called upon to do far more complex things, under far more challenging conditions.

On a basic mathematical level, every single line of a software program offers a chance for some possible screwup. Software does not sit still when it works; it "runs," it interacts with itself and with its own inputs and outputs. By analogy, it stretches like putty into millions of possible shapes and conditions, so many shapes that they can never all be successfully tested, not even in the lifespan of the universe. Sometimes the putty snaps.

The stuff we call "software" is not like anything that human society is used to thinking about. Software is something like a machine, and something like mathematics, and something like language, and something like thought, and art, and information... but software is not in fact any of those other things. The protean quality of software is one of the great sources of its fascination. It also makes software very powerful, very subtle, very unpredictable, and very risky.

Some software is bad and buggy. Some is "robust," even "bulletproof." The best software is that which has been tested by thousands of users under thousands of different conditions, over years. It is then known as "stable." This does not mean that the software is now flawless, free of bugs. It generally means that there are plenty of bugs in it, but the bugs are well-identified and fairly well understood.

There is simply no way to assure that software is free of flaws. Though software is mathematical in nature, it cannot be "proven" like a mathematical theorem; software is more like language, with inherent ambiguities, with different definitions, different assumptions, different levels of meaning that can conflict.

Human beings can manage, more or less, with human language because we can catch the gist of it.

Computers, despite years of effort in "artificial intelligence," have proven spectacularly bad in "catching the gist" of anything at all. The tiniest bit of semantic grit may still bring the mightiest computer tumbling down. One of the most hazardous things you can do to a computer program is try to improve it -- to try to make it safer. Software "patches" represent new, untried un"stable" software, which is by definition riskier.

The modern telephone system has come to depend, utterly and irretrievably, upon software. And the System Crash of January 15, 1990, was caused by an improvement in software. Or rather, an attempted improvement.

As it happened, the problem itself -- the problem per se -- took this form. A piece of telco software had been written in C language, a standard language of the telco field. Within the C software was a long "do... while" construct. The "do... while" construct contained a "switch" statement. The "switch" statement contained an "if" clause. The "if" clause contained a "break." The "break" was supposed to "break" the "if" clause. Instead, the "break" broke the "switch" statement.

That was the problem, the actual reason why people picking up phones on January 15, 1990, could not talk to one another.

Or at least, that was the subtle, abstract, cyberspatial seed of the problem. This is how the problem manifested itself from the realm of programming into the realm of real life.

The System 7 software for AT&T's 4ESS switching station, the "Generic 44E14 Central Office Switch Software," had been extensively tested, and was considered very stable. By the end of 1989, eighty of AT&T's switching systems nationwide had been programmed with the new software. Cautiously, thirty four stations were left to run the slower, less-capable System 6, because AT&T suspected there might be shakedown problems with the new and unprecedently sophisticated System 7 network.

The stations with System 7 were programmed to switch over to a backup net in case of any problems. In mid-December 1989, however, a new high-velocity, high security software patch was distributed to each of the 4ESS switches that would enable them to switch over even more quickly, making the System 7 network that much more secure.

Unfortunately, every one of these 4ESS switches was now in possession of a small but deadly flaw.

In order to maintain the network, switches must monitor the condition of other switches -- whether they are up and running, whether they have temporarily shut down, whether they are overloaded and in need of assistance, and so forth. The new software helped control this bookkeeping function by monitoring the status calls from other switches.

It only takes four to six seconds for a troubled 4ESS switch to rid itself of all its calls, drop everything temporarily, and re-boot its software from scratch. Starting over from scratch will generally rid the switch of any software problems that may have developed in the course of running the system. Bugs that arise will be simply wiped out by this process. It is a clever idea. This process of automatically re-booting from scratch is known as the "normal fault recovery routine." Since AT&T's software is in fact exceptionally stable, systems rarely have to go into "fault recovery" in the first place; but AT&T has always boasted of its "real world" reliability, and this tactic is a belt-and-suspenders routine.

The 4ESS switch used its new software to monitor its fellow switches as they recovered from faults. As other switches came back on line after recovery, they would send their "OK" signals to the switch. The switch would make a little note to that effect in its "status map," recognizing that the fellow switch was back and ready to go, and should be sent some calls and put back to regular work.

Unfortunately, while it was busy bookkeeping with the status map, the tiny flaw in the brand-new software came into play. The flaw caused the 4ESS switch to interacted, subtly but drastically, with incoming telephone calls from human users. If -- and only if -- two incoming phone-calls happened to hit the switch within a hundredth of a second, then a small patch of data would be garbled by the flaw.

But the switch had been programmed to monitor itself constantly for any possible damage to its data. When the switch perceived that its data had been somehow garbled, then it too would go down, for swift repairs to its software. It would signal its fellow switches not to send any more work. It would go into the fault recovery mode for four to six seconds. And then the switch would be fine again, and would send out its "OK, ready for work" signal.

However, the "OK, ready for work" signal was the very thing that had caused the switch to go down in the first place. And all the System 7 switches had the same flaw in their status-map software. As soon as they stopped to make the bookkeeping note that their fellow switch was "OK," then they too would become vulnerable to the slight chance that two phone-calls would hit them within a hundredth of a second.

At approximately 2:25 p.m. EST on Monday, January 15, one of AT&T's 4ESS toll switching systems in New York City had an actual, legitimate, minor problem. It went into fault recovery routines, announced "I'm going down," then announced, "I'm back, I'm OK." And this cheery message then blasted throughout the network to many of its fellow 4ESS switches. Many of the switches, at first, completely escaped trouble. These lucky switches were not hit by the coincidence of two phone calls within a hundredth of a second. Their software did not fail -- at first. But three switches -- in Atlanta, St. Louis, and Detroit -- were unlucky, and were caught with their hands full. And they went down. And they came back up, almost immediately. And they too began to broadcast the lethal message that they, too, were "OK" again, activating the lurking software bug in yet other switches.

As more and more switches did have that bit of bad luck and collapsed, the call-traffic became more and more densely packed in the remaining switches, which were groaning to keep up with the load. And of course, as the calls became more densely packed, the switches were much more likely to be hit twice within a hundredth of a second. It only took four seconds for a switch to get well. There was no physical damage of any kind to the switches, after all. Physically, they were working perfectly. This situation was "only" a software problem. But the 4ESS switches were leaping up and down every four to six seconds, in a virulent spreading wave all over America, in utter, manic, mechanical stupidity. They kept knocking one another down with their contagious "OK" messages. It took about ten minutes for the chain reaction to cripple the network. Even then, switches would periodically luck-out and manage to resume their normal work. Many calls -- millions of them -- were managing to get through. But millions weren't.

The switching stations that used System 6 were not directly affected. Thanks to these old-fashioned switches, AT&T's national system avoided complete collapse. This fact also made it clear to engineers that System 7 was at fault.

Bell Labs engineers, working feverishly in New Jersey, Illinois, and Ohio, first tried their entire repertoire of standard network remedies on the malfunctioning System 7. None of the remedies worked, of course, because nothing like this had ever happened to any phone system before.

By cutting out the backup safety network entirely, they were able to reduce the frenzy of "OK" messages by about half. The system then began to recover, as the chain reaction slowed. By 11:30 pm on Monday January 15, sweating engineers on the midnight shift breathed a sigh of relief as the last switch cleared-up.

By Tuesday they were pulling all the brand-new 4ESS software and replacing it with an earlier version of System 7. If these had been human operators, rather than computers at work, someone would simply have eventually stopped screaming. It would have been obvious that the situation was not "OK," and common sense would have kicked in. Humans possess common sense -- at least to some extent. Computers simply don't. On the other hand, computers can handle hundreds of calls per second. Humans simply can't. If every single human being in America worked for the phone company, we couldn't match the performance of digital switches: direct-dialling, three-way calling, speed-calling, callwaiting, Caller ID, all the rest of the cornucopia of digital bounty. Replacing computers with operators is simply not an option any more.

And yet we still, anachronistically, expect humans to be running our phone system. It is hard for us to understand that we have sacrificed huge amounts of initiative and control to senseless yet powerful machines. When the phones fail, we want somebody to be responsible. We want somebody to blame.

When the Crash of January 15 happened, the American populace was simply not prepared to understand that enormous landslides in cyberspace, like the Crash itself, can happen, and can be nobody's fault in particular. It was easier to believe, maybe even in some odd way more reassuring to believe, that some evil person, or evil group, had done this to us. "Hackers" had done it. With a virus. A trojan horse. A software bomb. A dirty plot of some kind. People believed this, responsible people. In 1990, they were looking hard for evidence to confirm their heartfelt suspicions.

And they would look in a lot of places. Come 1991, however, the outlines of an apparent new reality would begin to emerge from the fog.

On July 1 and 2, 1991, computer-software collapses in telephone switching stations disrupted service in Washington DC, Pittsburgh, Los Angeles and San Francisco. Once again, seemingly minor maintenance problems had crippled the digital System 7. About twelve million people were affected in the Crash of July 1, 1991.

Said the New York Times Service: "Telephone company executives and federal regulators said they were not ruling out the possibility of sabotage by computer hackers, but most seemed to think the problems stemmed from some unknown defect in the software running the networks."

And sure enough, within the week, a red-faced software company, DSC Communications Corporation of Plano, Texas, owned up to "glitches" in the "signal transfer point" software that DSC had designed for Bell Atlantic and Pacific Bell. The immediate cause of the July 1 Crash was a single mistyped character: one tiny typographical flaw in one single line of the software. One mistyped letter, in one single line, had deprived the nation's capital of phone service. It was not particularly surprising that this tiny flaw had escaped attention: a typical System 7 station requires ten million lines of code.

On Tuesday, September 17, 1991, came the most spectacular outage yet. This case had nothing to do with software failures -- at least, not directly. Instead, a group of AT&T's switching stations in New York City had simply run out of electrical power and shut down cold. Their back-up batteries had failed. Automatic warning systems were supposed to warn of the loss of battery power, but those automatic systems had failed as well.

This time, Kennedy, La Guardia, and Newark airports all had their voice and data communications cut. This horrifying event was particularly ironic, as attacks on airport computers by hackers had long been a standard nightmare scenario, much trumpeted by computer-security experts who feared the computer underground. There had even been a Hollywood thriller about sinister hackers ruining airport computers -- Die Hard II.

Now AT&T itself had crippled airports with computer malfunctions -- not just one airport, but three at once, some of the busiest in the world.

Air traffic came to a standstill throughout the Greater New York area, causing more than 500 flights to be cancelled, in a spreading wave all over America and even into Europe. Another 500 or so flights were delayed, affecting, all in all, about 85,000 passengers. (One of these passengers was the chairman of the Federal Communications Commission.)

Stranded passengers in New York and New Jersey were further infuriated to discover that they could not even manage to make a long distance phone call, to explain their delay to loved ones or business associates. Thanks to the crash, about four and a half million domestic calls, and half a million international calls, failed to get through. The September 17 NYC Crash, unlike the previous ones, involved not a whisper of "hacker" misdeeds. On the contrary, by 1991, AT&T itself was suffering much of the vilification that had formerly been directed at hackers. Congressmen were grumbling. So were state and federal regulators. And so was the press.

For their part, ancient rival MCI took out snide fullpage newspaper ads in New York, offering their own longdistance services for the "next time that AT&T goes down." "You wouldn't find a classy company like AT&T using such advertising," protested AT&T Chairman Robert Allen, unconvincingly. Once again, out came the full-page AT&T apologies in newspapers, apologies for "an inexcusable culmination of both human and mechanical failure." (This time, however, AT&T offered no discount on later calls. Unkind critics suggested that AT&T were worried about setting any precedent for refunding the financial losses caused by telephone crashes.)

Industry journals asked publicly if AT&T was "asleep at the switch." The telephone network, America's purported marvel of high-tech reliability, had gone down three times in 18 months. Fortune magazine listed the Crash of September 17 among the "Biggest Business Goofs of 1991," cruelly parodying AT&T's ad campaign in an article entitled "AT&T Wants You Back (Safely On the Ground, God Willing)."

Why had those New York switching systems simply run out of power? Because no human being had attended to the alarm system. Why did the alarm systems blare automatically, without any human being noticing? Because the three telco technicians who should have been listening were absent from their stations in the power-room, on another floor of the building -- attending a training class. A training class about the alarm systems for the power room!

"Crashing the System" was no longer "unprecedented" by late 1991. On the contrary, it no longer even seemed an oddity. By 1991, it was clear that all the policemen in the world could no longer "protect" the phone system from crashes. By far the worst crashes the system had ever had, had been inflicted, by the system, upon itself. And this time nobody was making cocksure statements that this was an anomaly, something that would never happen again. By 1991 the System's defenders had met their nebulous Enemy, and the Enemy was -- the System.

Go to the previous, next section.

Copyright © 1998, Software Engineering Laboratory
National Technical University of Athens