“The work required painstaking concentration, faultless attention to minute details, and the ability to recognize nearly invisible patterns.”

 

By Walter S. Topp

WHEN THE Japanese Navy’s powerful aircraft carrier strike force approached Midway Atoll on the morning of June 4, 1942, three U.S. Navy carriers were waiting in ambush. The American fleet’s presence that morning was, of course, no accident. The role of Navy codebreakers in intercepting and reading Japanese radio signals, which enabled Pacific Fleet commander Chester Nimitz to surprise the Japanese at Midway, is well-known.

Though details of U.S. communications intelligence during World War Two remained classified for many years, the work of codebreakers in uncovering Japanese plans at Midway was initially hinted at in a newspaper article by reporter Stanley Johnston within days of the battle. While hundreds – and by war’s end, many thousands – of military and civilian personnel participated in America’s radio intelligence effort, a mere handful performed the extraordinary intellectual feat of manually deciphering the enemy’s coded messages. Although decoding machines were in use throughout the war, the navy’s critical codebreaking work in the first months of 1942 was performed almost entirely by hand.

No Machine Could do This

Throughout the Pacific War, the Imperial Japanese Navy used a series of manual book-based codes for operational message traffic. While Japanese diplomats used a machine-derived code – the famous ‘Purple’ machines – the navy, with many thousands of units sending radio messages, preferred their manual system.

Japanese naval radio messages were transmitted using a modified version of Morse code. Messages that were intercepted by the allies were recorded as a series of digits; it was up to codebreakers to turn the rows and rows of numbers into words and phrases. There was no machine that could do this. Although rudimentary tabulating equipment might help detect patterns in the seemingly random digits, it was up to individual cryptanalysts to deduce the meaning of each enciphered word. They did this by studying thousands of messages, finding sequences of numbers that appeared in previous messages and recognizing patterns. This, coupled with their knowledge of Japanese naval procedures, strategy, and the Japanese language itself, enabled the codebreakers to make educated guesses.

The work required painstaking concentration, faultless attention to minute details, and the ability to recognize nearly invisible patterns. Historian David Khan, in his 1996 book The Codebreakers, wrote, “This work was the most excruciating, exasperating, agonizing mental process known to man.”

Progress was incremental. Each tiny success in uncovering the meaning of a code group – as the numbers that represented words or phrases were called – was carefully recorded and shared with other cryptanalysts as they struggled to read other messages. It was a slog. There was no cinematic moment where a flash of inspiration suddenly revealed the hidden secrets of a particular message. The daily experience of cryptanalysts was frustration and a gloomy sense of failure.

Ultimately, though, the efforts paid off. U.S. Navy successes at Coral Sea and Midway in early 1942 – a time when the Imperial Japanese Navy outclassed the U.S. Pacific Fleet in numbers of ships, training, tactics, and combat experience – were made possible by information gained through American radio intelligence, comprised of codebreaking, traffic analysis, direction finding, and intelligence analysis.

Felonious Beginning

That the U.S. Navy’s pre-war radio intelligence effort – in typical American fashion, undermanned, under-equipped, under-resourced, and under-appreciated – should have evolved into an organization that would change the direction of the Pacific War was something of a miracle.

The United States was one of several nations that conducted communications intelligence activities during World War One – the first conflict in which radio was widely used. Before the war, America had no organization that could intercept another nation’s radio signals. In 1917, as the U.S. entered the conflict, the U.S. Army established America’s first code-breaking office. The Navy did not follow suit, relying instead on Britain’s Royal Navy for code-breaking support. During the final days of the war, however, the Navy set aside $100,000 in a secret Office of Naval Intelligence (ONI) account to fund future code-breaking efforts.

It wasn’t the Navy or even the Army that scored America’s first big code-breaking victory. Instead, it was the U.S. Cipher Bureau, established in 1919 by the State and War Departments and capably headed by cryptologist Hebert O. Yardley. After a year’s effort, the Bureau broke Japan’s primary diplomatic code, giving the U.S. a priceless advantage during the 1921-1922 naval disarmament talks.

Although the Navy benefitted greatly from the Bureau’s success, the service was in no hurry to develop its own code-breaking capability, especially since there were quicker ways to penetrate an adversary’s communications. By 1920 the secret ONI fund had financed a series of break-ins at the Japanese consulate in New York City. Later, in 1923, a similar operation struck codebreaking gold when Navy agents managed to steal a Japanese codebook from the luggage of a visiting naval officer. The Navy responded by establishing a small code-breaking section to exploit the find and mount additional attacks on foreign codes and ciphers.

From that felonious beginning, the Navy’s code-breaking capability slowly grew. During the 1920s and 30s, many of the people who would later lead the branch’s wartime effort entered the field. Notable among them were the indomitable Agnes Meyer Driscoll, who trained most of the Navy’s key wartime cryptanalysts during the interwar period, and the brilliant Joseph J. Rochefort, who would lead the organization that uncovered Japanese plans at Midway.

During this period, under the leadership of Laurence F. Safford and Rochefort, the Navy developed a network of radio intercept stations, devised methods of deciphering intercepted messages, and built a small staff of trained cryptanalysts and linguists. Depression-era spending limits and the fact that intercepting another nation’s radio communications was actually illegal during these years constrained the Navy’s efforts. Still, by the mid 1930’s the Navy had created the largest cryptological branch within the American military.

Building the Foundation

Under Safford and Rochefort, the Navy developed a three-pronged cryptologic program comprised of direction finding, traffic analysis, and cryptanalysis, or code-breaking. The program was managed from Navy headquarters in Washington D.C. and included radio intelligence stations at Pearl Harbor and the Philippines. Promising young officers were sent to Japan on three-year assignments to learn Japanese language, customs, and culture. This knowledge would later prove invaluable as these officers attempted to glean Japanese intentions from the fragmentary information available through radio intelligence.

As cryptanalysis was not considered a career-enhancing endeavor, the Navy had some difficulty attracting and keeping talented officers in the field. While Rochefort spent most of the 1930s studying Japanese and learning cryptanalysis, making him one of the most, if not the most, qualified cryptanalysts in the U.S. military, the Navy made sure he remained a well-rounded officer by periodically taking him away from communications intelligence work and assigning him to sea duty. This was the pattern for other cryptanalysts, as well.

Still, during the 1920s and 1930s, the Navy succeeded in building a foundation that would support a vastly expanded and spectacularly successful communications intelligence effort during the Pacific War. That expansion began in earnest in the late 1930s, as war with Japan became ever more likely, and accelerated after the Germans invaded Poland in 1939.

By the time of the Pearl Harbor attack, the Navy’s communications intelligence infrastructure in the Pacific included listening stations on Guam, the Philippines, Oahu, and in Washington state. A network of high-frequency direction-finding stations at numerous sites throughout the region was also established, along with radio intelligence centers at Pearl Harbor and Cavite in the Philippines. A third radio intelligence center in Washington D.C. directed radio intelligence activities in the Pacific. A British radio intelligence group in Singapore shared information with the Americans, as well.

During this period, the three stations – Cavite, Pearl Harbor, and Washington DC – worked well together. Later, personality clashes and a struggle for supremacy within the Navy would hinder cooperation and result in Rochefort spending a year in command of a self-propelled floating drydock, but in the months between Pearl Harbor and Midway, cooperation was excellent.

It Wasn’t All Codebreaking

The Cavite unit, located in Manila, had grown to nearly 70 officers and men by the end of 1941. At Pearl Harbor, Rochefort led an organization that included 100 officers and enlisted men. Nearly all were assigned to direction-finding and traffic analysis, with only a handful available for codebreaking work.

As the United States and Japan edged closer to war in 1940 and 1941, the Navy relied primarily on radio direction finding and traffic analysis to monitor Japan’s war preparations.

Traffic analysis was the core of radio intelligence and it involved learning everything possible about an intercepted message without actually decoding it. By analyzing the identities of senders and receivers, the volume of radio activity, command relationships between senders and receivers, traffic patterns that matched previous operations, types of units being messaged, and other information, traffic analysts could uncover significant details about an adversary’s operations.

Sensitive direction-finding (DF) antennas located throughout the Pacific could receive radio signals from extremely long distances and identify the precise bearing of the radio transmitter. If a signal was received at two or more DF sites, technicians could plot the bearings from the sites and accurately locate the transmitter.

Though valuable, traffic analysis and direction finding could not match the potential of codebreaking, which promised to reveal far more information about Japanese activities and intentions.

Red, Blue, and Purple

Thanks to the theft of a Japanese code book in 1923, by the mid-1930s the Navy had already made considerable progress against the Japanese Red Code – so named because the Navy bound copies of the purloined codebook in red.

When the Japanese replaced the Red Code with a similar cypher – the Blue Code – the Americans were able to use their knowledge of the earlier encryption to break the Blue Code. By 1939, the U.S. had penetrated the Blue Code and several minor codes used for weather messages, shipping information, and harbormaster operations.

In late 1940, a cooperative effort by U.S. Navy and Army codebreakers also succeeded in solving the primary Japanese diplomatic code. Unlike the manual Red and Blue codes, the diplomatic code – known as Purple – was a machine code, in that messages comprised of code groups were further enciphered by an electro-mechanical coding machine. Somehow, American cryptanalysts, despite never having seen the original machine, managed to deduce the operation of the coding machine through mathematical analysis of intercepted messages. They even built a working copy of the device that could strip away the encipherment of the code groups. Once the encipherment was removed, the Americans could attack the code groups manually using standard cryptanalysis techniques.

An Assembly-Line Process

So, how did manual code-breaking really work? How did collaboration, pattern recognition, record-keeping, trial and error, and perseverance lead to victory at Midway?

Beginning in June, 1939, Japan was using a new naval operations code, which the Americans called JN-25. Though based on the earlier Red and Blue Codes, JN-25 was more complex, providing three distinct layers of protection.

First, each word or numeral in the message was replaced by a five-digit number, called a code group. There were more than 33,000 individual code groups, each representing a word, phrase, letter, or number. Earlier codes had used four-character code groups.

Second, the code groups, were further enciphered, or superenciphered, by adding random numbers to each group. These numbers were called ‘additives’ and 30,000 of them were provided in a separate book. The additive values were added to the code groups, changing the numbers and disguising the underlying code groups. The message drafter randomly selected a starting point in the additive book and then used the additives in sequence. A different additive value was applied to each code group.

Finally, the message contained the starting point in the additive book where the sequence of additives began, so the message recipient knew where to find the string of additives. The receiver of the message subtracted the additives to reveal the underlying code groups, and then looked up the code groups in the code book to uncover the message content.

The task for the Navy’s radio intelligence team was to intercept the message, convert the Morse code to numbers, remove the additives to reveal the code groups, and figure out what the code groups meant. It was an assembly-line process where intercepted messages were catalogued, assessed, evaluated, and deciphered in turn by radiomen at the intercept stations and DF sites, and cryptanalysts, translators, linguists, data processing assistants, and intelligence analysts at the radio intelligencer centers.

Intercepted messages were first examined for information that could be used in traffic analysis, including the message sender, recipients, call signs used, length of the message, time of day it was sent, circuit used, message format, radio direction-finding information and other external cues. After traffic analysis, the intercepts were sorted, duplicated, and provided to cryptanalysts who had the seemingly impossible task of identifying the additives, stripping them off to reveal the code groups, and deciphering the underlying code groups.

The Staring Process

Because details of America’s codebreaking effort remained classified for decades after the war, few published sources have discussed the actions cryptanalysts took to decipher the additives and code groups. But even in the few accounts that are available, codebreakers struggled to describe the process, although they all agree that it was grueling.

Rochefort said he would begin with “the staring process.”

“You look at all the messages that you have,” he said. “You line them up in various ways; you write them one below the other; you write them in various forms and you stare at them. Pretty soon you’d notice a pattern; you’d notice a definite pattern between these messages. That is the first clue.”

Some codebreakers would uncover additives by making up potential additives and subtracting them from a column of enciphered numbers pulled from messages. If the results were divisible by three – as all code groups were, as a check the Japanese used to guard against coding or transmission errors – then the codebreaker would know that they had discovered an additive.

Codebreakers arranged new messages on large sheets filled with numbers pulled from earlier messages to look for patterns. If they found a repeated sequence, they would pull the earlier messages and examine them for information that could help uncover the new message.

Sometimes codebreakers would make educated guesses about the meaning of words or phrases and replace code groups with those guesses – a process called ‘cribbing.’

Having searchable records of thousands of earlier messages was critical to the process. So was the careful preparation of charts, graphs, and tables to identify repeated sequences or patterns. But in the end, it sometimes came down to a flash of insight where the codebreaker suddenly saw what they had been looking for.

That insight, of course, would mean that a single additive, or – if additives had already been stripped off – a single code group might be identified. But there were more than 30,000 additives and more than 33,000 code groups, so reading a single word was usually a very small step forward. Plus, the Japanese periodically replaced their code groups and additive books with updated versions, sending American codebreakers back to square one.

Still, the American cryptanalysts carried on. Codebreakers at all of the radio intelligence centers worked together to create and share lists of the additives and code groups they had identified. Any flash of brilliance or inspiration occurred as part of a large-scale cooperative process.

“Codes are broken not by solitary individuals,” wrote Liza Mundy in her 2017 bestselling book Code Girls, “but by groups of people trading pieces of things they have learned and noticed and collected; little glittering bits of numbers and other useful items they have stored up in their heads like magpies, things they remember while looking over one another’s shoulders, pointing out patterns that that turn out to be the key that unlocks the code.”

As more and more additives and code groups were identified, that knowledge helped uncover additional additives and code groups. The work was cumulative, and it was greatly assisted by painstaking maintenance of voluminous files of intercepted messages, identified additives, and deciphered code groups. The use of electro-mechanical sorting machines – rudimentary data processing devices – including punchers, sorters, and tabulators, also helped speed up the process. When additives were stripped from an intercepted message, the uncovered code groups were hand-punched on IBM cards, recorded, and the cards were hand-sorted. The meaning of deciphered code groups was also stored.

Decrypted messages were also recorded by hand on index cards, with key information underlined. Multiple copies of the cards were made and the copies filed by in separate batches according to the underlined information. Later, when cryptanalysts were working a message, they might recall a similar message and they would have a way to find it.

The greater the number of messages intercepted and worked, the more information became available, and the easier the task became. But the idea that at some point these codes were “broken,” and the Americans could read entire messages is false.

One Word Out of Five

The Japanese introduced the original version of their JN-25 operational code in June of 1939. As the version was significantly more complex than the code it replaced, it took the Americans more than a year to decrypt any significant information, but by November 1940 they were beginning to produce intelligible text from JN-25 intercepts.

In December of 1940, however, the Japanese introduced a revised version (JN-25a), which set U.S. cryptanalysts back again. The Americans were not able to read anything in the revised code until late 1941.

By December, 1941, American cryptanalysts could read less than 10 percent of the Japanese code groups. Messages that did not contain any of the recovered code groups would remain a total blank. So, in the critical months before Pearl Harbor, Navy radio intelligence was restricted to traffic analysis and direction finding. On Dec. 4, 1941, as the Japanese carrier striking force bore down on the Hawaiian Islands, the Japanese introduced another revised version of JN-25.

Following the Pearl Harbor attack, the U.S. Navy reorganized its codebreaking operations, and assigned Rochefort’s Pearl Harbor unit to lead the JN-25b effort. Before that, Rochefort’s station had been assigned to different Japanese codes, while stations in Washington DC and the Philippines worked on JN-25. At that point, Washington D.C. and the Philippines had made only minor inroads into JN-25, and though they had determined the structure of the code and uncovered several thousand code groups, they were not yet able to read Japanese intercepts.

Shocked by the scale and speed of Japanese victories in the first weeks of the war, Rochefort and his unit, assisted by Washington and Cavite, redoubled their efforts against JN-25. By April 1942 – a month before the Battle of the Coral Sea and two months before the Battle of Midway – Rochefort’s cryptanalysts were able to read approximately one fifth of the code groups in the JN-25b code.

It was enough.

By combining information gained through traffic analysis, direction-finding, fragments of decrypted message traffic, and their understanding of Japanese language, military culture, capabilities and strategic intentions, the Americans were able to identify critical details of Japan’s Port Moresby operation and, more importantly, discover the existence of a larger operation being planned that would eventually be revealed as an invasion of Midway.

Traffic analysis and direction finding helped identify and locate units that were preparing for the operation, while cryptanalysis uncovered tantalizing shards of movement orders, requests for supplies, casualty reports, and other operational messages. Linguists and intelligence analysts pored over the partial decrypts, connecting the dots and filling in as many blanks as they could. Rochefort, especially, was adept at deriving Japanese intentions from the fragmentary clues available.

Pacific Fleet Intelligence chief Edwin Layton later wrote that “Rochefort’s sixth sense in assembling seemingly unrelated information in partially decrypted enemy messages and turning the puzzle into an accurate picture of enemy intentions” was the key to the American success at the Battle of the Coral Sea.

The battle also boosted Pacific Fleet Commander Chester Nimitz’s trust in radio intelligence.

The Next Big Thing

Even as the Coral Sea fight was raging, Rochefort was pushing his unit to uncover details of the next big operation. By the middle of May they were solving as many as forty percent of JN-25 code groups. Since the groups that were solved tended to be the code groups used most frequently, cryptanalysts were able to determine the meaning of – or at least, make a reasonably informed guess at the meaning of – a high percentage of intercepted messages.

Of course, only a fraction of Japanese radio traffic was intercepted. The number of messages being transmitted was far too high at that stage of the war for the Americans to intercept enough to provide a complete picture of Japanese plans. During the month before Midway, when the Americans were desperate to uncover additional details of Japanese plans, Navy radiomen intercepted no more than 60 percent of Japanese traffic and cryptanalysts decoded parts of just 40 percent of the messages copied.

Through the month of May, Rochefort’s team slowly discerned the outline of the Japanese Midway operation. Indications flowed to the intelligence analysts in seemingly unremarkable bits.

The volume of message traffic was extremely high, and traffic analysis placed key Japanese fleet units in home waters. Partially decrypted messages described the merging of several Japanese carrier groups into a single striking force. Other messages arranged refueling rendezvous,’ ordered destroyers to meet up with a carrier striking force, curtailed shipyard periods, mentioned a landing force and an occupation force, and requested charts for the Aleutians and the area around Midway.

Each scrap of information filled in another piece of the enormous puzzle that Rochefort and his team were struggling over. No message was completely understood. But words or phrases from many hundreds of messages, information from traffic analysis, and knowledge of previous Japanese operations were laboriously knitted together to create a recognizable picture of Japanese intentions.

By mid-May Rochefort could say with certainty that the Japanese Navy’s mighty Second Fleet was assembling an invasion force at Saipan. Meanwhile, at least four large carriers – the Akagi, Kaga, Hiryu, and Soryu – were to rendezvous in Japan’s Inland Sea, then head toward an unknown destination after May 21.

Based on the use of a geographic designator that they knew had been used in previous message traffic to identify Midway, the Americans were virtually certain that the intended target was Midway. Nimitz and Admiral King in Washington agreed. While there was always a chance that some of the information that codebreakers had deduced was transmitted as part of a Japanese deception campaign, Nimitz would act based on Rochefort’s assessment.

Okay, But When?

But so far, Rochefort had not been able to uncover the planned dates of the Japanese attack. Without knowing when the Japanese force intended to strike Midway, Nimitz would be unable to place his three carriers in the proper position.

So, during the last week of May, Rochefort’s team began a meticulous review of all messages that had been intercepted in the period May 19-20. While these messages had already been decoded, portions remained unbroken. Somewhere in the still-coded fragments was the date and time of the attack.

On May 25, the Americans discovered a partially decrypted message that included the code groups for ‘Midway’ and ‘attack’ and also included a time-date group that had been separately enciphered in a highly complex and rarely used process that Navy cryptanalysts had not been able to solve.

Miraculously, two of Rochefort’s cryptanalysts, Joseph Finnegan and Wesley Wright, working through the night, were able to deduce the structure of the table that the Japanese had used to encipher the date/time group. The date of the planned strike at Midway was June 4, while the Aleutians would be struck on June 3.

This was the key. The U.S. Navy would be able to place its available striking power – the undamaged carriers Enterprise and Hornet and the damaged but still operational Yorktown – northwest of Midway, in position to launch a surprise air attack against Yamamoto’s carriers as soon as they could be located.

By the evening of June 4, American carrier airstrikes had destroyed all four of the Japanese carriers. The American victory would not have been possible without the contributions of the radio intelligence organizations.

Rochefort, in one of the great understatements of the war, explained simply, “We felt that we had earned our pay.”

Walter Topp is a former U.S. Navy officer, emergency manager, police officer, and newspaper reporter. He is currently a writer and is working on several history and emergency management projects. He speaks regularly on emergency management topics but prefers writing about military history.

Sources:

Layton, Edward; And I Was There; William Morrow and Company; NY; 1985.

Carlson, Elliot; Joe Rochefort’s War; Naval Institute Press; Annapolis, MD; 2011.

Mundy, Liza; Code Girls; Hachette Books; NY, Boston; 2017.

Kahn, David; The Codebreakers; Scribner; NY; 1967.

Prados, John; Combined Fleet Decoded; Random House; NY; 1995.

Haufler, Hervie; Codebreakers’ Victory: How the Allied Cryptographers Won World War II; Open Road Integrated Media, NY, 2003.

National Security Agency Central Security Service; Pearl Harbor Review: JN-25; National Security Agency Central Security Service > About Us > Cryptologic Heritage > Center for Cryptologic History > Pearl Harbor Review > JN-25 (nsa.gov); Retrieved 5.22.2020.

Rutherford, Heather: Codebreakers: From the Civil War to Midway to Virtual Reality; Chips, The Department of the Navy’s Information Technology Magazine; October-December 2014; https://www.doncio.navy.mil/Chips/ArticleDetails.aspx?ID=5628 Retrieved 5.22.2020.

Station Hypo Started Reading Japanese JN-25b; Blog post, StationHypo.com; https://stationhypo.com/2016/03/05/march-5-1942-station-hypo-started-reading-japanese-jn-25/#more-421 Retrieved 5.22.2020.