TWELVE YEARS after decimating an entire Republican Guard tank formation without loss, Herbert Raymond McMaster returned to Iraq. It didn’t take him long to discover that the pitched battles he and John Nagl waged against Saddam Hussein in 1991 had gone the way of the musket. This was a different sort of war.

As the director of Central Command’s advisory group in 2003, McMaster visited every brigade in Iraq. When he came across the 101st Airborne Division in Mosul, he was pleasantly surprised to find Maj. Gen. Petraeus experimenting with counterinsurgency. The two hadn’t spoken in a while. Six years earlier, Fred Kaplan recounts in The Insurgents, McMaster’s intellectual honesty caught Petraeus’s eye. Both officers had written their PhD dissertations on the Vietnam War. Petraeus chose not to publish his for fear of insulting the commanders who held sway over his career; McMaster did not share that compunction. Dereliction of Duty bears the subtitle “Lyndon Johnson, Robert McNamara, the Joint Chiefs of Staff, and the Lies That Led to Vietnam.” Not exactly subtle. Petraeus was an aide to the chairman of the joint chiefs at the time. He called McMaster and instructed him to reach out directly should he ever come under fire for what he’d written. The need never arose, but after reconnecting in Mosul, they stayed in touch. McMaster would experience his own watershed soon enough.

It was the spring of 2005, and Tal Afar was in shambles. Colonel McMaster arrived at the helm of the Third Armored Cavalry Regiment to find the city, thirty-five miles from the Syrian border, beset with sectarian violence. Iraqi and foreign-fighter jihadists, along with elements of the local Sunni population, ran riot. Al Qaeda in Iraq, ISIS’s forebear, targeted the minority Shia with heinous violence, the kind of theatrical brutality all too familiar today. A Shia police force fanned the flames with indiscriminate reprisal killings. In Insurgency and Counter-Insurgency in Iraq, Ahmed S. Hashim recalls that “Tal Afar looked like one of those forlorn humanoid settlements on a distant and barren planet in the outer reaches of the solar system.” On multiple occasions, the U.S. military ousted the insurgents. But each time they came back. The city became an insurgent sanctuary, a convenient support base for launching attacks on Mosul, which markedly deteriorated after Petraeus’s departure.

In the April 10, 2006 issue of the New Yorker, George Packer painted a vivid picture of McMaster’s pioneering operations in Tal Afar. On his own initiative, McMaster employed classic counterinsurgency tactics in a bid to “clear, hold, build.” (Among the books McMaster assigned his regiment before deploying was Learning to Eat Soup with a Knife, by John Nagl. Combatting insurgency is “messy and slow, like eating soup with a knife,” T. E. Lawrence wrote in Seven Pillars of Wisdom.) Clearing and holding went well enough. He collaborated with the Eighty-Second Airborne and special-operations units to dispossess the jihadists of their strongholds. He ordered his soldiers to establish neighborhood outposts and to dismount their armored vehicles while on patrol (shifting the emphasis from protecting themselves to protecting civilians); to curtail gratuitous night raids (which embitter and alienate); to use minimum necessary force when possible (to avoid collateral damage); and to treat the locals with a dose of dignity and compassion. “Every time you treat an Iraqi disrespectfully, you are working for the enemy,” he instructed. Trust yields intelligence, and intelligence saves lives. James Mattis, McMaster’s current counterpart at the Pentagon, adopted a similar philosophy as commander of the First Marine Division in Anbar Province during the early years of the war. Tom Ricks documented Mattis’s “first, do no harm” approach in Fiasco, a 2007 Pulitzer finalist.

Building was another challenge entirely. McMaster worked to restore basic services, stood up a local security force and encouraged municipal workers to return by paying their wages. He rather shrewdly funneled reconstruction funds through tribal sheikhs. Appearances matter; people were more likely to welcome the return of order and infrastructure if they could plausibly deny affiliation with the American occupiers. Sowing trust among the locals proved most daunting. “When we came to Iraq, we didn’t understand the complexity—what it meant for a society to live under a brutal dictatorship, with ethnic and sectarian divisions,” McMaster told Packer. The Jordanian terrorist Abu Musab al-Zarqawi understood the complexity. Under his leadership, Al Qaeda in Iraq had brought Tal Afar to the brink of civil war. Zarqawi’s brand of terror was calculated and effective. McMaster knew that if he couldn’t find a way to end the cycle of revenge killings, his regiment’s progress would evaporate. Taking death squads off the street was but a Band-Aid.

And then the rubber met the road. As Packer observed, “Shiite sheikhs accused the Sunnis of tolerating the presence of terrorists, and Sunni sheikhs accused the Shia of making unwarranted generalizations about them.” McMaster tasked an exceptionally competent squadron commander with reversing Zarqawi’s damage, through diplomacy at the community level. Lt. Col. Chris Hickey had to become an expert in tribal politics overnight—in his words, “to switch the argument from Sunni versus Shia, which was what the terrorists were trying to make the argument, to Iraqi versus takfirin.”

This mismatch is representative of so many failures throughout the occupation. A soldier was forced to play a role better suited to an anthropologist. In The Mission, Dana Priest illustrates how, by the turn of the century, the United States had become over-reliant on its military to carry out a broadening set of objectives, at the cost of the State Department and its “shriveling” resources. In 2006, Praeger reissued David Galula’s 1964 book Counterinsurgency Warfare, with a new foreword by John Nagl. The French counterinsurgent argues that the responsibilities of the soldier, the policeman and the civil servant are drawn into a state of mutual dependence by COIN’s inherently political character.

The position McMaster put Hickey in wasn’t uncommon. Before he was a COINdinista, Peter Mansoor led the First Brigade, First Armored Division. “I was a brigade commander on the ground in 2003–4 and almost never saw a representative of the Coalition Provisional Authority. We desperately needed civilian expertise to help us with reconstruction and to reform the local governments,” the retired colonel told me. “There are certain aspects of counterinsurgency that really only civilians can provide.”

Nagl had a similar experience. “I became a Sunni-Shia religious, cultural and political expert—‘expert’ in quote marks!—on the fly because there was nobody else to do it,” he lamented. “When I was on the ground in Iraq trying to untangle tribal politics, I was despondent that I didn’t have help from the State Department, USAID or anybody. And I became convinced that investment in diplomats would literally have saved my soldiers’ lives.”

Even more comprehensively than Petraeus in Mosul, McMaster supplanted chaos with stability in Tal Afar. His accomplishments were hailed as a wild success. And rightfully so. His was the first systematic counterinsurgency operation of the Iraq War—conducted, as Kaplan points out, “with total independence from headquarters”—at a time when the mere utterance of the “i-word” invited opprobrium. But Tal Afar was one city, and McMaster one man. The insurgents cleared from one town would simply move to another, creating a “balloon-squeezing phenomenon” that prevented security from improving country-wide. It’s what Galula called an “accidental mosaic.” Furthermore, the very nature of military deployments precludes the kind of continuity that might have seen McMaster achieve more in Tal Afar. He got his orders and moved on. On February 22, 2006, Al Qaeda in Iraq blew up the Al-Askari Shia shrine in Samarra. The jihadists got the civil war they wanted.

Back in Washington, the war of ideas Petraeus sparked in 1986 was coming to a head. After years of surreptitious politicking, chain-of-command bending and painstaking intellectual salesmanship, Petraeus and McMaster were finally making inroads. In September 2006, Petreaus installed McMaster and Mansoor into a secret advisory group the joint chiefs had assembled to rethink Iraq. The “council of colonels” was afforded the opportunity to dispense with ceremony and tell the most senior officers in America’s military the hard truths they couldn’t countenance: Iraq was roiled in an insurgency. Clean victory and hasty withdrawal were pipedreams. Killing the enemy would not be enough. Only an injection of new commanders, resources and doctrine could salvage the crisis. Meanwhile, a network of influential experts, scrupulously cultivated by Petraeus, was working to convince President Bush to sign off.

They were on the cusp. Petraeus and his cadre of doctrinal disobedients put pen to paper. On December 15, 2006, they published FM 3-24, the COINdinistas’ true statement of purpose. The paradigm-shifting counterinsurgency field manual, which cites McMaster’s exploits in Tal Afar and Mattis’s in Anbar Province, turned the American way of war upside down. It codified hearts-and-minds COIN. Corporals and lieutenants would have to unlearn much of what the military had taught them. An insurgency’s center of gravity is the civilian population. The public is the prize; win them over and the insurgents lose support. FM 3-24 made its authors look more like agents of change than petulant insubordinates. They had broken free of the fortress-cloister.