The birth of Excalibur

Only weeks before the 2024 General Election was called, the Labour History Group held a meeting in the House of Lords. It was intended to be the first in a planned series of discussions about Labour’s General Election victory of 1997.

The advertised topic was “Excalibur” and the role it played in the Party’s media operations in the run up to and during that year’s great Labour victory.  In essence Excalibur was a gigantic electronic political database. It was the first of its kind in Europe and joint-first in the world.

The room was packed. There was plainly a great deal of interest.

For me it was sobering to think what was once fresh and vivid in one’s mind, because one was part of it, was now a topic of historical interest. Tempus fugit.

However, the subject also had a degree of topicality based on media reports which suggested Keir Starmer was planning to upgrade Labour’s campaigning ability by establishing “Excalibur 2”.  

At the time of the meeting in the Lords I didn’t know if he had or not but much of the reporting and chatter on the origins of “Excalibur 1” had been irksomely wide of the mark so for some time I had been thinking about “putting the record straight”. The only question was how and when.

The other listed speakers at the History Group’s meeting that night were Adrian McMenamin, Faz Hakim, Jo Gibbons and Richard Elsen. They had been part of the wider ’97 effort but two core members of the dedicated Excalibur team, Jayant Chavda and Daniel Beard were also present. All were heroes. Some of the poor bloody infantry of the ’97 campaign.

A delay seemed appropriate

But with another General Election upon us, publishing something about the ‘97 election as the ‘24 one was just getting underway would have been a little odd, to say the least.

A delay seemed appropriate.

That delay is over.

The progenitor of Excalibur was Bill Clinton 

The US Democrats’ triumph in November 1992 gave the British Labour Party great encouragement. 

Following the huge shock of John Major’s unexpected victory in April of that year, it was an enormous relief to see left-of-centre parties could still win elections at least somewhere. Throughout the Labour Party everyone wanted to know more, in depth, in detail. How had the Democrats done it, politically and organizationally?

Activists to America

There is a well-established tradition of Labour people going to the USA to help the Democrats during their Presidential elections. 1992 was no exception.

The accounts that came back from America coalesced into a narrative about how the Clinton people had used new technology in novel ways to help project their politics. There was definitely something in this, but not a lot. For the Democrats, the new technologies were really going to come into their own four years later when Clinton was up for re-election.

Read on, but first…

Back to the UK

Following Neil Kinnock’s election as Leader in 1983, “modernization” had become the watchword. He set about overhauling Party policies and the machinery which was meant to help project them to the electorate.

John Smith was elected Leader in July 1992. He made clear he would support OMOV (one member, one vote for internal party elections) but in September 1993, much to the dismay of the modernizers, modernization was put on ice.

Tony Blair won the Leadership in July 1994. He revived modernization and put it on steroids. New Leader. Soon to be “New Labour”. New methods.

Tony wanted the Party to be seen to be learning from, associated with and embracing a sparkling Clintonesque modernity. In addition, and as part of the total Blairite package, Tony and his advisers demanded a more streamlined and efficient way of making campaign decisions. In turn this required a better and thoroughly reliable way to fact-check Government and Tory claims, while also ensuring Labour’s positions were solidly based. After so many years of Tory rule Whitehall press officers and Whitehall research resources had effectively been co-opted into Conservative Central Office. The Labour Party was completely outgunned. This was no longer acceptable. It had to change.

A slight historical detour is appropriate at this point

Life before Google

In what were, for practical purposes, pre-internet days, if you wanted information of any sort you had to acquire it and organise it yourself. Google did not exist. Search engines were beginning to appear but, to put it mildly, findable material was limited and eclectic.

The stuff of politics and political intelligence was still to be found in speeches, articles and books by Ministers, by other politicians and academics, Government reports, enquiry findings, expert and media commentary, Think Tank documents, press releases, stories which had appeared in local and regional newspapers but maybe hadn’t gone national and all this supplemented by a Rolodex of phone numbers, filing cabinets and folders stuffed with cuttings.

In other words paper-based sources ruled. However, typically, these written sources were scattered to the winds. You needed to track them down or get someone to do it for you then identify and digest the relevant bits, often within minutes, or at best hours. Ideally you would want to be able to respond within the same news cycle, an idea that was to become central to the notion of “rapid rebuttal”, an idea which had crossed the Atlantic.

Managing or getting hold of all these paper-based data could be a haphazard and often frustrating business, not infrequently leading to sub-optimal results. Yet with varying degrees of comprehensiveness, it was exactly the same in both Walworth Road (Labour HQ) and Smith Square (Conservative Central Office). Most of Fleet Street, broadcast media, national and foreign, were in the same position.

Could tech help?

Back to the main narrative.

The messages about a marriage between tech and politics that had been brought back from America by Labour people who had been there to help Clinton in 1992 were essentially anecdotal and most definitely incomplete.

I was never clear whether it was Philip Gould or Patricia Hewitt who suggested the Labour Party send a mission to America to confirm the nitty-gritty of what the Clinton people had actually done with tech in ’92 and find out as much as possible about what they were planning for the re-election effort in ‘96. At his first Party Conference as Leader, in October 1994, that is what Tony asked me to do.

Tony and I had known each other for many years. Under the aegis of Jack Straw and Mo Mowlem Barry Cox managed the money and I had run the internal Party effort for Tony’s Leadership election campaign. He knew I was one of the comparatively few people on the left who worked with and was comfortable around computers and new technology.    

Forward with Folletts

At that time Ken Follett was the Party’s principal fundraiser. He and his wife Barbara had been enthusiastic modernisers, key allies of Neil Kinnock. Tony asked Ken to help arrange a visit to the USA. Ken agreed. Philip Gould and Patricia Hewitt both had good contacts and Ken was a huge celebrity over there. We had no problem getting people to meet with us. 

Ken, Barbara, Patricia and I flew to DC on 18^th December 1994. The Clinton crowd wanted to know more about the Blair project, and it became clear they also wanted to meet Ken. He signed several books people brought with them to the meetings we had. It was a great mix.

We had lunch at the White House with George Stephanopoulos, delivered Christmas presents sent over by Philip Gould, and, crucially, met with the leading people who were working on the ‘96-re-election effort, including Bob Schrum, James Carville and Paul Begala. James Carville is the person usually credited with the idea of “rapid rebuttal” as a core technique, an essential part of any strategy for dealing with the media in a political campaign.

He saw rapid rebuttal as crucial to both a defensive and an offensive strategy

“it’s difficult for your enemy to attack you if you always have your fist in his mouth”.  

And that was how “rapid rebuttal” became established in the UK’s political lexicon.

Finding Excalibur

Before going to the USA, I had researched available software packages to help organize the paper-based data we knew we were going to need in what lay ahead.

“Excalibur” was not a title invented by a spin doctor, although its poetic appeal is obvious. It was simply the brand name of an “electronic filing system”, developed by a company in Karlsbad, California. 

Excalibur was on my list before we got on the plane. They had an office near DC. A demo was arranged. Eric Berman of the Democratic Party National Committee came with us. He revealed they were planning to go with Excalibur as the cornerstone of their ‘96 rebuttal operation.

That pretty much sealed the deal. The Labour Party would go with Excalibur as well. We could help and support each other as our projects moved along.

On our return from the USA, I drew up an implementation plan with costings. Initial estimates suggested it was going to take £300,000 to get us started and roughly a million pounds would be needed to put it all together and sustain it up to the anticipated date of a General Election. Ken was going to front the fund-raising. Sir Richard (Dickie) Attenborough had agreed to contribute as had several other wealthy individuals.

Calamity

Disaster struck. For entirely unrelated reasons Tony decided to drop Ken as the Party’s main fundraiser.

By this time Jonathan Powell had become the head of Tony’s office. Immediately before taking up his post he had been based in the British Embassy in Washington DC so he too had good contacts with the Democrats. Jonathan had grasped the significance of the Excalibur idea. Unfortunately, the Labour Party’s main financial backers – the Trade Unions – didn’t. They were not willing to sink what they thought was a considerable sum of money into an unproven technology project.

I was left with a fully worked out blueprint but no means of implementing it. Higher-ups in the Party probably thought Excalibur was a dead duck but with Jonathan’s blessing and encouragement, on a voluntary basis I carried on looking for the money.  I visited senior people at The Guardian. They liked the novelty of the idea of building a major, electronic political database and letting the Labour Party have access but in the end Hugo Young vetoed it saying the whole thing was too obviously “big p” political. 

I saw other companies and organizations. Either the money was beyond them, or they also thought the idea too weird, too risky or, again, too “big p” political.

I went to dinner

During all this I went to dinner at Mo Mowlem’s house. Everyone around the table was a Labour insider. I told them the story behind my “blueprint without any money”. One of the guys said he would introduce me to the owner of the New Statesman, multi-millionaire Philip Jeffrey. Philip was, like me, a Leeds lad and a Leeds United supporter. Unlike me he was rich, having made his pile in DIY and paint products.  

We met a few days after the dinner at Mo’s. I explained the concept and the background. Quick as a flash he said “Tony must have it. I’ll pay.”  No paperwork. No application form. The deal was done in minutes.

Philip Jeffrey employed me. I was never on the Party’s staff or integrated into the Party’s official hierarchy or processes. On reflection that may have been a mistake, but I hired staff and assembled all the kit to build the system from scratch. I stayed in touch with the Democratic Party HQ but as it became clear Clinton was going to coast to victory over Bob Dole, they took their foot off the gas. We didn’t.  In the end the UK’s political database was almost certainly larger than the Americans’.

Glitches and hitches

As with every major technology project all kinds of glitches, hitches and delays had to be overcome. The Party’s leadership, though by no means techno-phobic, were not exactly relaxed in its presence.

 “How to build and use a very large, electronic political database. For dummies”

had yet to be written.

Using massive and extraordinarily expensive Fujitsu machines, we began scanning and digitising pages from the wide range of sources mentioned earlier. Excalibur converted the results into searchable text. It allowed us to query a vast array of information simultaneously, not unlike the way today’s AI and search technologies operate. But that’s now. This was then.

Later, the Labour Party developed structured databases to streamline operations. “Streamlining” in this context meant Excalibur could identify and retrieve relevant data in usable form at even greater speeds while also minimising the need to sift through irrelevant data. However, these streamlining advances were only possible because of the groundwork laid in the initial build-and- data-acquisition phase.

We were experimenting. Learning as we went along. People with a low awareness of tech don’t always understand that’s how these things go. They get impatient. They expect everything to come together and be perfect a lot faster than was ever going to be possible. Also, at this point, no major Party big wig felt they owned Excalibur or wanted to be too closely identified with it, likely in case it failed.

“Arms to Iraq” changed everything

The internal perception of Excalibur started to change when the Scott Report into arms to Iraq finally appeared.

The trigger for the Scott Report was the collapse of the Matrix Churchill Old Bailey trial in 1992.

The 5 volume 17lb Scott Report had taken three years to prepare. It was published on 15^th February 1996. Robin Cook was only allowed to have sight of it three hours before it was released to MPs and the public at 3.30.p.m. that day. By then the Government had had it for over a week, perhaps more. The Secretary of State was scheduled to make a statement in the Commons sometime shortly after 3.40.p.m.  The Opposition would be expected to respond immediately.

Robin Cook in a dark place

Around mid-day, on his own, Cook had to go to a windowless room where a copy of the Scott Report had been placed at his disposal. He was not allowed to take it out of the room or to photocopy any part of it. He could make his own notes by hand but that was it.

However, Cook did not exactly walk naked into that gloomy space. Before 15^th February Excalibur had digested every previous reference to the Matrix Churchill trial and arms to Iraq that had appeared in the public domain.  Jayant Chavda on the Excalibur team had been working with Cook’s office developing likely lines of attack. The Excalibur sourced aide-memoires which Cook carried in his pocket guided him in what to look for.

In the Commons, about 4.00.p.m Cook rose to speak in what was hailed as a brilliant example of the best kind of House of Commons oratory. But the larger value of Excalibur only became evident over the next few days. By midnight on 15^th February, the whole of the Scott Report had been taken apart (literally) then scanned, meaning everything in it was now searchable. The Labour Party had put itself ahead of the Government, all the media outlets and everybody else for that matter, in terms of having an ability to find and cross reference statements or particular words in super-fast time.

That “Press Pack”

At first the government narrative dominated the media. Alongside the Scott Report they had issued a “Press Pack” . Over 40 pages long, it was filled with crisp quotes and selective extracts which were intended to “help” journalists get to the heart of what the Government wanted them to write. But Excalibur swiftly enabled Labour to demolish their preferred account. The public argument started to be transformed.

This is what The Guardian leader said about the Press Pack on 17^th February.

“Once upon a time senior Whitehall press officers had sufficient clout to intercede and require a government minister to redraft statements or press releases which contained inaccuracies or exaggerated claims. Now it seems even the cabinet secretary is powerless to intervene when desperate ministers produce - and release under departmental imprint - totally distorted summaries of a crucial official inquiry. The government's press pack released with the report contained 13 separate releases from government departments each filled with fibs, fabrication, falsehoods and just plain lies.”

Elsewhere in the paper that day Patrick Wintour reported as follows

“Labour and the Liberal Democrats yesterday despatched a dossier of quotes from Sir Richard Scott's report on the arms-to-Iraq affair to every Tory MP in an attempt to persuade them of the extent of Sir Richard's criticisms of the Government.

 Labour's shadow foreign secretary Robin Cook and the Liberal Democrat spokesman Menzies Campbell, holding a rare joint press conference, urged Tory MPs to study the report and recognise the former Foreign Office minister, William Waldegrave, had been found guilty of deliberately misleading Parliament more than 30 times.”

And this was possible because

“A dozen researchers working overnight at Labour's media headquarters gathered a damning collection of quotes from the 1,800-page report, including Sir Richard's statement that ministers breached their own rules on parliamentary conduct.”

This was Excalibur doing its work with the researchers aided and abetted by the likes of Charlie Falconer QC and James Goudie QC who toiled with them into the wee hours in Millbank Tower. Excalibur’s part in Cook’s and the Party’s political success was acknowledged within Party circles. Its value was better understood.

Excalibur strikes again

There was another famous moment nine months later when Labour issued a point-by-point rebuttal of a Tory attempt to repeat the 1992 “Labour tax bombshell” scare.

A not insignificant part of the media coverage which followed was about the speed and effectiveness of Labour’s rebuttal, betokening Labour’s new found campaigning prowess. The Party was definitely frightening the horses and as a result acquiring a bit of a swagger itself. Blair’s methods were very visibly working.

Tory MPs getting worried

Tory MPs in marginal seats had been edgy for a while. This surprisingly modern stuff from the Labour Party wasn’t helping. That message went up the line to No. 10 and Central Office. The Tory Party Chairman, Brian Mawhinney, asked young Danny Finklestein to check out if they could acquire and build something like Excalibur, only better. They wanted to soothe Tory nerves, showing the troops their leaders were on the case.

Finklestein reported that Excalibur was the best in the field. Mawhinney told him to look again and find something else because they did not want to be seen to be simply copying Labour. This cost the Tories valuable time. Finklestein came back and said it was still Excalibur. The Tories bought an Excalibur system. Imitation is the sincerest form of flattery, but it did them no good.  They had left it too late. Their Excalibur system never actually worked in the way intended before the 1997 General Election but even if it had it’s hard to imagine it would have made any real difference. The Tories were sunk long before.

Things move up a gear

Within the Labour Party any doubts which might have existed about Excalibur’s usefulness or potential had now been utterly vanquished. Internal politicking moved up a gear. Who within the Party machine would own this shiny new object and bask in its reflected glory?

During the General Election campaign proper, Adrian McMenamin was in day-to-day charge of Excalibur. Adrian knew about tech so that put him at an advantage compared with almost everyone else. I’m not sure how the chain of command worked above Adrian. Alastair Campbell, Peter Mandelson, Philip Gould and Matthew Taylor were in there somewhere. I was gone by then but someone like Adrian probably should have been brought in and put in charge sooner to help with crafting, shaping and organizing Excalibur’s formidable power. I had secured the money and put the system together. Getting it to work the way the Party needed it to, that was a different challenge.

For the first time in the Party’s Post-War history, in an area of fundamental importance to politics Labour was ahead of the Tories and every major media outlet.  Nobody else had anything like Labour’s Excalibur. Today almost everyone does, thanks to the internet, search engines and now AI.

New smartphones have more computing power and greater access to information than the “ur-Excalibur” ever had.  As AI grows in importance it will become easier and easier to locate, analyse and digest a vast range of information sources. But deciding what you do with the product will remain a human task for the foreseeable future. Nuance and timing matter in politics. In that department computer code will struggle.

And then what?

After the election was over, I anticipated the Party would be short of money and would therefore find it difficult to sustain Excalibur in the way it had done hitherto. An out-of-date database is potentially a lethal weapon. I mentioned this to Ken Follett and, possibly based on what I had told him about The Guardian’s original interest, he said he would be willing to buy the whole Excalibur system from the Party, maintain it in the way it needed to be maintained and sell back access to the Labour Party at favourable rates while also selling access to media outlets on a commercial basis.

I relayed this information to Tom Sawyer, General Secretary of the Party. Tom came back and said the idea had been turned down because Ken Follett remained persona non grata.

I gather Excalibur was, nevertheless, still used in some form or other in the election of 2001, but it was not available at all to the Party in 2005 or 2010. Not only had it been moth-balled, apparently its contents had become inaccessible.

New Labour triumphant

In the run up to and during the 1997 General Election campaign, Excalibur was a vital part of Labour’s campaigning armoury, but perhaps more importantly it had contributed to and helped consolidate a modernist culture within the Party. The swagger referred to above and the self-confidence Labour had acquired were important not just in unsettling the Tories and impressing journalists. They were also important to the Labour Party itself. Excalibur was serious, concrete and reassuring proof of “actually existing Blairism”. Cutting edge. Looking to Labour’s future, not its past.