Where’s the usual excitement about an imminent change of government in the UK?

London: Why are so many voters so unsatisfied with Britain’s main parties? In the opinion polls, despite Labour’s strong lead, their combined support is low by historic standards. Barely two-thirds of people say they will vote for them. One poll from this weekend predicted the Conservatives’ worst ever election result: only 98 seats. Meanwhile Labour’s membership is falling fast. Also, unusually, the main parties’ leaders are unpopular at the same time. While Rishi Sunak is one of the least-liked prime ministers ever, Keir Starmer looks likely to succeed him with his personal ratings firmly negative and trending downwards. The unacceptable replaced by the unpalatable.

This is not how Britain is meant to move towards a new political era. In the run-up to 1964, 1979 and 1997 – our three most mythologised modern changes of government – there was widespread excitement about the administration in waiting. There was also keen anticipation across the right before the 2010 election, until Cleggmania and a muddled Conservative campaign produced a hung parliament. In all four cases, a common desire to be rid of a tired old government was at least partially transformed into enthusiasm for the likely new one.

Yet since Starmer became our probable next prime minister more than two years ago, as Boris Johnson’s administration began to disintegrate, such a transfer of political energy has stubbornly refused to happen – and nor has a Conservative revival. Some of the reasons for this stasis are fairly obvious: Sunak and Starmer’s limitations as public figures; the lack of serious Tory policies and eye-catching Labour ones; the rise of Reform UK and the revival of the Greens and Liberal Democrats.

A less noticed factor is that the Commons has become ever more fragmented, with more than a dozen parties represented, as well as more than a dozen independent MPs, most of whom have effectively been ejected from the main parties. The absence of Diane Abbott and Jeremy Corbyn, in particular, has made Labour blander and less appealing to leftwing voters. Meanwhile, the Tories’ seemingly endless shift to the right is alienating their more centrist supporters. That some voters are switching off from this repetitive, ideologically limited two-party drama is hardly surprising.

However, the disconnect between the main parties and many of their supporters goes much deeper than the unsatisfying politics of the past few years. Its origins are in the turbulent 1980s, when so much of today’s society was created.

In that decade, the Conservatives became the party of deregulated capitalism, above all else, thanks to Margaret Thatcher’s economic reforms. Meanwhile, Labour began to be less associated with economic equality than before, and more with multiculturalism and social liberalism, through Labour-run bodies such as the controversial but influential Greater London Council (GLC). These respective party roles and identities have been challenged periodically since, but they have endured and profoundly changed the country. Britain today is ever more accepting of diversity, and ever more damaged by market ideology. In effect, since the 80s, Labour and the Tories have each won one big, society-shaping, battle.

But have they won the victory their supporters most wanted? Research into the underlying attitudes of Tory and Labour voters suggests otherwise. In 2020, a report by the thinktank The UK in a Changing Europe found that Conservative voters were far more likely than Tory MPs to believe that “big business takes advantage of ordinary people”, that “management will always try to get the better of employees”, and that “working people do not get their fair share of the nation’s wealth”. This surprisingly leftwing outlook suggests that, in the eyes of many Tory voters, Thatcher’s economic revolution was either misguided, went too far, or both.

The report found a similar Labour tension over social values. Labour voters were more likely than Labour MPs to back the death penalty, to believe that “young people don’t have enough respect for traditional British values”, and that “schools should teach children to obey authority”. On this evidence, the conversion of all Labour supporters to social liberalism remains far from complete.

What many Labour voters do still strongly desire, the report suggested, is a more pro-worker, more equal economy – while many Tory voters still want a more ordered, less permissive society. In other words, both groups yearn for the country that existed before Thatcherism and GLC-style liberalism began the great unravelling of paternalistic postwar Britain.

The Tories’ culture wars and Starmer’s constant calls for “an economy that works for working people” are attempts to satisfy these yearnings – and also acknowledgments of the need to reconnect with alienated supporters. Yet neither party has produced policies of the scale and systematic quality that an economic or social counter-revolution would need. Under the stern, methodical Starmer, Labour may have the seriousness and work ethic to change the country – certainly more so than Sunak’s blustering, manic Tories – but truly transformational governments are much readier to take on vested interests than Starmer has so far appeared to be. It’s likely that many leftwing and rightwing Britons will go on feeling in their gut that the main parties have let them down.

For more than 30 years, more people have been switching parties from election to election than in the postwar period. Weakening class identities and strengthening individualism have fed this volatility. So have party-fracturing events such as the Brexit and Scottish independence referendums, as well as Britain’s perceived decline and the increasing hardship many individuals are suffering. Millions of voters are rushing from one party to another, looking for a saviour.

This flux and instability is not confined to the main parties. Last month, the political analyst James Kanagasooriam, the first person to identify the “red wall”, examined the attitudes of supporters of the Lib Dems, Greens, SNP, Plaid Cymru and Reform UK, alongside those of voters for the main parties. “Each party’s [policy] platform isn’t that well calibrated to its voters,” he concluded. For what attracts people to a party: “Vibes seem to be the order of the day.”

After 14 years of disastrously over-promising Tory premiers, and Labour policies that have never been enacted, many Britons have stopped listening to what politicians say they will do in office. But being guided by a party’s “vibes” instead feels unsuited to a time when concrete solutions are needed for so many crises. Until the electoral system changes, Britain is probably stuck with Labour and the Tories. They could connect much better with voters. But voters could also try connecting better with them.

Andy Beckett is a Guardian columnist. His new book, The Searchers: Five Rebels, Their Dream of a Different Britain, and Their Many Enemies, will be published in May