Technological change has profoundly altered the employment structure of many advanced capitalist democracies. The most pervasive trend is a stark contraction in middle-skilled jobs dominated by repetitive routine tasks. Decreasing job opportunities in routine work confronts the lower middle class with unknown levels of economic vulnerability and threatens the sheltered position in society, which they have achieved during the post-war 20th century. What are the political consequences of the increasingly bleak perspectives for this crucial pillar of society? I argue that individual employment trajectories are key to understand the political repercussions of modernization and occupational change. While remaining in the contracting job environment (fearing decline) increases the demand for identity politics, losing a job in routine work (experiencing decline) prompts an economic response. Panel data from Germany, Switzerland and the UK allow to follow individual employment biographies and electoral behavior over time. Based on a marginal structural model, a tool for causal inference from longitudinal data, I demonstrate that it is indeed fear of social decline rather than the actual experience of economic hardship that drives support for conservative and right-wing populist parties. Support for these parties withers as soon as voters leave their routine jobs - for the better or the worse. The finding that economic hardship is not the root cause for supporting right-wing populists implies that the proposed remedy of more welfare will be an ineffective instrument to counter the recent surge of populist movements observed in much of the developed world.