When the black death hit Norway in 1349, 2/3 of the population was wiped out. The whole structure of society broke down. Most of the state and church was just gone. It took 300 years to get everything back in operation because most people that could read and write had died.
I don't know how you define "long term", but for several generations society didn't work, there were no justice and people really suffered.
Think about how many highly specialized jobs are needed to run modern-day society. Think about how much of the required knowledge and skill to do those jobs would be lost, not to mention available person-hours. We could be set back a century or more, but without the cheap and plentiful natural resources that got us to where we are today.