Figurative use comes from
《南史·檀道济传》: when the Liu-Song general Tan Daoji was arrested on false charges, he hurled his headdress to the ground and cried
乃坏汝万里长城 - 'You are destroying your own Great Wall!' Thereafter
万里长城 became shorthand for an irreplaceable defender of the state.