First attested in the Ming novel 《封神演义》 by Xu Zhonglin, chapter 43: 吾等又不是凡夫俗子 ('We are not ordinary mortals'). The phrase contrasts ordinary humans with sages or immortals.
usage
Often self-deprecating ('I'm just an ordinary person, what would I know') or used to contrast someone with the truly elite or extraordinary.