In this reading 氓 is bound to 流氓 (流氓), which means both 'hooligan/rogue' and the behavior of harassing or molesting someone (耍流氓 = 'to act indecently'). It is a common, fairly strong everyday insult.
亡 means 'to flee / be lost'. Combined with 民 (民, the people), the original picture is displaced wanderers — rootless people, which later colored the 'rogue/drifter' sense.
