红豆 here is not the kidney/adzuki bean but the bright-red seed of the coral bean tree (
相思子 /
红豆树), traditionally gathered in Guangdong and Guangxi. Still sold today as keepsakes and wedding gifts. The whole poem is memorized by almost every Chinese schoolchild.