番 marks fruit/plants introduced from abroad (cf.
番茄 'tomato',
番薯 'sweet potato'). Here
番荔枝 is 'foreign lychee' — the bumpy green skin vaguely resembles lychee's, but the flesh tastes like sweet custard. Also called
释迦 (shìjiā) in Taiwan for its resemblance to Buddha's topknot.