Right
番 supplies the sound fān with no drift.
番 originally depicted an animal's clawed footprint pressed into a field - by extension 'one turn around, a round.' Same phonetic appears in
翻 (flip),
播 (broadcast),
蕃 (luxuriant). For
幡 the picture suits well: a banner that turns and flaps in the wind.