This Xià reading appears almost only in 厦门 (the city Xiamen, historically romanized 'Amoy'). For the 'tall building' meaning, the same glyph is read shà.
The cliff-shelter radical wraps the top and left, picturing a roof or eave overhanging a structure, which is fitting for a word meaning a large building.
