木 is the indexing tree radical — a horizontal-and-vertical drawing of a trunk with branches above and roots below. It places
橘 (mandarin orange) firmly in the fruit-tree family alongside
桃 (peach),
李 (plum),
梨 (pear),
柚 (pomelo). The radical signals: this names a tree, not just the fruit.