In imperial Chinese courts, a drum was beaten to signal the start and end of proceedings. The
退堂鼓 (tuìtánggǔ) — literally 'retreat-court drum' — was struck to dismiss the session. The idiom captures the idea of ending things before they are properly concluded, i.e., backing out.