替罪羊 is a calque from the Western concept of the scapegoat (from the Hebrew Bible). The Chinese version literally means "substitute-sin-goat." The concept entered Chinese through biblical translations.
memory
替 (substitute) + 罪 (sin/guilt) + 羊 (goat/sheep) = a goat that takes on sins for others. Picture a goat wearing a prison uniform.