Then again, this isn't entirely your fault either.
Tips
usage
A discourse marker that walks back or qualifies what you just said, very common in conversation. Often paired with 不过 or 但: 不过话说回来…, 但话说回来…. Its near-twin 话又说回来 means almost the same thing - both signal 'switching to the other side of the argument'.