Yes, but the greatest danger is the "Penalty Game", as the winner can inflict a punishment on the loser, depending on the magic being poured into the game.

Or it can be baked into the Duel.

Imagine every Life Point lost, say, a family member becomes more of a cowardly crybaby.

Or they could be told to forget you.