"W-with me?" Rher or Per'kele or whatever had been trying to get him to fight and kill everyone, but he never gave in. "...no, not when I wasn't in real danger."

On the battlefield in the middle of a war wasn't really the same thing, his life had really been on the line then. But it wasn't here, so he thinks he should have just left.