Not sure if this is how it works for everyone but if I challenge Fighter B and he returns my challenge I lose the desire to jump him unwillingly later on. If he has done something to earn the challenge (which is usually the case, not just because he exists) and then ignores the challenge then he'll probably be killed a few times because of it.
Too many times young fighters are mouthy, rude and disrespectful and expect no repercussions for it and that is their mistake. Not the bigger one. Sometimes even being a city enemy is enough to warrant a slaying because well that's what warring enemy cities do. It's usually up to the individuals on how far they want to go to punish someone of smaller size.
It's RARE that a small fighter is repeatedly harassed for no obvious reason and can always ask the bigger fighter to lay off. This usually works 95% of the time unless the bigger fighter is completely heartless. Sadly most would rather play the I defy you martyr role and then act the victim later.
Written by my hand on the 17th of Leaflost, in the year 1250.