In many [possibly most] UI situations, the UI hows something fairly obvious. Partner makes a slow penalty double, and it is clear he would be happy to hear it removed with an unsuitable hand - and the reverse is that a quick, happy double is one that partner would like you to leave in whatever.
In this case, the first question is: What does a slow 5
It could be that he was considering defending 4
, perhaps taking it out because of fear that it might make. Alternatively it could be that he is wondering whether 5
is enough, considering a more encouraging bid.
The problem as far as deciding what to rule is that these two views work in different ways. If he was considering passing, then this suggests that 6
now is wrong: if he was considering doing more, then this suggests that 6
So I agree with the TD. South is on a guess, and the hesitation from partner does not really help [apart from the fact that he is not allowed to use it].
Suppose South had passed rather than bid 6
, and North had a hand that was borderline whether to defend 4
, so 6
does not make. Now the defence would be asking for an adjustment to 6
While I think this ruling fairly clear, others might disagree on the bridge judgement. For that reason, an appeal is not frivolous.