How could you be ruining the hosts gank squad? If the host is ganking he'd keep you to also help gank. If the host was letting invaders gank you, he'd keep you so they could gank you...
I send blues home regularly when there is already a blue, and only one invader, and they are having a duel. The second blue always rushes into the fight so I just banish immediately in this case. If there's a second invader waiting with me, or the invader that is fighting has asked me to let the blues try to gank them (yeah, there are some invaders with serious skill/balls; *cough* supramatti) I'll let you stay.
It's not personal, and if you message me your hate after, I'll explain what's going on and let you stay next time you get summoned (I remember you).
And I don't let them gank you either, so why should I let you gank them?
PSN smellerbee2000 SL 154 crucifixion woods