No, I don’t think so. If we look back to Catelyn II, we see a Brienne who is ride-or-die (or more accurately, ride-and-die) for Renly. So he’s not going to stop her from leaving.
But a hundred picked men is going to be noticeable – they’re going to be wearing his livery and fight under his banner, so it visibly commits him to Renly’s cause no matter what.