I'd get rid right now. Some councils remove nests for free, but if not would pay a professional to do the deed.
We had a nest in the hedge, and let it be, thinking it was nice and natural and they weren't doing any harm.
Then, whilst working in the garden, hammering some posts in, the vibratiions must have upset them. The whole swarm gave chase and we had to run for it. Me and the dogs got away clean, but OH got stung.
So we got a man in and he 'treated' the nest and removed it.
Understand you not wanting to harm them, we didn't either, but learnt the painful way.