I guess I would spray them. This is done at NIGHT, when they are all in the nests.
You take a flashlight, aim it at the nest, leave it sit and walk away to spray. Wasps will head for the light when disturbed, YOU are not close! Get a good brand wasp spray, that can shoot for a distance. Soak the nest down.
For the siding, you may try spraying, but often best to call an exterminator. I tried spraying our siding last year, but never got the nest area, wasps kept appearing in the house. If this happens this year, I will call an exterminator instead of wasting my spray and time doing it myself.
Bees usually are NOT agressive, but wasps are. Wasps come in all sizes, tiny to quite large. Try not to spray those wasps or bees that are not in your way or trying to get you. With Bee Colony Collapse going on, wild bee hives are getting to be real scarce and there is nothing to pollinate the plants of crops and flowers. Wasps do some of that pollinating, killing them ALL off is harmful to the ecology.
If you have never seen them, there is a HUGE wasp, the Cicadia Wasp that is active in mid-summer. They nest in the ground, but are NOT going to hurt you! We have a couple nests around here and they are quite fascinating, have NEVER stung anyone. The kids got quite excited watching them fly about, interact with other wasps, and even annoyed them a bit with no problems. I caught them bothering the wasps, and asked if kids WANTED to have GIANT stings? They knew about regular wasps, and these are 2" long, with bigger stingers, more poison to hurt! Well that talk ended the wasp harassment, but the kids did continue to watch them. We walk the horses by twice a day, shake the ground, NEVER had any of the big wasps react in any way to people or animals around them. The cats catch and eat a few. Ours are a reddish color, bigger than those shown below.
http://en.wikipedia.org/wiki/Eastern_cicada_killer