Right now I've got 5 adults and 5 young. All in raised wire and wood hutches. 4 are under a harbor freight cloth covered carport. The rest are under tarps in two old dog pens. I'm across the Chesapeake from you so my weather's similar.
I wouldn't do a colony. I realize wire and materials are expensive. I'm building a hutch now and sweating how much I'm spending. Everything that's been posted as negatives I agree with.
Snow and rain, they're off the ground and dry. Snow here is generally light until it isn't. The tarps and wire might buckle, plywood roof won't.
I had wasps once in the hutch. Didn't bother the rabbit or me, winter I pulled it out.
Winter you wrap the sides with heavy plastic or tarps on the prevailing wind side.
In separate cages or hutches you can tell who is eating or drinking, control your breeding, handle the kits.
On the ground I would imagine you'll have more problems with ticks, mites and probably fleas.
Summer time definitely sucks it's been close to 100 under the carport couple of times. Freeze some soda bottles, ceramic or granite tiles. They lay on them or not but I give them the chance. I change the water twice a day.
On the ground they will eat everything nothing will grow for long. You'd have too have different pens like having different pastures. In a cage they relieve themselves in the same spot everyday. You'll have to dig that out occasionally on the ground and put down lime to stop the flies.
I wanted to do a colony myself but I feel in the long run hutches/ covered cages are the way to go.
You commented about netting. Use wire. Any kind of net will rot in the ground or in the sun. When you least expect it you'll get a hole. Same with chicken wire.
Any wood you use can't be treated. If they can reach it they chew it and everything I've read that's bad for the animal, dogs included.