Our son was up the allotment two days after he was born in early October last year. OK, I didn't get a lot done that day, but he's now 9 months old and we've had our best allotment year ever.
Before he's mobile, you'll be able to get lots done. You need to be ready to drop what you're doing and feed, change nappies, comfort etc. but you can still work away. Light duties on the lottie were great for my wife - a nice, easy, satisfying way for her to get mobile and strong again. She had a big cut, and found it difficult to walk, but could push a hoe (sorry to go all Mumsnet detailed there).
If two of you can go, it's easier.
We planned our work this year really well and it helped hugely. Got the seed order in and wrote down what needed to be sown and transplanted when, stuck it up on the wall, and used it to keep us going. Much easier than trying to remember or having to look at a dozen different packets and work out what you need to do next.
It's also really good for baby to be outdoors early - it helps them to learn the difference between day and night, if they're warm enough they'll happily lie there, good for their eyes as they can focus on distant things, and your lottie neighbours will be all gooey-eyed.
On other occasions, as the inevitable stress of the huge changes took its toll, the lottie was a great place for me to escape to.
Life doesn't and shouldn't come to a stop because of the baby. He came into OUR life, not the other way around.
And one top tip - if you haven't already, buy and read "Sleep" by Beatrice Hollyer. Our son was sleeping all night by 8 weeks, and now sleeps 13 hours a night, largely thanks to this book.