I suspect everyone goes through a phase of wondering if it's worth it.....
I've been lucky enough not to have major setbacks since I took a half plot on about 4 years ago, and I added the other half plot this year, which has been quite a lot of work, although some of that was 'one off' like doing a major clearance of it, and planting a fruit tree cordon.
I don't have rabbits as a problem though most definitely slugs are, particularly in the last wet month. It can be very tough to do a lot of work then get no reward. Sounds like maybe growing as many plants on before planting them out might help, as would netting to a degree, though you can't net the whole plot! My biggest challenge has been clubroot, and no matter what I've done it's been a problem to a varying degree every year for brassicas. Some areas are worse than others. Mare's tail is our other general challenge, and having got fairly well on top of it on my half plot, I'm now having to start again on the other half.....never ending.
The guy across the path from me has moved much of his plot to 'no dig' and seems to be doing very well with it, with virtually no mare's tail and very few slugs. He has put a lot of compost down for it, and is pretty meticulous about where he sources it.
Good luck and I hope things improve for you.