Thats awful! So sorry to hear your story. That site manager isn't living up to their job title at all!!! Sure, there may be nothing in the rules about using weedkiller, but that irresponsible vandalism of what is your property.
Is the site manager a good friend of the people who were using the weedkiller (visualising 'old boys' clubs and looking out for their own kinda nonsense)
Either way, I wouldnt let this drop if I were you... The council will have rules about negatively affecting neighbours even if the site manager is blind to it.
I grow organically on my plot- i would have a pink fit if one of my neighbours was letting weedkiller spray onto my plot- I wouldn't want those chemicals in my soil... If nothing else, if you were unaware that the spray had gone onto your soil, you could have been unwittingly trying to plant things on there before it had broken down. Bleaurgh!
Surely that would be a health hazard??! If they're not capable of being responsible with a toxin, they shouldnt be allowed to use it willy nilly!
I don't blame you for wanting to pack it in- the site managers response is almost more disappointing than the act of vandalism.... but why let them get away with it? If you say nothing, then nothing changes! If you escalate it to the council, then things might improve- for everyone on the plot, as people who are so careless to others are bound to do it again, and annoy someone else.
Sigh rant grumble....