I hate to suggest this --- but have you thought about giving up your garden plot? I'm assuming you are doing city-type gardening where you don't actually own your plot? Yes, giving it up would look like he "won" but at the same time you have to look out for yourself. Clearly, your gut is screaming out "danger!" to you and you need to listen to that! Are there other garden plots available, even though they may not be near where you live?
If you don't want to give up the plot, do you have a friend or relative who can accompany you now and then while you garden? This person won't so much be there for protection but rather to act as a witness to this man's behavior. Also, document everything and take pictures. If you can come up with some concrete proof of what he's done, your next step would be to get a restraining order out on him.
As for talking to the other gardeners --- one thing you need to know about narcissists is that they are very manipulative. No doubt, he has charmed the others. If they have turned against you, I'm thinking that he has spoken to them and badmouthed you to them. So, talking to them most likely won't work. If anything, they will end up taking his side.
I wish you luck as you deal with this. Personally, if it were me, I would give up the plot. No one needs to live in such fear!