As long as you are not operating early or late, I wouldn't worry about it. And hopefully, the end is in sight.
Yep, neighbors can be a royal PAIN!!! In order to give any opinion, a bit more info would be nice: Size of property and sizes of properties around you? Developed or undeveloped area? You in the boonies or near lots of population? Wealth of area? NIMBY is alive and well the further up the economic ladder you go, the worse it gets. With those questions about the only other thing is have you introduced yourself to the neighbors and explained what is going on?
I take it you don't require permits to do this work? Some places, changing the grade requires approval from the governing bodies. I imagine you don't or the neighbors would be reporting you. The neighbors may be concerned, not about the noise, traffic etc., but that the water may end up on their property. It has to go somewhere.....just sayin.' What did the police have to say? You mention retaliation....How would you retaliate, nothing illegal I hope?
In this case, it isn't the police you may need to worry about if you aren't breaking sound ordinance or something stupid. That's just a civil infraction. You might need to worry about the EPA. They'll get you, AND make you put things back to the way they were. About a year ago I was hauling sand from one corner of our property to another. I was working on the loader, and a guy drove up and started talking, nicely and respectfully. He says "I think you're taking sand from my property." Uh-oh... We didn't have a survey done when we bought it, as I THOUGHT I knew the boundary. Apparently not quite. I was unknowingly driving on his land (a remote corner of it), but where I was taking sand was about right on the border. I thought the fine and cease and desist order was coming. Ended up the guy is SUPER cool. He didn't care about what had been done, but he asked that I not operate heavy equipment on his side for liability reasons. No problem! These days we keep to ourselves, but we're both on very friendly terms. Good neighbors.
Don't know the laws in your area, BUT around here, activity like that will land you into the jaws of the EPA and they will have a field day with you!----- Make positive that you have ALL permits in order from ALL govt. agencies involved in your area ---- That POS neighbor only has to call EPA and it's all over except the shouting for you! ----Back up and get everything in order my friend!
We don't deal with such crap here. Also, runoff from above you is their problem unless someone was stupid and built your home in a natural creek bed. Make them take care of it and see if they complain even louder. This is one instance where hiring legal counsel my be to your advantage.
Those interested in reading the final EPA ruling, dated 1-23-2020, can read about it here. Stormwater runoff has different rules in different states, some even tax it! Here in Florida, those above you from where the water comes from, have no responsibility to mitigate anything. If you are impacted by the stormwater runoff, you deal with it. That's the law. One thing is though if the stormwater crosses city or county owned land, you can get them to do something about it. While it did take years, the city finally removed the sediment buildup in the swale on the right-of-way adjacent to my property that caused it to flood during heavy downpours.
Minor note about EPA rulings, they are somewhat dependent on the President and party in power, and can quickly change with power swings.