Because you live in New York, knowing about and using some crime-prevention tactics would be a good way to protect yourself even if they aren't threatening, harrassing or assaulting you.
Download and print a map of the streets of your neighborhood and the neighborhoods where you walk. On the map(s), highlight or mark where the nearest police stations, fire stations, hospitals and businesses (especially those with security officers). Study your map and try to memorize where these locations are, and keep the map in your pocket when you are out walking.
When you are harrassed by anyone (especially those who follow you), walk quickly to the nearest location that you know has security officers and tell them that you believe that one or more individuals are harrassing you and, you believe, are violating the city's and state's hate-crime laws. You don't need to call for police (because it will take them forever to arrive), but make sure that the person harrassing you sees where you go and that you are talking to others. They will avoid following you into a building (especially a police station), and will ignore you.
After they see you reporting their harrassment enough times, they will probably find more amusement throwing rocks off the piers. Meanwhile, you have probably recruited a few business workers in your neighborhood who will: 1) get to know you, even slightly, 2) know enough about you to look out for you when they see you pass by, and 3) if you do report any of this to police officers, they will likely step up their presence around your area. In other words, you will end up recruiting others to help you, even if they don't know it.
But, the best advantage this practice gives you is the ability to see your own surroundings and how they can help you, even if it is to avoid some name-calling for a few minutes.
Nobody deserves to fear walking home. Walking with friends, or at least knowing people on your route is a great way to defend yourself against harrassment.