Normally, my advice (and I get asked a lot considering I packed it in pretty early), is to go towards something and not run away from anything. I think that applies to jobs, where to live and life in general. However, it does have to be balanced with your mental health and if the current situation is really getting to you, then it could be time. But, I don't think you just want to wake up one morning and say "now what?" Especially if you have financial or other responsibilities.
My wife and I call it reinvention. We've done it four times, now. When we've felt we were "done" with our location, we had already picked out where we wanted to be next, well in advance. We didn't lose friends, just gained a whole lot more and discovered new things, places and people. That said, we did that after leaving our careers at relatively early ages (for us, not for most of you guys who are considerably younger). Although we were pretty financially stable, it worked out even better in the ensuing years. New opportunities presented themselves.
One last point on big life decisions like this. Our thinking about "what's next" always started a couple of years prior so we had a target. If you don't have one, you not only don't know which direction to go but, you may not even be sure when you get there. If you're lucky (or smart) like Roy, not having a plan may work out fine as long as you're outgoing, know how to make connections with people and are generally open to new opportunities when they come across your path. So, think about what you really like to do, what turns you on and what you're really good at that could point you in the right direction.
Good luck with all of this. It's a big step but, could be the first step of the best thing that happens to you.
p.s. based on what I recall about your physical size, I'd discard the idea of being a jockey