I had a very similar apprehension of how it would be if I'd gone to university, so I didn't go. I might have gone if they'd offered me a place in my home town on a relatively easy course, but the idea of massive academic expectations in a new town miles away from my girlfriend, my friends, and all the things I was used to, just seemed way too risky and uncomfortable.
I don't know if the situation is different in the UK than in the US, and I wouldn't want to advise anybody to go or not to go to university, but in my own case I've never felt any great regret about ducking out. I got myself a job that was (arguably) below my maximum academic potential but it got the money coming in and it was enough for me to get a mortgage on a reasonably nice house (my wife at the time was on a similar economic level so we were a double-income couple). Of course that was a long time ago and maybe it was easier to make ends meet in those days. But to this day I've never felt badly short of money, in spite of abandoning my first career and going back to the same kind of work after a break of a couple of years.
I was good at my job(s), which were in universities, though I wasn't an "academic" as they call the higher-up ones in the UK, I was "technical staff." I enjoyed a lot of the work, it was good and nerdy and I got to use clever equipment. Frankly I didn't envy the academics, especially in my later years in the workplace, they seemed to have to work too hard and there seemed to be an expectation on them to "do whatever it took," which to me smacked of working too hard. I was happier with reasonably fixed working hours, I didn't like the idea of giving up my free time in the hope of the bosses condescending to reward me at some point if they felt like it. I worked quite hard myself when I got immersed in the science of it, but there is a limit, and I've never really defined myself by my employment. My jobs were pretty safe from redundancy and getting the sack. So I felt I was in a lower-stress environment - I'm not saying I never got stressed out, far from it, but compared to an academic post I think I had it pretty easy in terms of pressure.
One wife suggested I tried to get a degree by working on it in my spare time. But I didn't take her advice. Again, I just didn't want to bust a gut to "better myself," I didn't want to sacrifice the spare time in which I could mess about with my special interests where I was my own boss. The subject would have been biochemistry, and although it interests me, the thought of fathoming and understanding the whole subject just seems too hard. I hated my last few years at school, and performed progressively worse than previously, for much the same reason. Don't get me started on the notion of taking on student loans.