I think they worry about it because society holds such "age milestones" out as real things, and people have a habit of comparing themselves to others, and another habit of assuming a particular set of achievements (university degree, driver's license, career, spouse, money, brand new car, house ownership etc) is the only worthwhile path to take. I guess in my case it helps that I'm English, because in England there might be a bit more respect for people who choose eccentric lifestyles than there is in the USA, and an eccentric lifestyle is the only lifestyle I can imagine myself ever being happy with.
Sadly I'm not completely immune to such social mores, and I have sometimes felt like I was being left behind or under-achieving, but the greater part of me rejects it all as complete rubbish, and I like myself better when I connect with that part of myself, which is most of the time. Sometimes I think I'd be better off if I didn't know what year I was born, that way I wouldn't have felt scared of getting older every time I hit one of those arbitrary boundaries such as 30. I think the scariest one was 50, which seemed really like the beginning of the "old age followed by illness and death" stage. 30 was a bit of a shock, I felt slightly as if I wasn't quite in the club of young people any more, but it didn't make that much difference.