Yeah, I definitely feel that, both the anxiety and fear of rejection. It never feels great, but in my experience, the more you chase it, the sillier you feel after it still doesn't pan out.
I think the best thing to do would be say something like "Just let me know when you're free to catch up" and then dial yourself back a bit. The thing is, even if you give him the benefit of the doubt (he could be genuinely busy, he could be shy, have anxiety, double booked himself etc), it still can set a dangerous pattern early on that your time isn't as valuable, and those patterns can be hard to break.
In my experience (and probably speaking from both sides of this), the quickest way to make someone commit one way or the other is to stop meeting them more than halfway. Even if you really, really like this guy, your boundaries and your time are valuable and you deserve the respect of clear communication. Some guys just need that reminder. Either because they're not totally serious and they need a reminder of "Oh this is another human being" or they're just generally immature and haven't had many people call them out on their behaviour etc.
I had this on again/off again thing with a girl a few years back, and she would string me along on planning dates for WEEKS. We'd go on a date, it would be great, and then nothing. Cancelled dates or not enough money to go for coffee, etc etc. So I'd take the hint and stop messaging, and she'd chase after me again. I'd suggest a date and the same thing would happen. Eventually I just called it what it was and said I'm getting clear signals that she's just not interested and she acted genuinely surprised. I put the ball in her court to plan a date and it never happened.
About a year later, we bumped into each other and caught up and she told me all about her severe depression, and how that's what she was dealing with etc. Now obviously I empathise with this, because I've dealt with it on and off myself over the years. But I knew immediately that I'd done the right thing. Because even if we'd started dating, that could've been the start of a super toxic patter, of one person going above and beyond for a person who won't even communicate etc.
Anyway, long winded way of saying there's nothing wrong with protecting yourself in these instances. It's not selfish or stand offish when it comes to dating, just practical. And then when the right person comes along, all that stuff should feel easy and natural.
Good luck with it all!