that has nothing to do with X-Men.
If he really wanted to direct X-Men, he would have said yes over Black Panther 3. Directors moving to other IPs is usual. even inside the comicbook industry.
so he clearly wasnt that interested in the X-Men after all. This is a "once in a lifetime" oportunity, to direct Marvel Studios's first X-Men film. So Im sure many directors in Holywood would kill to get this chance..... while others, more established likely feel this "has been done already" so preffer new concepts and original films over another version of the X-Men, the third version on the big screen, to be precise (this plays a factor for d*** sure)