Language is important. Which is the issue with carving out these caveats that ignore the issue and give safe haven to bigots. If you fill up an auditorium in Britain with 1000 over 40s cis British men and women, and 50 of them would be considered "not transphobic" by someone with an understanding of transphobia, would this fundamentally change the struggle for trans folk in Britain? It's so bad in Britain, the Council of Europe put it in writing.
Your concern is for those who aren't struggling and imo, gives cover to those who argue they aren't bigots, but are concerned for "women's sports", "women's rights", etc., while trying to rip rights from women, men and NB folk, including children. Just because every Confederate solider didn't own a slave or considered themselves racist, didn't change the struggle. Those who are allies, don't see the phrase "all older white folk" and think they are being attacked. They understand the struggle.