I should clarify, TDKR was a cultural phenomenon, coming off the back of TDK, but if anything, I think that was the narrative on the film, that everyone was hyped for it, coming off TDK and the Joker, and when people finally saw it....they liked it, but it didn't set the world on fire.
I'd say TDKR is akin to AOU in that sense.
And yeah, it's "easier" for Infinity War and Endgame to bank on the character development, goodwill, and buildup of the other films, but this isn't about being a fair or equal comparison. And besides, it's not like the reception and impact of those two films was a gimme. Just look at how Star Wars squandered everything it had going for it.