Keaton felt that way after the first movie, and especially after the second time. He must be sick of the heavy workout and critics slamming each film. Just trying to be fair. But I do think it's funny since Bale was on and off as this character for 9 years, while Ben hasn't appeared in a full on Batman movie yet.
If he doesn't continue, he will be known as the older Batman who appeared in those weird messy team up movies that featured a bunch of DC superheroes. And that will be his legacy, because let's face it, Reeves' trilogy will most likely take a lot from the current costume design and just make it better. Which means "comic book looking Batfleck" will not be a exclusive thing for Ben's version. People will remember the quality and look of the films, as well as his acting/physique. It baffles me to think that Ben could be remembered as one of the greats, or even the greatest of all time when in 5 years from now, the talk of the town will be "the younger version in the prequels is much better".