The two aren't as mutually exclusive as you think. Guys with outstanding physiques can also move really well.
Below is Scott Adkins in his signature role of Boyka. I know he is the most hated actor around these parts but using him as an example because as Boyka, he is pretty big - extremely muscular and also insanely shredded but his movement and agility are never in doubt and are beyond compare. All those impressive back flips and flying kicks and lightening like movements you see are all done by him for real without a shred of wire work or CGI or editing to hide anything.
So you there are men out there, even actors out there, who can look huge and cut and still move insanely fast. So it is possible to have both.
Possible, but not necessary. Adkins does all those flips and parkour movements himself it's true. I am well aware. He performs wu-shu like stunt work without wires? I am also aware of that. Now I don't know what his actual martial arts, meaning non-stunt work, training is. I do know what the physiques of guys whom I would trust in an actual street fight looked like in their prime through personal experience and study. Paul Vunak, Marc "Crafty Dog" Denny and Larry Hartsell. All incredible martial artists that are/were experts at street fighting. Not sport matches. Not exhibition skills. And none of them, even at their peak looked like Adkins.
To be a top flight person with the skills to survive a street fight the training is much more wide ranging than simply have a great looking agility one acquires in doing the more acrobatic techniques of martial arts, thought the underlying attributes of an Adkins type are very important, like great cardio, balance, speed, ect. For the true street fighter martial artist you have to also develop the range familiarity and skills within each range necessary to accomplish ones goal, which is actually NOT about winning the fight per se, but surviving it. That's the real win. Getting to go home, though to do that it may be that you HAVE to crush your opponent totally.
But to get to that point you had better have a wide array of skills, which are of course buttressed by your attributes: Wind, strength, coordination, ect. But the training for this doesn't in turn give you an Adkins styled body for sure. Knife fighting, stick fighting, boxing/kickboxing, close quarters/trapping/standing grappling, ground grappling, improvised weaponry, unarmed against blades, unarmed against weapons, one on one scenarios, two on one scenarios, mass attacks, "dirty tricks"( i.e. spitting, biting, pulling hair, eye gouging, ect.) and more. As you can imagine, as with any skill this takes constant honing. Constant. Now if I am to believe that a man like Batman is training to be a street fighter then this is what he's doing. And in my experience, this does not leave a lot of time for hitting the iron to the extent that it gives you the Adkins type body. The men I've known that train like this are in great shape, but they don't look like Adkins or Michael Jai White. The flight time needed to develop the reactions for a successful "win" in a street fight scenario don't allow for it.
Ben can look very lean, but for myself he doesn't need the mass of an Adkins. Even if it's true for Adkins, for the most part in a real world scenario most guys that size carrying around that extra bulk, it helps them be extra protected true, but they also are slower than a lean machine type. Affleck is actually to my eyes in the type of shape a good street fighter should be in. Especially an older (meaning over 30) one.