Cain certainly wasnt a mass-monster by any means but he definately did not have 'a normal mans physique' and clearly put in the gym-time. Hell, one of the jokes in the pilot episode referred to Clark as 'looking like Mr Hardbody' and it wasnt entirely inaccurate. He wasnt tiny, he just wasnt as big as some, kind of like the way Sly and JCVD were never as huge as Arnold and Dolph but did you ever see anyone label them as tiny?
Despite your view to the contrary, I've spent years in the gym; enough to learn that it isnt all about sheer muscle-mass. It should always be about the quality of the muscle, not the size of it - any idiot can just get huge and I find the whole 'I gotta be massive' thing a bit of a younger persons attitude. More seasoned gym goers tend to evolve from this.