The men shown with Bane in the beginning and in the sewers were most likely League of Shadow members(although it becomes confusing during that coup Bane was involved with...maybe Bane had some of his own men?), but once Blackgate prison was destroyed, Bane's army grew with just criminals, rapists, etc.