I don't believe the immorality of racism inheres in whether or not one happens to be part of the so-called power structure, though I can see how such a belief can be used to critique only those in power and allowing others to skate free. I've even encountered the idea that what people call "institutionalized racism" is or ought to be the sole definition of racism.
So you reply that Lavar Ball is just a ****-talker who has no power to hire and fire, and that's why he's not part of the power structure? Well, on that logic Don Imus should never have been fired for his "nappy headed hos" remark in 2007. He had no power to hire or fire, right? He wasn't an executive, right? And being that Imus was a shock jock, wasn't his job explicitly all about ****-talking, as opposed to Ball, who's just doing it to satisfy some animus against white players?
By the way, while Googling institutionalized racism I came across this
interesting quote from Ben Shapiro: