It is, metaphorically if not actually, a rape (though I suspect if it had been an actual physical rape, neither Batman nor Gordon would have been able to play by the books). It's actually interesting that the book came in for a lot of criticism for the way it objectifies Barbara yet this is one of Alan Moore's recurring themes, the ways in which women are constantly objectified by culture and society and their brutal hierarchies of power.
I've loved The Killing Joke for 20 years now, and I think there are two elements, apart from the perfection of the joke itself, that elevate it to genius. One is the way Jeannie dies in the memory section: electrocuted by a baby bottle warmer she's testing. That's just so completely absurd. It's like (of course) the punch line of a very cruel joke. The other is what happens to Barbara, because it has to be horrifying, in order for Gordon's decision to play by the books, and Batman's decision to renew his offer of help to the Joker, to be as heroic as they are. Or as insane? If this had just been a standard kidnapping plot where she's tied up in a basement somewhere, it would probably just be a good suspense thriller.
But like many comic book manifestations of the Joker, this one is fairly asexual, even in the memory sequence, where Jeannie's insistence that he's good in the sack might seem more like wishful thinking in reconstructing the memory. (I have elsewhere argued that Harley Quinn was invented for the same reason Aunt Harriet was, in response to a gay scare: after Grant Morrison's Arkham Asylum, which did sexualize the Joker, someone must have decided that if he was to be sexualized, however, nastily, it had to be heterosexually.)
So, in short, no: Barbara is not physically raped. But she and her father are emotionally violated by being forced into situations of extreme vulnerability where any element of choice is removed.