If my brother and his friends came at me with knifes and I'm armed with a gun, and I give him fair warning, offer him forgiveness and a place in my life, and he still insists on trying to kill me, it's not dishonourable of me to shoot him before he can end my life. Renly had every intention of killing Stannis in the books, and even makes jokes that he'll try not to mutilate his corpse too much.
Besides, Stannis in the books had no knowledge that Renly was killed by Melisandre. She told him that she had seen Renly's death in the flames and at some point later had sex with Stannis at some point, but there's no indication that he was aware of what happened. He never admits to killing Renly, even when pressed by his closest friend.
Renly was himself, a vainglorious warlord who starved half the realm and fully prepared to usurp the Throne even while Robert was alive. The show glamourised him quite a bit, but in the novels he was an utter bastard who happened to be charismatic enough to hide his greed and ambition.