I'd agree with you for last season SV Fan, but the setup they have here, Felicity and Ollie's non-relationship is deeply tied into the future of Queen Consolidated and Ollie's identity. And now that Olicity is an actual thing instead of just a tumblr phenomenon, their interactions actually have weight for their future interactions. Now the cat is out of the bag. They delivered on this set up already by having Felicity reject Ollie doing the precise kind of teasing you don't appreciate. Think about that, Ollie gave Felicity a big gif-ready kiss and the reaction was 'we're done.' The more he pushes her away, the more she ends up working for Palmer and that changes a lot of things for Star City. It's really good times out here.
I like love interests coming from the supporting cast sometimes, I think that's more realistic, to develop feelings for someone you deal with on a regular basis than someone who comes swooping in, and likewise, in most situations depicted on TV, it doesn't make sense for them to disappear from your life just because you two break up.