It'd be damn interesting fight for sure. Also, I wonder if season one Ollie would put up a tougher fight since he'd be going to kill. Just straight for the jugular with him. Season two Ollie would hold back a little bit, but only a little.
Bruce did take on quite a few of the League in Batman Begins, when Ollie had Canary to back him up. It's not really comparable, but I guess in trying to gauge their skills against people who are highly skilled as well (since we can't really say organized criminals are anywhere near their level) Far as I remember, Bruce already knew a couple martial arts in the films. I think if you're worried about speed, his TDK/TDKR outfit would be the one for him since he sacrificed defense for speed when he switched.
The movies did show that Batman pretty much would decimate his opponents. But, again so many there as with Arrow are people who aren't even in the same ballpark in skillset. The only ones that he struggled with were Ra's and Bane, but I feel like Bane was a separate deal anyway. Could be too close to call for me, but I won't deny that Ollie is an absolute beast. I'm eager to see exactly what he learned, in the series, really.