Usually I would put something like this down to fanboyism but I actually think you are onto something here. Del Toro apparently signed for multiple movies, something like 6 was the number I believe, and I just cant see The Collector being that prominent a character in the MCU, GOTG and then maybe ONE more appearance would be it for TC I think. But Thanos, he could be in 10 MCU movies let alone 6 as he is always coming back to life and thinking up new schemes.
Plus getting someone like BDT to The Collector just seems redundant when someone like Terence Stamp would be a much better fit for the character.