Here's a quick scenario I'd like to run by you; would it have been a retcon if they had done something more linear to explain the connection? Something like:
- Quantum, which already connects Le Chifre and Greene, was mortally wounded as an organization by Bond after Casino Royale and Quantum of Solace.
- We find out that Silva had support and networking with another organization in Skyfall, not that he was working for anyone, but that they supported him logistically so he could move up his timetable.
- Bond kills the guy in the opening, and escapes the Rome meeting to find Mr. White.
- Mr. White reveals that one of Quantum's surviving members managed to steal the remaining resources from White by allying himself with numerous more mundane criminal enterprises, and supplied Silva with aid for the sake of damaging and distracting MI-6. Thus, the Spectre of Quantum was born.
Because that kind of scenario seems like a way to retcon a connection between Skyfall and the rest but keep it more logical, with the escalation of damage between the participating parties leading to the Bond-Bloefeld feud, as opposed to trying to make all previous players chess pieces of a spiteful step-sibling.