Again, when you solely look at Ned Stark's emotions during Game of Thrones,
he never feels Jon Snow is threatened (if Robert was truly all about murdering Targaryens) but the heated arguments between Robert and Eddard and him dreaming about Lyanna when it came to Daenerys really tell you something.
Not to mention Eddard Stark never thought about Rhaegar Targaryen for years, you'd think he would think about the Last Dragon if he was raising his child.
Further, Catelyn asks about Ashara Dayne in Game of Thrones because you had rumors circulating about Ned and Ashara and what does Ned respond with? He says and I quote "Never ask me about Jon", Why would he say that when the question was about Ashara Dayne? Unlessss.

Ned even forbids the discussion of Ashara Dayne in Winterfell lol.