In the show for example, when Ned and Jon are about to part on the Kingsroad and Jon asks about his mother, Ned starts to get broken up about it. Not the reaction you would expect if it was some random woman, it had to be someone Ned deeply cared for.
This is a great article about it on Tower of the Hand. You can change the scope on the site to prevent yourself from seeing spoilers. You'll need to change it to at least A Storm of Swords, but I don't think there's anything that will ruin it for TV viewers, the majority of the material is the backstory about the Tournament of Harrenhal that Meera and Jojen tell Bran (which will probably be cut from the show).
http://towerofthehand.com/essays/chrisholden/jon_snows_parents.html