That's understandable. Harry Potter is still a different kind of beast then Star Wars. Especially since the books started out as children's books and slowly grew into young adult/adult fantasy novels. But it's impacted this generation in a huge way. Especially since the Harry Potter generation has grown up, so HP is moving into the adult aspect. I'm 21, and I grew up with Harry Potter, and there are millions of people who grew up with it just like me, in addition to all the younger kids who have been turned on to it by the movies.
Now, with Star Wars, you had films that not only impacted a generation, but also revolutionized the film industry. Harry Potter hasn't done that. The films have been solid, but nothing revolutionary like Star Wars was. But there's no denying that Harry Potter has been, and still is, a huge pop culture event.