Isn't this the entire problem?
Take a closer look at Raiders of the lost Ark. I quote Indy:
Oh, Marcus. What are you trying to do, scare me? You sound like my mother. We've known each other for a long time. I don't believe in magic, a lot of superstitious hocus pocus. I'm going after a find of incredible historical significance, you're talking about the boogie man. Besides, you know what a cautious fellow I am.
It's quite a point and theme in Raiders of the Lost Ark that Indy
does not believe in anything superstitious or magical. He says in the film over and over that he thinks religious powers are fake. It is clear that the opening of the ark is the first time that Indy has ever seen anything supernatural and it is what makes him a believer. Yet in George Lucas's screwed up continuity how is this possible when just one year earlier Indiana Jones watched Mola Ram utter voodoo curses and magically rip out a man's heart through the powers of the Sankara stone? In Temple of Doom he sees numerous acts of supernatural and magic take place yet a year later in Raiders he goes on several rants about how he does not believe in the supernatural or the power of religious artifacts.
Temple of Doom does not work as a prequel, plain and simple. It screws up the continuity of the character and his core beliefs. And if you watch Temple first then you will also miss out on small little nods to Raiders such as Indy going for the gun, if you don't watch Raiders first the joke won't make a lick of sense.