Not to count it just utterly made no sense in the way they jumped from 3-4. With the normal SS form it was a difficult transformation. It also took a long while to master, the energy it consumed was pretty enormous and Goku had to force himself and Gohan to stay in that form to get used to it. Then at Cell games Gohan achieved the second form, but Vegeta still hadn't mastered the 1st form like Gohan and Goku had, and 2 was entirely new.
Then the time jump happens and Goku and Vegeta have achieved the second form, but still haven't mastered it. The form probably still consumes large amounts of energy, even worse than the 1st forms, and Gohan hasn't progressed at all. Goku's third form increased his strength and speed dramatically, but still seemed to be in it's infant stage. It drained him so completely he couldn't stay in it long, and it was a desperation transformation.
GT comes along and they all stay in the first form and never use the second from what I can tell. SSJ3 seems like a waste as Goku did as well in his normal state against bebi if not better than the 3rd form. Heck they didn't explore that form at all. Then he achieves this fourth form which doesn't resemble any part of the previous ones, but it's not an energy burner in the least, seems to have absolutely no downsides, and yet (like stated above) sometimes seems weaker than a normal form Goku or Pan.
Gluon hit it on the head, the series basically blanded out any meaning the transformations held, and took away any downsides which made them a desperation move that upped the excitment. Which fits in perfect with the straw that broke the camels back statement, the 4th form was just the one that became too much.