Why the huge gap between movies?

I doubt there was intent to make a sequel before 15 years. Previously John Carpenter had only ever made one sequel and that was Halloween II... from the shoddy screenplay, it's likely this was only done because at the time, Halloween was the highest grossing independent film ever and money was the goal. Escape From New York was a great film and rather successful, but not as much as Halloween. I think the sequel was only made because John Carpenter was in a slump and tried (and failed) to replicate past success.

Some possible reasons:

*Carpenter swore he'd never make another movie for the big studios after having a bad experience with Big Trouble In Little China in 1986. They Live and his other films were done with smaller companies. I'm not sure he did another "big" movie until Escape From L.A.

*The rights to Escape From New York and Snake are owned by the studio, not Carpenter, so having sworn off working with them, there was no way he could do a sequel, assuming he even wanted to. It was completely in their hands.

*Kurt and Carpenter are good friends, and they've both said EFNY was the movie that took each of their careers to the next level. Even if the studio had wanted to do a sequel with a different director, Kurt probably would have refused to do it without Carpenter.

*Russell has said he considers Snake his favorite and most iconic character. He really wanted to play him again. Supposedly, he was the driving force behind getting the movie made, and bringing Carpenter back into the fray to direct it.
In other words, maybe he was the only person involved who was really enthusiastic about seeing it get made, and that's why it didn't for so long.

Halloween 2 was not a carpenter film

They were busy doing other movies and John Carpenter doesn't like doing sequels he made an exception for the character of Snake because him and Kurt had other ideas on how to add to the character in modern times plus they wanted to work together again