Mario is an archetype, he is effectively the father of the adventure game. If they made a Mario game that didn't follow that archetype (as cliche as it is), I think it would ultimately fail in the market. People like Mario games because they're all basically the same. I think that's why Sunshine didn't do well (among other reasons) -- Peach wasn't kidnapped for the whole game.
Watch for Super Paper Mario in October. You can play as Bowser and Peach, so assumedly Peach hasn't been kidnapped, nor is Bowser the ultimate enemy. This could spell trouble for sales, but the fact that it's a traditional style Mario side-scroller might counteract any ill effect.
Yeah, as much as people complain about cliches and the same ideas being recycled over and over again in games, it really isn't such a bad thing. Sometimes, the game just needs a couple of new spins and its good to go. By this I mean like tweaking the storyline a bit or adding some new weaponry or moves. The vital part of the process is that the storyline and gameplay basically remain the same. That way, not only is the new game easier to pick up, but it more easily retains the original fun of its predecessor. Mario has done this pretty well over the ages (with the exception of Sunshine, true) and that's why the Mushroom Kingdom still exists. Even Donkey Kong, Metroid Prime and PokeMon have been rehashed time and time again with good success.
That's why games can't deviate too much from the original storyline/gameplay--they just stop being as fun as the original. Sometimes a series can't even deviate from its original graphics. I don't mean to beat a dead topic here, but look at what happened in Zelda Wind Waker. I enjoyed WW, but I know there's a lot of people out there still seething at Miyamoto for the injustice.
I think it depends on what you want to do with the next version of the game. Is it a sequel/prequel? If so, then the game must stay with the type of genre and story as before. If not, and you want to simply use the familiar characters and use them in a new way, then you can change anything.
I don't see the big deal with this archetype and formulaic sequel business. The essential archetypical qualities of anything Mario isn't the story at all, rather the elements and interactions with and of the elements present in the game worlds. Since when has the Mario universe been concerned with consistency in regards to story? Nintendo is building a universe from the bottom up, so anything goes. The important thing to them is that they establish memorable, whimsical objects and characters, and then lick referential riffs off of that for a while, most often within different genres (platformers, sports, RPGs) while commenting on and adding something new to the genre.
As for Sunshine, well, I'll admit I wasn't a big fan at first. Maybe they tried introducing too much to the collective universe at one time without including enough familiar elements. Another thing is the overlevel (the town) wasn't nearly as interesting as the castle in M64. Still, Sunshine grew on me. The water cannons really added a different kind of spatial complexity and freedom and agility that wasn't present in Mario 64. However, Mario 64 to me still the purest 3d experience in gaming.