How much world building is too much? I vote that it ought to be a spice rather than a side dish in a story. To my mind, the whole point of world building is to assist in telling the story. Too much and it's distracting. Too little and the story isn't grounded.
Me saying that world building is spice might not reassure you much based on my candy preference.
In terms of my stories, any tale with elements we'd consider outside the norm of our modern world requires more world building than a story set in an office building in downtown Seattle. (Where the elevators aren't going to come to life and begin requiring blood sacrifices to continue delivering your characters safely to their destinations day in and day out...But see, maybe this is my problem because this is the only way a story in an office building in downtown Seattle would be interesting to me -- which may indicate a need to increase meds. Dunno.)
My issue with world building is that I'd prefer you not be able to see it. Or if you do, that you don't readily identify it. In a perfect world, it would be like a great bowl of soup - you know I tossed a bunch of spices in there - you can totally taste and appreciate them, but the flavors have blended into something greater than the sum of its parts and now you can't pick one spice out from the others as you sip. That's how I'd like world building to meld with a story. I'd rather not tell you 'hey, magic exists, k?' I'd rather you pick that up from the characters, the dialog, the action - anything but me laying it out. This means I sometimes get notes from my editor asking for more world building detail because occasionally the world building plays to motivation. Not that I've heard (and fixed!) this really, really recently. Ehem.
The more outside of our day to day experience a story world happens to be, the more information is going to have to slip into the world building. Science fiction stories that aren't earth-based (or earth-derived - as in humans have left earth and even though it's centuries later and they're now living on planet Zurg, they're still humans with earth-derived perspectives) require extra world building chops. You may be required to come up with different languages, different ways of looking at life and death, different ways of relating to the unfamiliar. The hard part is integrating that so the world building supports your story soufflé without making it so heavy that it falls.
A friend's one line answer to the world building question was "If you need an index, you've done too much."
If world building is spice, then the real question is do you like your stories 0 star spicy? Or 5 star spicy?