me it was "non-genre fiction."
To me, "non-genre fiction" is a term in the thesaurus along side words like "boring" and "stale."
Books are all categorical. They all fit somewhere. Where "Romance" tells you "there's kissing and stuff in here," and "Sci-fi" says "dude, it takes place in space," the term "Mainstream" is just a catch-all term synonymous with "it's selling like hotcakes**."
**I work at a VFW. On Monday mornings, I wait tables. We do not sell that many hotcakes. I do not understand this particular phrase.
Harry Potter, YA. Hunger Games, YA. Sookie Stackhouse Books, UF. Twilight, UF/R/YA. Fifty Shades of Gray, Erotica. Inferno, Thriller. A Game of Thrones, Fantasy. World War Z, Apocalyptic Horror.
Each genre of fiction has it's own tropes, themes, and styles. A genre novel that breaks out and goes Mainstream --admittedly usually and/or arguably-- minimizes those unique facets while maximizing the character and broadening the scope of the plot.
RWA will tell you that their romances must have a Happily Ever After, an HEA...in certain categories. Even the grand RWA recognizes that such fairy tale endings are not the right ending every time, and that it is not always believable in this modern world where traditional marriage isn't always the lovers' goal. RWA has adopted the Happily For Now ending allowable for some categories. The stories are changing because our lives are changing.
Let it be known:
Paranormal Romance is Urban Fantasy where the hero and heroine save the world, then make love. (A generalization, I know, but roll with it.) Urban Fantasy may or may not have romance in it, but if it does, the hero and heroine do it before they save the world. (Another generalization, I know, but roll with it.) Sci-Fi often has world-building that reflects a current social concern taken to an extreme and shown to work/tear down an alien society. Hard Sci-Fi has cutting edge technology, pushed farther, faster, and often looks at the worst-case scenario ramifications of that technology. Horror can be haunted and ghostly, it can be horrible monsters, or can be us at our worst.
Think about that.
Mainstream has many examples of all of those. Yes, each genre novel that experiences a Mainstream Breakout has embraced it's quirks...but if you want to give the mainstream market a genre idea, you have to make it rapidly understandable without overpowering the story or stopping all the action to give the reader a lesson in how to understand the world. It seems to me that the writer must make the characters big and sympathetic. And they must have crafted a plot that is so big, so important, that the big, sympathetic character he or she created truly struggles to surmount it, a plot so rich with emotional impact for the character that he/she learns deep lessons about himself/herself in the process. Think of it like a recipe. Reduce the "genre aspects" so that they feel like the natural setting for the character and plot. Then double the character, and triple the plot. Yield: Story with broad (read: mainstream) impact and appeal.