There are advantages and disadvantages to both. More often than not, being a prolific and popular author generates significant in-the-moment income. It can also leave room for an author to experiment with other styles on the side, knowing they don’t have to rely on those financially.
On the other hand, it’s rare for those kinds of books to have enduring success outside of the ones put out by “branded” authors with large publishing houses. Even then, a book’s commercial appeal may be fleeting, and the same author who was the It Writer last year might see sales plummet if their next novel doesn’t meet expectations.
Authors who write for themselves or for the art are often less frustrated with their craft. Not having to worry about mass-market appeal takes of a great deal of pressure. A classic doesn’t become so overnight, and authors focused solely on their art has less concern for this.
However, there’s no guarantee that such a book will ever be widely read, and sales may remain low indefinitely as it fades into obscurity. Those writers often need another source of income, especially if they are self-published or with a small press. Some smaller publishing companies may turn down works they think won’t have much appeal.
Not to be that person, but I think what’s best for an author really does depend. Yes, earning a living via writing makes a difference, and if someone needs to write for commercial success in order to do so, there’s no shame in that.
As a parallel, my grandfather made his living as an artist. He spent a good deal of his time making popular ceramics, such as dishes, vases, and the like. They were well-made and certainly had his personal touch, but they were not necessarily meant as “art for art’s sake.” Alongside that, he also painted and made mosaics. When he partially retired, he stopped making dishes and vases and concentrated on mosaics. (He is an example of when art for art’s sake can be successful, however: he created a mosaic for a local synagogue, built right into the wall.)
Personally, I’ve discovered that I don’t want to be a career author. Writing along popular lines isn’t for me. For some folks, it wouldn’t compromise their art, but it certainly would for me. That doesn’t make me right and others wrong, nor the other way around. It’s a preference.
The reality is, I’m not a commercially successful author. I don’t have enough book sales to live off the profits. I don’t do well forcing myself into a specific set of genre expectations. Even people who veer off the well-worn paths of genre fiction tend to know when and how to stray and still hold the audience. I have almost no ability to do so and little desire either.
And that is okay! As much as I would love if people liked my writing and were drawn to it, I’m aware that I’m not for everyone. I don’t have it in me to conform to popular genre expectations and have enough energy left for my preferred style. So I am one of the people who holds another job and writes on the side. Suits me perfectly fine.
Whatever path you choose is up to you. There isn’t any shame in either style. I would never advise an author which way to go. The only thing I’d suggest is to be sure whatever you choose leaves you room to be happy and satisfied with your writing.