Pursue your passion. Follow your dreams. These common phrases, etched on everything from mugs to throw pillows, urge people to build their lives around what they love most. But what should people do when their interest in a passion starts waning?

Harvard Business School Assistant Professor Jon M. Jachimowicz says many people are afraid to give up a career that has lost its luster, often because they’re concerned that others will judge them harshly. This can prevent workers from leaving jobs, making them feel trapped in unfulfilling careers. These fears might even stopping them from speaking out about inhospitable working conditions.