You cannot afford to allow the challenges of fear, or the baggage of your past to kill your passion for success, neither can you allow what people think of you.
Being passionate in fulfilling your goals are often referred to by others as being aggressive, or a demonstration of pride. I am here to tell you; shut your ears to gainsayers while in pursuit of your goals.
Your passion is that which motivates you to overcome the obstacles and the challenges that associated success. Kick those obstacles and challenges to the curbside and pursue.