We tend to hold ourselves back. This is true, but perhaps in more ways than you think. The Bhagavad Gita tells us that there is the true ego, which is our consciousness, and the false ego, which is an identity that preserves the sense of our superiority. This ego is the stumbling block.
There is a huge difference between ego and good self-esteem. The ego pits you against everyone, certain that you are the best. Self-esteem appreciates your strengths but knows your weaknesses while wishing everyone else growth and good fortune. The ego will project your insecurities onto others, making the purest intentions seem wicked. It makes everything into a competition where the winner can only be you when in truth you lose most of all.