We are to love one another and as much as possible live at peace with each other (John 15:12 & Romans 12:18). Galatians 1:8 says if someone comes preaching a different gospel let them be accursed. 1 Thessalonians 2:4 says we speak to please God and not man.
There are souls that need to come into the kingdom. How will they hear unless someone speaks? You have a voice, use it, before that gets taken away as well.