When in Rome ...
When everyone around you pronounces the b and v differently, you may choose to sound like them. When you're in the rest of the places where the b and v are pronounced the same, you may choose to sound like the rest of us.
Kidding aside, no one is going to think you sound terrible. You'll have an accent (try as you might), yes, but from what I hear, accents are adored by all. Open your mouth and sing your song.