Irony here is that I had a really similar situation. I actually enjoyed chemistry somewhat, but I was terrible at it. I wound up doing just fine (I did a little better than average) and did reasonably enough overall in VCE (uh, 90+ ATAR from recollection). I also now have a minor in chemistry, so... yeah, things worked out just fine for me.
If you're intending on doing medicine and some fields, you'll be closing those doors, as Sdfg said. But the good news is that there aren't actually that many areas that require VCE chemistry as a prereq. Off the top of my head, there's undergrad med, biomed and maybe some engineering (chemical engineering) and health sciences, so if you're not interested in those, then it might be time to consider the move. What sort of area are you intending to delve into post-VCE?
If you're interested in my experiences, please feel free to PM me.