I'll be honest. I absolutely HATE HSC physics now.
Real world physics is alright
Hahahaha and you don't need physics in your degree
But its definitely true, HSC physics goes through some pretty abstract ideas, most of which my first year uni physics course didn't even touch.
However, I totally understand what you are going through- I had 13 units at the beginning of my HSC year but I chose to drop chemistry, even though I really liked the subject and keep 4U math to stay on the "ideal" 11 units.
Through this experience, I'd say that:
1. 4U math is for the hard core math enthusiasts - You will be doing ALOT of math, EVERYDAY in order to even pass the subject. So be prepared for that if you do end up keeping the subject. However, it does go through heaps of helpful content for math heavy degrees like Engineering. Which leads me to my next point
2. Keep in mind the university degree you are after - There is usually a list of recommended subjects which will make your uni life a lot easier because you have background knowledge on a few of its courses.
3. Most importantly, pick something you ENJOY. Otherwise you will regret it so much when you have to spend 5 hours puzzling out the subject's content, during HSC and trials study
4. Keep scaling on the back burner. It counts for nothing if you don't do well in the subject
Hopefully this helps
I wish you all the best in your studies