as everyone is telling me that spec gets better as you go on
"Everyone" is not you. Your decision should be an individual one based on consultation with people who know you
well and those who are qualified to provide such advice (e.g. teachers, careers counsellors). Stuff "everyone" else and what they think. You obviously dropped it for a reason - don't let heresay change your mind.
The number one objective in year 12 is to pick subjects you enjoy or are interested in. If you like eco, chances are you'll do better in it and succeed. I'd even argue prerequisites for a uni course are secondary to interest/enjoyment areas (I know they're important but i've seen the toll this has on mental health and obviously you have to get through year 12 first before you even consider other things).
Should I be stressing so much
No. As long as you're happy with the decision you made (not taking into account influencing opinion/outside noise).
Depending on your other subjects you can still obtain a 99 without spec.
It really is as simple as that.