You aren't doing anything wrong; you're doing as well as you can possibly expect given the length of time you've been studying Spanish and the amount and type of practicing speaking & listening you've been able to do.
And you've got it: mastering speaking and listening requires practice and more practice, supported by continuing to study grammar and vocabulary and continuing to develop your reading and writing skills.
|