Certainly the advice to learn independent of courses is valuable. Searching out the great texts and studying them will almost always prepare you better than a course. I speak from an American perspective where course evaluations and various other pressures invariably force programs to diverge from what is intellectually optimal. Unless, you are at some elite school... I think this advice applies to both math and physics.
That said, we need to either solve original problems and publish papers of substance or, the more standard path, we earn degrees. My opinion, it is much easier to learn higher math online than physics. The trade craft of physics, it is communicated in person far more effectively. In my experience, the physics I learned in person I probably would not have learned as well from books. Physics has customs and traditions which are easier to absorb when amongst physicists. Mathematics also has such customs, but, I think is better defined.
So, I would advise seeking out physics degrees while studying math in parallel, perhaps partly just as independent study. At some point, you may have to switch to a math degree as you go on, but to begin, I think physics needs some face-to-face interaction.
A bit on my background: I earned a B.S. in math and physics. Then, for graduate school, I found a theory program in physics where I could take a course on string theory. For 2 years I took most graduate physics courses, earned a M.A. in physics, and more importantly, I gained some understanding of physicsy supermath. The thing that really helped me was not a course, rather, I chose to prepare a seminar where there was threat of questions from professors (gulp...) Thankfully, a string theory student who was further along got me started and after a few hundred hours of spinor index calculations I felt like I understood something. I gave the seminar, survived, and as it happens, it served to form the basis for my PhD work in math (which was a lot of fun). There are many paths to take and the possibilities for study seem endless, so, pick something, and enjoy the journey.