The truth (in my opinion) is that "simple math" (where is the boundary between "simple" and "difficult" maths?) probably is not enough, in particular if you want to do theoretical physics.

University mathematics can be daunting for beginners; it was for me in the first semester, then I had adapted. I have heard that many others have similar experience. Here some perseverance and endurance in the face of hardship may be required. I (physicist) had my mathematics lectures together with the mathematicians, and I have never regretted this. With such a background you should be able to acquaint yourself with any field of mathematics you may require at some time in the future.

It seems to me that an important aspect here is why you want to study physics. To call yourself a physicist and hold a degree? Or do you really want to understand? The latter case not only requires an understanding of the physical concepts as such but also of the maths used to describe them. In my opinion you should understand the proofs of the mathematical theorems you are relying on when discussing physics. Of course, you will hardly be required to learn everything within a year.

As for rigour:

In Newtonian mechanics, deriving, in the context of a set of point masses, expressions for change of total momentum, change of angular momentum, motion of the centre of mass involves rigour.

In statistical physics, deriving the expressions for probabilities in the various ensembles involves rigour.

The singularity theorems in gravitation involve rigour.

Wightman axioms and PCT theorem involve rigour.

And there are many more examples.