Considering that I actually force myself to wait a long time before I watch a new season, no.
On a lot of shows, they like to set the new season "one year later" or "eleven months later", so it would throw me off a little if I just watch the new season immediately.
Now, as to whether if I have the problem of remembering the story, yeah, I definitely recognize the problem. It helps when the new season recaps the previous season (as most American serials do), but even then, there are intricate little details that you sometimes forget. And it's not just different seasons. With Les Revenants, sometimes, I'm forced to remember the details between different episodes if I don't binge-watch them because each episode contains so much complexity.
An interesting note is that I'd often encountered this problem not just with live action serials, but with anime as well. With the complexity of some anime, it can also get a bit jarring to step into a new season not remembering the previous one.
I guess a good solution is to read up the Wikipedia notes to help jog your memory. Or search YouTube for the various clips from the previous season to catch up.