Listening the the Next Picture Show episode about it (you all should listen to that podcast if you aren't already) helped clarified the problems I have with this.
First of all, the Spielbergianity of it all. It feels hollow to me : it's made to look like it's the early 80's (the cars and the clothes, especially Driver), but it's clearly not a period piece, so... why ? I feel like the similarities with those movies are all surface-level : clearly the focus on family and fatherhood sets it very much apart from Close Encounters.
Then we get to the fatherhood allegory : don't get me wrong, it's the best part of the film, Michael Shannon is excellent and it's the only aspect in which the film comes close to managing all its threads together ("I like worrying about you" is a great line), but why is Kirsten Dunst there ? If you're going to have the mother present, she needs to be more of a character than she was, and if you want to make a film about fatherhood (not parenthood), why have her here at all ? She's not bad, but she felt like she came from another film entirely.
Then there's the actual sci-fi aspect. The problem here is that the reveal is completely underwhelming : a bunch of CGI buildings, that's what's making people form a cult ? Admittedly the evidence of another plane of existence would be enough, but the film leads up to it and wants you to be awed by it... and I really wasn't.
It feels like a hodge-podge of various elements (moody thriller, family drama, positive sci-fi, exploration of spirituality) which never coalesce in anything bigger, and that makes every little flaw bigger, like the ridiculously easy to breach roadblock, the ending which doesn't acknowledge the fact that thousands of people must have witnessed those things.