It was The VVitch with hallucinations. The ending is almost the same, just a spoonful of Rosemary's Baby added.
When Toni Collette tries to get Gabriel Byrne to throw the book into the fireplace, I thought for sure Byrne was going to reveal he was possessed by the demon this whole time. I don't understand that scene. Why is Collette pleading like she can't do it herself only to then do it herself when Byrne refuses? She didn't know the consequences because if she did she wouldn't have tossed it in.
Why did the son break his nose? I don't understand that scene at all. What was actually happening and how was the demon benefitted by breaking the nose?
By far the most effective moment was the fate of the daughter, though it set me up for a much different movie, one that would make horror out of random tragedy instead of supernatural weirdness. I read the director said this isn't a horror film but a "tragedy that curdles into a nightmare." If that were true, I believe he would've used less ants.