I think until you're in such extreme circumstances it's unrealistic to say what you would or would not do. So without knowing one way or another I think it would be presumptuous for me to say "he went too far when he did x".
But to look at it another way, it's fair to say he's doing things he would absolutely not be doing if it weren't for the crazy circumstances. He would never kidnap and beat up Dano if it weren't for his children being missing.
Can you say the same of Dano's character? Would he still be viciously strangling his own dog without the slightest bit of remorse if it were just any other day? I believe he would. That didn't look like first-time animal cruelty to me. If it was, it was a tremendous coincidence Jackman happened to be sitting there in his truck witnessing it. More likely he abuses the dog every day.
Now which character is beyond redemption, the one who commits torture regardless of the circumstances, or the one who commits torture only in these extreme circumstances?
One? Both? Neither?
My feeling, based on all there is to know, is that neither character is beyond redemption. But that's just a feeling, and it's based on incomplete information. We're not deep enough in either characters' head to know for sure whether they WANT redemption (or in Dano's case whether he even has the mental capactiy to want it).
In Jackman's case I don't think he wanted to do what he did. He clearly didn't enjoy it. I think if his kids had turned up after the first day of torture and cleared Dano's character of having anything to do with it, I think Jackman would regret his actions. His character didn't seem so disconnected from humanity to me that he wouldn't feel bad about torturing an innocent person. I think he would. And maybe at the end of movie he does. We don't get to see his reaction to finding out the whole truth, or how he comes to terms with his actions.
I'll say this, there wasn't enough there to for me put his character beyond redemption. It was up in the air the whole time. And it's a question that the conclusion of the film did not resolve for me. So in summary, no he was not beyond redemption.
If he had killed Dano it would make it that much harder to come back from (assuming he wanted to come back from it). If he did want to, than I don't think you could say he is beyond redemption, just that much further from getting it. But again, it all depends on his wanting redemption, and that's something we never get know, just deduce. If he had killed Dano it would be harder to make that deduction because the action is so much more extreme. Or it might stretch a person's willingness to suspend judgement too far.
It's a good question.