Haven't read the books. But he could have left it with Kalesi with her rule, Tyrion in power, John Snow rising to prominence. Will all those people continue? Will they take up fighting again? I've seen several movies that end without a definitive conclusion. It is unusual, but it can be done.

You're absolutely correct that he could have aimed towards that ending. But that's why he's having problems. Because even with just those three characters, it's hard to "force" them into a place where that actually happens while still being consistent with the initial conceit of the first five novels: that these characters will act like real people but living in a world with fantasy elements rather than medieval Europe. You haven't read the books, and I don't want to spoil anything in case you pick them up - but it doesn't make sense that you would have all three of those things happen, nor is there enough "time" in only two novels (consistent with the pacing of the earlier ones) to get to that point.

The criticism of the TV show, which is deserved, is that the writers ended up rushing everything and forcing the characters to make irrational or dumb choices in order to get them to where they needed to be during the last two seasons. I think Martin has written himself into a position where anything he tries to do in the next two novels will suffer from the same faults.

As you point out, you can instead finish a work without a conclusion, but it's hard to do it well - and it's impossible to do it well if you haven't started the movie on that basis, with that kind of inconclusive ending in mind. It's unusual and requires a lot of craft, so that the audience is left with an inconclusive ending that feels intentional rather than just feeling (correctly) that the work didn't have an ending at all. Those types of works will often have something else, other than plot, that ends up working as a through line that makes the work cohere even if there isn't a conclusion - a theme, a philosophical question, or a character arc. I don't think Martin's first five novels in the series are built with that in mind.