And…?
That’s my point. At every instance in human existence someone said ‘don’t’ Because “it’s too hard” or a variation of that.

And...that's why we shouldn't do it.

You're absolutely correct that we should never avoid doing something that's worth doing just because it is hard. But that doesn't mean we should ever do something that isn't worth doing just because it's hard, either.

It's the reason we don't have undersea colonies today. We probably could establish an undersea colony, using current or near-future tech. It would be insanely difficult, but 1000x less difficult than a space colony. But we don't. Because it just isn't worth it to do it. It if were less difficult - if the humans who lived in an undersea colony could survive with less privation and less risk of instant death and less resources to persevere - then we might attempt it. But it's not.

The same is true of space colonization (not exploration). Given the stuff you'd have to do in order to make it work (including a lot of what are essential human experiments on babies and children), it's not worth it. Any more than it's worth establishing an undersea colony.