My biggest concern is that their profit margins are sufficiently obscene that their very very rich clients have a huge motivation to roll their own. That would take a lot of time and money, but doesn't seem impossible.

Already happening; and its taking lots of capital but Google is making in roads with its TPU chips.

Hacker News: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

Quote:
Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects.

To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale Nvidia system based on 144 “Blackwell” GPU chiplets with an aggregate of 20.7 TB of HBM memory look like a joke."

Nvidia may have the superior architecture at the single-chip level, but for large-scale distributed training (and inference) they currently have nothing that rivals Google's optical switching scalability.

tecmo
...