So the actual cost to consumers is likely to rise not only from the increase in price due to the tariff, but also to price increases (or higher availability) of goods free of the tariff/tax. Consumers not only pay a higher price for the taxed goods, but may be forced to pay higher prices for alternative goods as well.

That is true. Tariffs are like all taxes. They shift behavior from what the parties would have done absent the taxes, which results in a loss of utility to both buyers and sellers of the good in question. Producers make less, consumers pay more, and the government receives revenue - and neo-classical economy theory teaches that the amount of revenue to the revenue will be smaller than the combined loss to consumers. That difference is the "deadweight loss" to all the market participants.

But the idea with tariffs is that the producers are foreign firms. So while the net effect on all market participants (including the government revenue) is a deadweight loss, the net effect on all domestic participants (including the government revenue) doesn't have to be a deadweight loss. If the tax incidence on producers is very high compared to consumers, it is possible that the domestic parties collectively will be better off. Consumers pay higher prices, but the higher prices they pay could theoretically lower than the revenues collected by the government.

Again, if people are having trouble seeing how tariff supporters can convince themselves that this might be a win-win, consider it a mirror to the "carbon tax plus rebate" argument that used to be made by progressives before they gave up on the idea of a carbon tax in any form. There, the idea was that as long as you structured the carbon tax and rebate dividend appropriately, enough of it could be forced onto a very small group of "other" people (in that case, the rich) so that you could have significant taxes on everyone else without causing them any economic pain. Even though that would have forced "consumers to not only pay a higher price for the taxed goods [fossil fuels rather than imports], but may be forced to pay higher prices for alternative goods as well [alternative energy rather than domestic goods]."

Same lyrics, different tune. People tend to exalt neo-classical micro-economic models of the markets inconsistently, depending on whether the results support or oppose their preferences...