2026-04-06 07:57 UTC
@MichaelTBacon @alienghic @cwebber Re: chips... That post about Azure is quite interesting but not related to AI accelerators. The core workload for these neural nets is almost totally unrelated to what a general purpose CPU does, and for large models the majority of energy consumption is DRAM and interconnect (e.g. not the arithmetic). So the answer to how much more customized they can get is "a lot" if you frame the problem as how to avoid interconnect and DRAM usage. Wafer scale etc.
Replies (1)
-
@mirth@mastodon.sdf.org 2026-04-06 08:00
@MichaelTBacon @alienghic @cwebber Between improved models, app designs, and hardware, it seems costs would come down. Do you have a source for operating costs being 10x greater than the current API pricing, or typical developer usage being 2B tokens a month? I have access to some developer usage data and I'd guess devs using every day average under 10M tokens/dev/month. I'm sure there are some wild outliers, but that is a typical problem many subscription businesses manage.