Flash next only needs 64GB for core model inference. If you really wanted it. Ne... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		bitexploder 3 days ago \| parent \| context \| favorite \| on: Notes on gotchas while migrating 35kb preprompts f... Flash next only needs 64GB for core model inference. If you really wanted it. Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm. Maybe older GPUs work.
		help

hedgehog 3 days ago [–]

128GB on a Ryzen 395 is enough but not by a lot. Around 90GB for weights plus 10GB to 20GB for KV cache and checkpoints. I don't know about 64GB of VRAM for less than around $2500 by itself.

bitexploder 3 days ago | [–]

Tesla V100? Would have to be a 3-ish bit quant in 64GB ram

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact