Not the OP, but I've actually made something like that to help people figure out where to live in London based on the criteria they care about -- here it is https://findmyarea.co.uk/?search_type=areas
Would be great to see similar tools for other cities!
In principle it should be possible to do the same for any city. The hardest part is getting the data and massaging it into shape, which can be somewhat tedious. London is a fairly "easy" city for this due to the large amount of open data that's readily available from the ONS and the like.
This is really cool! Appreciate sharing this work and the explanation.
You mentioned that massaging the data into shape as one of the problems, which I think is possibly one of the best applications of LLMs in my opinion. Creating a pipeline of a data feed (still hard) into an LLM which outputs JSON with the fields of interest would be amazing.
The code never advanced beyond some hack scripts and a Jupyter notebook. Cleaning it up and perhaps making it into web app has been on my todo list for the past 2-3 years. Maybe this reminder will be the kick I need to actually do something about it.
Sorry I don't know how hackernews notifications work, didn't see this reply. But if you see this here's another reminder for you to clean this up into a public demo :)
I just found about this project from this comment, absolutely excited to try this out.
As someone who's never used any of the infrastructure tools, I'm thinking of pyinfra as a way to run shell commands + install dependencies on hosts (declaratively?) on a bunch of hosts via ssh.
Inventory is to sort of take a self-defined inventory on a bunch of hosts?
One final question on usage, would it be possible to sync or reference files from the machine running pyinfra on the remote hosts? Or would that have to be done indirectly via running shell commands to sync?
I think distillation in the original sense isn't being done anymore but finetuning on outputs from larger models like GPT-4 is a form of distillation (top-1 logit vs all logits and a curated synthetic data instead of the original dataset)
On quantization though its still weird how just the weights are quantized in methods like gptq / int8 while there are other methods which quantize the activations as well. There's also the matter of KV cache still being in original 16bit precision regardless which is also unsolved here. Do you have any thoughts or insights into this?
It’s not clear to me what’s happening on the distillation front. I agree no one is doing it externally, but I suspect that the foundation model companies are doing it internally, performance is just too good.
There’s a bunch of recent work that quantizes the activations as well, like fp8-LM. I think that this will come. Quantization support in PyTorch is pretty experimental right now, so I think we’ll see a lot of improvements as it gets better support.
The KV cache piece is tied to the activations imo- once those start getting quantized effectively, the KV cache will follow.
1) i actually think that’s too high, i bet it’s more like 30%. My logic is that they have to have _some_ margin, but LLMs are too expensive to have typical software margins. Total speculation though.
2) It generally tracks pretty well unless the model is gaming the metric (training on the test set, overfit to the specific source of data, etc). The relative rankings will typically match in both.
3) alas, not with the mild winter North America’s having. They only stop below -5C or so. I am lucky though. The woodpecker stopped attacking my house and started attacking my neighbor’s. Even worse, it used to be a downy woodpecker,and it’s now been replaced by a pileated one (think: Woody).
There are some unique assumptions being made in parts of the gist
> 10: Cost Ratio of OpenAI embedding to Self-Hosted embedding
> 1: Cost Ratio of Self-Hosted base vs fine-tuned model queries
I don't know how useful these numbers are if you take away the assumptions that self-hosted will work as well as API.
> 10x: Throughput improvement from batching LLM requests
I see that the write up mentions memory being a caveat to this, but it also depends on the card specs as well. Memory Bandwidth / TFLOPs offered by say 4090 is superior while having the same amount of VRAM as 3090. The caveat mentioned with token length in the gist itself makes the 10x claim not a useful rule of thumb.
> This means it is way cheaper to look something up in a vector store than to ask an LLM to generate it. E.g. “What is the capital of Delaware?” when looked up in an neural information retrieval system costs about 5x4 less than if you asked GPT-3.5-Turbo. The cost difference compared to GPT-4 is a whopping 250x!
In a narrow use-case of a strict look-up. This seems to exaggerate the cost difference while having completely different trade-offs.
I've been looking for a course like this! Especially great given how much of the recent progress in training large models is made possible with the aid of flash attention and fused kernels
In my case It was link sharing, no torrents, and at that time that didnt constitute -public publishing- as the link linked to some 3rd party host, and it was also user generated content.
FWIW they've only won some fringe cases where they scared the person into accepting the charges.
