> I would always pick an incremental batch load with a cursor value over a CDC connection.
What's the benefit vs. something like Postgres's logical replication for CDC? IMO, the hard part of CDC is maintaining consistency in the face of potential network issues or downstream slowdowns. One is forced to choose between scylla: generate excess trx logs if replication slows, and charybdis: lose consistency. I don't see how an open transaction helps here?
> Q: Are the releases aligned with pre-training efforts?
> A: There used to be a time not that long ago, maybe half a year, distant past, where the models would align with RL runs or pretraining runs ... now the naming is by capability. GPT5 is a capable model; 5.1 is a more capable model
> I also think it’s important to notice that a lot of these challenges they happen with humans too. The concept of prompt injection isn’t that different from social engineering, right? When somebody calls in and says, “Oh, I forgot my password, can you just help me this one time?”
I wonder if the error propagation problem could be solved with a “branching” generator? Basically at every token you fork off N new streams, with some tree pruning policy to avoid exponential blowup. With a bit of bookkeeping you could make an attention mask to support the parallel streams in the same context sharing prefixes. Perhaps that would allow more of an e2e error minimization than the greedy generation algorithm in use today?
Having the data structures is nice and all, but using them is kind of painful. They are certainly second class.
Having to use accessor functions or destructuring macros instead of just a period or -> is often annoying too. The lack of syntax has cons as well as pros.
Writing a reader macro that allows for something like...
[some-numbers 0]
...to get the first (many programming languages make this mistake, using 0 to refer to the first element of a collection, so we can forgive CL for this) element. But I'm curious how you can write...
(object -> slot)
...without getting an error about OBJECT not being a valid function or macro.
The 1962 dated Lisp 1.5 Programmer's Manual already describes a 0 based array feature. Lisp was clearly one of the historic instigators of zero based array, rather than just playing along.
Yes, but the various Lisps that Common Lisp is the more-or-less common subset of are (were?) all 0-indexed. Between easy heap implementation (left is (ash index 1), right is (1+ (ash index 1)), parent is (ash index -1)) and easy last element selection (nth seq (length seq)) I prefer 1-indexing, but I realize that's an unpopular opinion.
A late reply but it's worth addressing one way of doing this. First, your concern about object not being a valid function or macro isn't relevant at read time. Second, note that Lisp already has similar syntax: '(1 . 2) is essentially (cons 1 2). Implementing this type of syntax is not a privilege of the implementation alone. You're allowed to redefine your own reader for left paren. In SBCL:
You can write `(set-macro-character #\( 'sb-impl::read-list)` and everything continues to work just fine. You can also jump-to-source and modify it if you want -- though it's cleaner to just copy it out to your own project, that's what I did for a quick hack/proof of concept. Essentially I added before the existing (when...) which handles the special dot syntax:
(when (and (eq firstchar #\-)
(eq (peek-char t stream t nil t) #\>))
(read-char stream t) ; actually read the nextchar > to discard it
(let ((next-obj (read stream)))
(sb-impl::flush-whitespace stream rt)
(return `(slot-value ,@listtail ',next-obj))))
I won't claim this is good or proper, but it shows that it's quite feasible. We've turned (foo -> bar) into (slot-value foo 'bar).
Personally I wouldn't use this even if it was more properly/carefully implemented. (There's really no reason to replace the default left-paren reader, and no reason we have to have a space surrounding the "->". One thing I like about the infix reader macro package https://github.com/quil-lang/cmu-infix is that it doesn't care about spaces, I can write #I(1+1 + 4) and get 6.) I'm quite happy putting my class in its own package, and thus getting the primary tab-completion behavior I care about. e.g. "(ma:<tab>" could complete to "(math:" and then "(math:v<tab>" could complete to a list of options like "vector-x" "vector-y" or so on. I also like the somewhat unusual approach of naming my accessors with a dot prefix, e.g. (.x vec) and (.y vec), or even (math:.x vec) if I haven't imported the symbol.
Sparse attention essentially combines 3 types of attention optimizations:
1. Compression of the query input vectors to reduce the size of the KV cache
2. Selectively computing uncompressed attention on a subset of tokens based on the compressed blocks with the highest attention scores
3. Using sliding window for local attention at full resolution
> Both Full Attention and sparse attention models are pretrained on 270B tokens of 8k-length texts, followed by continued training and supervised fine-tuning on 32k-length texts with YaRN to achieve long-context adaptation. Both models are trained to full convergence to ensure fair comparison.
> our experiments adopt a backbone combining Grouped-Query Attention (GQA) and Mixture-of-Experts (MoE), featuring 27B total parameters with 3B active parameters
Evaluated on MMLU, MMLU-PRO, CMMLU, BBH, GSM8K, MATH, DROP, MBPP, and HumanEval. NSA outperforms full attention on 7/9.
Beats out H2O, InfLLM, Quest, Exact-Top, and full attention on LongBench
Perfect retrieval on 64k needle-in-a-haystack
The CoT eval is less convincing, but outperforms the FA on AIME24.
Training speed of 2-9x vs. FlashAttention
Decoding speedup of 4-12x vs. full attention ["expected"? Didn't see comparison to other attention mechanisms]
Great to see this is alive and progressing! I believe Ohm started life in Alan Kay’s research group, to build a graphical OS and office suite in 10k lines of code. I found this talk immensely inspiring https://m.youtube.com/watch?v=ubaX1Smg6pY
Very close! Alex Warth created OMeta (https://en.wikipedia.org/wiki/OMeta) as part of the STEPS project. Ohm was designed as a kind of successor to OMeta, but was created after STEPS.
What's the benefit vs. something like Postgres's logical replication for CDC? IMO, the hard part of CDC is maintaining consistency in the face of potential network issues or downstream slowdowns. One is forced to choose between scylla: generate excess trx logs if replication slows, and charybdis: lose consistency. I don't see how an open transaction helps here?