If there was documented, provable harm done to the residents from the lead poisoning (not saying there isn’t, I just don’t know), they could surely win a civil case claiming negligence by the city and/or water treatment facilities/etc… There may also be laws or regulations in Jackson or Mississippi that were broken. Perhaps there aren’t, but there maybe should be. Which are all points brought up by Engelhardt:
> The Constitution does not provide redress for every governmental wrongdoing. Rather, the remedy for Plaintiffs’ injuries lies in pursuing tort claims, electing representatives who will better manage the public-water system, and petitioning their representatives for other remedies. And Plaintiffs have not been left to go at it alone. The State of Mississippi and federal government have worked for years, and continue today, to improve the City of Jackson’s drinking water quality through regulations, investigations, and compliance plans. These already-established-and-undertaken avenues are the proper course for rectifying the lead contamination. We will not expand the Fourteenth Amendment to create novel theories of constitutional liability.
And now people who don’t take the time to actually read, or just can’t understand, are gonna be yelling and screaming about how the courts are corrupt, etc… (some courts may very well be corrupt! But this would not be a good example why).
> If there was documented, provable harm done to the residents from the lead poisoning (not saying there isn’t, I just don’t know), they could surely win a civil case claiming negligence by the city and/or water treatment facilities/etc
I would never understand this system. So to get repatriation on damage done a citizen has to mount an entire legal initiative. Even if the guilt is already proved. It just feels like a system where departments of government will only sue to fill their own budget next year.
This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way.
Also, what is the difference between “target model” and “target-model,” if any? I feel like half the instances of that phrase included the hyphen and half didn’t.
You're correct that it needs to run the full model to "verify" a token, but LLM inference benefits from batching - it's much faster to run twice in parallel than sequentially. So the draft model runs ~2-5 tokens ahead, and the full model then runs ~3-6 batches in parallel using those tokens, and can skip ahead by however many results match.
If you have some other source of parallel data (lots of users, many separate tasks) then speculative decoding might not provide any benefit.
I just finished overhauling our speculative decoding implementation for Mixlayer, so maybe I can help.
I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediate tokens even during prefill.
So for example, let's say you prefill the prompt "The quick brown fox" (and for the sake of simplicity, let's say each word is a single token). The model outputs a tensor that is [4, $vocabulary_size]. The first dimension is a token index into the input and the 2nd dimension assigns a probability to each token in the vocabulary. So even during prefill, we can look at the prediction logits for all of the intermediate tokens. That is, we can look at what the model would have predicted after "quick" and "brown", not just the tail token "fox".
In the single token autoregressive case, we just look at the next token prediction for "fox". But in the speculative decoding case we can use this information to compare the distribution of the draft model against the target model. In the greedy decoding case (ie, no sampling) we just make sure the highest probability token matches in draft and target. If we have sampling params like temperature and top-P, we have to apply something called Leviathan rejection sampling to the distribution. This basically allows us make sure the distribution is the same even if the exact probabilities are not and accept or reject draft tokens on that.
The target model is the original LLM that is large and expensive. It can verify candidate tokens in a single forward pass. It means you give all the context + candidate tokens that passes in parallel in the backbone, then you pass the language head (a matmul transformation to produce the token distribution) on all the candidate tokens and you can keep or drop tokens based on how many "quality" you want.
You need the next token to begin the next decode. So if you can get what might be the next token in half the time, you can kick off the next decode before the true decode for this token has finished.
If the draft was wrong you can kill the speculative decode, and you haven’t lost anything except for idle time
It’s true we can’t show the user the token until we have the true decode finished, but we can launch more work internally before we’re certain
> Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with.
Yes, but you can do it in parallel.
Suppose you predicted the tokens "D E F" in the sequence "A B C D E F". To "generate" the last token (F), it must know all preceding tokens (A B C D E). To "generate" the next-to-last token (E), it must know all preceding tokens (A B C D). And so on.
Assuming the prediction is correct, it can then run the "generation" for tokens D, E, and F at the same time. At the end, after all these tokens were "generated", it compares each token with the prediction; if the "generation" result was "D H F" it knows it has to discard the last two predicted tokens (and output "D H"), if the "generation" was "D E H" it knows it has to discard the last predicted token (and output "D E H"), etc.
And the most important part is that you can do it in parallel for each layer of the model. That is, you run "A B C D E F" through the first layer, then through the second layer, and so on; you only have to load the model weights from memory once for each layer. Instead of reading the full weights for all layers once for D, then once for E, then once for F, you only read them once for "D E F", and if the prediction was correct, you output three tokens by the (memory read) price of one (you still had to do the same amount of compute, but AFAIK LLMs tend to be more memory-bound than compute-bound).
> it must perform it's normal autoregressive decoding to know what is the correct token in order to have something to compare with
Correct except for the word "autoregressive". When you have to verify a sequence of tokens (which were autoregressively generated by the cheap model), you can do each token in parallel. This amortizes the cost of loading the weights from vram to the processors (the primary cost in LLM serving) across those tokens. Cost here is wall clock time, as well as power.
The autoregressive decoding that generates this batch of tokens is delegated to the cheaper model where the cost of loading the weights is lower and so not amortizing it is fine.
Verification means, how close is each token in this sequence to the one I would have output. You keep the longest prefix that is close enough for your liking.
The point is that radio, then television, then internet, then smartphones, then AI, each one of which is as impactful on the world as the examples you gave, emerged within a couple hundred years ago years, rather than a couple thousand (and the last few within a couple decades).
Not quite verbose but they tend to repeat the same few ideas multiple times with varied wording/imagery. You keep scrolling because you think you’re gonna see something new and by the end you’ve realized you just read the same thing 4 times over.
That “overbearing assumption” is made by practically every other X-media editing app, for good reason — you can save your work and continue/revert stuff later. If all you’ve ever used is MS paint then I guess it’s an unexpected thing, but any other image/video editing app, digital audio workstation, 3D modeling software, etc… work exactly the same way. It would be unexpected to me if “save as” exported a flat file.
I guess the idea is you can do edits via non-destructive layer effects and compositing a stack of layers, where somewhere amongst the stack is your original unmolested data. Though that has not been my experience with Gimp - it's far more likely to want to to commit to a rasterizing decision far earlier than Photoshop would.
I don't care. 99% of my uses of an image editor are for ad hoc edits. I never want to clutter my disk with project files on the off-chance I want to preserve layers or whatnot. I have zfs snapshots and cloud backups should I want to revert but the odds I want to revert an ad hoc image edit are approximately nil - I can never recall needing to do so.
I don't use Gimp as a 3D modeling software or a digital audio workstation. I use it to add text to a family photo. The primary storage medium is jpeg. I don't want or need anything else.
The real issues with Gimp start when you try to work with selections, transforms, cuts, crops and so on. Photoshop muscle memory doesn't transfer and it is so incredibly clunky.
> I use it to add text to a family photo. The primary storage medium is jpeg. I don't want or need anything else.
So you open your photo, add your text, press Ctrl+E, choose a filename or press Enter to overwrite your old version, and press Enter to accept the default jpeg settings.
I guess it could save a couple of keypresses by auto-overwriting your original file without asking, or not asking about jpeg settings. But it doesn't seem like an outrageous workflow, considering that other people will want those options.
I do feel the same way as you about Photoshop muscle memory, having got quite good at it in the '90s and never really having caught up in Gimp.
I can empathize. I don't really care about the save vs export thing, as I have adapted to that and just go directly to export in the menu.
But, as a long time but infrequent user, I really dislike the changes towards "non-destructive" UX that just seems to spawn endless layers and other forms of pixel buffer limbo that confuse me. Lately, it feels like I'm being slow walked by a passive-aggressive tool that wants to waste my time and mental energy.
I wish there was a "novice" or "casual" user setting option to go back to a much simpler UX, where I am operating immediately on the selected layer. Where a pasted object can be dragged but anchors as soon as I touch something else, etc. Where filters preview but then "apply" immediately. Where undo can revert some recent changes but otherwise the effects accumulate destructively into the current layer.
A naive user may never have to learn about layers, but they are still there when desired. I don't want a layer spawning on their own. I'll make the new layer before I start changing things, and I'll think deliberately how I want that layer filled. Then, I'll resume actions that immediately mutate it.
1) When you apply a filter, there's a "Merge filter" checkbox. If you check it, filters will be merged down immediately like in 2.10. The setting is remembered, though currently the unique Color filters are remembered separately from the generic ones.
2) If you go to Edit -> Keyboard Shortcuts and search for "Paste as Floating Selection", you can bind Ctrl + V to it so that you get the 2.10 floating selection behavior.
I was a straight A student all through high school; first year of college I got Bs and Cs, couple As, and failed one class. Next three years I got my shit together and was back to straight As, including retaking the class I failed. I would have appreciated this lol
> The Constitution does not provide redress for every governmental wrongdoing. Rather, the remedy for Plaintiffs’ injuries lies in pursuing tort claims, electing representatives who will better manage the public-water system, and petitioning their representatives for other remedies. And Plaintiffs have not been left to go at it alone. The State of Mississippi and federal government have worked for years, and continue today, to improve the City of Jackson’s drinking water quality through regulations, investigations, and compliance plans. These already-established-and-undertaken avenues are the proper course for rectifying the lead contamination. We will not expand the Fourteenth Amendment to create novel theories of constitutional liability.
And now people who don’t take the time to actually read, or just can’t understand, are gonna be yelling and screaming about how the courts are corrupt, etc… (some courts may very well be corrupt! But this would not be a good example why).
reply