I find that writing spaced-repetition prompts in the way the article describes to have too small of a return on investment: Reviewing them is quick, but it takes a lot of effort to craft good ones from a text.
Good authors take care with their prose and use nuance to get their point across, which is often lost in the translation to a few bullet points. This subtlety is often the meat of the text, and the high-level points are the bones that hold it all together: essential, but not where the real value is.
My Anki deck is mostly cloze deletions automatically extracted from ebooks, where I've imported the entire text ~20 lines at a time. I've deliberately set up the review schedule so that it's hard for me to memorize particular phrases: 1 week for the first review and 4 weeks for the second, with a 1 day delay for misses. This forces me to remember what role the passage plays in the wider narrative (argument for non-fiction) to successfully predict the blanked-out word.
With this setup, I'm noticing new connections between different sections of the same book, and between different books, every time a card comes up for review: My understanding of the world has changed since I last saw it, and so I notice different subtleties than I did the first time around.
Instead of viewing them as a chore, I'm now actually excited to do my Anki reviews every morning-- where they used to make me mentally exhausted, they're now a warm-up that gets my brain started for the day.
Though this engagement itself aids memorization...
I'm not a fan of spaced repetition, instead preferring to understand material and find connections (associations) integrating it within my existing knowledge. This makes it usable, not just recallable. It's holographic, in that I can also recover it from other things I know, like the shape of the missing jigsaw piece, or a theorem or proof.
However, I've come to realize that the very act of engagement to understand is, in itself, aiding memorization.
A key insight for me was when I realized that spaced repetition isn't really about memorization, it's about skill maintenance.
The skill most people use formal spaced-repetition methods for is factual recall, to be sure, but the underlying principles can be applied to any human endeavour. My Anki deck, for example, is designed to exercise reading comprehension. For cooking, I don't try to memorize recipes like the article demonstrates; I do, however, try to revisit and make the dishes in my repertoire semi-regularly to reinforce the operational cues that are impossible to put into words. It's similar for all of my other hobbies: I naturally tend to have a period of intense engagement with them at first that trails off as my confidence builds. I seem to revisit most of them on the same sort of exponential schedule without really trying, as if my subconscious understands when it's time to have a refresher.
This is very interesting! Can you say more about the task you perform when you see these prompts? How do you decide how to "grade" your response? Presumably it's not just whether or not you can remember the deleted word(s)?
Also: are you blanking out individual words? Phrases? Sentences? An example would be interesting if you're willing to share!
Edit: Though my examples below are all fiction, that's half or less of my current reading list. I also have philosophy, research papers, long-form academic books, popular science books, and the like.
-----
For reference, here's the next new card for one of the books I'm reading:
Bleak House
Charles Dickens
CHAPTER III
---------------------------------------------------------------------
coach gave me a terrible start.
It said, "What the de-vil are you crying for?"
I was so frightened that I lost my voice and could only answer in a
whisper, "Me, sir?" For of course I knew it must have been the
gentleman in the quantity of wrappings, though he was still looking
out of his window.
"Yes, you," he said, turning round.
"I didn't know I was crying, sir," I faltered.
"But you are!" said the gentleman. "Look here!" He came quite
opposite to me from the other corner of the coach, brushed one of his
[...] furry cuffs across my eyes (but without hurting me), and showed
me that it was wet.
"There! Now you know you are," he said. "Don't you?"
"Yes, sir," I said.
"And what are you crying for?" said the gentleman, "Don't you want to
go there?"
(The prior cards include the text up to and including the line starting "But you are!")
My response, having never seen the passage before, was "coat's"; the correct word is "large". The substitution doesn't meaningfully alter the passage: The only thing we know about the gentleman at this point is that he's a stranger whom Esther has just met in the coach, wearing a large coat (cloak?). This is the first time he speaks in the book.
I selected "again" because this is a new card; if this had been a review, I might have deemed it acceptable. The bar for similarity varies depending on the book the passage came from, my mood on any particular day, and how much I care about the contents of the passage. The Rime of the Ancient Mariner, being poetry, doesn't get any latitude: I must remember the word exactly. Some of the tedious battle descriptions in Le Morte d'Arthur get a pass no matter how badly they go wrong.
If a card is causing me trouble, there are a few corrective actions I can take: In some instances I'll edit the card to blank out a more meaningful word than the one randomly selected, which is an easier and more engaging review task. If I can't figure out the blanked word from context, I'll look it up in a dictionary; this happens most often with my second-language texts. And in extreme cases, I have enough coverage of each book that permanently dropping card or two won't cause problems.
The idea for this came from the paper[1] that introduced the concept of Cloze deletions. They were originally envisioned as a readability score. Instead of calculating statistics about the words used, like traditional methods, Taylor proposed to blank out every Nth word and measure how many could be guessed correctly by a reader. Readability and comprehension are two sides of the same coin. They both rely on observing the relationship between text and reader. If used to evaluate the text, this is called readability; if to evaluate the reader, comprehension.
I'm intrigued. I've read your comments in this thread, and I can't comprehend how you decide what to cloze on. Is this something that's decided by your script? Do you have multiple clozes per note? Why not cloze on "wrappings"?
If you have a blog post describing your process, I'm sure I, and many others, would love to read it.
I have so many questions, I am intrigued! Why are you doing this? Are you reviewing whole books in this fashion, as in the whole text? How long does that take? Do you read those books first? Are you reviewing in order of the order the text was written in? How much time do you spend reviewing?
First, the mechanics: I'm reviewing entire books that I haven't read before; I have a Python script that will split up a text file and produce a CSV file for Anki to import. New cards are set to come up in the original order, but reviews follow Anki's scheduling algorithm. Because I only have 1-2 new cards per book per day, the effect is similar to having several bookmarks: One is the most current, the second trails the first by a week, and the third by a month, etc.
A "normal" length book, like a genre fiction or popular science book, generally produces about 1000 cards like the one above. At a pace of 1 new card per day, that's about a three year commitment. It's a lot easier to have more books going than to increase the pace on any single book; the brain likes variety. I've got about 20 books going at once, which is around an hour of reading every day, including both reviews and new material. Overall, that works out to an average pace of 1 "standard" book every two months.
As for why, it started as an experiment to fix several problems I was having at the same time: My preexisting Anki deck was running dry, but still contained items that I wanted to keep reviewing; I needed a source of low-effort cards to keep the review habit going. I also had a long list of books that I should read someday but that day never seemed to be getting any closer; I decided to force the issue.
And finally, I had been unable to figure out how to make flashcards for literature at all. What series of questions / prompts can you write that captures the essence of something like this:
It is an ancient Mariner,
And he stoppeth one of three.
'By thy long grey beard and glittering eye,
Now wherefore stopp'st thou me?
The Bridegroom's doors are opened wide,
And I am next of kin;
The guests are met, the feast is set:
May'st hear the merry din.'
He holds him with his skinny hand,
'There was a ship,' quoth he.
'Hold off! unhand me, grey-beard loon!'
Eftsoons his hand dropt he.
He holds him with his glittering eye—
The Wedding-Guest stood still,
And listens like a three years' child:
The Mariner hath his will.
The Wedding-Guest sat on a stone:
He cannot choose but hear;
And thus spake on that ancient man,
The bright-eyed Mariner.
(...)
Short of memorizing the poem word for word, it's hard to imagine any set of prompts that would adequately capture this: The events are almost incidental compared to the cadence and sound of the words. Or, using the Bleak House passage from above: If this gentleman becomes a recurring character in the book, I'm bound to learn more about his personality as the story progresses. If he exhibits some small behavior in this scene that foreshadows his character development later, how am I to notice it? If I have simply noted that he and Esther first meet on this trip and never revisit the actual text, I'll not have the opportunity to make that connection.
Thank you for your detailed response. I'm really tempted to try this as I'm already in the habit of using anki on a daily basis. Would you be willing to share your Python script?
If I may, how do you ensure you're only getting one or two cards per book? Do you create a deck per book, or is all in one deck and there's some trickery in the review settings you do?
Additionally, it's really interesting to me that there is a non-linear approach for the reviewing, what's the impact on your enjoyment of books? And, does this approach give you better retainment of details?
Finally, is the length of text set, and how did you determine what it should be?
Really interesting how you extract cloze deletions from ebooks! It sounds very similar to what I'm building with Traverse.link, a social spaced repetition app with connected cards.
You can import an article (working on ebook support), and it'll turn the paragraphs into cards, to which you can then add cloze deletions and free form prompts.
I enjoyed this article however the examples given were markedly poor. The article talks a lot about distilling knowledge and what it is to “know” a topic, then the example flash cards about the article ask things like “vague prompts result in ______ recollection”, which is trivia about word selection. Although I suppose this just indicates how difficult prompt writing can be, perhaps a section on “what it means to answer a prompt” was necessary. Consider as an alternative: “what is the problem with vague prompts?”
From the article:
"Unfortunately, most spaced repetition interfaces treat each prompt as a sovereign unit, which makes this kind of high-level revision difficult."
This is why I've built Traverse (https://traverse.link/), a spaced repetition app with hierarchical and interlinked markdown flashcards, so you study connected knowledge rather than bare facts
If you are using spaced repetition, or thinking about using spaced repetition with these kind of hand-crafted notes, you might be interested in Mochi[0], which uses markdown note-cards and kind of mixes the Zettelkasten concept and spaced repetition into one app.
What is the advantage of a proprietary SaaS SRS webapp as opposed to open source Anki? (note that Anki has support for, for example, LaTex, which makes collections in topics such as math highly useful).
I read a couple pages of the friendly article, waiting and waiting for the missing vocab word to be filled in.
I reluctantly opened the link that was an introduction to spaced repetition.
I opened the Wikipedia article for spaced repetition.
I searched on DuckDuckGo for "Anki prompt" and "Anki what is a prompt"
I opened the Wikipedia article for Anki.
What the hell is a prompt? Is it the question on the flashcard? Does Anki show you pop-ups periodically? There's no onboarding ramp for someone who doesn't know what spaced repetition is to read this article about 'prompts'
Prompts is a piece of jargon, unfortunately confusing but crucial to anyone trying to memorize anything using an SRS. A prompt could take the form “What is a X” or be a blank “the dog ____ over the log” or just be a foreign word like 中国. So in some ways its a more general concept than a “question” but calling it the front side of a flashcard is probably clearer.
The problem is always forgetting what is uncommon jargon and what your readers probably already know. We all have our bubbles (tech folk notoriously so).
Ironically the solution already exists but no one uses it because of the editorial overhead: hyperlinking text to their definitions. Also, there are many ‘types’ of linking and they all look the same (blue underlined text by default). I’d love a more subtle “context” link to be natively supported. Maybe one day the semantic web will save us all...
I’m not even sure if it’s particularly jargon-y but it might be more familiar as a verb: “When the witness paused, the officer prompted her to continue, saying “Any detail, no matter how small, might help us find the kidnapper.’”
The article has example prompts on its own content, served with the author's platform Orbit, I found these to be a useful (though meta) illustration of what he means by prompts.
The word “prompt” has nothing to do with SRS, I learned it in grade school English class. The “prompt” is the thing given by the lesson to prompt you to write onwards. Although to be fair I didn’t actually find this definition in the two dictionaries I checked, which was very surprising to me.
Creating and generating good questions and answers is a skill. Too vague, you don't know if your answer is incorrect or not. If your answer require too much, you're never going to get the answer right, and so forth.
"Questions" is too specific, as prompts are often not actually phrased as questions. E.g. cloze deletions, where you fill in a missing word, or vocabulary for language learning, where the prompt is a word and the answer is the translation.
I went to read the article expecting it to be about making command line prompts that would help users remember them by acting as spaced repetition lessons.
Exactly my thoughts. The article is all pseudoscientific hogwash.
I skimmed through to understand what it's about or at least to know what a "prompt"/"spaced repetition" is. I still don't know, but I bet it's something very trivial.
I find it peculiar that you can confidently dismiss something as pseudoscientific hogwash while simultaneously admitting that you don’t know what it is.
It doesn't explain what "spaced repetition", "memory", "understanding", or "experiments" are either. It is written for people who already use spaced repetition, and aims to help them improve one aspect: writing prompts. It is not intended as an elementary introduction to spaced repetition, and so it doesn't bother to explain basic concepts.
Your objection is a little like complaining that a computer science textbook aimed at post-grads doesn't bother to explain what a variable is. Some background knowledge is assumed.
A prompt in this context is just the side of the flashcard that prompts you to remember the "other side"/answer. They can be questions, sentences with words missing, or many other things. E.g. for learning a language a prompt could be the English word, prompting you to remember the translation. Gwern calls them "questions" in his article, but "prompt" is more general, given that they're often not phrased as questions.
There are many examples of prompts and answers in the main article, with blue backgrounds, and prompts about the article's content with red backgrounds.
Man, those two comments sound uncharitable. In the first paragraph Andy states:
> This guide assumes basic familiarity with spaced repetition systems.
The words "prompt"/"spaced repetition" are basic jargon for such systems.
But to make this comment at least a bit helpful, too: An example of a "prompt" in a spaced repetition system is the front of a flashcard. It's "spaced repetition" because you look at them multiple times, spaced out over time.
Your persistent desire to expose your ignorance and belligerence to us all on this subject is quite fascinating. Spaced repetition as a mnemonic tool and pedagogical technique has decades of scientific research behind it and a simple search on Citeseer leads me to more than a thousand citations. It is neither trivial or pseudo-scientific.
Good authors take care with their prose and use nuance to get their point across, which is often lost in the translation to a few bullet points. This subtlety is often the meat of the text, and the high-level points are the bones that hold it all together: essential, but not where the real value is.
My Anki deck is mostly cloze deletions automatically extracted from ebooks, where I've imported the entire text ~20 lines at a time. I've deliberately set up the review schedule so that it's hard for me to memorize particular phrases: 1 week for the first review and 4 weeks for the second, with a 1 day delay for misses. This forces me to remember what role the passage plays in the wider narrative (argument for non-fiction) to successfully predict the blanked-out word.
With this setup, I'm noticing new connections between different sections of the same book, and between different books, every time a card comes up for review: My understanding of the world has changed since I last saw it, and so I notice different subtleties than I did the first time around.
Instead of viewing them as a chore, I'm now actually excited to do my Anki reviews every morning-- where they used to make me mentally exhausted, they're now a warm-up that gets my brain started for the day.