The 100% learned issue, and suggested solutions

LessPaul · July 17, 2020, 1:31pm

First of all, Clozemaster is my favorite learning tool. With over a million points gathered I’ve spent a fair amount of time using Closemaster and learning it’s strengths and weaknesses.

This all has to do with limitations of the scheduling system. If you are using the site in multiple choice mode, this might not affect you as much, as the default levels (0/25/50/75/100%) with 0/1/10/30/180 are not bad for word recognition, and the 100% words only loop back twice a year.

However, for those of us who are using the type-your-answer mode and are focusing on word production rather than recognition, the intervals are far too widely spaced. I currently have mine at 1/2/8/20/60 days, with the 60 probably being a bit too long.

As a result “100% learned” words not only occur sooner, but they also repeat at a much higher rate. This is not too bad for words that are a still a bit foggy. However once words have hit 100% and are well known, they become a burden showing up every (for me) 60 days.

So yes, I can individually ignore them. However this is not only tedious, but not a good use of my study time. Also – when I ignore a word, it evaporates from my words learned, a metric that I enjoy for its motivation.

I can see multiple solutions to this issue. These are not only stand alone solutions, but are also mutually compatible (i.e. could be implemented in tandem.)

Add a “correct streak metric” and a “suspend 100% after correct answers” option. This would prevent it from re-scheduling, but not remove it from my words learned.
Add a user settable incremental percentage for 100% learned words. Say I set my increment to 50%. With my 60 day interval, the next time the word comes up it is 60+30=90 days. Then next time 90+45=135 days, etc.
Add more than the current number of levels. Rather than 0/25/50/75/100 make it 0/10/20/30/40…/100.

Lastly, of course a true SRS algorithm such as SM2 would be an incredibly elegant, and arguably the best solution. However that would likely require a major restructuring of the current codebase. The above 3 suggestions would likely be far easier to implement.

Since all of these issues probably only matter to the more active users, I can see them being limited to Pro, and also a good impetus to move to Pro.

mike · July 17, 2020, 8:15pm

Thanks for the post and feedback! Very much appreciated and glad to hear

This topic was also discussed in One-year review of Clozemaster Pro - #8 by alanf - Clozemaster - Clozemaster. Will continue that discussion here, FYI @alanf_us @kadrian @AlexG71 @adrianxu @Tarob and thanks for all your input there.

What we’re thinking at the moment is that we’ll use the easiness factor from the SM2 algorithm to determine the next review due date for correctly answered “100% Mastered” sentences. The SM2 algorithm is outlined here.

So! For “100% Mastered” sentences that are answered correctly, we’ll show three buttons in place of the “Next” button - Hard, Normal, and Easy. These sentences will have a base “easiness factor” of 2.5, and we’ll use the SM2 easiness factor equation

EF’ = EF+(0.1-(5-q)*(0.08+(5-q)*0.02))

where EF’ is the new easiness factor, EF is the current easiness factor, and q is the quality of the response - 3 for Hard, 4 for Normal, and 5 for Easy. The minimum value for EF’ will always be 1.3.

The next review interval, I(n), will then equal I(n-1)*EF', where n represents the nth repetition.

Technically we should be able to do this just be adding an easiness factor value for each sentence and using the current easiness factor and current next review date to determine the previous interval, which can then be used to determine the next interval when a “100% Mastered” sentence is answered.

All this is to say we’re aiming for a mix of a true SRS and option 2 you suggested

It will likely take a few more weeks to get this up on the web, and likely a bit longer after that to get it implemented in the mobile app. If you have any thoughts or think of any shortcomings/improvements to the above approach please be sure to let us know. Thanks again!

punk · July 17, 2020, 10:05pm

Please consider keyboard input for the buttons.

Shortcuts

1 - Easy
2 - Normal
Enter - Normal (to keep the current workflow going)
3 - Hard

For me something like this would work well, but if someone has a better idea, please suggest.

kadrian · July 19, 2020, 4:54am

Me too. I find overall sentences played more motivating than score these days. I’d prefer that sentences I don’t want to review again because they’re too easy didn’t disappear from my “sentences played” count.

alanf_us · July 19, 2020, 2:14pm

I’m very glad to see you actively considering this change, which would fix what I consider Clozemaster’s biggest remaining weakness. The SM2-based algorithm that you propose, which is also used by Anki, would be great, even if not necessarily perfect for everyone. At Anki, the parameters are exposed, but their functions are not intuitively obvious from their names (“Starting ease”, “Easy bonus”, “Interval modifier”), and I’ve never quite figured out how to configure the intervals exactly the way I want. But my configuration is good enough for me, and better for me than the default configuration. I also like the approach better than the one that Clozemaster uses currently.

I see that there are many algorithms out there beyond SM2 (SM3, SM5, SM17, SM18, etc.), and involved discussions of which one is best (for instance, https://supermemopedia.com/wiki/SuperMemo_or_Anki), but I suspect that (a) personal opinions will differ and (b) any one of these algorithms will do a satisfactory job.

cazort · July 19, 2020, 4:07pm

I really like that people are critically discussing and reflecting on the algorithm. I suspect there are a lot of changes that could be made to improve it and make it more effective for learning, even though I think it is already pretty good as it is.

I’m not sure that I feel similarly about the original point raised here though.

In my experience, with the default repetition intervals, at the rate I work through exercises naturally, in the long-run I end up with a lot of very easy words thrown in. If anything, I have wanted to have some words repeat less often. I also feel like the system doesn’t adequately separate between the 100% words that I know REALLY well, and the ones that I might have a risk of forgetting several months out. I’m not sure if this is possible or not given the current setup, but that is my only complaint about the system right now.

I tend to feel that you learn more if you’re “thrown in over your head” a little bit, which means that you’re continually learning new words. Yes, maybe you slightly forget some old words, but you’re also forging ahead…and keep in mind that most of those words you will eventually encounter in the context of other sentences, which will passively reinforce them.

If you make the system too easy, you risk it becoming like Duolingo, where it is just sort of rote and you can pour hours and hours into it and not really get that much better at the language. Ideally I think Clozemaster challenges you and continually throws new stuff at you. If you aren’t getting 100% of the stuff correct, it doesn’t matter. When learning a language, you really need to be challenging yourself mentally or you are not going to be learning at an optimal pace.

And I don’t know about you, but I really care about efficiency. One reason I love Clozemaster and prefer it to tools like Duolingo is that I think I can put in a small amount of time and get a lot out of it.

LessPaul · July 19, 2020, 10:25pm

Mike – An SM2 style solution is darn near ideal, and I could not be happier to hear this. I don’t know how I managed to miss the original thread, but I did.

My only question is how current progress will migrate into the SM2 model.

EDIT: Rereading above as I understand it the SM2 algorithm is in effect for only 100% words and thus migration is a simple matter of enabling the feature. Sounds good.

Again, thanks!

mike · July 21, 2020, 11:24am

Correct! Though the simple matter bit remains to be determined

@cazort thanks for the feedback!

Same! Like @LessPaul mentioned, and apologies this wasn’t clear in my post above - this change will only be for already 100% mastered sentences that you answer correctly again. Otherwise the system will remain the same. The idea is to simply stretch out the review intervals for mastered sentences.

If anything it might end up making things more difficult in the sense that you’ll have longer intervals for those sentences and will end up seeing more new sentences (keeping you a bit more “in over your head” and learning new words more often like you mentioned ).

So! Here’s a minor obstacle I’m hoping to get everyone’s input on: when/how to show these Hard/Normal/Easy controls after answering. Screenshots will show what we’re thinking for the flow at the moment (note - this will only be for 100% mastered sentences in spite of what’s shown here).

Essentially what’s being shown here is that, after answering an already 100% mastered sentence correctly, you’d then see the Hard/Normal/Easy controls. After selecting one of those options, you’d then see the normal sentence controls and Next button.

The downside is that it’s an additional click for these sentences. The upside is that Hard/Normal/Easy can set the initial next review due date and then there’s still the option to manual reset/master and use the sentence editor to change the next review due date if necessary. If we tried to show the manual master/reset and other controls at the same time as the Hard/Normal/Easy options, it’d get a bit confusing with what next review date should be shown, how changes should be handled, etc.

If you think the approach in the screenshots is fine please let us know, and if you think something else might work better please let us know as well! We still might end up going with the above approach, but curious to hear what you think.

Thanks again for all the feedback!

morbrorper · July 21, 2020, 11:45am

I wouldn’t mind having Hard, Normal and Easy buttons all the time. For a non-mastered sentence, “Hard” would keep the current mastered percentage, “Normal” would increase it to the next level, and “Easy” would set the sentence to 100 % mastered.

Or is this maybe too complicated?

LessPaul · July 21, 2020, 1:28pm

Personally, I feel the third screen being shown for all answers is a needless interruption of the review process.
How about Hard/Normal/Easy and a fourth option such as Edit? (or Modify, Change, Reschedule, Manage or ??)
So if Hard/Normal/Easy is selected I move directly to the next question. If I select “Edit” I am presented with the third screen and can further modify the results.

kadrian · July 21, 2020, 2:12pm

I’d strongly prefer a layout that requires one click only, with the default being “Normal” as it is now.

What about showing the Hard-Normal-Easy buttons on the same screen as the other sentence controls, underneath them? Hitting any of those buttons, or Next, or Return, could then take you on to the next sentence.

Mupfel · July 21, 2020, 7:38pm

I also vastly prefer only one screen. Efficiency

punk · July 22, 2020, 12:38am

A simple solution would be to show the normal date as a default.
Then, if the user selects 1 or 3, the date is updated, no big deal in my opinion.

Alternatively, instead of:

You could list the 3 possibilities, example:

Next review (easy): 2021-03-18
Next review (normal): 2021-01-18
Next review (hard): 2020-11-18

This lets the user know the exact date for whichever option he decides to pick.

Mupfel · July 22, 2020, 1:04pm

Hm, I think showing all three dates within in app might just clutter the (smartphone) screen.

I would prefer if the screen always had the buttons for voice
replay, hint, notes etc.

Contrary to morbroper, I would prefer if we had the Easy/Normal/Hard buttons only for mastered sentences.
First because I think it might be easier to implement
Mostly because I would like to have clean, simple interface that focuses on learning and fun. If I want complexity, I can always use anki…
Actually, before clozemaster I was using Anki for about 2-3 years, but in the end it was so annoying. It simple felt like homework.

mike · July 28, 2020, 12:06pm

Thanks for all the feedback!

This feature is now up!

These buttons and the variable next review interval only apply for already 100% Mastered sentences that are answered correctly again.

There shouldn’t be any extra clicks (went with one screen as discussed above), and the edit sentence modal shows the next review dates (good idea @punk). No new settings either. @morbrorper we may make it an option in the future to have these buttons available all the time like you suggested.

For those interested in SRS, we’re using an initial easiness factor of 2.0.

Hopefully this feature will help spread out reviews for mastered sentences a bit / prevent mastered sentences from backing up review queues, as well as help prevent leaderboard issues when 100% Mastered is set to a short review interval.

This feature is only available on the web for the moment. Curious to hear if you have any feedback / notice any issues, and then we’ll work on getting them added to the mobile app in an upcoming release. Any questions please be sure to let us know as well.

Thanks again!

kadrian · July 28, 2020, 1:03pm

For those of us who don’t understand algorithms, what does that mean in simple terms? Roughly how much is a bit sooner and a bit further in the future? Does this algorithm now completely override whatever interval you set for 100% mastered sentences, or does it start from it and make adjustments over time from there?

It’s a good day for updates here - I’m really appreciating the change that lets us prioritise 25% reviews!

LessPaul · July 28, 2020, 1:03pm

First of all, THANK YOU! This is a terrific enhancement of Clozemaster!

I am assuming the ability to adjust the easiness factor is in the works, since I am not seeing a new setting.

And I assume the first time the SM2 algorithm I(n) = I(n-1) *EF’ picks up the interval “I” it is from that word’s current interval, rather than the value set in the interval settings screen.

In other words – assume I have been tinkering with my scheduling – which I have – and some words have a different 100% intervals (60/90/120 days), the algorithm will be applied to whichever interval the word currently possesses.

jamesp999 · July 28, 2020, 2:15pm

So to make this simpler, just hitting enter will result in the interval increasing?

AlexG71 · July 28, 2020, 7:49pm

Thanks for this enhancement

A big drawback of CM for me in the past was that there were just too many reviews, once you had mastered a substantial part of the database.

I’ve been tinkering with the review settings and noted that the manual review interval setting for mastered sentences is now doubled, when you check the Normal review date in Edit mode, for a worked sentence (I assume this is due to the initial easiness factor of 2.0 and this changes over time?).

However, the differentials for the review intervals of Easy and Hard were only a few days different than Normal.

For some reason, I imagined the difference would be greater than that and it doesn’t really space the reviews out very much for sentences classed as Easy, Normal, Hard going forward.

Will the algorithm change the differentials between Easy, Normal, Hard over time for any given mastered sentence?

Just my opinion of course but I would have liked to have been able to choose the initial review date in the settings, as the starting point, before the algorithm kicked in, in order to deal with the mastered elements of my collections that I am less familiar with e.g. I am a lot more familiar with the mastered words in my 100 Most Common Words, than I am with those in my 50,000 Most Common.

Would setting the initial easiness setting to 1.0 have achieved a match between the manually set review date and the Normal review date?

I only have a basic understanding of algorithms, so apologies in advance for any silly questions or remarks !!!

mike · July 28, 2020, 10:37pm

Apologies for the confusion! I should have explained more thoroughly.

In simpler terms - each 100% Mastered sentence has a multiplier starting at 2, and an initial review interval equal to your 100% Mastered review interval setting. Easy causes the multiplier to increase, Hard causes it to decrease, Normal keeps it the same. This multiplier is then multiplied by the current review interval to get the next review interval. In other words it starts from your 100% Mastered review interval setting and makes adjustments over time from there.

So! Let’s say your 100% Mastered review interval setting is 180 days (the default), and you have a sentence at 75% Mastered. You answer it correctly, it goes to 100% Mastered, and it’s set to review in 180 days.

The next time you see this sentence and answer it correctly, this new feature comes into play. It’ll start with a multiplier of 2 and a review interval of 180 days. If you choose Normal, its multiplier will stay at 2 and you’ll see it again in 360 days. If you choose Easy, its multiplier increase to 2.1 and you’ll see it again in 378 days. If you choose Hard, its multiplier decreases to 1.86 and you’ll see it again in 335 days.

Let’s say you choose Easy. 378 days go by. You answer this sentence correctly again And you choose Easy again. Its multiplier goes to 2.2 and you’ll see it again in 832 days.

So for this particular sentence, its review intervals will go 1, 10, 30, 180, 378, 832 (assuming you’re using the default review intervals and select Easy twice).

At the moment the plan is to have the Hard/Normal/Easy buttons as the only way to adjust the easiness factor for a given sentence.

Unfortunately no - the initial interval I is taken from your interval settings. This is simply a result of the data we track at the moment.

Correct! You can simply hit enter, essentially ignoring the other buttons, and the review interval for an already 100% Mastered sentence will increase by 2x each time you answer it correctly.

Correct!

Yes - I think this is demonstrated by the example above, but please let me know if it’s still unclear or I’ve misunderstood.

Thanks for all the feedback! Curious to hear what you think given the above information, and please let us know if anything’s still unclear, there’s anything that was missed, or you have any further questions.