The 100% learned issue, and suggested solutions

[mike]

Apologies for the confusion! I should have explained more thoroughly.

In simpler terms - each 100% Mastered sentence has a multiplier starting at 2, and an initial review interval equal to your 100% Mastered review interval setting. Easy causes the multiplier to increase, Hard causes it to decrease, Normal keeps it the same. This multiplier is then multiplied by the current review interval to get the next review interval. In other words it starts from your 100% Mastered review interval setting and makes adjustments over time from there.

So! Let’s say your 100% Mastered review interval setting is 180 days (the default), and you have a sentence at 75% Mastered. You answer it correctly, it goes to 100% Mastered, and it’s set to review in 180 days.

The next time you see this sentence and answer it correctly, this new feature comes into play. It’ll start with a multiplier of 2 and a review interval of 180 days. If you choose Normal, its multiplier will stay at 2 and you’ll see it again in 360 days. If you choose Easy, its multiplier increase to 2.1 and you’ll see it again in 378 days. If you choose Hard, its multiplier decreases to 1.86 and you’ll see it again in 335 days.

Let’s say you choose Easy. 378 days go by. You answer this sentence correctly again And you choose Easy again. Its multiplier goes to 2.2 and you’ll see it again in 832 days.

So for this particular sentence, its review intervals will go 1, 10, 30, 180, 378, 832 (assuming you’re using the default review intervals and select Easy twice).

@LessPaul

I am assuming the ability to adjust the easiness factor is in the works, since I am not seeing a new setting.

At the moment the plan is to have the Hard/Normal/Easy buttons as the only way to adjust the easiness factor for a given sentence.

@LessPaul

And I assume the first time the SM2 algorithm I(n) = I(n-1) *EF’ picks up the interval “I” it is from that word’s current interval, rather than the value set in the interval settings screen.

Unfortunately no - the initial interval I is taken from your interval settings. This is simply a result of the data we track at the moment.

@jamesp999

So to make this simpler, just hitting enter will result in the interval increasing?

Correct! You can simply hit enter, essentially ignoring the other buttons, and the review interval for an already 100% Mastered sentence will increase by 2x each time you answer it correctly.

@AlexG71

(I assume this is due to the initial easiness factor of 2.0 and this changes over time?).

Correct!

@AlexG71

Will the algorithm change the differentials between Easy, Normal, Hard over time for any given mastered sentence?

Yes - I think this is demonstrated by the example above, but please let me know if it’s still unclear or I’ve misunderstood.

Thanks for all the feedback! Curious to hear what you think given the above information, and please let us know if anything’s still unclear, there’s anything that was missed, or you have any further questions.

[Ceid]

@mike

For those interested in SRS, we’re using an initial easiness factor of 2.0.

I’d appreciate the option to disable this feature altogether and go back to the single preset for frequencies of review. I already have enough difficulties using the website and app with my disabilities (something that recent changes have not helped) and am content with having a preset that covers all sentences. I see no need to spend my learning time making micro-decisions and navel-gazing like this and I really dislike Clozemaster thinking they need to micro-manage my studies by managing reviews like this.

This is also disproportionately punitive for those of us who do smaller courses with fewer sentences–it’s not our fault Clozemaster hasn’t improved or expanded those courses and for us Pro users, while we have the Cloze-collection, that’s an incredibly time-consuming option unless you’re just lazy and settle for whatever you can find online to upload. I’ve been working on sentences to add Cloze-collections to some of these courses, sentences that are challenging or cover things I’m weak on, and I can personally attest to what a time-consuming pain it is. I really don’t think it’s fair to end up not being able to review those sentences as frequently as I wish after all that work. Seriously, no one has mastered the Irish copula or Celtic initial mutations in just 4 rounds. Please just let me study my languages without a bunch of unnecessary hassle.

Additionally, if this is in anyway an effort to police users who have been accused of cheating (and I suspect it is, at least in part, for good reasons), then again, I will express my strong disapproval of changes that make using Clozemaster harder, more complicated or less rewarding for everyone so to punish a couple of maybe-cheaters based on inflammatory accusations from just a few, not necessarily neutral people who’ve complained the loudest. Online multiplayer games do this all the time and it’s always bad in the long run for regular users, and it’s depressing to think Clozemaster is going down that route. If the Clozemaster team really thinks leaderboard cheating is a problem, why not let give us the option to opt out of the leaderboard, let us keep streaks and levels for the motivation and let us review sentences at whatever frequency we feel we need? I would personally love that.

[kadrian]

@mike

You can simply hit enter, essentially ignoring the other buttons, and the review interval for an already 100% Mastered sentence will increase by 2x each time you answer it correctly.

The problem I see with this is that answering a sentence correctly doesn’t necessarily mean you really know the answer with the fluidity required to speak a language. In French I use the colour and text box hints to get around not being able to tell from the sentence which gender or tu / vous verb form is wanted, and that means I can see whether I’m right or not before hitting enter. If the text goes red, I change things until it’s green! Using multi-choice, you can often identify the answer even if you couldn’t bring it to mind without help. Automatically doubling the review time for correct answers is going to push the reviews for those sentences out way too far (especially, as @Ceid mentioned, for languages with small collections).

@mike

If you choose Easy, its multiplier increase to 2.1 and you’ll see it again in 378 days. If you choose Hard, its multiplier decreases to 1.86 and you’ll see it again in 335 days.

I agree with @AlexG71 that I expected to see a much bigger difference between Easy and Hard. These differences are so small that it’s hardly worth clicking the Easy and Hard buttons rather then just hitting Enter. I’d rather see Easy sentences appear MUCH less often than normal ones (half as frequently, or less), and Hard ones MUCH more often (twice as often or more).

[jamesp999]

First, I think post-100% being able to increase the intervals is very important, and I think the easy medium hard buttons are a great addition. I do agree with the above posters though that there isn’t much of a difference between E,M, and H. I would expect that Easy would double, maybe normal would increase by 1.5, and maybe hard would reduce the interval by 25%-50%?

The key is just having the option of increasing the intervals. I’ve been using clozemaster for several years and TBH, when I had played over 50k sentences in portuguese the idea of having to play every year on average ~150 reviews a day just to keep my head above water was overwhelming to the point that I just gave up on reviews. Today I had some motivation to start clearing out some of my 13k reviews backed up, knowing that at least over time the rate of reviews would decrease.

[morbrorper]

After trying this out a bit I’m leaning towards 50% - 100% and 200%, for Hard, Normal and Easy.

Either that, or I will consider changing my 100% interval down to 90. I am sadly expecting my retention capability to decrease as I’m aging

[mike]

Thanks for all the feedback!

@kadrian

I agree with @AlexG71 that I expected to see a much bigger difference between Easy and Hard.

@jamesp999

I do agree with the above posters though that there isn’t much of a difference between E,M, and H. I would expect that Easy would double, maybe normal would increase by 1.5, and maybe hard would reduce the interval by 25%-50%?

Good points! Sounds like we’ll need to change the initial values a bit until we come up with a better fit. The thinking is that Hard/Normal/Easy would always result in the next review interval increasing, just by different amounts. If you wanted to reduce the interval, you’d reset to 0% Mastered or change the percent mastered when editing the sentence after answering (pencil icon).

Here’s the change we’re considering: rather than 2x for the initial multiplier, we go with 1.5x, and we spread out Easy/Hard a bit such that the initial multiplier for Hard is 1.2x and Easy to 1.7x.

So for example if your default 100% Mastered is 180 days, the next review intervals would be 216 days for Hard, 270 for Normal, and 303 for Easy. Note that the multiplier for that sentence would change based on your response for the next time you answer that sentence, with 1.2x always being the minimum.

Still not a huge difference initially, but it would continue to grow each time you answer the sentence correctly and depending on your responses.

There’s definitely some complexity here, but no need to get bogged down in the details too much. The gist is that the 100% Mastered review interval will continue to grow for a given sentence each time you answer it correctly, and at any point you can still reset a sentence back down to 0% Mastered (or back to 25/50/75% by editing the sentence after answering).

@morbrorper

After trying this out a bit I’m leaning towards 50% - 100% and 200%, for Hard, Normal and Easy.

This would be a different approach - rather than a dynamic / always increasing next review interval, we could instead go with static 50%/100%/200% of your 100% Mastered review interval setting. Curious to hear if this would be preferable - it’d certainly be easier to explain but perhaps not as effective.

This change is aiming to reduce reviews building up / review overload, as well as improve the spaced-repetition system Clozemaster uses thereby improving retention/efficiency. A static max review interval, in other words seeing the same sentences at the same interval once you reach a certain point, doesn’t seem quite right or as efficient as it could be.

So! What do you think? Note that these options only apply for already 100% Mastered sentences.

• Always increasing next review interval
• Static 50/100/200% of your 100% Mastered review interval setting
• Neither - revert / go back to the way it was! Always your 100% Mastered review interval setting

Our goal with this change is to make Clozemaster more effective and more useful, so we’re also happy to revert if it turns out not to fit that criteria.

Thanks again!

[LessPaul]

I answered 50/100/200 by accident. Apologies for messing up the poll.
My answer would be Always increase, but with a slightly smaller EF. I can alway edit to 50 or 100 for those I feel need further adjustment.

— And I assume the first time the SM2 algorithm I(n) = I(n-1) *EF’ picks up the interval “I” it is from that word’s current interval, rather than the value set in the interval settings screen.

– Unfortunately no - the initial interval I is taken from your interval settings. This is simply a result of the data we track at the moment.

Actually, I find this preferable.

[Koeppchen]

I voted for always increasing, for example always 200% of the previous interval, this is the implementation at the moment and from my point of view this makes sense.
Choosing between Easy normal and hard is bad in the ui because the ui looks different for <100% and =100%.

[LuciusVorenusX]

@mike

The gist is that the 100% Mastered review interval will continue to grow for a given sentence each time you answer it correctly, and at any point you can still reset a sentence back down to 0% Mastered (or back to 25/50/75% by editing the sentence after answering).

No, no a thousand times NO! Have this for the people who want it; it’s fine to have it as an option, I have no objection to that whatsoever. However what works for me is repetition and plenty of it. The idea of having words reappear a year or two from now is absolutely NOT something that I want to see happen for me. True, the initial review cycles are in my control, and they are WAY, WAY shorter than the default cycles, but if I say 30 days, I don’t want the system deciding “Oh, you got that one right so we’ll make it 45 days (even though you wanted to see it again within 30), now we’ll make it 60 days, etc, etc”. Nor do I want to have to reset the question to 0% mastered and have it mixed in with the ones that I genuinely haven’t seen.

I understand that some people want to see longer repetition times and as I say, as an option I’m fine with that. But I have no interest at all in having my Closemaster exercises stretched out for years.

[blogscot]

Currently, learners can configure their review intervals for 0% 25% 50% 75% and 100% mastered. Could a similar method of configuring Hard, Normal, and Easy settings be developed to allow learners to choose their preferred ratios which would apply to each setting, i.e. I’d choose 0.25, 1 and 2. Others might choose 1.5, 2 2.5. To prevent gaming the system I’d suggest any ratio less than 1 gets its score set back to zero.

[kadrian]

@blogscot

Could a similar method of configuring Hard, Normal, and Easy settings be developed to allow learners to choose their preferred ratios which would apply to each setting, i.e. I’d choose 0.25, 1 and 2. Others might choose 1.5, 2 2.5.

If it wouldn’t be too hard to implement and explain, this might suit lots of people. I’d love to push the review interval for easy sentences WAY out, to 10 or 20 times the interval for “normal” sentences, so playing reviews would be a more effective use of time once you’re very familiar with the easier material.

Another solution for this would be to ignore those sentences permanently, but I often don’t use that feature because I find the “sentences played” count motivating. Would it be possible to keep ignored sentences in our statistics?

[LuciusVorenusX]

@kadrian

If it wouldn’t be too hard to implement and explain, this might suit lots of people. I’d love to push the review interval for easy sentences WAY out, to 10 or 20 times the interval for “normal” sentences, so playing reviews would be a more effective use of time once you’re very familiar with the easier material.

I think that’s a really good idea. There aren’t a LOT of sentences that I would use it on because in Italian Fluency Fast Track has over 11,000 sentences, so even with my relatively short review cycle and doing at present a few hundred sentences per day, it takes quite some time for the most basic sentences to reappear. Even those I don’t REALLY regard as a waste of time because as long as I’m hearing the language and writing in the language (with NO ENGLISH, thankfully!) then I’m happy because everything is being reinforced. But pushing the most basic of basic ones out would probably mean clearing the way for some of the more complex ones.

[Ilraon]

Firstly, thank you Mike (and whoever’s working with you) for trying to deliver a nice experience to your users. I believe we all highly appreciate your taking the time to discuss improvements with the community.

Now, onto my small thoughts.
I am not convinced by the user experience regarding the “hard” button. If seems to me incompatible with the fact the sentence is 100% mastered. Intuitively, if I find a sentence to be hard, I don’t want to review it in over 180 days; I can be sure I’ll have forgotten by then. In my opinion, two buttons would be enough.

Additionaly - and that’s a separate thought, it seems to me that users are essentialy eager to have two things: 1) keep the review time more or less normal ; or 2) push the review way further in time when it feels like reviewing the sentence is a waste of time.
So, maybe pushing the “normal” button shouldn’t change the interval, and pushing the “easy” button should multiply it by whatever factor, possibly 2 by default?

You mentioned setting the multipliying factors for those two could appear in settings, and that’s one way to settle the decision. This would allow the configuration I just described, by choosing “1” for normal. Pressing enter would by default press “normal”, and people like @Ceid (whose post I think made several good points), could keep studying the way they want to.

As a sidenote - fairly unrelated, if you’ve completely played a collection, and you play it anyway, you’ll review words even though they’re not ‘ready’.
I’ve never used that myself, but I think I might when I feel like forcingly reviewing a collection at a random time, even though its review pile hasn’t stacked up.

I like the idea of the interval increasing, since it can really get out of proportions with the daily number of reviews. I personally find almost humoristical that the review times might be larger than a year - it seems to me unlikely that I’d even be using clozemaster on the same collections at that point, but based on the fact some people are really long-time, loyal users, it seems to be relevant.

[mike]

@LuciusVorenusX

Closemaster

---

Thanks for all the additional feedback! Given the poll results and all the feedback, this feature is now opt-in:

You should be able to go to your Dashboard > Review Settings top right, and then select how you’d like to handle correctly answered 100% Mastered sentences - dynamically always increasing next review, static 0.5/1/2x your 100% Mastered review interval, or the default / way it was originally which always uses your 100% Mastered review interval setting. While we can’t please everyone all the time, making this feature opt-in seemed like as cloze as we could get to a win-win-win

@blogscot

Could a similar method of configuring Hard, Normal, and Easy settings be developed to allow learners to choose their preferred ratios which would apply to each setting, i.e. I’d choose 0.25, 1 and 2.

Good idea! We may be able to add this in the future. Will also have to give some more consideration to points like you mentioned.

Any other thoughts or feedback please be sure to let us know. Thanks again!

[jamesp999]

thanks so much for finding a way to provide a balance of user experiences to please everyone.

[LuciusVorenusX]

@mike

@LuciusVorenusX

Closemaster

Oh Mike, Mike, Mike… the S is diagonally adjacent to the Z key and after doing several hundred exercises that day that particular tyop could have just as easily have ended up being Cloxemaster or Cloaemaster given that the Z is a key that my fingers seldom have reason to visit as its Scrabble score can attest. Actually now that I think of it, it could easily have ended up as “Vkixwnsdywt” by that time of night. You’ll note that I use the word “Cloze” correctly in every other post! (As far as I can recall.)

Thanks for making this opt in. That should make everyone happy.

[Mupfel]

Mike, thanks a lot!
I really enjoy increasing intervalls, but it’s also great to have the options.

One question: what happens in the app? On the website I set the option to increasing, will the intervalls on the app be also increasing now?

[AlexG71]

Thanks Mike!

Hopefully this will please as many people as possible.

One small point to mention … since the changes made for the 3 options, the shortcut keys don’t seem to work for the first two options i.e. 1 for Hard, 2 For Normal, 3 for Easy aren’t working. Also the normal ‘default’ shortcut of ‘enter’ is also not working.

At moment I either have to tab to the options, or mouse click.

Can you take a look when you’ve got a second? Thanks!

[mike]

@Tarob

One question: what happens in the app? On the website I set the option to increasing, will the intervalls on the app be also increasing now?

For now already 100% Mastered sentences answered correctly in the app will still always use your 100% Mastered review interval setting. We’ll aim to get the review setting and Easy/Normal/Hard controls added in the next update to the app. Giving it a bit more time to see if we get any further feedback - updating the website is faster/easier than the app

@AlexG71

shortcut keys don’t seem to work for the first two options i.e. 1 for Hard, 2 For Normal, 3 for Easy aren’t working. Also the normal ‘default’ shortcut of ‘enter’ is also not working.

This should now be fixed - thanks for letting us know! And please let us know of course if you’re still seeing the issue or notice any others.

[LessPaul]

Am I correct in my understanding?

"Use Hard/Normal/Easy buttons after answering to set next review to 50/100/200% of 100% Mastered review interval" setting always calculates from the 100% master days setting.
Example: Assume my 100% is set to 100 days. No matter how many times I answer the question, when I press 3, it will be scheduled 200 days from today.

"Use an always increasing next review interval with Hard/Normal/Easy buttons after answering to modify the rate of increase" setting will continue to lengthen the existing interval.
So the new interval will be longer than the current interval every time I select 1, 2, or 3.