What makes for "good" multiple choice options?

mike · July 28, 2023, 1:16pm

Thank you to all of you who recently participated in the survey we sent out via email! One of the most requested features/improvements was “more challenging multiple choice options”.

Here’s what we’re considering at the moment:

For each sentence, come up with a list of

X of the most “similar” words
Y words of the same part of speech
and Z random words

then select from that list at random when playing that sentence.

X of the most “similar” words

By “similar” words we mean if the answer is “dinero”, you might see “diera”, “ligero”, “dieron”, using Levenshtein distance to find words that require the fewest changes to get from one word to another. Here are a few more examples:

estamos → estemos, estados, estas, estarás
nadie → nació, nace, radio, nariz
padre → pare, madre, padres, pase
después → espías, espuma, deseas, desnuda

If you think it might be helpful to see any other examples please let me know.

Y words of the same part of speech

estamos → tenemos, podemos, hemos, somos (plural first person present finite)
nadie → nada (exact match for singular negative pronoun), él, lo, que (other pronouns)
padre → bueno, solo, dios, dinero (masculine singular)
después → no, aquí, ayer, más (adverbs)

A concern here is whether having all the options be the same tense for estamos, for example, is too exact and gives away the part of speech - perhaps the list should also include some that are verbs of any tense.

Z random words

Random words used as the multiple choice options is the current approach. It seems like it’s still helpful to have some random options.

So the list for estamos, for example, might end up being

estemos, estados, estas, estarás (“similar”) + tenemos, podemos, hemos, somos (same part of speech) + hazlo, físico, afuera, competición (random)

and so if you were playing {{Estamos}} aquí., you might see multiple choice options

Estamos, Hazlo, Estemos, Podemos
or
Hemos, Somos, Estamos, Físico
or
Afuera, Estamos, Hazlo, Tenemos
or
Estamos, Afuera, Estados, Estas

What do you think? Would any other examples be helpful? Anything you think it might be helpful to include besides “similar” + same part of speech + random?

Any and all feedback and ideas welcome as always! Thanks again!

Jogboh · July 28, 2023, 3:32pm

All the options mentioned would be an improvement on 1 correct + 3 random.

I’d go for something like 1 correct, 1 similar, 1 same part of speech, 1 random, if that’s not too intensive on the servers.

rinkuhero · July 28, 2023, 3:55pm

This is something I’ve wanted for a long time because right now most of the words completely don’t fit in the blank, they aren’t even the same part of speech most of the time.

What you can do is have each wrong alternative be chosen in a different way. For instance, let’s say you have 3 wrong answers, as you do now. One of those wrong answers can be the same word, but a different form of it (e.g. different tense if the correct answer is a verb, different gender/plurality if it’s an adjective or noun). The second wrong answer can be a word that is related in meaning to the correct answer, but is a different word (for instance, cat instead of dog, or orange instead of apple, like the same part of speech and same category of thing, but not the same thing). The third wrong answer can be a word that sounds similar, but has a different meaning (for instance, having a similar length, and similar first letter or two, but different meaning entirely). For instance, “support” instead of “supply” or “cylinder” instead of “cymbol”.

What I describe sounds too much like a formula though, and once someone catches on to that formula it’s easy to figure out the right answer. So what I’d also suggest is to have more options, not just four. This is likely harder to implement than word choice, but it’d be the easiest way to make it more challenging, and each of the wrong answers can be chosen in a random way from different ways of choosing wrong answers.

Even if you don’t add more choices than four, what you could do is have several different ways of choosing wrong answers, and then for each wrong answer, choose one of the ways of determining a wrong answer at random.

Another thing to do is have the wrong answers relevant to a category. For instance, when doing the words by frequency, you can guess which answer is correct often just by it being a shorter word, since most of the most frequent words are short. Like if someone is doing the category called 500 most frequent words, the wrong answers should also all be in that 500 most frequent word list, it shouldn’t have complicated words as wrong answers in that category, since those are clearly not simple and common words.

Another thing to do is, for the similar looking words category, you don’t need to have all the wrong answers actually be real words. For instance, if the answer is “Break”, you don’t have to have the wrong answer just be “Bread” and “Brook” and “Brake” and “Broke” for similar looking words, but you can occasionally also make up nonsense words as wrong answers that look similar, like “Breek” and “Berak”.

alanf_us · July 28, 2023, 10:50pm

I just want to say up front that other than my very first few days on the site, I’ve played text input rather than random choice. While text input doesn’t match the real-world situation that I want to simulate (being able to come up with the words I want in conversation), it’s far, far closer to it than trying to select a word from a predetermined set is. But interested as I am by why people would play random choice in the first place, that’s not the subject of the thread.

@rinkuhero, I think your analysis is good. I like the idea of including nonsense words. One might argue that this would make the nonsense words stick in your head, but I’m not sure this would actually happen.

My additional contribution is this idea: the choice “I don’t know” should always be provided. (It doesn’t even have to be a button that takes up as much space as the other ones.) If this were available, you could be honest with yourself about the fact that you don’t know a word, and you would eliminate the possibility that you could simply guess the right one (and cheat yourself out of the repetitions you should be doing) based on nothing but chance and/or the unsuitability of the other choices.

rinkuhero · July 28, 2023, 11:10pm

I like the I don’t know idea yeah, if you do add more options than 4 that’d be a good addition.

As for why someone would want to do multiple choice, for me it’s just a time issue. There are 100,000 sentences to master on this site. You know how long that would take if you did them all by typing them in? When I do text input, a set of 50 might take me half an hour. When I do multiple choice, a set of 50 takes me about 4 minutes. It seems a lot more feasible to master everything if you are doing multiple choice. And sure, I might learn less, but I treat this site as a game, something to be completed. If it has gamification elements, I’m going to try to figure out the best way to beat the game. It’s not like I am learning nothing while doing multiple choice either, I favorite interesting sentences, I learn more about how words are used, and I get exposure to a lot of different sentences. To me, reading 50 new sentences and understanding most of them in 4 minutes is worth more to me than going less fast. If someone did typing in exclusively, they will literally never master every sentence in their target language in their lifetime, even if they did it hours a day and even if they set the duration of 100% to “never”, there’s just that many sentences (at least in Spanish, there’s over 100,000 sentences. I want to read them and understand them in a reasonable amount of time.)

rinkuhero · July 29, 2023, 4:47pm

Another thought… sometimes two of the multiple choice answers can reasonably work. It’s probably impossible to eliminate that, and going to the translation lets you know which of the two is correct, but it leads to some funny alternatives sometimes, like this (the answer is 2, but 4 works fine too, and the only way to know is to check the translation).

I’m not sure exactly how you can prevent this from happening though, besides to select non-generic sentences that wouldn’t work very easily with random words.

alanf_us · July 29, 2023, 7:06pm

Thanks for the explanation. I think we’re working with different goals. My primary goal is to be able to speak the languages I practice here at a certain level and to have Clozemaster help me as much as possible to get there, not to work my way through all the sentences in a particular language pair. The number of sentences and the criteria for selecting them for Clozemaster are a consequence of factors I can’t control (how many sentences exist on Tatoeba, how many people were available to take care of importing them into Clozemaster, how they chose the set). Nobody consulted me during this process, so it would be a big surprise if there were exactly as many sentences as I needed, no more and no fewer.

If I extrapolate correctly from what you’ve written, and if we assume that you play text input for ten hours a week, and if we’ve both done the math correctly, it would take you 115+ years to play (let alone master) all 100,000 sentences. I agree that that’s not going to happen. But even if you played multiple choice, it would take you 15+ years, which also seems unrealistic to me. So since selecting multiple choice over text input is probably not going to make the difference between your finishing all the sentences or not finishing them, it seems to me that you might as well use other criteria (and probably already do) to determine which mode you play in. Maybe pushing a button feels like more fun than typing, and fun is primarily what you’re after. In that case, multiple choice makes sense. Personally, I’m fine with typing in general, and I’ve set up keyboard layouts that let me do it easily in the different languages I play, on the different devices I use, so taking everything all together, text input works better for me.

For reference, here’s my math, based on your figures:

Text input:
50 sentences/30 minutes
1.66 sentences/minute
(100000 sentences)/(1.66 sentences/minute) = 60,241 hours

Multiple choice:
50 sentences/4 minutes
12.5 sentences/minute
(100000 sentences)/(12.5 sentences/minute) = 8000 hours

Text input:
(60,241 hours/(10 hours/week))/(52 weeks/year) = 115+ years

Multiple choice:
(8000 hours/(10 hours/week))/(52 weeks/year) = 15+ years

rinkuhero · July 29, 2023, 7:29pm

I think you forgot to convert minutes to hours? If you do that conversion, it should be about 1000 hours vs about 130 hours. But you also need to see each sentence 4 (or on average, 5, since you’ll get some wrong) times before it’s mastered. So it’d be 5000 hours vs 650 hours. To me, 650 hours is a reasonable amount of time. It’s certainly long, but it’s about as much time as I spent playing certain videogames, like Starcraft, or Civilization 5. 5000 hours is more like the amount of time I spent in high school.

I’m also about slightly less than 1/5th of the way through the sentences right now, and I’ve been playing for about 150 hours, so it seems to line up with the actual time I’ve spent on the site (though because of the timer bug, where it used to show negative time, I’m not sure about the accuracy of the time I’ve spent on the site. But 150 hours feels like about what I’ve spent on it so far.)

But I didn’t mean that completing the site is my only goal, just that it’s one of the goals. If I wasn’t learning as I was doing it, or having fun, I wouldn’t be doing it. It’s not the only site I use to learn either (I also use Busuu, Duolingo, Anki, and a few others). So because I’m using the site as one tool among many, I am not worried about thoroughly learning as much as I can from any one site. I trust other sites will fill in whatever I miss here from doing them multiple choice vs text entry.

Regardless, I do think that even with multiple choice, 100,000 sentences may be too many. I wouldn’t be against reducing it, and the dev has talked about it before (saying they are thinking of curating things more closely). Reducing the total number of sentences may make it feasible to go through several different languages.

alanf_us · July 29, 2023, 10:35pm

Indeed, I forgot to convert from minutes to hours. Thanks for catching that and pointing it out so politely. Thanks also for explaining your overall reasoning.