Fabulous! Great idea!
1 Like
some of the answers are definitely more useful than others. I have no idea if this is possible, but it would also be awesome if we could ask a follow-up question to the chatbot.
2 Likes
What has annoyed me about ChatGPT (other than the massively overbaked hype) is that it is lauded as some huge breakthrough in AI, but in reality it’s little more than smoke, mirrors and search engines. It has no comprehension of the content it generates, and does little enough fact checking.
As expected when I looked through simple sentences in FFT in Italian, its explanations were fine. As soon as it encountered something that required actual understanding, though (and that didn’t take many questions), it fell in a heap.
For example:
“Ti ho chiamata perché mi serve un favore” is an Italian phrase that can be translated as “I called you because I need a favor” in English.
True enough.
Let’s break it down word by word for an easier understanding:
Clearly you and I have different definitions of “understanding”, ChatGPT.
- “Ti” means “you” (used as an object pronoun referring to the person being called, specifically a female).
Utter nonsense. “Ti” is just an informal form of “you”. It applies equally to males and females. In this case it DOES refer to a female but it certainly ISN’T because of the word “Ti”.
- “ho” means “I have”; it comes from the verb “avere” (to have) and must be used in compound tenses.
Must be? MUST be? In every single one of them? Gee, good thing I don’t have to worry about all those pesky verbs that require Essere for the passato prossimo form, then.
- “chiamata” means “called”; it comes from the verb “chiamare” (to call) and should agree in gender and number with the subject it refers to (in this case, a female person).
Again, utter nonsense. The fact that it IS in the feminine singular form tells us that the “ti” above is referring to a female in the singular. But, I should point out… it is agreeing with the OBJECT, not the SUBJECT.
Specifically, It agrees in gender and number when a direct pronoun like “ti” is used before the verb in a sentence like this. When you DON’T have such a pronoun, then:
I called Mario.
Ho chiamato Mario.
Luisa called Mario.
Luisa ha chiamato Mario.
Oh, gee, look… the form of chiamare doesn’t actually change.
- “perché” means “because”.
Finally something we agree on, because even ChatGPT and its over-hyped search engine can’t screw up A1 level language.
- “mi” means “me” or “to me” (used as an indirect object pronoun).
- “serve” means “needs” or “is needed”; it comes from the verb “servire” (to need or to be necessary).
- “un” is an indefinite article that means “a” or “an.”
- “favore” means “favor.”
Ditto.
So the entire phrase “Ti ho chiamata perché mi serve un favore” means “I called you because I need a favor” when referring to a female person.
This is disturbingly ambiguous. The person being called would be a female person in this case, but the person doing the calling does not have to be.
What narks me about this is that it will confuse the living heck out of a newbie because it’s giving them information that is just plain wrong, given the subtleties and nuances of languages like Italian.
And Italian is a (relatively) logical language. I don’t even want to THINK about what this thing is doing in French FFT.
As long as students work from the assumption that ChatGPT will probably have no idea what it is talking about for sentences which extend beyond a single clause, the explanation won’t be completely useless. But it should NEVER be trusted or assumed to be correct. Always assume the opposite and
check, and it can still be useful.
9 Likes
I suspect that Clozemaster is using the still very imperfect GPT 3.5, as GPT-4 is much more expensive.
Here is GPT-4’s output:
Much sounder, but point 3 is still not an ideal explanation and it remains a bit of a missed opportunity to present a basic grammar rule.
After a couple of weeks of use, I would rate GPT-4 as a “massively useful” foreign language assistant. However, users need to understand that, counterintuitively, it has no sense yet of fact, truth or things like “verification”.
Caution and critical thinking must still by applied to its ramblings.
4 Likes
I’m a simple soul and much prefer my books, trusty translator and more importantly, coming here to ask and discuss with other humans. Old-fashioned I’m afraid but doing ok;-)
10 Likes
Thanks for all the input! This is super helpful.
Not sure yet - it’s currently only available for the first 1,000 or so sentences for the French, Italian, and German Fast Tracks, and a bit more for the Spanish Fast Track. We used a different prompt for the Spanish Fast Track, and the output for the others seems more useful so far, so we’ll likely regenerate the Spanish explanations soon.
We’re looking for feedback like in this thread in deciding whether to keep expanding the explanations. It’s also a bit tricky where to draw in the line in terms of for which languages to make it available.
Interesting idea! It might still be too expensive for now, but hopefully prices will keep coming down.
Thanks for the feedback! That’s all helpful to know. We’ve done our deepest dive reviewing and fact checking Spanish so far, and those explanations have tended to have no issues (aside from perhaps sounding a bit robotic in some cases) in >90% of cases. To your point, however, that’s why we wanted to make it clear these explanations are generated by ChatGPT. Despite the downsides, we still thought it seemed more useful to offer the explanations in most cases than not have them at all, hence this experiment 
Doch, we’re using GPT-4, only the best for Clozemaster users , but perhaps we should be trying a different prompt 
Agreed! If you think there’s perhaps some way we should better present the explanations to make that clear, or perhaps some way to help verify, we’re open to ideas.
At the moment explanations seem to be more helpful than not, and we’ll keep the implementation requiring some action to view them for anyone who prefers not to use them. We’ll likely continue expanding explanation availability both in terms of collections and languages, and work on getting them added in the mobile app.
Any other feedback or thoughts on the above please be sure to let us know. Thanks again!
4 Likes
I like it! But yeah, maybe a prominent disclaimer that the ChatGPT explanations are AI-generated and may not be entirely correct wouldn’t hurt.
3 Likes
It is surely old-fashioned, but again, you are doing ok ;^D
1 Like
Hello, Mr. @mike ! Is it possible to add this Chat robot thing to the Arabic course, if it will? Thanx!
BTW, the tatoeba translations in most courses are the worst (!!!). It would be great if the AI could do it it/herself. I’m pretty sure any other translation would be better - I often use Google Translator and Reverso Context and they never fail to me. Again Tjanx.
1 Like
Heeeelp! ChatGPT stopped working on Chinese!!!
1 Like
I love this feature but it is unlimited on mobile and limited to 20 per month on the website. Why? I’m a paying customer and should get full access on both mobile and web!
1 Like
Thanks for the comment! We generated explanations for a large number of sentences for some of the more popular languages and collections, on the order of thousands of sentences. Those are available for both the web and mobile apps.
Then we realized generating explanations for millions of sentences, many of which might not even need an explanation (very simple/easy sentences for example), would probably get a bit expensive, so right now we’re experimenting with a way to generate explanations “on the fly”/as needed for sentences that don’t yet have them in the web app. That’s what has a limit - both to help control cost, see usage, and see if anyone would be willing to pay to generate more to help defray the costs. And that’s what’s only available in the web app at the moment.
@Roberto.Melgaco thanks for letting us know about Arabic. ChatGPT should be working for Chinese - please let us know if you’re still seeing any issues. Thanks!
4 Likes
Mike - any other plans to apply AI to your collections? I figured some fairly simple ones would be:
- spelling and syntax corrections (such as the incorrect case, person or plural/singular)
- Removing person names and city names from being used as Clozes (maybe with the exception when the city has a different name between the languages)
- making sure these person/city names are not selected during import
- filling in more alternate correct answers
It could really be a major upgrade I think (I would not be surprised at all if you’re already working on these…)
Anyhow keep up the good work!
3 Likes
We’re aiming for all 4 you mentioned, though likely with human involvement (for example trying to use AI to identify problematic sentences then having a human make/approve the actual changes).
An option to use AI to generate sentences/content for custom collections is also an idea we’re exploring, as well as attaching images to sentences that can be toggled like translations.
Thanks!
2 Likes
Looking forward to these new enhancements!
Ah hell, I missed that this was a feature, otherwise I wouldn’t have gotten a Pro subscription. I jumped ship from DuoLingo precisely because of their use of GPT and presenting it as a replacement for human-verified information. As a computational scientist, I cannot support the use of GPTs as a source of fact, because its design inherently has no mechanism to optimize for truthful results.
Crucially, it also has no mechanism to truthfully cite what part of its training set contributed the most to its output, which makes it impossible to use GPTs as a starting point for acquiring verifiable information.
This is not only a problem of veracity and continuing language-learning, but it’s also a broader ethics problem: We know that OpenAI has used an undisclosed amount of copyrighted text in its training sets, which presents an intractable ethical problem with using ChatGPT in its current state. Use of ChatGPT enriches OpenAI, which has made its business on the corporate-level appropriation of other’s works without compensation or attribution. Given how language textbooks are often written by people who don’t see huge returns on their work, lost sales can translate to material harm to the people who compiled this information in the first place.
On both practical and ethical grounds, I can’t justify having a Clozemaster Pro subscription, if some of my money might go toward this. I definitely understand what @mike says about the prohibitively difficult task of providing human-written explanations for millions of sentences, but I’d much rather just ask the forums when I have something I’m confused about.
2 Likes
Thanks for the post! I saw your email making the same points as well. Happy to provide a refund if you prefer, we don’t want Clozemaster to be a source frustration.
I appreciate your points. We definitely try to make it clear that the explanations are from ChatGPT. Looking at our messaging again now, we can probably make it clearer that you need to do your own verification/fact checking whenever using anything from ChatGPT. Aside from that, they do seem to be relatively useful despite the shortcomings, and a number of people have written in to say the same. It’s also an optional add-on feature and only accessible if you choose to tap the Explain button, so not being forced upon anyone if they don’t want to use them.
As far as the ethics and copyrighted material, that’s prickly. If it does spit out any copyrighted material from textbooks or anyplace else that comes to our attention, we’ll of course remove it, same as any of the other content on Clozemaster.
I realize you likely won’t be swayed here, but I did want to provide a reply. We’re happy to provide a refund like I mentioned, just let us know via support@clozemaster.com.
1 Like
ChatGPT is an excellent addition to the program. It explains the grammar well, identifying all the parts of speech. When I’m playing a sentence and want to evaluate the structure, I hit the Explain button. When I run across a sentence I want to evaluate further, the majority is already understood and I focus on the part(s) needing review (saves time).
Also, ChatGPT is pretty good with idiomatic expressions (which can be a challenge!)
Great addition!
Thanks for your reply and your understanding, and I’m sorry to @ you with something like this. I really have enjoyed the rest of Clozemaster, particularly in light of the increasing reduction in human-verified information on Duolingo. But as you guessed, I have some hard limits around use of ChatGPT.
OpenAI unfortunately is not following professional ethics standards for the field. It not only fails to cite and compensate the sources of its data, they’re also using datasets that were specifically compiled for non-commercial use, which contain unredacted personal medical data and information about children. The ethical standards in my field absolutely forbid that kind of behavior, standards set after historical abuses of patients that have led to tragedy in the past.
There are ethical ways to get these things implemented. OpenAI has repeatedly refused to take the necessary steps to do so, which is honestly tragic to watch. My field has been using ML for years before GPTs became popular, and I was really excited to see the technology grow, until it became clear that many of the commercial ML firms had no intention of being safe or respectful with their training sets.
I appreciate that people see ChatGPT as just another new computational tool that they wish to explore. But for these reasons, I can’t condone its use, unless and until OpenAI restructures their training set harvesting policies.
1 Like
Can we get the explain feature on Galician from Spanish?
Thanks!
1 Like