That isn’t actually what’s important. It’s the frequency of the token, which could be as simple as single characters. The frequency of those is certainly not zero.
LLMs absolutely can make up new words, word combinations, or sentences.
That’s not to say chatgpt can actually give you good windows keys, but it isn’t a fundamental limitation of LLMs.
I’ve never ever, in many hours of playing with ChatGPT as a toy, had it make up a word. Hallucinate wildly, yes, but not stogulate a word out of nothing.
I’d love to know more, though. How does it combine new words? Do you have any examples of words ChatGPT has made up? This is fascinating to me, as it means the model is much less chained to the training data than I thought.
A lot of compound words are actually multiple tokens so there’s nothing stopping the LLM from generating the tokens in a new order thereby creating a new word.
That isn’t actually what’s important. It’s the frequency of the token, which could be as simple as single characters. The frequency of those is certainly not zero.
LLMs absolutely can make up new words, word combinations, or sentences.
That’s not to say chatgpt can actually give you good windows keys, but it isn’t a fundamental limitation of LLMs.
Okay, I’ll take your word for it.
I’ve never ever, in many hours of playing with ChatGPT as a toy, had it make up a word. Hallucinate wildly, yes, but not stogulate a word out of nothing.
I’d love to know more, though. How does it combine new words? Do you have any examples of words ChatGPT has made up? This is fascinating to me, as it means the model is much less chained to the training data than I thought.
A lot of compound words are actually multiple tokens so there’s nothing stopping the LLM from generating the tokens in a new order thereby creating a new word.