Repetitive token generation error and fixing issue

[sid2983]

I was trying ChandraOCR for EP patent pdf documents. But I am mostly getting this Repetitive token generation error and fixing loop , and its still generating repetitive tokens after a default max retrial limit of 6.

[sid2983]

@VikParuchuri , @zanussbaum any suggestions/tips to fix this issue temporarily.

[Backdrop9019]

I’ve run into a very similar issue as well. Increasing the repetition penalty can sometimes help reduce the repeated-token behavior. However, in my experience, this issue tends to occur on difficult documents, and even when the repetition is suppressed, it doesn’t necessarily lead to better overall results

[sid2983]

Yes, certainly.
I have tried adding frequency penalty values ,but increasing the value above 0.3 also leads to inaccurate token generation sometimes,and lowering it down doesn't work sometimes.
For failsafe, I have added penalty to trigger when repetitive token occurs. But it would really be good, if root model related issues can be found and fixed, possibly by retraining and releasing new version of model.
@VikParuchuri @zanussbaum

[tarun-menta]

Can you share the doc? We can take a look and make sure this is fixed, or address in a new model release

[WarriorOfLiberation]

Can you share the doc? We can take a look and make sure this is fixed, or address in a new model release

Hi, this happen whenever I try to run Chandra on any Indic language(Hindi/odia/Telugu), repetetion penalty fixes this somewhat but then the output is hallucinated.
While the same documents do work on the playground

[tarun-menta]

We've been working on updates behind the scenes for the model, and Indic languages are a one of those. The model on the API is a little more up-to-date than OSS.

[WarriorOfLiberation]

Got it, Any plans to oss the model which does understand indic languages, since that seems to be the MOAT of surya/chandra for me personally.

…

On Sat, 21 Feb 2026, 00:55 Tarun Menta, ***@***.***> wrote: *tarun-menta* left a comment (datalab-to/chandra#62) <#62 (comment)> We've been working on updates behind the scenes for the model, and Indic languages are a one of those. The model on the API is a little more up-to-date than OSS. — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZWJ5GYPBPKW5CO37GVMWBL4M5NQ7AVCNFSM6AAAAACR7YU3UWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMZWGY3TMNBRGU> . You are receiving this because you commented.Message ID: ***@***.***>

[tarun-menta]

Coming soon :)

[sp-000-sim]

Can you share the doc? We can take a look and make sure this is fixed, or address in a new model release

Sure, this is the file referenced in the above chat screenshot . Please try with it.

EP0398795B1.pdf

sorry for the delay!!