Weblate
423 Locked error when creating a new component with languages and doing auto-translate
#14604
-
Describe the issueThe following describes a sequential process of creating a new component and uploading the main POT file with new languages and auto translating them. Additionally, you can see the output within our custom logs. 1.) We create a new component with ✅ This is successful. 2.) We install an addon with ✅ This is successful. 3.) We create languages in the freshly created component with ✅ This is successful. 4.) We upload the main POT file with ✅ This is successful. 5.) We auto translate the created languages with machine translation with ❌ The response is a But, I found this ticket: #4666, especially the comment #4666 (comment). Relevant code: Lines 104 to 116 in f595bb9 Is there a chance to find out what caused the lock? Why is it 120 seconds? I already tried
Steps to reproduce the behaviorSee above. Expected behaviorNo lock error ScreenshotsNo response Exception tracebackNo response How do you run Weblate?Docker container Weblate versionsNo response Weblate deploy checksNo response Additional contextNo response |
Beta Was this translation helpful? Give feedback.
Replies: 30 comments
-
|
Weblate locks internally the component for some operations, for example to avoid concurrent manipulation with the files. Retrying later should work. You should be able to see what is going on in the server logs. |
Beta Was this translation helpful? Give feedback.
-
|
Command to get the log: As you can see, there are other components (e.g. This is our server log from the first But from previous calls (I do not know from which request, but I think when uploading the new source file) I can see some locks with Is there a chance to get the timeout of the lock with the Another question: In the access logs, I see that some crawlers access the edit page of strings. Does this also lead to a lock? For reference, I have found the following locks which have more than 5 seconds timeout: Lines 98 to 105 in 6763676 weblate/weblate/utils/backup.py Lines 39 to 41 in 6763676 Lines 114 to 116 in f595bb9 |
Beta Was this translation helpful? Give feedback.
-
|
The timeout applies when acquiring the lock and waiting while other process holds the lock. The locking happens on component or repository level, so when components share a single repository they will wait for a single lock when Weblate is working with the repository. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, we have this shared repository: https://translate.owlinfra.de/projects/shared-glossaries/real-cookie-banner/:
So, in this case, when e.g. WordPress Real Cookie Banner (Backend, PHP) and WordPress Real Cookie Banner (Frontend, JavaScript) call the |
Beta Was this translation helpful? Give feedback.
-
|
Yes. |
Beta Was this translation helpful? Give feedback.
-
|
This is only a glossary and is used for the DeepL support for glossaries (#10519). As this glossary is not affected by the e.g. |
Beta Was this translation helpful? Give feedback.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
-
I'm confused now, you get the locking when calling autotranslate, so how is the glossary not affected by it?
Locking is necessary to avoid concurrent operations on the underlying repository. For database operations, we're slowly progressing towards row level locking, but we're not yet fully there for some code paths. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I am getting the locked error when I call the
But at this time, the Glossary is only used for read operation. We have now added a workaround to just wait two minutes on an 423 error, but we are still running into the issue. Is there a chance to get the timeout of the lock with the Retry-After header? |
Beta Was this translation helpful? Give feedback.
-
|
Why do you think it's the glossary that is being locked? There is no locking involved when reading glossaries to be used in DeepL. There are two kinds of lock which can influence this:
#13606 will make Weblate tell in the error message what kind of lock is causing this error. |
Beta Was this translation helpful? Give feedback.
-
Because of #13345 (comment), sorry, if I understood something wrong.
Nice, thanks for your efforts!
So, if I understand correctly, the REST API could send a response already back to our CI pipeline even when the repository is not yet unlocked / commits are pending? If yes, would it be useful to have a something like a |
Beta Was this translation helpful? Give feedback.
-
|
Ah, sorry, I misunderstood your question then. You should be able to see in the server logs what is going on while you get this error. The committing might be it, but there might be a different reason as well. |
Beta Was this translation helpful? Give feedback.
-
|
How can I test those changes? We are currently using docker. What are your thoughts about the |
Beta Was this translation helpful? Give feedback.
-
|
We have no clue how long the lock will be held, so I don't see a reasonable way to produce the PS: We should really revisit locking. The single lock to prevent all consistency issues on component level is probably not a viable approach. I've created #13623 to track this task. |
Beta Was this translation helpful? Give feedback.
-
What do you think about this? This would ensure committing is done directly within the request and a response is only sent back when it has finished. I also found When I think about all this, I would bring in another question: Would it make sense to provide a REST API to "pause" commits so the CI job would look like this:
|
Beta Was this translation helpful? Give feedback.
-
|
The commits are done only when needed. That currently translates to changing an already pending string (util #8770 is implemented). But as mentioned before, it might be something different from committing in your case. Check server logs, what is actually happening at that time. |
Beta Was this translation helpful? Give feedback.
-
|
In case you rely on translation propagation, #13665 might have addressed the root cause of this issue. |
Beta Was this translation helpful? Give feedback.
-
|
Do you have an ETA when #13665 will be released? I did not yet test it as I am running within Docker and there is no updated docker image for "unreleased" fixes/betas. |
Beta Was this translation helpful? Give feedback.
-
|
The |
Beta Was this translation helpful? Give feedback.
-
|
Hey @nijel We have updated to the latest version 5.10, and we now get the following email when this error occurs: But: Our retry-mechanism which listens for t he |
Beta Was this translation helpful? Give feedback.
-
|
Ah, the handler was mistakenly removed. Will be back after #13903. |
Beta Was this translation helpful? Give feedback.
-
|
Custom exception handler works again. Does the provided email with stack trace provide any useful information about the locking issue? |
Beta Was this translation helpful? Give feedback.
-
|
Not really. Did you look into server logs what happens at this time? |
Beta Was this translation helpful? Give feedback.
-
|
This issue has been automatically marked as stale because there wasn’t any recent activity. It will be closed soon if no further action occurs. Thank you for your contributions! |
Beta Was this translation helpful? Give feedback.
-
|
This might have been addressed by e3ea2a5 which could have caused the lock to be hold way too long in some cases. Does upgrading to 5.10.3 change anything? |
Beta Was this translation helpful? Give feedback.
-
|
@nijel thanks for the update! We have updated to 5.10.4, but the issue persists with the following log (this is within our CI/CD and we have implemented a retry-mechanism as a workaround and retry every 10 seconds for at least 15 times): Log
Within this CI job, the autotranslate is done sequential for every language within a single component, in parallel for two projects. |
Beta Was this translation helpful? Give feedback.
-
|
Are there many components in the project and translation propagation enabled? I've recently fixed #14189 which could be causing massive slow-downs in such a situation as well. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, every project contains many components, as we are running Weblate in a "real" Continuous Localization scenario. Every branch in our Git repository also gets an own dedicated component: https://translate.owlinfra.de/projects/wordpress-real-cookie-banner-frontend-javascript/ (this is also why #10362 would be so important for us). Translation propagation is enabled for all components within the project. |
Beta Was this translation helpful? Give feedback.
-
|
In that case, changes queued for 5.11 should make the situation much better. Still, the server logs might show why the component is actually locked. In case it is Weblate committing changes, you might want to adjust your automatic translation triggers to avoid repeatedly translating the same strings. It might also be caused by enforced checks marking strings as needing editing. |
Beta Was this translation helpful? Give feedback.
In that case, changes queued for 5.11 should make the situation much better.
Still, the server logs might show why the component is actually locked.
In case it is Weblate committing changes, you might want to adjust your automatic translation triggers to avoid repeatedly translating the same strings. It might also be caused by enforced checks marking strings as needing editing.