training stopped at epoch 1

[rajpratyush]

can you tell me how long does it take for the training process to complete?

i am using a google colab notebook. and it has been stuck at epoch 1 since last 20 mins

[hitvoice]

I just run the training on an RTX-4000:

03/12/2022 04:51:19 [Data loaded.]
03/12/2022 04:51:26 Epoch 1
03/12/2022 04:59:32 > epoch [ 1] updates[  2707] train loss[4.39872] remaining[0:00:00]

03/12/2022 05:00:20 dev EM: 54.285714285714285 F1: 65.42046261815145
03/12/2022 05:00:22 [new best model saved.]
03/12/2022 05:00:22 Epoch 2

It's about 10 minutes after the data were loaded.

[rajpratyush]

I m using linux without gpu though i hv a gpu but havnt setup cuda in it and dont know how to setup too, to train it or colab notebook.

Could you post up a pretrained model

[rajpratyush]

Linux gets hanged while in prepro.py but incolab it toom more than 2.5 hours still stuck at epoch 1

[hitvoice]

On CPU it's supposed to be slow. I haven't tested on CPUs before. And I do not have any experience with Colab. Did you complete your prepro.py?

[rajpratyush]

On colab i did completed but not on cpu it was always getting stuck at dev part

[hitvoice]

If you interrupt the kernel, what's the stack trace? It may help by indicating where it was stuck.

[rajpratyush]

Actually the system got hanged but would you mind if we get up on a short online meeting if you are free

[rajpratyush]

@hitvoice My Google Colab notebook is successfully working now. Shall I put up a pull request for those who want to implement it?

[hitvoice]

Sorry for the late reply. Pull requests are definitely welcomed!