improvement: gpu interfacing on training and inference

[akgohain]

right now, initating a training or inference job in their respective modules provides no feedback (not even a loading bar) on successful job start. the only feedback is in the terminal....

we should implement some degree of user feedback for these workflows, pooling from data given from tensorboard/pytorch in addition to log files

the degree of which and specifics should be discussed on this thread

[samcraftt]

@akgohain Starting training runs the main.py script in pytorch_connectomics, which doesn't really have any progress information outputted to begin with. Would we need to make changes inside the pytorch_connectomics repo then?

[akgohain]

@samcraftt good find. i think the current submodule commit we pool from for that is a bit old. does the main.py on the current pytc repo have anything of use? if not, we still might not need to touch the original repo and maybe can put a tqdm wrapper over the loop

[samcraftt]

@akgohain I'll look into it