Welcome to TomoTwin Discussions!

[thorstenwagner]

👋 Welcome!

We’re using Discussions as a place to discuss ideas or questions around TomoTwin. :-)

If you have certain problems with TomoTwin we can also start here and if needed convert it into an issue.

Lets get in touch :-)

[akas0018]

Hi Thorsten,

Is it possible to get Tomotwin to work on Mac m1 ?

Thanks

[thorstenwagner]

Mh, havent tried it yet. But pyTorch recently added M1 support. So it might be possible but is not a priority right now.

[stmonash]

Hi Thorsten,

Great software ! Working perfectly on a Linux environment.
This is my first time using Napari and I quite like it!
I already finished a full workflow of particle detection inside a 3D reconstruction.
Pretty happy with the result, with some extra polishing it will most probably give nice particle positions.

I wanted to know if it is possible to increase the number of CPUs when running the "tomotwin_locate.py findmax" command.
It is currently limited to 4 jobs and I have 12 references (so in theory I could run up to 12 jobs simultaneously).
Just to be clear, this is certainly not a computer limitation.
I could not find any information in the tomotwin_locate.py file.
Is there an option for that?
Or do I have to specify that during the compilation?

On a more general comment, how can we know more about the parameters we feed to the software during the different steps?

• For instance what are the -b and -s parameters in the embedding steps?
• Also, why do we have to specify "findmax" when we run the "tomotwin_locate.py" command. I reckon this might indicate that other parameters are available, like a "findmin" option maybe.

Thanks in advance,
Cheers,
Sylvain

[stmonash]

Well, I just had a look at the computation I started yesterday, and it seems that at some point tomotwin uses more than 4 CPUs (see first attached screenshot).

But see how 3 jobs are much older than the other ones? Those are the ones I started yesterday, there is one missing though, because yesterday I could see 4 of them... And I guess that when at least one of the jobs finished, then it unleashed the whole potential of the computer.

Though I do not know why there are so many instances of tomotwin running.... I count 27 of them, give or take.
But it varies over time, right now it is back to 4 instances (second screenshot).

Is this a normal behaviour?
Thanks,
Sylvain

[thorstenwagner]

Thanks for getting in touch!

Pretty happy with the result, with some extra polishing it will most probably give nice particle positions.

Cool! We are always curious how TomoTwin does in the wild. If you want, please share some details how the results look like. If its confident, you can also drop us a PM.

I wanted to know if it is possible to increase the number of CPUs when running the "tomotwin_locate.py findmax" command. It is currently limited to 4 jobs and I have 12 references (so in theory I could run up to 12 jobs simultaneously). Just to be clear, this is certainly not a computer limitation. I could not find any information in the tomotwin_locate.py file. Is there an option for that? Or do I have to specify that during the compilation?

You can list all options with:

tomotwin_locate.py findmax -h

the option --processes might be want you are looking for:

  --processes PROCESSES
                        Number of parallel processed references. More processes require more memory. (default: 4)

However, it's only parallelized over the references. So a single reference won't be processed faster by using more processes.

On a more general comment, how can we know more about the parameters we feed to the software during the different steps?

* For instance what are the -b and -s parameters in the embedding steps?

Does the output of the following command answer you question?

tomotwin_embed.py tomogram -h

* Also, why do we have to specify "findmax" when we run the "tomotwin_locate.py" command. I reckon this might indicate that other parameters are available, like a "findmin" option maybe.

The findmax procedure is one way to detect the peaks in the 3D heatmap. Other procedures are also possible (like using thresholding and connected components analysis), therefore I capsuled it into a seperate command. So in the future there might a command like

tomotwin_locate.py threshold ...

Does that answer your questions? :-)

[stmonash]

Hi Thorsten,

Cool! We are always curious how TomoTwin does in the wild. If you want, please share some details how the results look like. If its confident, you can also drop us a PM.

Yeah, at some point I will definitely be able to share some results, I just want to play a bit with the software first.

tomotwin_locate.py findmax -h

I will try that as soon as I can (the server is currently down), thanks heaps!

However, it's only parallelized over the references. So a single reference won't be processed faster by using more processes.

Yep, this is what I figured, but since I had 12 references I knew I could use more CPU ;-)

Does that answer your questions? :-)

It answers them perfectly! Thanks for that!

[3P1L]

Hey! I'm really psyched to try tomotwin, but hit a little snag:

trying to load the umap embedding in napari leads to the error below.

Has anybody else seen this, too? O:-)

Thanks,

Philipp.

---

AttributeError Traceback (most recent call last)
File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/psygnal/_signal.py:980, in SignalInstance._run_emit_loop(self=<SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>, args=(False,))
979 try:
--> 980 caller.cb(args)
caller = <psygnal._weak_callback._StrongFunction object at 0x7fab70eecc10>
args = (False,)
981 except Exception as e:

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/psygnal/_weak_callback.py:291, in _StrongFunction.cb(self=<psygnal._weak_callback._StrongFunction object>, args=())
290 args = args[: self._max_args]
--> 291 self._f(*self._args, *args, **self._kwargs)
args = ()
self._f = <function FunctionGui.init.._disable_button_and_call at 0x7fab722deef0>
self = <psygnal._weak_callback._StrongFunction object at 0x7fab70eecc10>
self._args = ()
self._kwargs = {}

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/magicgui/widgets/_function_gui.py:218, in FunctionGui.init.._disable_button_and_call()
217 try:
--> 218 self.call()
self = <FunctionGui load_umap_magic(label_layer: napari.layers.labels.labels.Labels = None, filename: pathlib.Path = PosixPath('Tomo_009-dose_filt_embeddings.tumap'))>
219 finally:

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/magicgui/widgets/_function_gui.py:331, in FunctionGui.call(self=<FunctionGui load_umap_magic(label_layer: napari...Tomo_009-dose_filt_embeddings.tumap'))>, update_widget=False, *args=(), **kwargs={})
330 with _function_name_pointing_to_widget(self):
--> 331 value = self._function(*bound.args, **bound.kwargs)
self = <FunctionGui load_umap_magic(label_layer: napari.layers.labels.labels.Labels = None, filename: pathlib.Path = PosixPath('Tomo_009-dose_filt_embeddings.tumap'))>
bound = <BoundArguments (label_layer=None, filename=PosixPath('Tomo_009-dose_filt_embeddings.tumap'))>
self._function = <function load_umap_magic at 0x7faba4138790>
333 self._call_count += 1

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari_tomotwin/load_umap.py:34, in load_umap_magic(label_layer=None, filename=PosixPath('Tomo_009-dose_filt_embeddings.tumap'))
33 label_layer.features = umap
---> 34 label_layer.opacity = 0
label_layer = None
35 label_layer.visible = True

AttributeError: 'NoneType' object has no attribute 'opacity'

The above exception was the direct cause of the following exception:

EmitLoopError Traceback (most recent call last)
File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/magicgui/widgets/bases/_value_widget.py:65, in ValueWidget._on_value_change(self=PushButton(value=False, annotation=None, name='call_button'), value=False)
63 if value is self.null_value and not self._nullable:
64 return
---> 65 self.changed.emit(value)
value = False
self.changed = <SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>
self = PushButton(value=False, annotation=None, name='call_button')

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/psygnal/_signal.py:935, in SignalInstance.emit(self=<SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>, check_nargs=False, check_types=False, asynchronous=False, *args=(False,))
932 sd.start()
933 return sd
--> 935 self._run_emit_loop(args)
self = <SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>
args = (False,)
936 return None

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/psygnal/_signal.py:982, in SignalInstance._run_emit_loop(self=<SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>, args=(False,))
980 caller.cb(args)
981 except Exception as e:
--> 982 raise EmitLoopError(
caller = <psygnal._weak_callback._StrongFunction object at 0x7fab70eecc10>
args = (False,)
983 slot_repr=repr(caller), args=args, exc=e
984 ) from e
986 return None

EmitLoopError: calling <psygnal._weak_callback._StrongFunction object at 0x7fab70eecc10> with args=(False,) caused AttributeError: 'NoneType' object has no attribute 'opacity'.

[thorstenwagner]

Hi,

sorry to hear that you have issues :-)

To make sure that we are using the same napari version could you run:

mamba env update -n napari-tomotwin -f https://raw.githubusercontent.com/MPI-Dortmund/tomotwin-cryoet/main/conda_env_napari.yml --prune

In case you don't have mamba installed (which is much much faster), you can replace mamba with conda and wait longer ^^

After you did that run it again. I case it crashes again send me the packages.txt generated by:

conda activate napari-tomotwin
conda list >> packages.txt

Best,
Thorsten

[3P1L]

Thank you so much for the support! :-)

I actually did the update just before. But let me do it again!

The problem still persists. Here's the list of installed packages:

packages in environment at /home/philipp/miniconda3/envs/napari-tomotwin:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
abseil-cpp 20211102.0 hd4dd3e8_0
alabaster 0.7.12 pyhd3eb1b0_0
app-model 0.2.0 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyhd3eb1b0_0
arrow-cpp 11.0.0 h374c478_1
asciitree 0.3.3 py_2
asttokens 2.0.5 pyhd3eb1b0_0
attrs 22.1.0 pypi_0 pypi
autopep8 2.0.2 pypi_0 pypi
aws-c-common 0.6.8 h5eee18b_1
aws-c-event-stream 0.1.6 h6a678d5_6
aws-checksums 0.1.11 h5eee18b_2
aws-sdk-cpp 1.8.185 h721c034_1
babel 2.11.0 pypi_0 pypi
backcall 0.2.0 pyhd3eb1b0_0
blas 1.0 mkl
blosc 1.21.3 h6a678d5_0
bokeh 3.2.1 pypi_0 pypi
boost-cpp 1.73.0 h7f8727e_12
bottleneck 1.3.5 pypi_0 pypi
brotli 1.0.9 h5eee18b_7
brotli-bin 1.0.9 h5eee18b_7
brotlipy 0.7.0 pypi_0 pypi
brunsli 0.1 h2531618_0
build 0.10.0 pypi_0 pypi
bzip2 1.0.8 h7b6447c_0
c-ares 1.19.0 h5eee18b_0
ca-certificates 2023.05.30 h06a4308_0
cachey 0.2.1 pyh9f0ad1d_0 conda-forge
certifi 2023.7.22 pypi_0 pypi
cffi 1.15.1 pypi_0 pypi
cfitsio 3.470 h5893167_7
charls 2.2.0 h2531618_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.0.4 pypi_0 pypi
cloudpickle 2.2.1 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
comm 0.1.2 pypi_0 pypi
contourpy 1.0.5 pypi_0 pypi
cryptography 41.0.2 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
cython 0.29.36 pypi_0 pypi
cytoolz 0.12.0 pypi_0 pypi
dask 2023.6.0 pypi_0 pypi
dask-core 2023.6.0 py310h06a4308_0
dbus 1.13.18 hb2f20db_0
debugpy 1.5.1 pypi_0 pypi
decorator 5.1.1 pyhd3eb1b0_0
deprecated 1.2.14 pypi_0 pypi
distributed 2023.6.0 pypi_0 pypi
docstring_parser 0.15 pyhd8ed1ab_0 conda-forge
docutils 0.16 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
executing 0.8.3 pyhd3eb1b0_0
expat 2.4.9 h6a678d5_0
fasteners 0.16.3 pyhd3eb1b0_0
font-ttf-dejavu-sans-mono 2.37 hd3eb1b0_0
font-ttf-inconsolata 2.001 hcb22688_0
font-ttf-source-code-pro 2.030 hd3eb1b0_0
font-ttf-ubuntu 0.83 h8b1ccd4_0
fontconfig 2.14.1 h52c9d5c_1
fonts-anaconda 1 h8fa9717_0
fonts-conda-ecosystem 1 hd3eb1b0_0
fonttools 4.41.1 pypi_0 pypi
freetype 2.12.1 h4a9f257_0
freetype-py 2.2.0 pyhd3eb1b0_0
fsspec 2023.4.0 pypi_0 pypi
gflags 2.2.2 he6710b0_0
giflib 5.2.1 h5eee18b_3
glib 2.69.1 he621ea3_2
glog 0.5.0 h2531618_0
grpc-cpp 1.48.2 he1ff14a_1
gst-plugins-base 1.14.1 h6a678d5_1
gstreamer 1.14.1 h5eee18b_1
hdbscan 0.8.33 pypi_0 pypi
heapdict 1.0.1 pyhd3eb1b0_0
hsluv 5.0.3 pypi_0 pypi
icu 58.2 he6710b0_3
idna 3.4 pypi_0 pypi
imagecodecs 2021.8.26 pypi_0 pypi
imageio 2.31.1 pypi_0 pypi
imagesize 1.4.1 pypi_0 pypi
importlib-metadata 6.0.0 pypi_0 pypi
in-n-out 0.1.8 pyhd8ed1ab_0 conda-forge
intel-openmp 2023.1.0 hdb19cb5_46305
ipykernel 6.19.2 pypi_0 pypi
ipython 8.12.0 pypi_0 pypi
ipython_genutils 0.2.0 pyhd3eb1b0_1
jedi 0.18.1 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
joblib 1.3.1 pypi_0 pypi
jpeg 9e h5eee18b_1
jsonschema 4.17.3 pypi_0 pypi
jupyter-client 8.1.0 pypi_0 pypi
jupyter-core 5.3.0 pypi_0 pypi
jupyter_client 8.1.0 py310h06a4308_0
jupyter_core 5.3.0 py310h06a4308_0
jxrlib 1.1 h7b6447c_2
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.20.1 h143b758_1
lazy-loader 0.2 pypi_0 pypi
lazy_loader 0.2 py310h06a4308_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libaec 1.0.4 he6710b0_1
libboost 1.73.0 h28710b8_12
libbrotlicommon 1.0.9 h5eee18b_7
libbrotlidec 1.0.9 h5eee18b_7
libbrotlienc 1.0.9 h5eee18b_7
libclang 10.0.1 default_hb85057a_2
libcurl 8.1.1 h251f7ec_1
libdeflate 1.17 h5eee18b_0
libedit 3.1.20221030 h5eee18b_0
libev 4.33 h7f8727e_1
libevent 2.1.12 hdbd6064_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 13.1.0 he5830b7_0 conda-forge
libgfortran-ng 11.2.0 h00389a5_1
libgfortran5 11.2.0 h1234567_1
libgomp 13.1.0 he5830b7_0 conda-forge
libllvm10 10.0.1 hbcb73fb_5
libnghttp2 1.52.0 h2d74bed_1
libpng 1.6.39 h5eee18b_0
libpq 12.15 hdbd6064_1
libprotobuf 3.20.3 he621ea3_0
libsodium 1.0.18 h7b6447c_0
libssh2 1.10.0 hdbd6064_2
libstdcxx-ng 11.2.0 h1234567_1
libthrift 0.15.0 h1795dd8_2
libtiff 4.5.0 h6a678d5_2
libuuid 1.41.5 h5eee18b_0
libwebp 1.2.4 h11a3e52_1
libwebp-base 1.2.4 h5eee18b_1
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 hfa300c1_0
libxml2 2.9.14 h74e7548_0
libxslt 1.1.35 h4e12654_0
libzopfli 1.0.3 he6710b0_0
llvmlite 0.40.1 pypi_0 pypi
lmdb 1.4.1 pypi_0 pypi
locket 1.0.0 pypi_0 pypi
lz4 4.3.2 pypi_0 pypi
lz4-c 1.9.4 h6a678d5_0
magicgui 0.7.2 pyhd8ed1ab_0 conda-forge
markdown-it-py 2.2.0 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.7.2 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
mdurl 0.1.0 pypi_0 pypi
mkl 2023.1.0 h6d00ec8_46342
mkl-fft 1.3.6 pypi_0 pypi
mkl-random 1.2.2 pypi_0 pypi
mkl-service 2.4.0 pypi_0 pypi
mkl_fft 1.3.6 py310h1128e8f_1
mkl_random 1.2.2 py310h1128e8f_1
mrcfile 1.4.3 pypi_0 pypi
msgpack 1.0.3 pypi_0 pypi
msgpack-python 1.0.3 py310hd09550d_0
mypy-extensions 0.4.3 pypi_0 pypi
mypy_extensions 0.4.3 py310h06a4308_0
napari 0.4.17 pyh275ddea_0_pyqt conda-forge
napari-boxmanager 0.4.2 pypi_0 pypi
napari-clusters-plotter 0.7.2 pypi_0 pypi
napari-console 0.0.8 pyhd8ed1ab_0 conda-forge
napari-plugin-engine 0.2.0 pyhd8ed1ab_2 conda-forge
napari-skimage-regionprops 0.10.1 pypi_0 pypi
napari-svg 0.1.10 pyhd8ed1ab_0 conda-forge
napari-tomotwin 0.2.0 pypi_0 pypi
napari-tools-menu 0.1.19 pypi_0 pypi
napari-workflows 0.2.9 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.5.6 pypi_0 pypi
networkx 3.1 pypi_0 pypi
npe2 0.7.2 pyhd8ed1ab_0 conda-forge
nspr 4.35 h6a678d5_0
nss 3.89.1 h6a678d5_0
numba 0.57.1 pypi_0 pypi
numcodecs 0.11.0 pypi_0 pypi
numexpr 2.8.4 pypi_0 pypi
numpy 1.23.5 pypi_0 pypi
numpy-base 1.25.0 py310hb5e798b_0
numpydoc 1.5.0 pypi_0 pypi
openjpeg 2.4.0 h3ad879b_0
openssl 3.0.9 h7f8727e_0
orc 1.7.4 hb3bc3d3_1
packaging 23.0 pypi_0 pypi
pandas 1.5.3 pypi_0 pypi
parso 0.8.3 pyhd3eb1b0_0
partd 1.2.0 pyhd3eb1b0_1
pcre 8.45 h295c915_0
pexpect 4.8.0 pyhd3eb1b0_3
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 9.4.0 pypi_0 pypi
pint 0.22 pyhd8ed1ab_1 conda-forge
pip 23.2.1 pypi_0 pypi
platformdirs 2.5.2 pypi_0 pypi
ply 3.11 pypi_0 pypi
pooch 1.4.0 pyhd3eb1b0_0
prompt-toolkit 3.0.36 pypi_0 pypi
psutil 5.9.0 pypi_0 pypi
psygnal 0.9.1 pyhd8ed1ab_0 conda-forge
ptyprocess 0.7.0 pyhd3eb1b0_2
pure_eval 0.2.2 pyhd3eb1b0_0
pyarrow 11.0.0 pypi_0 pypi
pycodestyle 2.11.0 pypi_0 pypi
pycparser 2.21 pyhd3eb1b0_0
pydantic 1.10.8 pypi_0 pypi
pygments 2.15.1 pypi_0 pypi
pynndescent 0.5.10 pypi_0 pypi
pyopengl 3.1.1a1 pypi_0 pypi
pyopenssl 23.2.0 pypi_0 pypi
pyproject-hooks 1.0.0 pypi_0 pypi
pyproject_hooks 1.0.0 py310h06a4308_0
pyqt 5.15.7 py310h6a678d5_1
pyqt5 5.15.9 pypi_0 pypi
pyqt5-qt5 5.15.2 pypi_0 pypi
pyqt5-sip 12.11.0 pypi_0 pypi
pyrsistent 0.18.0 pypi_0 pypi
pysocks 1.7.1 pypi_0 pypi
pystardb 0.4.2 pypi_0 pypi
python 3.10.12 h955ad1f_0
python-build 0.10.0 py310h06a4308_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python-lmdb 1.4.1 py310h6a678d5_0
python_abi 3.10 2_cp310 conda-forge
pytz 2022.7 pypi_0 pypi
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
pyzmq 25.1.0 pypi_0 pypi
qt-main 5.15.2 h327a75a_7
qt-webengine 5.15.9 hd2b0992_4
qtconsole-base 5.4.3 pyha770c72_0 conda-forge
qtpy 2.2.0 pypi_0 pypi
qtwebkit 5.212 h4eab89a_4
re2 2022.04.01 h295c915_0
readline 8.2 h5eee18b_0
requests 2.31.0 pypi_0 pypi
rich 13.3.5 pypi_0 pypi
scikit-image 0.20.0 pypi_0 pypi
scikit-learn 1.3.0 pypi_0 pypi
scipy 1.10.1 pypi_0 pypi
setuptools 68.0.0 pypi_0 pypi
shellingham 1.5.0 pypi_0 pypi
sip 6.6.2 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_1
snappy 1.1.9 h295c915_0
snowballstemmer 2.2.0 pyhd3eb1b0_0
sortedcontainers 2.4.0 pyhd3eb1b0_0
sphinx 4.4.0 pyhd3eb1b0_0
sphinxcontrib-applehelp 1.0.2 pyhd3eb1b0_0
sphinxcontrib-devhelp 1.0.2 pyhd3eb1b0_0
sphinxcontrib-htmlhelp 2.0.0 pyhd3eb1b0_0
sphinxcontrib-jsmath 1.0.1 pyhd3eb1b0_0
sphinxcontrib-qthelp 1.0.3 pyhd3eb1b0_0
sphinxcontrib-serializinghtml 1.1.5 pyhd3eb1b0_0
sqlite 3.41.2 h5eee18b_0
stack_data 0.2.0 pyhd3eb1b0_0
superqt 0.4.1 pyhd8ed1ab_0 conda-forge
tbb 2021.8.0 hdb19cb5_0
tblib 1.7.0 pyhd3eb1b0_0
threadpoolctl 3.2.0 pypi_0 pypi
tifffile 2021.7.2 pyhd3eb1b0_2
tk 8.6.12 h1ccaba5_0
toml 0.10.2 pyhd3eb1b0_0
tomli 2.0.1 pypi_0 pypi
tomli-w 1.0.0 pyhd3eb1b0_0
toolz 0.12.0 pypi_0 pypi
tornado 6.2 pypi_0 pypi
tqdm 4.65.0 pypi_0 pypi
traitlets 5.7.1 pypi_0 pypi
typer 0.9.0 pypi_0 pypi
typing-extensions 4.7.1 pypi_0 pypi
typing_extensions 4.7.1 py310h06a4308_0
tzdata 2023c h04d1e81_0
umap-learn 0.5.3 pypi_0 pypi
urllib3 1.26.16 pypi_0 pypi
utf8proc 2.6.1 h27cfd23_0
vispy 0.11.0 pypi_0 pypi
wcwidth 0.2.5 pyhd3eb1b0_0
wheel 0.38.4 pypi_0 pypi
wrapt 1.14.1 pypi_0 pypi
xyzservices 2022.9.0 pypi_0 pypi
xz 5.4.2 h5eee18b_0
yaml 0.2.5 h7b6447c_0
zarr 2.13.3 pypi_0 pypi
zeromq 4.3.4 h2531618_0
zfp 0.5.5 h295c915_6
zict 2.2.0 pypi_0 pypi
zipp 3.11.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0

[3P1L]

Ah! Found it ... so the mask is necessary?

[3P1L]

But I also confirmed on another system that there's a problem with the PyQt bits

label_layer.visible = 1
AttributeError: 'NoneType' object has no attribute 'visible'

[thorstenwagner]

sorry, I totally forgot you!! Let me try to reproduce it...

[thorstenwagner]

So here is what I did:

napari path/to/tomo.mrc path/to/umap/tomo_label_mask.mrci

Then: Plugins -> Cluster UMAP Embeddings -> Select File (umap) -> Load

I dont see any problem here. And yes, you need to provide the label_mask!

[thorstenwagner]

Alright, with the next release there be pop-up message if parameters are not specified:

MPI-Dortmund/napari-tomotwin#9

[thorstenwagner]

But I also confirmed on another system that there's a problem with the PyQt bits

label_layer.visible = 1 AttributeError: 'NoneType' object has no attribute 'visible'

When does that happen? I need to reproduce it to fix it.

[thorstenwagner]

Lets we continue here:
MPI-Dortmund/napari-tomotwin#10

[hey-iain]

Hi Thorsten,

Thank you for making tomotwin! We really like it, and so far have used it succesfully on a small dataset for STA of a protein bound to liposome membranes (seems to do a good job, even though the model wasn't trained on this).

My test dataset was 10 tomograms, which was manageable. We now have 180 tomograms to process - and embedding is a major bottleneck. We will try the masking outlined in the latest version.
The main question - is it strongly recommended to keep stride at 2 voxels? Have other values been tested? Or could we sacrifice finer sampling for increased compute time (we find -s 2 -> -s 4 decrease compute time from 4h/tomo -> 20 min/tomo).

Thanks again,
Iain

[thorstenwagner]

Hi Iain,

Thanks for sharing your experience with TomoTwin.

We know that embedding is quite slow and a bottleneck. We are working hard on this issue. May I direct your attention to this PR:
#40

With this PR the embedding will be much faster. Additionally it much better scales with multiple GPUs (how many do you use?). The work is basically done, I'm just testing if everything works as it should because the internal libs have been updated. If you want, I can point you to the installation instructions for the beta and you give it a try? External testing would also help us :-)

Regarding the step:
If your target protein is big enough, then I think its safe to increase the stride. If its very small, chances are higher that you will miss the center of the protein while embedding.

Hope this helps you!

Best,
Thorsten

[hey-iain]

Hi Thorsten,

Thanks for the quick reply and all the helpful info.

With this PR the embedding will be much faster. Additionally it much better scales with multiple GPUs (how many do you use?). The work is basically done, I'm just testing if everything works as it should because the internal libs have been updated. If you want, I can point you to the installation instructions for the beta and you give it a try? External testing would also help us :-)

Generally I have been using 4 GPUs, but can request a max of 8 from our cluster (current plan was to split tomograms into 2 sets running simultaneously with 4 GPUs for each job) - so this may make things very fast. I would be happy to do some testing!

Regarding the step:
If your target protein is big enough, then I think its safe to increase the stride. If its very small, chances are higher that you will miss the center of the protein while embedding.

Ah OK - the protein is large (~ 30 x 15 pix in 2D in the 10A bin tomo), so maybe some wiggle-room there. Thank you!

Best wishes,
Iain

[julienbous]

Hi,
I try to get started with TomoTwin that I installed with conda (25.7.0) without apparent error/problems. Every tool used in the first tutorial worst nicely but when it come to the "TomoTwin clustering workflow" a module seems missing or misplaced: "---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/_command_registry.py:32, in CommandHandler.resolve(self=CommandHandler(id='napari-tomotwin.UmapTool', fu...me='napari_tomotwin.load_umap_widget:UmapToolQt'))
31 try:
---> 32 self.function = utils.import_python_name(self.python_name)
self.function = None
self.python_name = 'napari_tomotwin.load_umap_widget:UmapToolQt'
self = CommandHandler(id='napari-tomotwin.UmapTool', function=None, python_name='napari_tomotwin.load_umap_widget:UmapToolQt')
utils = <module 'npe2.manifest.utils' from '/home/cryosparc_user/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/manifest/utils.py'>
33 except Exception as e:

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/manifest/utils.py:260, in import_python_name(python_name='napari_tomotwin.load_umap_widget:UmapToolQt')
258 module_name, funcname = match.groups() # type: ignore [union-attr]
--> 260 mod = import_module(module_name)
module_name = 'napari_tomotwin.load_umap_widget'
261 return getattr(mod, funcname)

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/importlib/init.py:126, in import_module(name='napari_tomotwin.load_umap_widget', package=None)
125 level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)
level = 0
name = 'napari_tomotwin.load_umap_widget'
name[level:] = 'napari_tomotwin.load_umap_widget'
package = None
_bootstrap = <module '_frozen_importlib' (frozen)>

File :1050, in _gcd_import(name='napari_tomotwin.load_umap_widget', package=None, level=0)

File :1027, in find_and_load(name='napari_tomotwin.load_umap_widget', import=)

File :1006, in find_and_load_unlocked(name='napari_tomotwin.load_umap_widget', import=)

File :688, in _load_unlocked(spec=ModuleSpec(name='napari_tomotwin.load_umap_widge...te-packages/napari_tomotwin/load_umap_widget.py'))

File :883, in exec_module(self=<_frozen_importlib_external.SourceFileLoader object>, module=<module 'napari_tomotwin.load_umap_widget' from ...te-packages/napari_tomotwin/load_umap_widget.py'>)

File :241, in _call_with_frames_removed(f=, *args=(<code object at 0x7fab30598a80, file "/...ges/napari_tomotwin/load_umap_widget.py", line 1>, {'PlotterWidget': <class 'napari_clusters_plotter._new_plotter_widget.PlotterWidget'>, 'builtins': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, 'BlockingIOError': <class 'BlockingIOError'>, 'BrokenPipeError': <class 'BrokenPipeError'>, 'BufferError': <class 'BufferError'>, 'BytesWarning': <class 'BytesWarning'>, 'ChildProcessError': <class 'ChildProcessError'>, 'ConnectionAbortedError': <class 'ConnectionAbortedError'>, ...}, 'cached': '/home/cryosparc_user/miniconda3/envs/napari-tomo...twin/pycache/load_umap_widget.cpython-310.pyc', 'doc': None, 'file': '/home/cryosparc_user/miniconda3/envs/napari-tomo...site-packages/napari_tomotwin/load_umap_widget.py', 'loader': <_frozen_importlib_external.SourceFileLoader object>, 'name': 'napari_tomotwin.load_umap_widget', 'package': 'napari_tomotwin', 'spec': ModuleSpec(name='napari_tomotwin.load_umap_widge...te-packages/napari_tomotwin/load_umap_widget.py'), 'futures': <module 'concurrent.futures' from '/home/cryospa...n/lib/python3.10/concurrent/futures/init.py'>, ...}), **kwds={})

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari_tomotwin/load_umap_widget.py:9
8 from napari_clusters_plotter._new_plotter_widget import PlotterWidget
----> 9 from napari_tomotwin.load_umap import LoadUmapTool
10 from napari.utils import notifications

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari_tomotwin/load_umap.py:13
12 from napari.utils import notifications
---> 13 from napari_clusters_plotter._utilities import estimate_number_bins
14 from napari_tomotwin.anchor_tool import drag_circle_callback

ImportError: cannot import name 'estimate_number_bins' from 'napari_clusters_plotter._utilities' (/home/cryosparc_user/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari_clusters_plotter/_utilities.py)

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/app_model/backends/qt/_qaction.py:59, in QCommandAction._on_triggered(self=QMenuItemAction(MenuItem(when=None, group='3_plu...workflow (TomoTwin)'))), alt=None), app='napari'), checked=True)
55 def _on_triggered(self, checked: bool) -> None:
56 # execute_command returns a Future, for the sake of eventually being
57 # asynchronous without breaking the API. For now, we call result()
58 # to raise any exceptions.
---> 59 self._app.commands.execute_command(self._command_id).result()
self._command_id = 'napari-tomotwin:TomoTwin clustering workflow'
self = QMenuItemAction(MenuItem(when=None, group='3_plugin_contributions', order=None, command=CommandRule(id='napari-tomotwin:TomoTwin clustering workflow', title='TomoTwin clustering workflow (TomoTwin)', category=None, tooltip=None, status_tip=None, icon=None, icon_visible_in_menu=True, enablement=None, short_title=None, toggled=ToggleRule(condition=None, get_current=functools.partial(<function _get_current_dock_status at 0x7fabb08bbb50>, full_name='TomoTwin clustering workflow (TomoTwin)'))), alt=None), app='napari')
self._app = Application('napari')

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/app_model/registries/_commands_reg.py:247, in CommandsRegistry.execute_command(self=<CommandsRegistry at 0x7fabb0663dc0 (164 commands)>, id='napari-tomotwin:TomoTwin clustering workflow', execute_asynchronously=False, *args=(), **kwargs={})
243 except Exception as e:
244 if self._raise_synchronous_exceptions:
245 # note, the caller of this function can also achieve this by
246 # calling future.result() on the returned future object.
--> 247 raise e
248 future.set_exception(e)
250 return future

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/app_model/registries/_commands_reg.py:242, in CommandsRegistry.execute_command(self=<CommandsRegistry at 0x7fabb0663dc0 (164 commands)>, id='napari-tomotwin:TomoTwin clustering workflow', execute_asynchronously=False, *args=(), **kwargs={})
240 future: Future = Future()
241 try:
--> 242 future.set_result(cmd(*args, **kwargs))
future = <Future at 0x7fab30526ad0 state=pending>
cmd = <function Store.inject.._inner.._exec at 0x7fab304ff9a0>
args = ()
kwargs = {}
243 except Exception as e:
244 if self._raise_synchronous_exceptions:
245 # note, the caller of this function can also achieve this by
246 # calling future.result() on the returned future object.

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/in_n_out/_store.py:934, in Store.inject_processors.._deco.._exec(*args=(), **kwargs={})
932 @wraps(func)
933 def _exec(*args: P.args, **kwargs: P.kwargs) -> R:
--> 934 result = func(*args, **kwargs)
func = <function Store.inject.._inner.._exec at 0x7fab304ffc70>
args = ()
kwargs = {}
935 if result is not None:
936 self.process(
937 result,
938 type_hint=type_hint,
(...)
941 _funcname=getattr(func, "qualname", str(func)),
942 )

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/in_n_out/_store.py:804, in Store.inject.._inner.._exec(*args=(), **kwargs={})
797 logger.debug(
798 " Calling %s with %r (injected %r)",
799 _fname,
800 bound.arguments,
801 _injected_names,
802 )
803 try:
--> 804 result = func(**bound.arguments)
bound = <BoundArguments (plugin='napari-tomotwin', widget_name='TomoTwin clustering workflow', full_name='TomoTwin clustering workflow (TomoTwin)')>
func = functools.partial(<function _toggle_or_get_widget at 0x7fabb08bbac0>, plugin='napari-tomotwin', widget_name='TomoTwin clustering workflow', full_name='TomoTwin clustering workflow (TomoTwin)')
bound.arguments = {'plugin': 'napari-tomotwin', 'widget_name': 'TomoTwin clustering workflow', 'full_name': 'TomoTwin clustering workflow (TomoTwin)'}
805 except TypeError as e:
806 if "missing" not in e.args[0]:

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/_qplugins/_qnpe2.py:344, in _toggle_or_get_widget(plugin='napari-tomotwin', widget_name='TomoTwin clustering workflow', full_name='TomoTwin clustering workflow (TomoTwin)')
341 return None
343 # Get widget param name (if any) and check type
--> 344 widget_callable, _ = get_widget_contribution(plugin, widget_name) # type: ignore [misc]
plugin = 'napari-tomotwin'
widget_name = 'TomoTwin clustering workflow'
345 widget_param = _get_widget_viewer_param(widget_callable, widget_name)
347 kwargs = {}

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/plugins/_npe2.py:130, in get_widget_contribution(plugin_name='napari-tomotwin', widget_name='TomoTwin clustering workflow')
128 if contrib.plugin_name == plugin_name:
129 if not widget_name or contrib.display_name == widget_name:
--> 130 return contrib.get_callable(), contrib.display_name
contrib = WidgetContribution(command='napari-tomotwin.UmapTool', display_name='TomoTwin clustering workflow', autogenerate=False)
contrib.display_name = 'TomoTwin clustering workflow'
131 widgets_seen.add(contrib.display_name)
132 if widget_name and widgets_seen:

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/manifest/contributions/_widgets.py:49, in WidgetContribution.get_callable(self=WidgetContribution(command='napari-tomotwin.Umap...omoTwin clustering workflow', autogenerate=False), _registry=None)
46 def get_callable(
47 self, _registry: Optional[CommandRegistry] = None
48 ) -> Callable[..., Widget]:
---> 49 func = super().get_callable()
50 if self.autogenerate:
51 try:

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/manifest/utils.py:71, in Executable.get_callable(self=WidgetContribution(command='napari-tomotwin.Umap...omoTwin clustering workflow', autogenerate=False), _registry=<npe2._command_registry.CommandRegistry object>)
68 from npe2._plugin_manager import PluginManager
70 _registry = PluginManager.instance().commands
---> 71 return _registry.get(self.command)
_registry = <npe2._command_registry.CommandRegistry object at 0x7fabb38dbb50>
self.command = 'napari-tomotwin.UmapTool'
self = WidgetContribution(command='napari-tomotwin.UmapTool', display_name='TomoTwin clustering workflow', autogenerate=False)

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/_command_registry.py:138, in CommandRegistry.get(self=<npe2._command_registry.CommandRegistry object>, id='napari-tomotwin.UmapTool')
136 if id not in self._commands: # sourcery skip
137 raise KeyError(f"command {id!r} not registered")
--> 138 return self._commands[id].resolve()
id = 'napari-tomotwin.UmapTool'
self._commands = {'napari-tomotwin.UmapTool': CommandHandler(id='napari-tomotwin.UmapTool', function=None, python_name='napari_tomotwin.load_umap_widget:UmapToolQt')}
self = <npe2._command_registry.CommandRegistry object at 0x7fabb38dbb50>
self._commands[id] = CommandHandler(id='napari-tomotwin.UmapTool', function=None, python_name='napari_tomotwin.load_umap_widget:UmapToolQt')

File ~/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/_command_registry.py:34, in CommandHandler.resolve(self=CommandHandler(id='napari-tomotwin.UmapTool', fu...me='napari_tomotwin.load_umap_widget:UmapToolQt'))
32 self.function = utils.import_python_name(self.python_name)
33 except Exception as e:
---> 34 raise RuntimeError(
self.python_name = 'napari_tomotwin.load_umap_widget:UmapToolQt'
self = CommandHandler(id='napari-tomotwin.UmapTool', function=None, python_name='napari_tomotwin.load_umap_widget:UmapToolQt')
35 f"Failed to import command at {self.python_name!r}: {e}"
36 ) from e
38 return self.function

RuntimeError: Failed to import command at 'napari_tomotwin.load_umap_widget:UmapToolQt': cannot import name 'estimate_number_bins' from 'napari_clusters_plotter._utilities' (/home/cryosparc_user/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari_clusters_plotter/_utilities.py)"

Any suggestions to patch this?

Best,
Julien

[GavinR1]

Hi Julien,

The issue is with the napari_clusters_plotter plugin that we use some tools from. It seems in one of their latest releases they restructures the tools that we use (and deletes the _utilities.py) which causes problems. We're experiencing a few errors with napari at the moment that should be fixed an update soon.

For now could you do
pip install napari_clusters_plotter==0.8.1

And then try again? That should fix the plugin back to a version where the _utilities.py existed. If you run into other issues with the workflow from there I'd be curious to know as I am going through now to look for any more incompatibilities.

[julienbous]

Dear Gavin,
Thank you for providing your help and explanations. It work well now.
Best,
Julien