Interpretation error in TensorFlow

[pdeperio]

When testing new PositionReconstruction treemaker in newpatternlikelihood branch:

git clone https://github.com/XENON1T/hax.git
cd hax
git checkout newpatternlikelihood
source activate pax_head
python setup.py develop --user
export PATH=~/.local/bin:$PATH

haxer --main_data_paths /project2/lgrandi/xenon1t/processed/pax_v6.8.0 --input 171205_0400 --pax_version_policy v6.8.0 --treemakers PositionReconstruction --force_reload

get the following error:

  File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 1067, in _run
    + e.args[0])
dask.async.TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder_2:0", shape=(112, 100), dtype=float32) is not an element of this graph.

[tunnell]

This is really implementation specific. Different TensorFlow versions?

[feigaodm]

Tensorflow versions are different, I can try to use same version.

[pdeperio]

Just realized it works fine if we don't use --force_reload, which seems to want to generate the minitree twice, and fails on the second time. So hints towards an issue in initialization, rather than TF itself.

[feigaodm]

force_reload also introduces erros in notebook, without it, I generated the minitrees for 100 background files and it's fine. Sometimes minitrees failed to be generated, but can be regenerated if rerun. This is probably related to the mysterious problem in hax we saw long time ago, but it could be another one.