Create 2D graph of similarity matrix

[kezihlmann]

it would be nice to see the projects title in a two dimensional graph. this way one can see which projects are similar.

• I had to install a lot of libraries:
• write the name of the projects to the graph

Tutorial: https://towardsdatascience.com/visualising-similarity-clusters-with-interactive-graphs-20a4b2a18534

example:

[cporcellato]

I analyzed the results of the update_similarity_table and I would use the normal Bert (Bi-Encoder) as a Model to check the similarities beween projects. Below are the results based on the method used:

BI_ENCODER

[[1.         0.66499907 0.5774528  ... 0.46011084 0.6957389  0.60140073]
 [0.66499907 1.0000004  0.35996842 ... 0.42480966 0.70504665 0.6204318 ]
 [0.5774528  0.35996842 0.99999976 ... 0.40652382 0.47527164 0.48826885]
 ...
 [0.46011084 0.42480966 0.40652382 ... 1.         0.4223548  0.4681845 ]
 [0.6957389  0.70504665 0.47527164 ... 0.4223548  1.0000001  0.65181774]
 [0.60140073 0.6204318  0.48826885 ... 0.4681845  0.65181774 1.0000001 ]]

TF-IDF

[[1.         0.09933399 0.22905397 ... 0.10827627 0.1173323  0.03900618]
 [0.09933399 1.         0.05470752 ... 0.10100122 0.17434059 0.1081012 ]
 [0.22905397 0.05470752 1.         ... 0.00655724 0.013389   0.01951585]
 ...
 [0.10827627 0.10100122 0.00655724 ... 1.         0.10501308 0.07091517]
 [0.1173323  0.17434059 0.013389   ... 0.10501308 1.         0.05697075]
 [0.03900618 0.1081012  0.01951585 ... 0.07091517 0.05697075 1.        ]]

CROSS_ENCODER

[[0.98766941 0.3798261  0.58413547 ... 0.28300595 0.33832523 0.2818431 ]
 [0.41544628 0.99394834 0.30363074 ... 0.42977682 0.66484392 0.61225444]
 [0.64578456 0.38506973 0.99678814 ... 0.24817492 0.3930212  0.35898468]
 ...
 [0.32852244 0.5465104  0.24622783 ... 0.99363238 0.49296361 0.38691446]
 [0.4195208  0.70072752 0.38918504 ... 0.37852275 0.9810338  0.54162198]
 [0.392941   0.63447803 0.43372503 ... 0.3522211  0.49160165 0.97230613]]

About the plot_similarities.py file, after I installed the pydot library (requested by networkx), I got the following error for the update_similarity_table function.

Traceback (most recent call last):
  File "/home/claudia/Documents/women++/deploy-impact-22-openedu-c/NLP/update_similarity_table.py", line 97, in <module>
    update_similarity_table()
  File "/home/claudia/Documents/women++/deploy-impact-22-openedu-c/NLP/update_similarity_table.py", line 92, in update_similarity_table
    plot_similarities(similarities, df_1["title_en"])
  File "/home/claudia/Documents/women++/deploy-impact-22-openedu-c/NLP/plot_similarties.py", line 208, in plot_similarities
    fig = get_interactive_slider_similarity_graph(
  File "/home/claudia/Documents/women++/deploy-impact-22-openedu-c/NLP/plot_similarties.py", line 117, in get_interactive_slider_similarity_graph
    node_pos = nx.nx_pydot.graphviz_layout(G, prog='dot')
  File "/home/claudia/anaconda3/lib/python3.9/site-packages/networkx/drawing/nx_pydot.py", line 357, in graphviz_layout
    return pydot_layout(G=G, prog=prog, root=root)
  File "/home/claudia/anaconda3/lib/python3.9/site-packages/networkx/drawing/nx_pydot.py", line 406, in pydot_layout
    P = to_pydot(G)
  File "/home/claudia/anaconda3/lib/python3.9/site-packages/networkx/drawing/nx_pydot.py", line 302, in to_pydot
    any(
  File "/home/claudia/anaconda3/lib/python3.9/site-packages/networkx/drawing/nx_pydot.py", line 303, in <genexpr>
    (_check_colon_quotes(k) or _check_colon_quotes(v))
  File "/home/claudia/anaconda3/lib/python3.9/site-packages/networkx/drawing/nx_pydot.py", line 199, in _check_colon_quotes
    return ":" in s and (s[0] != '"' or s[-1] != '"')
TypeError: argument of type 'float' is not iterable