Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ jobs:
- name: Install More Packages
run: |
ps aux | grep memgraph
poetry install --all-extras
poetry install -E arrow -E dgl -E docker
poe install-pyg-cpu
poe install-dgl
poe install-tfgnn
Expand Down Expand Up @@ -148,8 +148,9 @@ jobs:
poetry-version: ${{ env.POETRY_VERSION }}
- name: Test project
run: |
poetry install --all-extras
poetry install -E arrow -E dgl -E docker
poe install-pyg-cpu
poe install-dgl
poe install-tfgnn
export TF_USE_LEGACY_KERAS="1"
poetry run pytest -vvv -m "not slow and not ubuntu and not docker"
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/build-packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,9 +122,10 @@ jobs:
- name: Install wheel and dependencies
run: |
python -m pip install dist/*.whl
poetry install --all-extras
poetry install -E arrow -E dgl -E docker
poe install-pyg-cpu
poe install-dgl
poe install-tfgnn

- name: Run Tests
run: |
Expand Down
24 changes: 24 additions & 0 deletions docs/how-to-guides/translators/import-python-graphs.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ hood](https://img.shields.io/static/v1?label=Related&message=Under%20the%20hood&
In this guide you will learn how to:

- [**Import NetworkX graph into Memgraph**](#import-networkx-graph-into-memgraph)
- [**Import DOT graph into Memgraph**](#import-dot-graph-into-memgraph)
- [**Import PyG graph into Memgraph**](#import-pyg-graph-into-memgraph)
- [**Import DGL graph into Memgraph**](#import-dgl-graph-into-memgraph)
- [**Import TF-GNN graph into Memgraph**](#import-tf-gnn-graph-into-memgraph)
Expand Down Expand Up @@ -80,6 +81,29 @@ Click **Run Query** button to see the results.

The NetworkX node identification number maps to the `id` node property in Memgraph. The `labels` key is reserved for the node label in Memgraph, while the edge `type` key is reserved for the relationship type in Memgraph. If no `type` is defined, then the relationship will be of type `TO` in Memgraph. You can notice that the node with the property `name` Kata and property `id` 2 doesn't have a label. This happened because the node property key `labels` was not defined.

## Import DOT graph into Memgraph

You can import DOT files by using `GraphImporter` with `graph_type="NX"`. DOT parsing uses `pydot` and NetworkX under the hood.

### Create and run a Python script

Create a new Python script `dot-graph.py` with the following code:

```python
from gqlalchemy.transformations.importing.graph_importer import GraphImporter

importer = GraphImporter(graph_type="NX")

# Import from a DOT file path.
importer.translate_dot_file("graph.dot")

# Or import directly from DOT content.
dot_data = 'digraph G { "A" [label="A"]; "B"; "A" -> "B" [fontsize="10"]; }'
importer.translate_dot_data(dot_data)
```

During DOT import, GQLAlchemy enriches parsed nodes and edges with DOT-specific metadata (for example `dot_type`, `attributes_json`, flattened `attributes_*` keys, `sequence`, and stable edge `id` values).

## Import PyG graph into Memgraph

### Prerequisites
Expand Down
1 change: 1 addition & 0 deletions docs/import-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ To import Parquet, ORC or IPC/Feather/Arrow file into Memgraph via GQLAlchemy, [
## Python graphs - NetworkX, PyG or DGL graph

To import NetworkX, PyG or DGL graph into Memgraph via GQLAlchemy, [transform the source graph into Memgraph graph](how-to-guides/translators/import-python-graphs.md).
DOT files and DOT strings are also supported through the NetworkX importer (`GraphImporter(graph_type="NX")`) with `translate_dot_file(...)` and `translate_dot_data(...)`.

## Kafka, RedPanda or Pulsar data stream

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,27 @@ Gets cypher queries using the underlying translator and then inserts all queries

- `graph` - dgl, pytorch geometric or nx graph instance.

#### translate_dot_file

```python
def translate_dot_file(path: str) -> None
```

Parses a DOT file into a NetworkX graph and imports it to Memgraph. This method is available when `graph_type="NX"`.

**Arguments**:

- `path` - Path to a DOT file.

#### translate_dot_data

```python
def translate_dot_data(dot_data: str) -> None
```

Parses DOT content from a string into a NetworkX graph and imports it to Memgraph. This method is available when `graph_type="NX"`.

**Arguments**:

- `dot_data` - Raw DOT graph content.

149 changes: 149 additions & 0 deletions gqlalchemy/transformations/importing/graph_importer.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@

from gqlalchemy.exceptions import raise_if_not_imported
import gqlalchemy.memgraph_constants as mg_consts
import networkx as nx
import json
import re
from collections import defaultdict
from typing import Any, Dict

try:
from gqlalchemy.transformations.translators.dgl_translator import DGLTranslator
Expand All @@ -31,6 +36,11 @@
except ModuleNotFoundError:
PyGTranslator = None

try:
import pydot
except ModuleNotFoundError:
pydot = None


class GraphImporter(Importer):
"""Imports dgl, pyg or networkx graph representations to Memgraph.
Comment thread
mattkjames7 marked this conversation as resolved.
Expand Down Expand Up @@ -74,3 +84,142 @@ def translate(self, graph) -> None:
memgraph = Memgraph()
for query in self.translator.to_cypher_queries(graph):
memgraph.execute(query)

def translate_dot_file(self, path: str) -> None:
"""Parses a DOT file to a NetworkX graph and imports it to Memgraph."""
self._raise_if_not_nx_importer()
raise_if_not_imported(dependency=pydot, dependency_name="pydot")

pydot_graphs = pydot.graph_from_dot_file(path)
if not pydot_graphs:
raise ValueError("Unable to parse DOT file.")

graph = self._normalize_dot_graph(self._graph_from_pydot(pydot_graphs[0]))
self.translate(graph)

def translate_dot_data(self, dot_data: str) -> None:
"""Parses DOT content to a NetworkX graph and imports it to Memgraph."""
self._raise_if_not_nx_importer()
raise_if_not_imported(dependency=pydot, dependency_name="pydot")

pydot_graphs = pydot.graph_from_dot_data(dot_data)
if not pydot_graphs:
raise ValueError("Unable to parse DOT data.")

graph = self._normalize_dot_graph(self._graph_from_pydot(pydot_graphs[0]))
self.translate(graph)

def _raise_if_not_nx_importer(self) -> None:
if self.graph_type != GraphType.NX.name:
raise ValueError("DOT import is supported only for NetworkX graph importer.")

def _normalize_dot_graph(self, graph: nx.Graph) -> nx.Graph:
"""Enriches raw DOT graphs with stable, Cypher-friendly metadata."""
normalized_graph = graph.__class__()
edge_ids = defaultdict(int)
sequence = 0

for node_id, data in graph.nodes(data=True):
node_properties = self._normalize_dot_properties(data)
node_properties["dot_type"] = "node"
node_properties["display_name"] = node_properties.get("attributes_label", str(node_id))
node_properties["source_graph"] = "dot"
node_properties["sequence"] = sequence
sequence += 1
if "id" not in node_properties:
node_properties["id"] = self._normalize_dot_value(node_id)
normalized_graph.add_node(node_id, **node_properties)

if graph.is_multigraph():
edge_iter = graph.edges(data=True, keys=True)
for source, dest, _edge_key, data in edge_iter:
edge_properties = self._normalize_dot_properties(data)
edge_properties["dot_type"] = "edge"
edge_properties["type"] = "DOT_EDGE"
edge_key = f"{source}->{dest}"
edge_index = edge_ids[edge_key]
edge_ids[edge_key] += 1
edge_id = edge_key if edge_index == 0 else f"{edge_key}#{edge_index}"
edge_properties["id"] = self._normalize_dot_value(edge_id)
edge_properties["points"] = [self._normalize_dot_value(source), self._normalize_dot_value(dest)]
edge_properties["sequence"] = sequence
sequence += 1
normalized_graph.add_edge(source, dest, **edge_properties)
else:
edge_iter = graph.edges(data=True)
for source, dest, data in edge_iter:
edge_properties = self._normalize_dot_properties(data)
edge_properties["dot_type"] = "edge"
edge_properties["type"] = "DOT_EDGE"
edge_key = f"{source}->{dest}"
edge_index = edge_ids[edge_key]
edge_ids[edge_key] += 1
edge_id = edge_key if edge_index == 0 else f"{edge_key}#{edge_index}"
edge_properties["id"] = self._normalize_dot_value(edge_id)
edge_properties["points"] = [self._normalize_dot_value(source), self._normalize_dot_value(dest)]
edge_properties["sequence"] = sequence
sequence += 1
normalized_graph.add_edge(source, dest, **edge_properties)
Comment thread
cursor[bot] marked this conversation as resolved.
Outdated

return normalized_graph

def _graph_from_pydot(self, dot_graph) -> nx.MultiDiGraph:
"""Builds a MultiDiGraph from a pydot graph."""
graph = nx.MultiDiGraph()

def _walk(current_graph) -> None:
for node in current_graph.get_nodes():
node_id = self._normalize_dot_value(node.get_name())
# Ignore pydot/graphviz pseudo-nodes used for defaults.
if node_id in {"", "node", "edge", "graph"}:
continue
Comment thread
mattkjames7 marked this conversation as resolved.

properties = {
self._sanitize_property_key(key): self._normalize_dot_value(value)
for key, value in (node.get_attributes() or {}).items()
}
Comment thread
cursor[bot] marked this conversation as resolved.
Outdated
if node_id in graph.nodes:
graph.nodes[node_id].update(properties)
else:
graph.add_node(node_id, **properties)

for edge in current_graph.get_edges():
source = self._normalize_dot_value(edge.get_source())
dest = self._normalize_dot_value(edge.get_destination())
if not source or not dest:
continue

properties = {
self._sanitize_property_key(key): self._normalize_dot_value(value)
for key, value in (edge.get_attributes() or {}).items()
}
graph.add_edge(source, dest, **properties)

for subgraph in current_graph.get_subgraphs():
_walk(subgraph)

_walk(dot_graph)

return graph

def _normalize_dot_properties(self, properties: Dict[str, Any]) -> Dict[str, Any]:
normalized_attributes = {
self._sanitize_property_key(key): self._normalize_dot_value(value) for key, value in properties.items()
}
normalized_properties: Dict[str, Any] = dict(normalized_attributes)
normalized_properties["attributes_json"] = json.dumps(normalized_attributes, sort_keys=True)

for key, value in normalized_attributes.items():
normalized_properties[f"attributes_{key}"] = value
Comment thread
cursor[bot] marked this conversation as resolved.
Outdated

return normalized_properties

@staticmethod
def _normalize_dot_value(value: Any) -> Any:
if isinstance(value, str):
return value.strip().strip('"')
return value

@staticmethod
def _sanitize_property_key(key: str) -> str:
return re.sub(r"[^0-9A-Za-z_]", "_", key)
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ exclude = '''
python = "^3.9"
pymgclient = "^1.5.1"
networkx = ">=2.5.1,<4.0.0"
pydot = ">=1.4.2,<5.0.0"
Comment thread
cursor[bot] marked this conversation as resolved.
Outdated
pydantic = "^2.3.0"
psutil = ">=5.9,<7.0"
dacite = "^1.6.0"
Expand Down Expand Up @@ -75,7 +76,7 @@ pre-commit = ">=2.15.0,<4.0.0"
[tool.poe.tasks]
install-dgl = "pip install dgl==2.4.0 -f https://data.dgl.ai/wheels/torch-2.4/repo.html --no-build-isolation"
install-pyg-cpu = "pip install 'torch>=2.4.0,<=2.5.0' torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.4.0+cpu.html --no-build-isolation"
install-tfgnn = "pip install tensorflow-gnn>=1.0.0,<2.0.0"
install-tfgnn = "pip install 'tensorflow-gnn>=1.0.0,<2.0.0' 'tf-keras>=2.16,<3.0'"
Comment thread
mattkjames7 marked this conversation as resolved.

[build-system]
requires = ["poetry-core>=1.0.0", "setuptools>=78.1.1"]
Expand Down
78 changes: 78 additions & 0 deletions tests/transformations/importing/test_import.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import json

import pytest

from gqlalchemy.transformations.importing.graph_importer import GraphImporter
Expand Down Expand Up @@ -43,3 +45,79 @@ def test_import_nx():
importer = GraphImporter(graph_type="Nx")
assert isinstance(importer.translator, NxTranslator)
importer.translate(None) # it should fail safely no matter what


def test_import_nx_from_dot_data(monkeypatch):
pytest.importorskip("pydot")

importer = GraphImporter(graph_type="Nx")
captured = {}

def _capture_translate(graph):
captured["graph"] = graph

monkeypatch.setattr(importer, "translate", _capture_translate)

importer.translate_dot_data(
'digraph G { "A" [label="A", shape="ellipse", color="blue"]; "B"; "A" -> "B" [fontsize="10", style="dashed"]; "A" -> "B" [fontsize="10", style="dashed"]; }'
)

assert "graph" in captured
assert captured["graph"].is_multigraph()
assert captured["graph"].number_of_nodes() == 2
assert captured["graph"].number_of_edges() == 2

node_data = captured["graph"].nodes["A"]
assert node_data["dot_type"] == "node"
assert node_data["source_graph"] == "dot"
assert node_data["display_name"] == "A"
assert node_data["attributes_label"] == "A"
assert node_data["attributes_shape"] == "ellipse"
assert node_data["attributes_color"] == "blue"
assert json.loads(node_data["attributes_json"]) == {"color": "blue", "label": "A", "shape": "ellipse"}

node_b_data = captured["graph"].nodes["B"]
assert node_b_data["dot_type"] == "node"
assert node_b_data["display_name"] == "B"
assert json.loads(node_b_data["attributes_json"]) == {}

edge_data_list = [edge_data for *_, edge_data in captured["graph"].edges(data=True)]
assert sorted(edge_data["id"] for edge_data in edge_data_list) == ["A->B", "A->B#1"]
for edge_data in edge_data_list:
assert edge_data["type"] == "DOT_EDGE"
assert edge_data["dot_type"] == "edge"
assert edge_data["points"] == ["A", "B"]
assert edge_data["attributes_fontsize"] == "10"
assert edge_data["attributes_style"] == "dashed"
assert json.loads(edge_data["attributes_json"]) == {"fontsize": "10", "style": "dashed"}

sequences = [data["sequence"] for _, data in captured["graph"].nodes(data=True)] + [
data["sequence"] for *_, data in captured["graph"].edges(data=True)
]
assert sorted(sequences) == [0, 1, 2, 3]


def test_import_nx_from_dot_file(tmp_path, monkeypatch):
pytest.importorskip("pydot")

importer = GraphImporter(graph_type="Nx")
captured = {}
dot_file = tmp_path / "graph.dot"
dot_file.write_text(
'digraph G { "S" [label="S", shape="ellipse"]; "T"; "S" -> "T" [fontsize="10"]; "S" -> "T" [fontsize="10"]; }'
)

def _capture_translate(graph):
captured["graph"] = graph

monkeypatch.setattr(importer, "translate", _capture_translate)

importer.translate_dot_file(str(dot_file))

assert "graph" in captured
assert captured["graph"].is_multigraph()
assert captured["graph"].number_of_nodes() == 2
assert captured["graph"].number_of_edges() == 2
edge_ids = sorted([edge_data["id"] for *_, edge_data in captured["graph"].edges(data=True)])
assert edge_ids == ["S->T", "S->T#1"]
assert captured["graph"].nodes["S"]["source_graph"] == "dot"