Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ jobs:
- name: Install More Packages
run: |
ps aux | grep memgraph
poetry install --all-extras
poetry install -E arrow -E dgl -E docker -E dot
poe install-pyg-cpu
poe install-dgl
poe install-tfgnn
Expand Down Expand Up @@ -148,8 +148,9 @@ jobs:
poetry-version: ${{ env.POETRY_VERSION }}
- name: Test project
run: |
poetry install --all-extras
poetry install -E arrow -E dgl -E docker -E dot
poe install-pyg-cpu
poe install-dgl
poe install-tfgnn
export TF_USE_LEGACY_KERAS="1"
poetry run pytest -vvv -m "not slow and not ubuntu and not docker"
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/build-packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,9 +122,10 @@ jobs:
- name: Install wheel and dependencies
run: |
python -m pip install dist/*.whl
poetry install --all-extras
poetry install -E arrow -E dgl -E docker -E dot
poe install-pyg-cpu
poe install-dgl
poe install-tfgnn

- name: Run Tests
run: |
Expand Down
32 changes: 32 additions & 0 deletions docs/how-to-guides/translators/import-python-graphs.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ hood](https://img.shields.io/static/v1?label=Related&message=Under%20the%20hood&
In this guide you will learn how to:

- [**Import NetworkX graph into Memgraph**](#import-networkx-graph-into-memgraph)
- [**Import DOT graph into Memgraph**](#import-dot-graph-into-memgraph)
- [**Import PyG graph into Memgraph**](#import-pyg-graph-into-memgraph)
- [**Import DGL graph into Memgraph**](#import-dgl-graph-into-memgraph)
- [**Import TF-GNN graph into Memgraph**](#import-tf-gnn-graph-into-memgraph)
Expand Down Expand Up @@ -80,6 +81,37 @@ Click **Run Query** button to see the results.

The NetworkX node identification number maps to the `id` node property in Memgraph. The `labels` key is reserved for the node label in Memgraph, while the edge `type` key is reserved for the relationship type in Memgraph. If no `type` is defined, then the relationship will be of type `TO` in Memgraph. You can notice that the node with the property `name` Kata and property `id` 2 doesn't have a label. This happened because the node property key `labels` was not defined.

## Import DOT graph into Memgraph

You can import DOT files by using `GraphImporter` with `graph_type="NX"`. DOT parsing uses `pydot` and NetworkX under the hood.

### Prerequisites

Except for the [**general prerequisites**](#general-prerequisites), install DOT parsing support:

```bash
pip install gqlalchemy[dot]
```

### Create and run a Python script

Create a new Python script `dot-graph.py` with the following code:

```python
from gqlalchemy.transformations.importing.graph_importer import GraphImporter

importer = GraphImporter(graph_type="NX")

# Import from a DOT file path.
importer.translate_dot_file("graph.dot")

# Or import directly from DOT content.
dot_data = 'digraph G { "A" [label="A"]; "B"; "A" -> "B" [fontsize="10"]; }'
importer.translate_dot_data(dot_data)
```

During DOT import, GQLAlchemy enriches parsed nodes and edges with DOT-specific metadata (for example `dot_type`, `attributes_json`, flattened `attributes_*` keys, `sequence`, and stable edge `id` values).

## Import PyG graph into Memgraph

### Prerequisites
Expand Down
1 change: 1 addition & 0 deletions docs/import-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ To import Parquet, ORC or IPC/Feather/Arrow file into Memgraph via GQLAlchemy, [
## Python graphs - NetworkX, PyG or DGL graph

To import NetworkX, PyG or DGL graph into Memgraph via GQLAlchemy, [transform the source graph into Memgraph graph](how-to-guides/translators/import-python-graphs.md).
DOT files and DOT strings are also supported through the NetworkX importer (`GraphImporter(graph_type="NX")`) with `translate_dot_file(...)` and `translate_dot_data(...)`.

## Kafka, RedPanda or Pulsar data stream

Expand Down
2 changes: 2 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ additional import/export capabilities, use one of the following install options:
```bash
pip install gqlalchemy[arrow] # Support for the CSV, Parquet, ORC and IPC/Feather/Arrow formats
pip install gqlalchemy[dgl] # DGL support (also includes torch)
pip install gqlalchemy[dot] # DOT graph import support (pydot)
pip install gqlalchemy[docker] # Docker support

pip install gqlalchemy[all] # All of the above
Expand Down Expand Up @@ -70,6 +71,7 @@ poetry install # No extras

poetry install -E arrow # Support for the CSV, Parquet, ORC and IPC/Feather/Arrow formats
poetry install -E dgl # DGL support (also includes torch)
poetry install -E dot # DOT graph import support (pydot)
poetry install -E docker # Docker support

```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,27 @@ Gets cypher queries using the underlying translator and then inserts all queries

- `graph` - dgl, pytorch geometric or nx graph instance.

#### translate_dot_file

```python
def translate_dot_file(path: str) -> None
```

Parses a DOT file into a NetworkX graph and imports it to Memgraph. This method is available when `graph_type="NX"`.

**Arguments**:

- `path` - Path to a DOT file.

#### translate_dot_data

```python
def translate_dot_data(dot_data: str) -> None
```

Parses DOT content from a string into a NetworkX graph and imports it to Memgraph. This method is available when `graph_type="NX"`.

**Arguments**:

- `dot_data` - Raw DOT graph content.

146 changes: 145 additions & 1 deletion gqlalchemy/transformations/importing/graph_importer.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@

from gqlalchemy.exceptions import raise_if_not_imported
import gqlalchemy.memgraph_constants as mg_consts
import networkx as nx
import json
import re
from collections import defaultdict
from typing import Any, Dict

try:
from gqlalchemy.transformations.translators.dgl_translator import DGLTranslator
Expand All @@ -31,6 +36,11 @@
except ModuleNotFoundError:
PyGTranslator = None

try:
import pydot
except ModuleNotFoundError:
pydot = None


class GraphImporter(Importer):
"""Imports dgl, pyg or networkx graph representations to Memgraph.
Comment thread
mattkjames7 marked this conversation as resolved.
Expand Down Expand Up @@ -69,8 +79,142 @@ def __init__(
def translate(self, graph) -> None:
"""Gets cypher queries using the underlying translator and then inserts all queries to Memgraph DB.
Args:
graph: dgl, pytorch geometric or nx graph instance.
graph: dgl, pytorch geometric, dot or nx graph instance.
"""
memgraph = Memgraph()
for query in self.translator.to_cypher_queries(graph):
memgraph.execute(query)

def translate_dot_file(self, path: str) -> None:
"""Parses a DOT file to a NetworkX graph and imports it to Memgraph."""
self._raise_if_not_nx_importer()
raise_if_not_imported(dependency=pydot, dependency_name="pydot")

pydot_graphs = pydot.graph_from_dot_file(path)
if not pydot_graphs:
raise ValueError("Unable to parse DOT file.")

graph = self._normalize_dot_graph(self._graph_from_pydot(pydot_graphs[0]))
self.translate(graph)

def translate_dot_data(self, dot_data: str) -> None:
"""Parses DOT content to a NetworkX graph and imports it to Memgraph."""
self._raise_if_not_nx_importer()
raise_if_not_imported(dependency=pydot, dependency_name="pydot")

pydot_graphs = pydot.graph_from_dot_data(dot_data)
if not pydot_graphs:
raise ValueError("Unable to parse DOT data.")

graph = self._normalize_dot_graph(self._graph_from_pydot(pydot_graphs[0]))
self.translate(graph)

def _raise_if_not_nx_importer(self) -> None:
if self.graph_type != GraphType.NX.name:
raise ValueError("DOT import is supported only for NetworkX graph importer.")

def _normalize_dot_graph(self, graph: nx.Graph) -> nx.Graph:
"""Enriches raw DOT graphs with stable, Cypher-friendly metadata."""
normalized_graph = graph.__class__()
edge_ids = defaultdict(int)
sequence = 0

for node_id, data in graph.nodes(data=True):
node_properties = self._normalize_dot_properties(data)
node_properties["dot_type"] = "node"
node_properties["display_name"] = node_properties.get("attributes_label", str(node_id))
node_properties["source_graph"] = "dot"
node_properties["sequence"] = sequence
sequence += 1
if "id" not in node_properties:
node_properties["id"] = self._normalize_dot_value(node_id)
normalized_graph.add_node(node_id, **node_properties)

for source, dest, data in graph.edges(data=True):
edge_properties = self._normalize_dot_properties(data)
edge_properties["dot_type"] = "edge"
edge_properties["type"] = "DOT_EDGE"
edge_key = f"{source}->{dest}"
edge_index = edge_ids[edge_key]
edge_ids[edge_key] += 1
edge_id = edge_key if edge_index == 0 else f"{edge_key}#{edge_index}"
edge_properties["id"] = self._normalize_dot_value(edge_id)
Comment thread
mattkjames7 marked this conversation as resolved.
edge_properties["points"] = [self._normalize_dot_value(source), self._normalize_dot_value(dest)]
edge_properties["sequence"] = sequence
sequence += 1
normalized_graph.add_edge(source, dest, **edge_properties)

return normalized_graph

def _graph_from_pydot(self, dot_graph) -> nx.MultiDiGraph:
"""Builds a MultiDiGraph from a pydot graph."""
graph = nx.MultiDiGraph()

def _walk(current_graph) -> None:
for node in current_graph.get_nodes():
node_id = self._normalize_dot_value(node.get_name())
# Ignore pydot/graphviz pseudo-nodes used for defaults.
if node_id in {"", "node", "edge", "graph"}:
continue
Comment thread
mattkjames7 marked this conversation as resolved.

properties = self._normalize_dot_attributes(node.get_attributes() or {})
if node_id in graph.nodes:
graph.nodes[node_id].update(properties)
else:
graph.add_node(node_id, **properties)

for edge in current_graph.get_edges():
source = self._normalize_dot_value(edge.get_source())
dest = self._normalize_dot_value(edge.get_destination())
if not source or not dest:
continue

properties = self._normalize_dot_attributes(edge.get_attributes() or {})
graph.add_edge(source, dest, **properties)

for subgraph in current_graph.get_subgraphs():
_walk(subgraph)

_walk(dot_graph)

return graph

def _normalize_dot_properties(self, properties: Dict[str, Any]) -> Dict[str, Any]:
normalized_attributes = self._normalize_dot_attributes(properties)
normalized_properties: Dict[str, Any] = dict(normalized_attributes)
normalized_properties["attributes_json"] = json.dumps(normalized_attributes, sort_keys=True)

for key, value in normalized_attributes.items():
prefixed_key = f"attributes_{key}"
while prefixed_key in normalized_properties:
prefixed_key = f"{prefixed_key}_attribute"
normalized_properties[prefixed_key] = value

return normalized_properties

def _normalize_dot_attributes(self, attributes: Dict[str, Any]) -> Dict[str, Any]:
normalized_attributes: Dict[str, Any] = {}
for key, value in attributes.items():
sanitized_key = self._sanitize_property_key(key)
unique_key = self._resolve_key_collision(sanitized_key, normalized_attributes)
normalized_attributes[unique_key] = self._normalize_dot_value(value)
return normalized_attributes

@staticmethod
def _resolve_key_collision(base_key: str, properties: Dict[str, Any]) -> str:
unique_key = base_key
suffix = 1
while unique_key in properties:
unique_key = f"{base_key}_{suffix}"
suffix += 1
return unique_key

@staticmethod
def _normalize_dot_value(value: Any) -> Any:
if isinstance(value, str):
return value.strip().strip('"')
return value

@staticmethod
def _sanitize_property_key(key: str) -> str:
return re.sub(r"[^0-9A-Za-z_]", "_", key)
6 changes: 4 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ exclude = '''
python = "^3.9"
pymgclient = "^1.5.1"
networkx = ">=2.5.1,<4.0.0"
pydot = { version = ">=1.4.2,<5.0.0", optional = true }
pydantic = "^2.3.0"
psutil = ">=5.9,<7.0"
dacite = "^1.6.0"
Expand All @@ -58,7 +59,8 @@ priority = "explicit"
arrow = ["pyarrow"]
tfgnn = ["tensorflow-gnn", "tensorflow-macos", "tensorflow", "tf-keras"]
dgl = ["torch", "dgl"]
all = ["pyarrow", "torch", "dgl", "docker", "tensorflow-gnn"]
dot = ["pydot"]
all = ["pyarrow", "torch", "dgl", "docker", "tensorflow-gnn", "pydot"]
torch_pyg = ["torch"]
docker = ["docker"]

Expand All @@ -75,7 +77,7 @@ pre-commit = ">=2.15.0,<4.0.0"
[tool.poe.tasks]
install-dgl = "pip install dgl==2.4.0 -f https://data.dgl.ai/wheels/torch-2.4/repo.html --no-build-isolation"
install-pyg-cpu = "pip install 'torch>=2.4.0,<=2.5.0' torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.4.0+cpu.html --no-build-isolation"
install-tfgnn = "pip install tensorflow-gnn>=1.0.0,<2.0.0"
install-tfgnn = "pip install 'tensorflow-gnn>=1.0.0,<2.0.0' 'tf-keras>=2.16,<3.0'"

[build-system]
requires = ["poetry-core>=1.0.0", "setuptools>=78.1.1"]
Expand Down
Loading