Skip to content

Add JSONL input support for tuple import #527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
aaguiarz wants to merge 3 commits into from
Closed

Add JSONL input support for tuple import #527

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3db5535
Add JSONL support for tuple write
aaguiarz Jun 29, 2025
cf53c70
Add condition example to JSONL tuples
aaguiarz Jun 29, 2025
c871f7d
Support .jsonl tuple files
aaguiarz Jun 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -702,7 +702,8 @@ fga tuple **write** <user> <relation> <object> --store-id=<store-id>
* `--condition-context`: Condition context (optional)
* `--store-id`: Specifies the store id
* `--model-id`: Specifies the model id to target (optional)
* `--file`: Specifies the file name, `json`, `yaml` and `csv` files are supported
* `--file`: Specifies the file name, `json`, `jsonl`, `yaml` and `csv` files are supported
* When using a `json` file, tuples can be provided as an array or newline-delimited (`jsonl`) objects
* `--max-tuples-per-write`: Max tuples to send in a single write (optional, default=1, or 40 if `--max-rps` is set and this flag is omitted)
* `--max-parallel-requests`: Max requests to send in parallel (optional, default=4, or `max-rps/5` if `--max-rps` is set and this flag is omitted)
* `--hide-imported-tuples`: When importing from a file, do not output successfully imported tuples in the command output (optional, default=false)
Expand Down Expand Up @@ -772,6 +773,7 @@ If using a `json` file, the format should be:
}
]
```
JSON files can also be provided in JSONL format (use the `.jsonl` extension), with one JSON object per line.

###### Response
```json5
Expand Down
4 changes: 4 additions & 0 deletions cmd/tuple/testdata/tuples_jsonl.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{"user": "user:anne", "relation": "owner", "object": "folder:product"}
{"user": "folder:product", "relation": "parent", "object": "folder:product-2021"}
{"user": "user:beth", "relation": "viewer", "object": "folder:product-2021"}
{"user": "folder:product", "relation": "parent", "object": "folder:product-2021", "condition": {"name": "inOfficeIP", "context": {"ip_addr": "10.0.0.1"}}}
32 changes: 32 additions & 0 deletions cmd/tuple/write_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,38 @@
},
},
{
name: "it can correctly parse a jsonl file",

Check failure on line 152 in cmd/tuple/write_test.go

View workflow job for this annotation

GitHub Actions / Lints 

File is not properly formatted (gofmt)
file: "testdata/tuples_jsonl.jsonl",
expectedTuples: []client.ClientTupleKey{
{
User: "user:anne",
Relation: "owner",
Object: "folder:product",
},
{
User: "folder:product",
Relation: "parent",
Object: "folder:product-2021",
},
{
User: "user:beth",
Relation: "viewer",
Object: "folder:product-2021",
},
{
User: "folder:product",
Relation: "parent",
Object: "folder:product-2021",
Condition: &openfga.RelationshipCondition{
Name: "inOfficeIP",
Context: &map[string]interface{}{
"ip_addr": "10.0.0.1",
},
},
},
},
},
Comment on lines +152 to +182

@coderabbitai coderabbitai Bot Jun 29, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Fix formatting in the test case.

The test case correctly verifies JSONL parsing functionality and includes a tuple with conditions to ensure comprehensive coverage. However, the formatting needs to be corrected.

Apply this diff to fix the formatting:

-		{
-name: "it can correctly parse a jsonl file",
-file: "testdata/tuples_jsonl.jsonl",
+		{
+			name: "it can correctly parse a jsonl file",
+			file: "testdata/tuples_jsonl.jsonl",
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
name: "it can correctly parse a jsonl file",
file: "testdata/tuples_jsonl.jsonl",
expectedTuples: []client.ClientTupleKey{
{
User: "user:anne",
Relation: "owner",
Object: "folder:product",
},
{
User: "folder:product",
Relation: "parent",
Object: "folder:product-2021",
},
{
User: "user:beth",
Relation: "viewer",
Object: "folder:product-2021",
},
{
User: "folder:product",
Relation: "parent",
Object: "folder:product-2021",
Condition: &openfga.RelationshipCondition{
Name: "inOfficeIP",
Context: &map[string]interface{}{
"ip_addr": "10.0.0.1",
},
},
},
},
},
{
name: "it can correctly parse a jsonl file",
file: "testdata/tuples_jsonl.jsonl",
expectedTuples: []client.ClientTupleKey{
{
User: "user:anne",
Relation: "owner",
Object: "folder:product",
},
{
User: "folder:product",
Relation: "parent",
Object: "folder:product-2021",
},
{
User: "user:beth",
Relation: "viewer",
Object: "folder:product-2021",
},
{
User: "folder:product",
Relation: "parent",
Object: "folder:product-2021",
Condition: &openfga.RelationshipCondition{
Name: "inOfficeIP",
Context: &map[string]interface{}{
"ip_addr": "10.0.0.1",
},
},
},
},
},
🧰 Tools
🪛 GitHub Check: Lints

[failure] 152-152:
File is not properly formatted (gofmt)

🤖 Prompt for AI Agents 
In cmd/tuple/write_test.go around lines 152 to 182, the test case formatting is
inconsistent, which affects readability. Adjust the indentation and alignment of
the expectedTuples slice and its nested fields to follow Go formatting
conventions, ensuring proper indentation for struct fields and nested maps to
improve clarity and maintainability.

{
name: "it can correctly parse a yaml file",
file: "testdata/tuples.yaml",
expectedTuples: []client.ClientTupleKey{
Expand Down
49 changes: 43 additions & 6 deletions internal/tuplefile/read.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"github.com/openfga/cli/internal/clierrors"
)

func ReadTupleFile(fileName string) ([]client.ClientTupleKey, error) {

Check failure on line 15 in internal/tuplefile/read.go

View workflow job for this annotation

GitHub Actions / Lints 

calculated cyclomatic complexity for function ReadTupleFile is 15, max is 10 (cyclop)
data, err := os.ReadFile(fileName)
if err != nil {
return nil, fmt.Errorf("failed to read file %q: %w", fileName, err)
Expand All @@ -20,12 +20,26 @@

var tuples []client.ClientTupleKey

switch path.Ext(fileName) {
case ".json", ".yaml", ".yml":
err = yaml.Unmarshal(data, &tuples)
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError(strings.TrimPrefix(path.Ext(fileName), "."))
}
ext := path.Ext(fileName)

Check failure on line 23 in internal/tuplefile/read.go

View workflow job for this annotation

GitHub Actions / Lints 

File is not properly formatted (gofmt)
switch ext {
case ".json":
err = yaml.Unmarshal(data, &tuples)
if err != nil {
err = parseTuplesFromJSONL(data, &tuples, "json")
}
if err == nil && len(tuples) == 0 {

Check failure on line 30 in internal/tuplefile/read.go

View workflow job for this annotation

GitHub Actions / Lints 

if statements should only be cuddled with assignments (wsl)
err = clierrors.EmptyTuplesFileError("json")
}
case ".jsonl":
err = parseTuplesFromJSONL(data, &tuples, "jsonl")
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError("jsonl")
}
case ".yaml", ".yml":
err = yaml.Unmarshal(data, &tuples)
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError(strings.TrimPrefix(ext, "."))
}
Comment on lines 23 to +42

@coderabbitai coderabbitai Bot Jun 29, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix formatting and improve error handling logic.

The implementation correctly handles both JSON array and JSONL formats for .json files, and adds proper support for .jsonl files. However, there are several issues to address:

  1. Formatting: The code has inconsistent indentation that needs to be fixed with gofmt
  2. Logic improvement: Consider reordering the empty check for better clarity

Apply this diff to fix the formatting and improve the logic:

-       ext := path.Ext(fileName)
-       switch ext {
-       case ".json":
-               err = yaml.Unmarshal(data, &tuples)
-               if err != nil {
-                       err = parseTuplesFromJSONL(data, &tuples, "json")
-               }
-               if err == nil && len(tuples) == 0 {
-                       err = clierrors.EmptyTuplesFileError("json")
-               }
-       case ".jsonl":
-               err = parseTuplesFromJSONL(data, &tuples, "jsonl")
-               if err == nil && len(tuples) == 0 {
-                       err = clierrors.EmptyTuplesFileError("jsonl")
-               }
-       case ".yaml", ".yml":
-               err = yaml.Unmarshal(data, &tuples)
-               if err == nil && len(tuples) == 0 {
-                       err = clierrors.EmptyTuplesFileError(strings.TrimPrefix(ext, "."))
-               }
+	ext := path.Ext(fileName)
+	switch ext {
+	case ".json":
+		err = yaml.Unmarshal(data, &tuples)
+		if err != nil {
+			err = parseTuplesFromJSONL(data, &tuples, "json")
+		}
+		if err == nil && len(tuples) == 0 {
+			err = clierrors.EmptyTuplesFileError("json")
+		}
+	case ".jsonl":
+		err = parseTuplesFromJSONL(data, &tuples, "jsonl")
+		if err == nil && len(tuples) == 0 {
+			err = clierrors.EmptyTuplesFileError("jsonl")
+		}
+	case ".yaml", ".yml":
+		err = yaml.Unmarshal(data, &tuples)
+		if err == nil && len(tuples) == 0 {
+			err = clierrors.EmptyTuplesFileError(strings.TrimPrefix(ext, "."))
+		}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
ext := path.Ext(fileName)
switch ext {
case ".json":
err = yaml.Unmarshal(data, &tuples)
if err != nil {
err = parseTuplesFromJSONL(data, &tuples, "json")
}
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError("json")
}
case ".jsonl":
err = parseTuplesFromJSONL(data, &tuples, "jsonl")
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError("jsonl")
}
case ".yaml", ".yml":
err = yaml.Unmarshal(data, &tuples)
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError(strings.TrimPrefix(ext, "."))
}
ext := path.Ext(fileName)
switch ext {
case ".json":
err = yaml.Unmarshal(data, &tuples)
if err != nil {
err = parseTuplesFromJSONL(data, &tuples, "json")
}
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError("json")
}
case ".jsonl":
err = parseTuplesFromJSONL(data, &tuples, "jsonl")
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError("jsonl")
}
case ".yaml", ".yml":
err = yaml.Unmarshal(data, &tuples)
if err == nil && len(tuples) == 0 {
err = clierrors.EmptyTuplesFileError(strings.TrimPrefix(ext, "."))
}
}
🧰 Tools
🪛 GitHub Check: Lints

[failure] 30-30:
if statements should only be cuddled with assignments (wsl)

[failure] 23-23:
File is not properly formatted (gofmt)

🤖 Prompt for AI Agents 
In internal/tuplefile/read.go around lines 23 to 42, fix the inconsistent
indentation by running gofmt on the code to ensure proper formatting.
Additionally, improve the error handling logic by checking for empty tuples
immediately after unmarshalling or parsing, before attempting alternative
parsing methods, to make the flow clearer and avoid redundant checks.

case ".csv":
err = parseTuplesFromCSV(data, &tuples)
default:
Expand All @@ -38,3 +52,26 @@

return tuples, nil
}

func parseTuplesFromJSONL(data []byte, tuples *[]client.ClientTupleKey, format string) error {
lines := strings.Split(strings.TrimSpace(string(data)), "\n")
for index, line := range lines {
trimmed := strings.TrimSpace(line)
if trimmed == "" {
continue
}

var tuple client.ClientTupleKey
if err := yaml.Unmarshal([]byte(trimmed), &tuple); err != nil {
return fmt.Errorf("failed to parse tuple on line %d: %w", index+1, err)
}

*tuples = append(*tuples, tuple)
}

if len(*tuples) == 0 {
return clierrors.EmptyTuplesFileError(format)

Check failure on line 73 in internal/tuplefile/read.go

View workflow job for this annotation

GitHub Actions / Lints 

error returned from external package is unwrapped: sig: func github.com/openfga/cli/internal/clierrors.EmptyTuplesFileError(extName string) error (wrapcheck)
}

return nil
}
Comment on lines 56 to 77

@coderabbitai coderabbitai Bot Jun 29, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Remove duplicate empty check and fix error wrapping.

The parseTuplesFromJSONL function has good line-by-line error reporting, but contains redundant empty file checking and an unwrapped error.

Apply this diff to fix the issues:

 func parseTuplesFromJSONL(data []byte, tuples *[]client.ClientTupleKey, format string) error {
 	lines := strings.Split(strings.TrimSpace(string(data)), "\n")
 	for index, line := range lines {
 		trimmed := strings.TrimSpace(line)
 		if trimmed == "" {
 			continue
 		}

 		var tuple client.ClientTupleKey
 		if err := yaml.Unmarshal([]byte(trimmed), &tuple); err != nil {
 			return fmt.Errorf("failed to parse tuple on line %d: %w", index+1, err)
 		}

 		*tuples = append(*tuples, tuple)
 	}

-	if len(*tuples) == 0 {
-		return clierrors.EmptyTuplesFileError(format)
-	}
-
 	return nil
 }

The empty file check is already handled by the caller, making this redundant. Also, consider using json.Unmarshal instead of yaml.Unmarshal for JSONL parsing to be more explicit about the expected format.

Committable suggestion skipped: line range outside the PR's diff.

🧰 Tools
🪛 GitHub Check: Lints

[failure] 73-73:
error returned from external package is unwrapped: sig: func github.com/openfga/cli/internal/clierrors.EmptyTuplesFileError(extName string) error (wrapcheck)

🤖 Prompt for AI Agents 
In internal/tuplefile/read.go lines 56 to 77, remove the redundant empty tuples
check at the end of parseTuplesFromJSONL since the caller already handles empty
file validation. Replace yaml.Unmarshal with json.Unmarshal to explicitly parse
JSON lines, and ensure error wrapping uses fmt.Errorf with %w for proper error
propagation.

Loading