Add DirectoryListener by dr0i · Pull Request #764 · metafacture/metafacture-core

dr0i · 2026-04-14T14:38:34Z

This PR works in principle.
Be aware that there is a bug regarding the WatchService that can result in a loss of awareness of created files, e.g. if these files are created too fast in a row, or e.g. by moving a whole directory with files in it to the directory which is listened to: that directory would be watched, but the files in there won't be recognized. So it's not as inotify in unix contexts - it only comes close to it.

You may want to test it like this:

./gradlew assembleDist (in the root of this branch to build the runner)
cd ./metafacture-runner/build/distributions/
tar xfz metafacture-core-702-addDirectoryListener-SNAPSHOT-dist.tar.gz
create a FLUX:

echo "default infile = FLUX_DIR + 'tmp';
infile|
listen-directory|
print
;"  > directoryListener.flux

mkdir tmp (creates the directory to listen to)
metafacture-core-702-addDirectoryListener-SNAPSHOT-dist/flux.sh directoryListener.flux to execute the FLUX
touch 1 2 3 4 5 6 7 tmp/ to create some files

You should see as output the names of the files with absolute path (which could be given in the FLUX to open-file) and some logs (which are not going to the piped flux-command (as e.g. open-file)) are printed to stdout.
(Interestingly , you see that even "touch"ing 7 files consecutively the WatchService is fast enough to observe the creation of these files.)

If you want the listener to go down: trigger it with the specially named file:
touch shutdownEtlNow

We may want to discuss if it's necessary to improve the behaviour by building some workarounds. One idea would be to just traverse the given directory every n-th second and notate the filenames, if any new appear, to a Map and push these down the pipe. This would guarantee to not miss one file (at the cost of not instantly getting the filename if one was created.)

dr0i · 2026-04-14T14:41:07Z

functional review: @TobiasNx (and maybe @fsteeg as this PR is supposed to be part of a workflow in the RPB context).
discussion (re bug and workaround): also @blackwinter
code review: @blackwinter (or @fsteeg )

TobiasNx · 2026-04-29T13:23:43Z

@fsteeg and I tested it. I followed your example:

default infile = FLUX_DIR + 'tmp';
infile|
listen-directory|
print
;

Your example flux with printing seems to work fine when creating or modifing files. The logging is a little too much.

tobias@hbz-hp:~/temp$ '/home/tobias/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh' directoryListener.flux
Add directory to watch: /home/tobias/temp/tmp
Event kind:ENTRY_CREATE. File affected: 1.
/home/tobias/temp/tmp/1Event kind:ENTRY_MODIFY. File affected: 1.

/home/tobias/temp/tmp/1Event kind:ENTRY_CREATE. File affected: 2.

/home/tobias/temp/tmp/2Event kind:ENTRY_MODIFY. File affected: 2.

/home/tobias/temp/tmp/2Event kind:ENTRY_CREATE. File affected: 3.

/home/tobias/temp/tmp/3Event kind:ENTRY_MODIFY. File affected: 3.

/home/tobias/temp/tmp/3Event kind:ENTRY_CREATE. File affected: 4.

/home/tobias/temp/tmp/4Event kind:ENTRY_MODIFY. File affected: 4.

/home/tobias/temp/tmp/4Event kind:ENTRY_CREATE. File affected: 5.

/home/tobias/temp/tmp/5Event kind:ENTRY_MODIFY. File affected: 5.

/home/tobias/temp/tmp/5Event kind:ENTRY_CREATE. File affected: 6.

/home/tobias/temp/tmp/6Event kind:ENTRY_MODIFY. File affected: 6.

/home/tobias/temp/tmp/6Event kind:ENTRY_CREATE. File affected: 7.

/home/tobias/temp/tmp/7Event kind:ENTRY_MODIFY. File affected: 7.

/home/tobias/temp/tmp/7Event kind:ENTRY_CREATE. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_MODIFY. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_CREATE. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_MODIFY. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_MODIFY. File affected: 1.

/home/tobias/temp/tmp/1Event kind:ENTRY_CREATE. File affected: shutdownEtlNow.
Event kind:ENTRY_MODIFY. File affected: shutdownEtlNow.

When changing print to write the process closes and the flux breaks when a file is modified or created.

tobias@hbz-hp:~/temp$ '/home/tobias/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh' directoryListener.flux
Add directory to watch: /home/tobias/temp/tmp
Event kind:ENTRY_CREATE. File affected: .1.swp.
Exception in thread "Thread-0" org.metafacture.framework.MetafactureException: java.io.IOException: Stream closed
	at org.metafacture.io.ObjectFileWriter.process(ObjectFileWriter.java:110)
	at org.metafacture.io.ObjectWriter.process(ObjectWriter.java:147)
	at org.metafacture.io.DirectoryListener$DirectoryWatcher.processFile(DirectoryListener.java:195)
	at org.metafacture.io.DirectoryListener$DirectoryWatcher.run(DirectoryListener.java:166)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: Stream closed
	at java.base/sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:51)
	at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
	at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:142)
	at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:223)
	at java.base/java.io.Writer.write(Writer.java:249)
	at org.metafacture.io.ObjectFileWriter.process(ObjectFileWriter.java:101)
	... 4 more

What I currently also do not understand is how I am able to recognize the filenames to further process the changed or created files. What is the output of listen-directory?

dr0i · 2026-06-16T08:08:20Z

What I currently also do not understand is how I am able to recognize the filenames to further process the changed or created files. What is the output of listen-directory?

Is there something missing in the description, something unclear? Did you read it? It begins with:
"Listens to a directory and passes occurring filenames to the receiver."

When changing print to write the process closes and the flux breaks when a file is modified or created.

Did you initialize writer correctly by given a path as destination? How exactly is your flux looking like?

TobiasNx · 2026-06-18T13:58:43Z

I added my test as runner example, so that you can check this: 759ce79

By the way, if the intended directory is a file instead the listener seems also to run. I am not sure if that is intended.

Documentation is fine. +1 for this.

TobiasNx · 2026-06-18T15:26:09Z

@dr0i unfortunatly you forced-pushed over my commit: 759ce79

dr0i · 2026-06-18T15:29:36Z

@TobiasNx can you add your test again - I've force pushed over it.

By the way, if the intended directory is a file instead the listener seems also to run.

I don't understand this. Do you mean, you create a directory and it's passed to receiver? That shouldn't happen!

- improve doc - test removing a file

TobiasNx · 2026-06-18T15:32:36Z

If you use instead mkdir tmp touch tmp so that no directory but a tmp file is created. Then run the script (but with print.

dr0i · 2026-06-18T15:39:47Z

If you use instead mkdir tmp touch tmp so that no directory but a tmp file is created

That's the expected behaviour - if you mean that this filename is passed. Because it is a filename, isnt' it?

TobiasNx · 2026-06-18T15:40:11Z

@@ -0,0 +1,5 @@
+default infile = FLUX_DIR + "tmp";


This path should point to a directory but it seems that it does not through an error if it points to a file instead of a directory.

I thought that this should be tmp/ and breaks if it points to a file.

#775 this pr shows the scenario when the input is a file-path instead of an directory path

TobiasNx · 2026-06-18T15:40:47Z

+default infile = FLUX_DIR + "tmp";
+infile|
+listen-directory|
+write (FLUX_DIR + "result.txt")


I would change this to print before merging. write is currently only used to highlight issue with the directory listener mentioned here: #764 (comment)

TobiasNx · 2026-06-18T15:41:42Z

This example should highlight the problem with the writing that I reported mentioned here: #764 (comment)

TobiasNx · 2026-06-18T17:44:21Z

That's the expected behaviour - if you mean that this filename is passed. Because it is a filename, isnt' it?

@dr0i see my inline comment and the upbranched PR #775

I would expect that if the input is a filepath instead of an directory path that an exception is thrown.

Also see my inline comment with regard to the problem when writing.

TobiasNx · 2026-06-19T09:42:24Z

In order to reduce the noise, I provide a structured list of all of my findings:

"What is the output of listen-directory?" -> @dr0i hinted to the documentation. So this is okay. ✔️
Default logging when using your example seems to be a little too much. Maybe okay:

Logging

tobias@hbz-hp:~/temp$ '/home/user/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh' directoryListener.flux
Add directory to watch: /home/tobias/temp/tmp
Event kind:ENTRY_CREATE. File affected: 1.
/home/tobias/temp/tmp/1Event kind:ENTRY_MODIFY. File affected: 1.

/home/tobias/temp/tmp/1Event kind:ENTRY_CREATE. File affected: 2.

/home/tobias/temp/tmp/2Event kind:ENTRY_MODIFY. File affected: 2.

/home/tobias/temp/tmp/2Event kind:ENTRY_CREATE. File affected: 3.

/home/tobias/temp/tmp/3Event kind:ENTRY_MODIFY. File affected: 3.

/home/tobias/temp/tmp/3Event kind:ENTRY_CREATE. File affected: 4.

/home/tobias/temp/tmp/4Event kind:ENTRY_MODIFY. File affected: 4.

/home/tobias/temp/tmp/4Event kind:ENTRY_CREATE. File affected: 5.

/home/tobias/temp/tmp/5Event kind:ENTRY_MODIFY. File affected: 5.

/home/tobias/temp/tmp/5Event kind:ENTRY_CREATE. File affected: 6.

/home/tobias/temp/tmp/6Event kind:ENTRY_MODIFY. File affected: 6.

/home/tobias/temp/tmp/6Event kind:ENTRY_CREATE. File affected: 7.

/home/tobias/temp/tmp/7Event kind:ENTRY_MODIFY. File affected: 7.

/home/tobias/temp/tmp/7Event kind:ENTRY_CREATE. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_MODIFY. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_CREATE. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_MODIFY. File affected: .1.swp.

/home/tobias/temp/tmp/.1.swpEvent kind:ENTRY_MODIFY. File affected: 1.

/home/tobias/temp/tmp/1Event kind:ENTRY_CREATE. File affected: shutdownEtlNow.
Event kind:ENTRY_MODIFY. File affected: shutdownEtlNow.

Using write after listen-directory breaks as in my added runner example: 759ce79

Exception

tobias@hbz-hp:~/temp$ '/home/tobias/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh' directoryListener.flux
Add directory to watch: /home/tobias/temp/tmp
Event kind:ENTRY_CREATE. File affected: .1.swp.
Exception in thread "Thread-0" org.metafacture.framework.MetafactureException: java.io.IOException: Stream closed
at org.metafacture.io.ObjectFileWriter.process(ObjectFileWriter.java:110)
at org.metafacture.io.ObjectWriter.process(ObjectWriter.java:147)
at org.metafacture.io.DirectoryListener$DirectoryWatcher.processFile(DirectoryListener.java:195)
at org.metafacture.io.DirectoryListener$DirectoryWatcher.run(DirectoryListener.java:166)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: Stream closed
at java.base/sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:51)
at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:142)
at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:223)
at java.base/java.io.Writer.write(Writer.java:249)
at org.metafacture.io.ObjectFileWriter.process(ObjectFileWriter.java:101)
... 4 more

If path for the listen-directory points to a non existing directory [as it would be if the path points to a file instead of an directory] the listener starts. In my opinion it should throw an exception that the directory is missing.
Additionally: I could improve the directory listener runner example after you finished working on this PR.

dr0i requested review from TobiasNx and fsteeg April 14, 2026 14:41

dr0i assigned fsteeg and TobiasNx Apr 14, 2026

dr0i linked an issue Apr 14, 2026 that may be closed by this pull request

Add possibility to listen on a directory #702

Open

fsteeg removed request for TobiasNx and fsteeg April 29, 2026 13:32

fsteeg assigned dr0i and unassigned fsteeg and TobiasNx Apr 29, 2026

dr0i requested a review from TobiasNx June 16, 2026 08:08

dr0i assigned TobiasNx and unassigned dr0i Jun 16, 2026

blackwinter reviewed Jun 16, 2026

View reviewed changes

Comment thread metafacture-io/src/main/java/org/metafacture/io/DirectoryListener.java

TobiasNx assigned dr0i and unassigned TobiasNx Jun 18, 2026

Add DirectoryListener (#702)

fd4423d

dr0i force-pushed the 702-addDirectoryListener branch from 759ce79 to 83ffeee Compare June 18, 2026 15:20

Use MetafactureLogger (#702)

87f9be4

- improve doc - test removing a file

dr0i force-pushed the 702-addDirectoryListener branch from 83ffeee to 87f9be4 Compare June 18, 2026 15:31

Add runner example #702

236e7f2

TobiasNx reviewed Jun 18, 2026

View reviewed changes

dr0i assigned TobiasNx and unassigned dr0i Jun 18, 2026

TobiasNx reviewed Jun 18, 2026

View reviewed changes

TobiasNx assigned dr0i and unassigned TobiasNx Jun 22, 2026

Uh oh!

Conversation

dr0i commented Apr 14, 2026 • edited by TobiasNx Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr0i commented Apr 14, 2026

Uh oh!

TobiasNx commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr0i commented Jun 16, 2026

Uh oh!

Uh oh!

TobiasNx commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx commented Jun 18, 2026

Uh oh!

dr0i commented Jun 18, 2026

Uh oh!

TobiasNx commented Jun 18, 2026

Uh oh!

dr0i commented Jun 18, 2026

Uh oh!

TobiasNx Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

TobiasNx Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

TobiasNx Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TobiasNx Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TobiasNx commented Jun 18, 2026

Uh oh!

TobiasNx commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dr0i commented Apr 14, 2026 •

edited by TobiasNx

Loading

TobiasNx commented Apr 29, 2026 •

edited

Loading

TobiasNx commented Jun 18, 2026 •

edited

Loading

TobiasNx Jun 18, 2026 •

edited

Loading

TobiasNx Jun 18, 2026 •

edited

Loading

TobiasNx commented Jun 19, 2026 •

edited

Loading