Implement with-out-str

[djblue]

This PR is a POC to support the following forms:

(with-out-str (println "hi"))
(binding [*out* *err*] (println "hi"))

Is this approach what we want for jank?

[jeaye]

Hmmm, ostreams are notoriously slow. That's why we don't use them for printing/formatting right now. I would prefer to explore different options which instead redirect stdout/stderr at a lower level. I'm not sure yet exactly how that look, but C folks are able to do this, so it must be possible.

[djblue]

Ohh, I didn't know std::ostream had performance problems 😅 Should our solution be based on jtl::string_builder instead?

[jeaye]

Ohh, I didn't know std::ostream had performance problems 😅 Should our solution be based on jtl::string_builder instead?

No, string_builder is not used for output. It's just for formatting. I think our solution should be based on dup. There's some interesting discussion here: https://stackoverflow.com/questions/5419356/is-there-a-way-to-redirect-stdout-stderr-to-a-string

[djblue]

Do we need that generic of a solution? I feel like since with-out-str only intercepts calls to clojure.core print functions, we can get away with something more specific. For example, in Clojure the following is what happens when you use Java's System/out directly:

% clj
Clojure 1.12.2
user=> (with-out-str (.println System/out "foo"))
foo
""

[jeaye]

Do we need that generic of a solution? I feel like since with-out-str only intercepts calls to clojure.core print functions, we can get away with something more specific. For example, in Clojure the following is what happens when you use Java's System/out direction:

% clj
Clojure 1.12.2
user=> (with-out-str (.println System/out "foo"))
foo
""

Ok, I chewed on this for a bit. I think we have two options here, depending on how robust we want things to be.

Option 1: If we only want to support with-out-str and not the ability to bind arbitrary things to *out* and *err*, we can use a jtl::string_builder. If *out* is non-nil, we expect it holds an opaque box with a string builder in it and we just write to that. Otherwise, we write as we normally would, with fwrite and putc.

Option 2: To support whole-program capturing and also the ability to specify custom "writers", we can have *out* hold the actual file number for stdout and have *err* hold stderr. Then we just fwrite to those. Later, we can provide a jank API for people to create their own file handles for easy plugging into our system. This not only captures everything, it also enables the full suite of output redirection, using POSIX file descriptors.

[jeaye]

If we do option 1, it'll be a stop-gap solution until we do option 2. I'm ok with this, if it unblocks you, but I think it's important that jank has a Clojure-idiomatic way to redirect to files and such. This is the way.

[djblue]

I like the idea of storing stdout in *out* and stderr in *err*, what would we store there for with-out-str? Is that what the dup stuff would be for?

open_memstream would be the answer for Linux/OSX I assume.

[djblue]

The current solution works for my Linux machine, and I assume it will also work for OSX, but Windows will need its own code path. I do wonder if it's time to implement jank.io/reader + jank.io/writer to hide the platform specific details?

[ikappaki]

Hi, just catching up on the conversation.

I got this working on windows using tmpfile() in place of open_memstream. It requires the CRT fix in CppInterOp (#803) since the FILE* crosses the JIT boundary.

Performance wise though, tmpfile() has overhead: every call creates a temp file, writes, seeks back, and reads. Since prn-str/print-str use with-out-str internally, that adds up in hot paths. Is that acceptable as a stop gap? There might be better Windows approaches for an in memory FILE* but I haven't looked into them yet.

Regarding Option 2 (dup2), I have some concerns about it for with-out-str specifically:

• Clojure with-out-str only captures what goes through *out*, Java's System.out is not captured. Capturing all C++ output would be a divergence from clojure semantics.
• My understanding is that dup2 operates on the process wide fd table. How does that interact with binding being per-thread? And with nested with-out-str, would each level need to save/restore the fd? What happens if an exception fires mid redirect? It seems like the fd management might get messy quickly.

I think @djblue's suggestion of introducing a jank.io/writer is worth exploring as the longer term path. If *out* held a jank writer object instead of a FILE*, with-out-str could just bind a string backed writer. No platform specific file APIs, no canonical type issues, no temp files.

Thanks

[jeaye]

Thanks for the helpful follow up, @ikappaki. I've followed up on the LLVM side, to ping Lang, but until that pans out, are you open to making the CppInterop change?

[ikappaki]

Thanks for the helpful follow up, @ikappaki. I've followed up on the LLVM side, to ping Lang, but until that pans out, are you open to making the CppInterop change?

Hi @jeaye, I've opened jank-lang/CppInterOp#4 for the interop change.

[ikappaki]

The change has been merged to main with #810, what is the next step for this PR please? Happy to help with the windows side of things.

[djblue]

@ikappaki if you can tackle the windows side, that would be awesome!

[ikappaki]

Hi gyus,

Sorry for the delay. I've added MS Windows support using tmpfile in djblue#2.

As discussed, this is a stopgap solution to get @djblue unblocked. The win implementation is inefficient because it relies on disk I/O, and neither the win nor POSIX implementations currently perform any error handling.

I'm looking forward to a proper writer based implementation in the near future. I'd be happy to help with that once @jeaye confirms the design.

thanks

[djblue]

Will bump test suite when jank-lang/clojure-test-suite#916 gets merged. Done.

[djblue]

Ok, I think this PR is ready for a final review pass 👍

[jeaye]

Works for me!