RFC 0018: Macro Methods

[Blacksmoke16]

Preview: https://github.com/crystal-lang/rfcs/blob/1195640eaf4f6780bb02e1839a35552adb42d04e/text/0018-macro-methods.md
PoC PR: TODO

A proposal based on the OP from crystal-lang/crystal#8835. Simple re-usable macro methods without the extra AST node monkey patching stuff.

[ysbaddaden]

Some personal observations:

1. ProcLitral#call is the main alternative (and/or complement) and shall be documented here. See ProcLiteral#call for macros crystal#16451.

2. Because of the proximity with def I think we should define macro defs on explicit types, for example macro def Foo.name and macro def self.name.

[BlobCodes]

• I think a dedicated syntax would be beneficial to separate builtin intrinsic functions on TypeNode and user methods.

  Maybe something like $function() and $Type.function()?

• Why should it only be possible to call macro defs within macros, and not other macros? Why are type restrictions only available on these new defs?

  The RFC differentiates macros and macro defs by stating that one returns an AST node and the other generates code, but the both are practically the same since you can always convert between the two.

  I think the following should be possible:

  macro fib(i : NumberLiteral)
    {% if i <= 1 %}
      {{ i }}
    {% else %}
      {{ fib(i-1) + fib(i-2) }}
    {% end %}
  end

  I think this would feel like a more holistic approach and allow existing libraries to make use of this feature more easily. These two functions would have exactly the same functionality:

  # This
  macro def test1(ARGS) : RET
    CODE
  end
  
  # Is always exactly the same as:
  macro test2(ARGS) : RET
    {{
      CODE
    }}
  end

  I think a very good source of inspiration would be the typst language. It's a very clean and modern TeX alternative and has this differentiation between "content mode" (code in square braces) and "code mode" (code in squiggly braces), but you can always switch back-and-forth between the two.

• I think the ability to pass blocks to macro defs could substantially complicate the implementation, so I think it should be moved to future possibilities. It would also be a bit confusing since blocks in normal macros accept ASTNodes (and the keyword yield is already being used to "paste" this ASTNode).

I actually wanted to post an RFC like this soon, only adding type restrictions and the ability to call macros weithin macros, but leaving the macro def syntax as a future consideration. I think I'll now wait how this RFC progresses.

[straight-shoota]

@BlobCodes that's an intriguing perspective.
Let's assume macro can be called from a macro expression and gets type restrictions (both seem feasible enhancements).
What's the difference between macro and macro def then? If it's just syntactic sugar for wrapping the entire body in {{ ... }}... 🤔

macro def format_name(name : StringLiteral) : StringLiteral
  name.underscore.upcase
end

macro format_name(name : StringLiteral) : StringLiteral
  {{ name.underscore.upcase }}
end

[Blacksmoke16]

• Is the "return value" just whatever is the last line wrapped in {{ }}? What would the expected behavior be for a macro that is doing something like:

macro generate_constant(name)
  CONST_{{ name.id }}
end

• Is the expectation that if you call one of these macros in a place its not expected to be, that it just fails or works but with undefined behavior?

• If we want macros to support next and return keywords and such too would that conflict with using {{ }} as well?

• Are we concerned at all that the API docs for these would just be mixed in with normal macros?

[BlobCodes]

Is the "return value" just whatever is the last line wrapped in {{ }}? What would the expected behavior be for a macro that is doing something like (...)

There is no explicit return value, the entire macro is executed as it works right now, its entire output is parsed into an ASTNode and this ASTNode is then returned to the caller.

If the given example macro was called with the name "foo", it would return Path("CONST_foo").
If the example macro was called with the name "A = 2", it would return Assign(@target = Path("CONST_A"), @value = NumberLiteral(2))

If we want macros to support next and return keywords and such too would that conflict with using {{ }} as well?

Yes, explicit returns would probably conflict with the current template syntax.

That's something that could be explored with macro defs.

Is the expectation that if you call one of these macros in a place its not expected to be, that it just fails or works but with undefined behavior?

Why should a macro result in undefined behaviour if it's called within a nested context?

[Blacksmoke16]

Why should a macro result in undefined behaviour if it's called within a nested context?

I was mainly thinking about my use case where some of my macro defs would be like:

macro foo(x)
  {%
    # Interacting with constants and stuff
    # Intended to be called within a macro expression at certain places in the code
  %}
end

So if you tried to use this macro outside of a macro expression where you're expected for it to run, what would it do? Probably error that the const is unknown? Maybe not a big deal in practice, but this is kinda why i liked macro def being a separate thing. Just easier, for me at least, mentally to know that "okay these macros i use in a macro context while this other kind of macro is for this other context" versus needing to possibly handle the case where a macro is used in a place it wasn't expected to be.

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/dependency-injection-with-crystal-macros/8684/2

[willhbr]

Foo.foo              # Calls regular macro (generates code)
{{ Foo.foo }}        # Calls macro method (returns AST node)

I would be in favour of having one lookup path, otherwise I think you'd end up with a lot of template macros that just call a macro def and splat the result into the program, ie:

macro def foo(args) : StringLiteral
  "from macro method: #{args.stringify}"
end

macro foo(args)
  {{ foo(args) }}
end

foo "hello!"

Since both types of macros are "returning" a syntax tree (one with a template, the other with an actual return value) I would like them to both insert that syntax tree into the program when called.

---

A possibility to consider is to only have macro defs and have a quote method that allows for templating:

macro make_class(name)
  puts "I'm making a class called #{name}"
  ast = quote do
    class {{ name }}
    end
  end
  puts "I've made this class: #{ast}"
  ast
end

That way the return value semantics are clearer. This would obviously be a breaking change so not really practical, but you could have a tool that converted code between the two.

Either way I'd love macro methods in whatever form they come in.

[Blacksmoke16]

otherwise I think you'd end up with a lot of template macros that just call a macro def and splat the result into the program, ie:

What would the use case/reasoning for this be? Because in this example you don't even need the macro def for it to work:

macro foo(args)
  {{ "from macro method: #{args.stringify}" }}
end

foo "hello!"

[willhbr]

I should've been clearer, I mean if I have a macro def that either does something you can't do with a normal macro, or is just a macro def because that's how I wanted to write it, to expose an API where it can be called like foo("some value") instead of {{ foo("some value") }} I'd have to wrap it in a regular macro and forward the arguments.

[Blacksmoke16]

Ahh, yea I'm not too concerned about that. There isn't anything you can do in a macro def that you can't do in a macro, at least logic wise. So if you have a single macro that does something and you need to paste its output into the program, you can do that with a normal macro. If you define both because you want to be able to share the logic in future macro calls, then it's working as intended, even if it looks strange when it's only used once.

[straight-shoota]

@willhbr Your comment got me thinking whether it might be an option to differentiate between the two modes by return type: a macro that returns a value must have a return type (like ASTNode). A template macro has no return type.

[Blacksmoke16]

I could go with that. Is a bit easier to explain too: a macro def is a macro with a return type, vs adding a whole new macro def concept. Should be fairly easy implementation wise, just have to tweak the parser to handle parsing restrictions on macros and key off if there is a return type to know what node to create.

It does make this a bit more similar to one another, but still fairly obvious which is which.

class Foo
  macro foo
    "from regular macro"
  end

  macro foo : StringLiteral
    "from macro method"
  end
end

What should be done about macro parameters. Can a normal macro have typed parameters like a macro method too?

[willhbr]

If a normal/template macro doesn't have a return type, would you still be able to call it from a macro def?

I don't know if this would actually end up being useful—it's hard to know without using it—but I think it would be useful to allow extracting parts of templates into separate methods to make code more readable/testable/etc.

I thought through rewriting my dependency injector macro if I had macro methods, and I came to this section:

def self.inject(scope)
  {{ @type }}.new(
    {% for arg in method.args %}
      {% if arg.restriction.nil?
           arg.raise "needs restriction on #{arg}"
         end %}
      {% if arg.restriction.resolve?.nil?
           arg.raise "Unable to resolve #{arg.restriction}"
         end %}
      {% if arg.restriction.resolve < Geode::Injector::Provider %}
        {{ arg.restriction }}.inject(scope),
      {% elsif arg.restriction.resolve < Geode::Injector::Lazy %}
        {{ arg.restriction }}.inject(scope),
      {% else %}
        scope.get({{ arg.restriction }}),
      {% end %}
    {% end %}
  )
end

I could pull out the verification into a separate macro def easily:

macro def validate_injectable_arg(arg)
  if arg.restriction.nil?
    arg.raise "needs restriction on #{arg}"
  end
  if arg.restriction.resolve?.nil?
    arg.raise "Unable to resolve #{arg.restriction}"
  end
end

But I'm not sure if I could do the same for the code that generates the actual code in the loop. It would want to be a template macro, so I could do this:

macro build_creation(arg, scope)
  %scope = {{ scope }}
  {% if arg.restriction.resolve < Geode::Injector::Provider %}
    {{ arg.restriction }}.inject(%scope),
  {% elsif arg.restriction.resolve < Geode::Injector::Lazy %}
    {{ arg.restriction }}.inject(%scope),
  {% else %}
    %scope.get({{ arg.restriction }}),
  {% end %}
end

Then the code generated by the main macro would look like this:

SomeInjectableType.new(
  build_creation(some_arg : HTTP::Context),
  build_creation(another_arg : User),
)

Which I guess is fine, but it does mean we've got another layer of macro expansion that needs to happen, and it's a bit harder to see what the final generated code is ({% debug %} won't show the fully expanded code in this case, right?).

It would be nice to be able to call any macro from any other macro and have the result inserted into the code, so I could do:

def self.inject(scope)
  {{ @type }}.new(
    {% for arg in method.args %}
      {% validate_injectable_arg arg %}
      {{ build_creation(arg, MacroId.new(:scope)) }},
    {% end %}
  )
end

Would I be able to create and return an actual AST from build_creation, without building a Crystal::Macros::Call myself, and then call that from another macro?

[zw963]

perfer macro def way, I can search this easier use rg for which macro is defined for another macros, it for readability.

Introduce this is for DRY when macro, eliminate duplicate code, but, I don't see any need for a block at this point—this is certainly not something that should be on the agenda for the current version.

(Translated with qwen3.5:35b)

[Blacksmoke16]

Would I be able to create and return an actual AST from build_creation, without building a Crystal::Macros::Call myself, and then call that from another macro?

I been thinking about this and that's a fair point. I'd have to see how this works in my POC branch. My current thinking is it would be the same if there is just a single ASTNode being returned, but having a macro able to inline the output of another macro feels like a different but related use case. You can of course expand a macro as part of the output of another macro so could maybe do something like:

def self.inject(scope)
  {{ @type }}.new(
    {% for arg in method.args %}
      {% validate_injectable_arg arg %}
      build_creation({{ arg }}, :scope),
    {% end %}
  )
end

Where build_creation is a normal macro that would expand as if it was inlined.