[feature] hashicorp hcl as configuration language

[harlequin]

Is your feature request related to a problem? Please describe.

No

Describe the solution you'd like

moon currently support yaml and pkl as configuration sources.

The hcl language from hashicorp would be an alternative.
With this we would get variables and conditions directly into the moon tasks definitions.

Another benefit would be that unlike pkl, no extra binary is needed to interpolate this syntax.

https://docs.rs/hcl-rs/latest/hcl/

What do you think?

[harlequin]

I have done some investigations today about my proposal to use hcl and the crate hcl-rs.

First steps

Include the hcl config loader in the schematics crate, like done for ron in the pull request #162

Afterwards created a moon adaption like in the pull request #2029.

In my sandbox I have tried a simple moon.hcl

tasks {  
  test {    
    command = "Write-Host Hello"
  }
}  

Which produces the intended output!

Hello

Usage of internal functions and arethmetics

tasks {  
  test {    
    command = "Write-Host ${ 1 + 1}"
  }
}  

Also here output is as expected

2

tasks {  
  test {    
    command = [for v in ["echo", "Hello", "World"]:v]
  }
}  

Also working

Hello World

Next try, custom functions

The hcl definition and implementation in the crate provides the possibility to define functions and variables within the context of the parser.

Therefore I have implemented a simple function called upper

fn upper(args: FuncArgs) -> Result<Value, String> {
    Ok(Value::from(args[0].as_str().unwrap().to_uppercase()))
}

With this we can now convert strings into uppercase like this

tasks {  
  test {    
    command = "Write-Host ${upper("Hello World")}"
  }
}  

Output

HELLO WORLD

Next, custom blocks

Many applications from Hashicorp which are using the hcl dialect provides blocks to define specials.
I have implemented the variables block definition, which succesfully parsed and providing a valid moon task definition.

variables {
  foo = "Hello World"          # sets a custom variable
  bar = "${foo} = ${1+1}"      # reads the variable and concat it with a sum of 1 and 1
}

tasks {  
  test {    
    command = "Write-Host ${upper(bar)}" # finally print this in uppercase
  }
}  

output

HELLO WORLD = 2

Next, secrets management with vault

With all this funcationallity I have tried also to create a custome function called vault, similar to vault function

fn vault(args: FuncArgs) -> Result<Value, String> {

    let mut settings: VaultClientSettings = VaultClientSettingsBuilder::default()
            .address("my_address")
            .token("my_token")
            .build()
            .unwrap();    

    // Create a client
    let mut client = VaultClient::new(settings).unwrap();
    // Todo: read mount and path from args[0].as_str().unwrap()
    let mount = "mount";
    let path = "path";
    

    let secret: HashMap<String, String> = block_on( kv2::read(&client, mount, path)).unwrap();
    
    // Todo: we have the hashmap which can be filtered 
    // with the second argument to read the key
    // args[1].as_str().unwrap()
    print!("{:?}", secret);
    Ok(Value::String("".into()))
}

I will not post the output, but it works ;)

Conclusion

With hcl and the crate we would have all possibilities to enahnce the moon ecosystem.
But what feels wrong at the moment is that the custom functions and variables are in the schematic crate.

From my point of view is to extend the schematic loader with some functions or members to extend the hcl context.
So that moon can provide these vault, upper and many other functions.

With this said, custom functions and variables would provide a method to get the item New token system done from the roadmap.

Additional the vault function would provide a good solution for the item Secrets management from the roadmap.

Vault itself is from hashicorp (IBM), but there is also the alternative like OpenBao

And, also from my point of view, we do not have external binaries dependencies like with pkl.

Source Code

I will put also the hacky (please don't judge) source code from schematics in this incident, so that you can get an overview how simple it was to introduce this.

# hcl.rs

use std::collections::HashMap;

use crate::config::error::ConfigError;
use crate::config::parser::ParserError;
use hcl::Value;
use hcl::{Body, Block};
use hcl::eval::{Context, Evaluate, FuncArgs, FuncDef, ParamType};
use miette::NamedSource;
use serde::de::DeserializeOwned;
use vaultrs::client::{VaultClient, VaultClientSettings, VaultClientSettingsBuilder};
use vaultrs::kv2;
use futures::executor::block_on;

fn vault(args: FuncArgs) -> Result<Value, String> {

    let mut settings: VaultClientSettings = VaultClientSettingsBuilder::default()
            .address("my_address")
            .token("my_token")
            .build()
            .unwrap();    

    // Create a client
    let mut client = VaultClient::new(settings).unwrap();
    let mount = "mount";
    let path = "path";
    

    let secret: HashMap<String, String> = block_on( kv2::read(&client, mount, path)).unwrap();
    
    // Todo: we have the hashmap which can be filtered 
    // with the second argument to read the key
    print!("{:?}", secret);
    Ok(Value::String("".into()))
}


// A function that is made available to expressions via the `Context` value.
fn upper(args: FuncArgs) -> Result<Value, String> {
    Ok(Value::from(args[0].as_str().unwrap().to_uppercase()))
}


fn parse_variables_block(body: &mut Body, ctx: &mut Context) {
    let all_blocks = body.blocks();

    for b in all_blocks {
        if b.identifier.to_string() == "variables" {            
            for attr in b.body().attributes() {

                let value = attr.expr().evaluate(ctx).unwrap();

                // IMPORTANT, add the variables, defined by the variables block.
                // this method can also be used to declare global variables or tokens from 
                // moon itself
                ctx.declare_var(
                    attr.key(),
                    value,
                );
            }
        }
    }

    let mut builder = Body::builder();
    *body = builder
        .add_blocks(body.blocks().into_iter().filter(|block| block.identifier.to_string() != "variables").cloned().collect::<Vec<_>>())
        .build();
}



pub fn parse<D>(name: &str, content: &str) -> Result<D, ConfigError>
where
    D: DeserializeOwned,
{
    let mut ctx = hcl::eval::Context::new();
    let mut body = hcl::parse(content).map_err(|error| {    
        ParserError {
            content: NamedSource::new(name, content.to_owned()),
            path: error.to_string(),            
            span: None,
            message: error.to_string(),        
        }
    })?;

    
    // Add all custom functions, like upper and vault
    // Can be used for all tokens from moon and much more
    
    // Create a definition for the `uppercase` function.
    let uppercase_func = FuncDef::builder()
        .param(ParamType::String)
        .build(upper);

    // Create a definition for the `vault` function.
    let vault_func = FuncDef::builder()
        .param(ParamType::String)
        .param(ParamType::String)
        .build(vault);



    // THIS IS IMPORTANT ctx can have the custom functions
    // Feels somehow wrong to define this here and not within moon
    ctx.declare_func("upper", uppercase_func);
    ctx.declare_func("vault", vault_func);

    // Parse the variables block and remove this block at end,
    // Because moon schemas are not aware of this definitions
    parse_variables_block(&mut body, &mut ctx);

    let de = body.evaluate(&ctx).map_err(|error| {    
        ParserError {
            content: NamedSource::new(name, content.to_owned()),
            path: error.expr().unwrap().to_string(),            
            span: None,
            message: error.to_string(),        
        }
    })?;
        


    let r = hcl::de::Deserializer::from_body(de);
    
    let result: D = serde_path_to_error::deserialize(r).map_err(|error| {    
        ParserError {
            content: NamedSource::new(name, content.to_owned()),
            path: error.path().to_string(),
            span: None,
            message: error.inner().to_string(),        
        }
    })?;

    Ok(result)
}

@milesj: You have rejected the pull request with ron for moon with a comment that you want to rework something. Can you maybe specify a little bit what do you mean exactly with it?

[milesj]

Not opposed to this, but it will have to wait for the schematic rewrite: moonrepo/schematic#160

[harlequin]

@milesj, just give me a ping whenever the schematic is done and I will rewrite my code and create a contribution

[milesj]

This will land in v2, but will not support functions at first.

#2233

[milesj]

Added in the upcoming v2.

• Alpha is available now: https://moonrepo.dev/blog/moon-v2-alpha
• More about v2: https://moonrepo.dev/blog/moon-v2.0