beanlabs/will: having cross-repo trouble using in bazel

[mhansen]

Hi there, I'll preface this by saying this is super low priority -- I'm not really using will.py for anything important. And I'm probably doing something very weird that nobody else is. Just wanted to report.

I'm trying to use bazel to run a bunch of beancount scripts. Mostly this is straightforward.

But I'm having some trouble using https://github.com/beancount/beanlabs/blob/master/beanlabs/will/will.py.

Problem number 1: will.py has a main function, but is not exposed as a py_binary in the BUILD rule. That's probably fairly straightforward to fix.

Problem number 2: Build visibility.

The BUILD file (https://github.com/beancount/beanlabs/blob/master/beanlabs/will/BUILD) refers to targets in //beancount:

py_library(
    name = "will",
    srcs = ["will.py"],
    deps = [
        "//beancount/core:account",
        ...
    ],
)

//beancount maybe used to be accessible when beanlabs was a subfolder of the main beancount repository. But now it's in a different repository, and we need to import and rename them via a WORKSPACE file like this:

git_repository(                                                                                                                                                                    
    name = "bc",                                                                                                                                                                   
    remote = "https://github.com/beancount/beancount.git",                                                                                                                         
    commit = "5adf7b9a3be99f631f4b8cd161287f1479dce377",                                                                                                                           
    shallow_since = "1616264925 -0400",                                                                                                                                            
)                                                                                                                                                                                  

And then access them with the renamed name, like "@//beancount/core:account".

So I'm doing this, by writing my own BUILD.beanlabs file:

load("@rules_python//python:defs.bzl", "py_binary")                                                                                                                                
                                                                                                                                                                                   
package(                                                                                                                                                                           
    default_visibility = ["//visibility:public"],                                                                                                                                  
)                                                                                                                                                                                  
                                                                                                                                                                                   
# Can't use the existing build rules because they assume beanlabs will be run from inside beancount                                                                                
py_binary(
    name = "will",
    srcs = ["//beanlabs/will:will.py"],
    deps = [
        "@bc//beancount/core:account",
        "@bc//beancount/core:account_types",
        "@bc//beancount/core:convert",
        "@bc//beancount/core:data",
        "@bc//beancount/core:getters",
        "@bc//beancount/core:realization",
        "@bc//beancount:loader",
        "@bc//beancount/parser:options",
        "@bc//beancount/parser:version",
    ],                                                                                                                                                                             
)                                                                                                                                                                                  

But then that fails due to BUILD visibility of the depended targets:

$ bazel test ...
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target '@bc//beancount/core:account' is not visible from target '@beanlabs//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target '@bc//beancount/core:account_types' is not visible from target '@beanlabs//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target '@bc//beancount/core:convert' is not visible from target '@beanlabs//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target '@bc//beancount/core:data' is not visible from target '@beanlabs//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target '@bc//beancount/core:getters' is not visible from target '@beanlabs//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target '@bc//beancount/core:realization' is not visible from target '@beanlabs//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: Analysis of target '@beanlabs//:will' failed

Indeed, the default visibility of beancount.core is fairly restrictive: https://github.com/beancount/beancount/blob/master/beancount/core/BUILD#L2

According to bazelbuild/bazel#3744, if you want visibility cross-repo, the only supported visibility is //visibility:public.

In the meantime, I'm working around this by specifying --nocheck_visibility. But it would be neat to avoid this :-)

I'm not sure, but I think there's an argument that beancount.core is part of the "public API" of beancount, depended on by anyone importing beancount, so maybe it should also be //visibility:public?

I'm guessing this beanlabs BUILD code is fairly vestigial (as there's no WORKSPACE in the repo).

I don't really expect you to fix this for me -- I'm happy to help contribute some fixes, just wanted to start a conversation about the desired state here.

[blais]

Hi Mark Bazel build not ready for cross repo work. Just use the old process or build locally + insert into PYTHONPATH, not sure why you're bothering with Bazel at this stage, the new code isn't really in play even in v3 (the old code path is still in use). Cheers,

…

On Sat, Sep 10, 2022 at 8:48 PM Mark Hansen ***@***.***> wrote: Hi there, I'll preface this by saying this is super low priority -- I'm not really using will.py for anything important. And I'm probably doing something very weird that nobody else is. Just wanted to report. I'm trying to use bazel to run a bunch of beancount scripts. Mostly this is straightforward. But I'm having some trouble using https://github.com/beancount/beanlabs/blob/master/beanlabs/will/will.py. Problem number 1: will.py has a main function, but is not exposed as a py_binary in the BUILD rule. That's probably fairly straightforward to fix. Problem number 2: Build visibility. The BUILD file ( https://github.com/beancount/beanlabs/blob/master/beanlabs/will/BUILD) refers to targets in //beancount: py_library( name = "will", srcs = ["will.py"], deps = [ "//beancount/core:account", ... ], ) //beancount maybe used to be accessible when beanlabs was a subfolder of the main beancount repository. But now it's in a different repository, and we need to import and rename them via a WORKSPACE file like this: git_repository( name = "bc", remote = "https://github.com/beancount/beancount.git", commit = "5adf7b9a3be99f631f4b8cd161287f1479dce377", shallow_since = "1616264925 -0400", ) And then access them with the renamed name, like "@//beancount/core:account". So I'm doing this, by writing my own BUILD.beanlabs file: ***@***.***_python//python:defs.bzl", "py_binary") package( default_visibility = ["//visibility:public"], ) # Can't use the existing build rules because they assume beanlabs will be run from inside beancount py_binary( name = "will", srcs = ["//beanlabs/will:will.py"], deps = [ ***@***.***//beancount/core:account", ***@***.***//beancount/core:account_types", ***@***.***//beancount/core:convert", ***@***.***//beancount/core:data", ***@***.***//beancount/core:getters", ***@***.***//beancount/core:realization", ***@***.***//beancount:loader", ***@***.***//beancount/parser:options", ***@***.***//beancount/parser:version", ], ) But then that fails due to BUILD visibility of the depended targets: $ bazel test ... ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target ***@***.***//beancount/core:account' is not visible from target ***@***.***//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target ***@***.***//beancount/core:account_types' is not visible from target ***@***.***//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target ***@***.***//beancount/core:convert' is not visible from target ***@***.***//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target ***@***.***//beancount/core:data' is not visible from target ***@***.***//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target ***@***.***//beancount/core:getters' is not visible from target ***@***.***//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: in py_binary rule @beanlabs//:will: target ***@***.***//beancount/core:realization' is not visible from target ***@***.***//:will'. Check the visibility declaration of the former target if you think the dependency is legitimate ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/beanlabs/BUILD.bazel:8:10: Analysis of target ***@***.***//:will' failed Indeed, the default visibility of beancount.core is fairly restrictive: https://github.com/beancount/beancount/blob/master/beancount/core/BUILD#L2 According to bazelbuild/bazel#3744 <bazelbuild/bazel#3744>, if you want visibility cross-repo, the only supported visibility is //visibility:public. In the meantime, I'm working around this by specifying --nocheck_visibility. But it would be neat to avoid this :-) I'm not sure, but I think there's an argument that beancount.core is part of the "public API" of beancount, depended on by anyone importing beancount, so maybe it should also be //visibility:public? I'm guessing this beanlabs BUILD code is fairly vestigial (as there's no WORKSPACE in the repo). I don't really expect you to fix this for me -- I'm happy to help contribute some fixes, just wanted to start a conversation about the desired state here. — Reply to this email directly, view it on GitHub <#7>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACSBE43BPCITY3AXB3P473V5UT47ANCNFSM6AAAAAAQJRVQRQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mhansen]

Thanks Martin. I've got an admittedly overengineered setup where I'm converting some historical hledger files to beancount and then running some tests on the hledger and some tests on the beancount (asserting that the numbers equal my tax returns for example), and some shellscript tests and generating a list of accounts with will.py, and I'm trying to tie together all these different languages with bazel. Is this silly? Yeah, probably. But it beats make, and lets me cache slow actions and get somewhat-hermetic results.

Not sure what the 'old process' and 'new code' is, sorry (is the old process an old build system, and the new code bazel BUILD)? But yeah, I was previously installing locally (maybe with pip, not sure) and inserting into PYTHONPATH, and that worked for a few years, the will.py python script wasn't really running hermetically under bazel so it all worked.

Is there any desire to get bazel ready for cross-repo work, or direction you'd like it to go (e.g. what do you think about making some of the modules visibility public and adding a WORKSPACE to this experimental git repo?) Totally understand if the focus is on v3 and this v2 stuff is sorta-deprecated, I'm happy to just continue with grabbing it from PYTHONPATH.

[blais]

On Mon, Sep 12, 2022 at 3:44 AM Mark Hansen ***@***.***> wrote: Thanks Martin. I've got an admittedly overengineered setup where I'm converting some historical hledger files to beancount and then running some tests on the hledger and some tests on the beancount (asserting that the numbers equal my tax returns for example), and some shellscript tests and generating a list of accounts with will.py, and I'm trying to tie together all these different languages with bazel. Is this silly? Yeah, probably. But it beats make, and lets me cache slow actions and get somewhat-hermetic results. Not sure what the 'old process' and 'new code' is, sorry (is the old process an old build system, and the new code bazel BUILD)? But yeah, I was previously installing locally (maybe with pip, not sure) and inserting into PYTHONPATH, and that worked for a few years, the will.py python script wasn't really running hermetically under bazel so it all worked. Is there any desire to get bazel ready for cross-repo work, or direction you'd like it to go (e.g. what do you think about making some of the modules visibility public and adding a WORKSPACE to this experimental git repo?) Totally understand if the focus is on v3 and this v2 stuff is sorta-deprecated, I'm happy to just continue with grabbing it from PYTHONPATH.

Just keep grabbing it from PYTHONPATH for now. will.py does not require any v3 stuff. When I switch the parser to use the C++ path it might be time to try to put something together, but it's way, way too early. Message ID: ***@***.***>

…

[mhansen]

OK, thanks. I'll keep using beancount from PYTHONPATH and won't try to use bazel to wire together beanlabs with the beancount repo. On Tue, 13 Sept 2022 at 11:05, Martin Blais ***@***.***> wrote:

…

On Mon, Sep 12, 2022 at 3:44 AM Mark Hansen ***@***.***> wrote: > Thanks Martin. I've got an admittedly overengineered setup where I'm > converting some historical hledger files to beancount and then running some > tests on the hledger and some tests on the beancount (asserting that the > numbers equal my tax returns for example), and some shellscript tests and > generating a list of accounts with will.py, and I'm trying to tie together > all these different languages with bazel. Is this silly? Yeah, probably. > But it beats make, and lets me cache slow actions and get > somewhat-hermetic results. > > Not sure what the 'old process' and 'new code' is, sorry (is the old > process an old build system, and the new code bazel BUILD)? But yeah, I was > previously installing locally (maybe with pip, not sure) and inserting into > PYTHONPATH, and that worked for a few years, the will.py python script > wasn't really running hermetically under bazel so it all worked. > > Is there any desire to get bazel ready for cross-repo work, or direction > you'd like it to go (e.g. what do you think about making some of the > modules visibility public and adding a WORKSPACE to this experimental git > repo?) Totally understand if the focus is on v3 and this v2 stuff is > sorta-deprecated, I'm happy to just continue with grabbing it from > PYTHONPATH. > Just keep grabbing it from PYTHONPATH for now. will.py does not require any v3 stuff. When I switch the parser to use the C++ path it might be time to try to put something together, but it's way, way too early. Message ID: ***@***.***> > — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAZYOOMYEPNF6HMYX3HFNTV57HPNANCNFSM6AAAAAAQJRVQRQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>