Use with Rust

[joshbenaron]

Is there a way to implement this in Rust? Or is this already done? I just want to do basic operations like add to a queue etc. Is there a rough high level api for key structures etc and what I need to store in order to do this?

[ccollie]

bullmq is distributed as a library, so no api is included, though its easy enough to set up a server with express/fastify. Check out bull-monitor for how they do this. They're oriented towards the UI but as part of the process they both provide an api for the frontend to use.

[ccollie]

BTW. Your use case is precisely one of the reasons I'm developing a graphql server on top of bullmq

[joshbenaron]

Well - I might not follow. But what I want to do is use a Rust http server. For each request I add a job to the queue. I simply want to add to the queue from the Rust server

[joshbenaron]

Surely adding an item to the queue is just adding specific key/values to Redis?

[ccollie]

You can certainly add jobs via direct access to redis, but that's incredibly brittle. If you're really insistent and are comfortable in calling redis from rust, check out

https://github.com/taskforcesh/bullmq/blob/master/src/commands/addJob-9.lua

Calling that should get you what you want. My question though is how are your workers implemented ? I'm unaware of any non-node projects using bullmq. If you're using node workers anyway, then why not use it to add jobs ?

[joshbenaron]

So the http server is rust - and yeah the workers are nodejs. I want high performance for the http server

[joshbenaron]

incredibly brittle

In what sense? @ccollie

[ccollie]

Its brittle because you're sneaking around the back of the api. Bullmq is implemented both in node and as scripts in redis, so it's important to know how they interact.

As far as the server being in rust, I honestly wouldn't worry about it, especially since all you'd be doing is pushing data to redis, which is IO bound in any event. Bull also has methods to do bulk pushes if you need to squeeze out extra performance.

If you really need speed, I'd implement the workers in rust, since bull allows for manual queue handling.

[ccollie]

To be more specific, if all you're doing in the http server is pushing data to redis, the performance differential would not justify the introduction of a new technology, IMO. Node is perfectly at home with IO bound processing, whereas rust could definitely be a benefit for workers if most of your queue tasks are CPU bound.

[joshbenaron]

I meant I want performance for the server in general. One route adds to the queue. i.e. there is some CPU bound stuff that the server does

[joshbenaron]

How would I go about manually processing jobs from Rust?

[ccollie]

I guess if you have other reasons for having a rust based server then that's understandable.

r.e. Manual Processing:
Looking at the docs here:
https://docs.bullmq.io/patterns/manually-fetching-jobs

I may have been overly optimistic, since this uses the bull js library itself. It seems like you would have to replicate the worker js code in rust and follow the directions in the link.

[joshbenaron]

Cool. I'll start with the adding first. Long term I'll make this into a nice Rust lib for people to use