Is it possible to get the deduplicated children inside parent?

[ThallesP]

i've a flow with a parent that depends on X children that will be duplicated so i've decided to use the jobId to prevent a lot of duplicated children but then the parent depends on data generated on those duplicated children, is there anyway to get those duplicate childrens' data?

and also, does it make sense to add a feature to getChildrenValues where it returns those duplicated children data? enabling it either by a parameter or the default way.

btw, im using nestjs

[roggervalf]

when you are deduplicating by jobId you will get only one job record, if you need data generated on those duplicated children you should not deduplicate them, but use different jobIds

[ThallesP]

when you are deduplicating by jobId you will get only one job record, if you need data generated on those duplicated children you should not deduplicate them, but use different jobIds

they'll generate the same data and I just don't see any reason to hit Redis with more 60k jobs where I can reduce it to 1k.
without taking into account on what the job do, it hits up my S3 account and 59k more requests will increase price.

[roggervalf]

hey, what confused me was the part where you mentioned

the parent depends on data generated on those duplicated children, is there anyway to get those duplicate childrens' data

when records are deduplicated, they are ignored, no extra data is saved about those records. Also you may want to take a look into our deduplication feature that was launched few weeks ago https://docs.bullmq.io/guide/jobs/deduplication

[ThallesP]

hey, what confused me was the part where you mentioned

the parent depends on data generated on those duplicated children, is there anyway to get those duplicate childrens' data

when records are deduplicated, they are ignored, no extra data is saved about those records. Also you may want to take a look into our deduplication feature that was launched few weeks ago https://docs.bullmq.io/guide/jobs/deduplication

yep, i want the duplicated jobs to be ignored (no extra job is created) but at the same time i would expect to get the data generated by that unique job created (in a flow context), does that make sense?

i'll try out deduplication, it might seems what i want.