if memory2 query returns an occupancygrid (or pointcloud?) how can we present it to a LLM model in a way in which it understands it?
write an eval, ask basic questions like
spatial understanding
model will guess - so we need a lot of very specific questions
"how many rooms there are in total"
"how big is that room down the hallway to the right?"
"where is the biggest room in this space"
"what type of space is this"
point placement
"where should we explore the space next?"
"place a point on a biggest/smallest room"
"place points marking hallways"
some (failed) attempt here of just encoding occupancygrid as a picture for VLM #822
if memory2 query returns an occupancygrid (or pointcloud?) how can we present it to a LLM model in a way in which it understands it?
write an eval, ask basic questions like
spatial understanding
model will guess - so we need a lot of very specific questions
"how many rooms there are in total"
"how big is that room down the hallway to the right?"
"where is the biggest room in this space"
"what type of space is this"
point placement
"where should we explore the space next?"
"place a point on a biggest/smallest room"
"place points marking hallways"
some (failed) attempt here of just encoding occupancygrid as a picture for VLM #822