server: full text search by recording contents

[brownman]

is there a way to filter/search the db for a keyword/tag ?

[ku1ik]

I am considering implementing site-wide search based on elasticsearch (that would include all the text from asciicast title, description and the whole terminal output).

[absorber]

👍

[coxley]

👍

[ku1ik]

ElasticSearch may be an overkill here. I'll look into implementing this using Postgres' built-in full text search.

[coxley]

FWIW my last team and I hacked on an internal asciinema and ES worked quite well but then again that infrastructure was already in place.

[amityahav]

yo, can u elaborate how you tackled this? if you remmber :D

[deeplook]

Any progress on this? This seems to be one of the most pressing missing features.

[edmundlaugasson]

Would appreciate both public and also private recordings search.

[m040601]

First of all thank you for all the work in this so much underrated tool and the website. Big fan for many years.

I was looking for some examples to teach some kids about terminal usage. In this day and age, with the attentiton grabbbing competition from flashy youtube "tuttorials" it's not an easy task. Their generation cant conceive of a "tutorial" without dramatic faces, flashy intros, music and sexy ladys.

So i was trying to search asciinema.org for some nice cool asciicasts. You know. With a bit of color. A bit of cool animations. Some ascii graphics . But. With basic tools everybody knows.. Not niche IT pros tools.

I ended up here because, me too, wondered why there was no kind of search on the website. And I mean no kind of search at all. Neither by title, content, keyword, description etc.

Such a basic feature missing. It would help make asciineman better known.

Well I then tried plan B. Use Google and enter something like

"some_keyword" site:https://asciinema.org/a

No. Nope. Zilch. Nada.

$ curl https://asciinema.org/robots.txt

# See http://www.robotstxt.org for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /

User-Agent: *
Disallow: /a/*.json
Disallow: /a/*.cast
Disallow: /a/*.png
Disallow: /a/*.svg
Disallow: /a/*.gif

# Allow Twitterbot access to casts images
User-Agent: Twitterbot
Allow: /a/*.png
Allow: /a/*.svg

Ahh ... so that explains it ...

A bit more googling and I ended up here.

I am considering implementing site-wide search based on elasticsearch (that would include all the text from asciicast title, description and the whole terminal output).

Wow. That's really old. Like, 2013.

And there are other users, wondering about the same in 2018, ....

So may I sincerely ask,

Is there any kind of non technical reason for this not happening ? Any thing related to prevent scraping or ??? Any non willingness of the owners not wanting "search" on the website ?

If not, and if this is just a technical feature missing, any updates on this ? Any time frame ?

Thanks in advance.

[ku1ik]

In a nutshell: this has always been on my list of things to add to the project but simply due to asciinema being my spare time project I never had a chance to get to this. There were always other priorities or just regular maintenance work getting in the way.

This is still one of my top priorities and I'd love to work on it soon. In order for me to dedicate more time to the project I've setup Github Sponsors program, which I hope with your help will provide me financial support to tackle many long waiting problems/features.

More on that here: https://github.com/orgs/asciinema/discussions/234

Thanks ❤️

[ku1ik]

Good news! I implemented recording search, which indexes titles, description, and full terminal session content :) It uses Postgres' native FTS support, so no extra components (such as ElasticSearch) are needed.

It's already on https://asciinema.org, and I'll make a new release of asciinema-server soon, for those who self-host.

Give it a go. The search box is in the page top nav bar, and you can focus it by pressing /. Also check the search tips - there's a link on the right side of the search results page. The syntax (explained in that tips popup) is just what Postgres websearch_to_tsvector supports. The results contain public recordings, and if you're logged in you'll also be able to find your own recordings, including unlisted and private ones.

[deeplook]

Very cool, thank you so much!

[cmprmsd]

Awesome stuff! This is a real milestone and I'm looking forward to use it locally! 🙀

If you are open for comments I'd suggest a feature enhancement. Having the ability to click into a video from the search results with a list of hits (and context or timestamps) and jump to those would help a lot on larger recordings.

We use asciinema for our terminal logging for which I created this tool https://github.com/cmprmsd/cinelog and jumping straight to the right spot in the recording is a nice time saver. Maybe you'd like to consider this as a later enhancement. 😊
PS: My search in Python slows down quite a bit for fulltext, but I have two logs, one with only the commands entered and one with the full recording. I also inject a marker for each command being executed via terminal hooks. Quite hacky but I tried my best to use your new marker feature at that time. 😅

[ku1ik]

Having the ability to click into a video from the search results with a list of hits (and context or timestamps) and jump to those would help a lot on larger recordings.

Yeah, I expect such functionality to be desired :) Not as straightforward to implement as it may seem though. On the player side the simplest it would be to use markers for this. On the server side we would need to augment the search index with timestamp information - right now as the basis for search the server uses plain text log + unique words from the alternate screen buffer (full screen programs). I'll keep that in mind for when I'll be looking for a challenge :)

We use asciinema for our terminal logging for which I created this tool https://github.com/cmprmsd/cinelog

I'll check it out for some good ideas!