spotlight: add support for localsearch indexing of home directories#2956
spotlight: add support for localsearch indexing of home directories#2956rdmark wants to merge 1 commit into
Conversation
🤖 Augment PR SummarySummary: This PR extends Spotlight (localsearch) indexing to cover user home directories defined via the Changes:
🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
The [Homes] spotlight option (default: inherits from [Global]) controls whether the basedir regex path is submitted to the search indexer, covering all user home directories beneath it. Because [Homes] volumes are resolved per-user at login and never pass through creatvol() in the master process, OPTION_SPOTLIGHT must be set explicitly during config parsing when Homes spotlight is enabled -- otherwise the localsearch dbus infrastructure is never started. Add INIPARSER_GETBOOL macro parallel to the other iniparser wrapper macros.
rdmark
force-pushed
the
2908-spotlight-not-enabled-when-only-home-volume-defined
branch
from
a246013 to
9eddda3
Compare
🔥 Spectest (AFP 3.4) - Flamegraph (AFP_ASSERT active)Commit: 🔥 Open interactive Flamegraph (SVG) 🔝 Top 10 leaf functions
|
andylemin
left a comment
andylemin
left a comment
There was a problem hiding this comment.
I like this requirement to specify spotlight for each home share separately - I think it is a better choice for individual users, where some users may not want localsearch indexing enabled when enabled globally.
|
@andylemin do you have a suggestion how to manage a per-user volume setting in practice? we don't have such a mechanism right now IINM. maybe something like a ~/.netatalk/afp.conf that afpd reads after each afpd fork? |
|
|
Hi @rdmark you are right.. there is currently only a single virtual homes share which covers all users.. (I don't use homes so forgot this). To provide per user override you would need the ability to optionally specify an individual [username] home mount with settings that override the common homes and global config. This would allow Spotlight to be enabled for only a single user if homes does not exist at all, or disabled if [homes] exists and enables. [username] > [homes] > [global] Whether you also allow those per username home mount settings to be defined in ~/.netatalk/afp.conf, IMO would just an extension of the same override logic. However, only implementing the username override in the main afp.conf keeps it simple and focused on value (patterns that are most likely to actually be used = only settable by the admin). And we still need to sort out Netatalk startup as non-root first. So definitely keep it simple and only in the main afp.conf for now. Ideally we should allow any property under [homes] to be override-able. There are clear use cases for different permission bitmaps for example. |
|
PS; Spotlight is working fantastically well with CNID after the bugfix. Love how fast it is! Less than 1 second to search >250k and display results. And searching >1M is indistinguishable (fraction of a second slower). 🚀 And performance regression is resolved 🫡 Just the 100 results limit that gets in the way - otherwise near perfect! |
|
filed #2963 to follow up on the granular home dir settings feature! I think we're very close to a non-root operation mode of netatalk -- what I'm thinking is adding logic to the netatalk controller daemon when detecting non-root uid, attempts to create ~/.netatalk/afp.conf with a suite of default options including server signature, vol uuid, vol dbpath with cnid scheme = sqlite ... and if the file already exists we read it as-is instead of course |
this was an unexpected lucky break, I had anticipated hours to debugging :-D with the PR at hand now indexing all home dirs by default, I'm starting to get cold feet about this idea of using localsearch as the default -- maybe I'm overthinking it but isn't there privacy implications of having a global-readable index that includes both filenames and fragments of file contents that any user with the right know-how can query? this concern goes for both localsearch and xapian. the xapian backend is arguably safer since it only indexes when a volume is mounted, and it should only be indexing contents of files that are global-readable in the first place (need to double check the logic, but this is what I remember) the localsearch indexer is running as root and proactively indexing all volumes paths listed in |
|
Hi @rdmark Yes, there are absolutely serious privacy risks here, which is where the granular per-user settings proposal was coming from. And at the very least, we need to ensure user A cannot get results for user B's home index! The safest solution is to not index any homes for now until we have a plan for per-user compartmentalisation of the index data? Indexes for shared volumes is an another potential issue. We also have to consider that shares may be restricted to certain users as well. Which means only those users should be able to get search results for those shares they are allowed to access.. I don't know the data structure for localsearch or xapian (to understand if there is a 'path' in the data to filter/restrict user queries only to their allowed paths)? This should be easy with the CNID backend as the "index" is already a table per volume, so search can be restricted per user only to tables they are allowed easily? |
Why would we try to auto-create the per-user ones? Best to let users create them if they want/need them. The default [homes] set by the admin should work for most, and these are only for overrides for users who need overrides (which should be the exception). I can see situations where users complain because homes are broken for some users and admins don't know why (because they don't know about the auto-created per user config), and that config is now stale after a config version/change or something? |
the use case I had in mind was that a normal user wants to run an ad-hoc AFP server, and runs this fails for a number of reasons, notably lack of write permissions to the netatalk localstatedir, which means it cannot edit afp_signature.conf or afp_voluuid.conf, or create a subdir in CNID/ to store the database but I see your point, the better approach is probably to instruct users who want to run an ad-hoc non-privileged AFP server to craft their own afp.conf in their home dir and then launch |
|
pulling this back to draft while we figure out the secure way to index home dirs! |
|
Before reinventing the wheel... how does Apple's Spotlight indexing work with a user's shared home folder on a Mac hosted server? Heck, how does Samba handle it? Looking on my Mac, it appears the index for a user's home is stored in |
Nice, looks like an entirely self contained index - which is the right way to go for both secure access and secure at rest.. @rdmark would this mean multiple localsearch processes (one per user)? |
|
yeah, multiple localsearch processes with multiple dconf db configuration files... potentially also multiple private D-Bus buses but I agree that for privacy's sake, an individual home dir index should be stored in the respective home dir (while making sure the index isn't indexed, e.g. skip dot files/dirs) -- should be straight forward to achieve with the xapian backend at least |
3 participants
The [Homes] spotlight option (default: inherits from [Global]) controls whether the basedir regex path is submitted to the search indexer, covering all user home directories beneath it.
Because [Homes] volumes are resolved per-user at login and never pass through creatvol() in the master process, OPTION_SPOTLIGHT must be set explicitly during config parsing when Homes spotlight is enabled -- otherwise the localsearch dbus infrastructure is never started.
Add INIPARSER_GETBOOL macro parallel to the other iniparser wrapper macros.