feat(mac): add a self-contained macOS port under mac/ - #282
Closed
Tongyun1 wants to merge 13 commits into
Closed
Conversation
Introduce a self-contained macOS version under mac/ (Avalonia, net10.0) without touching the existing Windows codebase. - Audio: SystemAudioCapture captures system audio from an input device (e.g. BlackHole) via SoundFlow/miniaudio, resampled to 16kHz mono. - Captions: WhisperCaptionSource transcribes locally via Whisper.net (Metal-accelerated), implementing the ICaptionSource abstraction as the macOS equivalent of the Windows LiveCaptions/UI Automation reader. - Translation: independent translation layer with Google (no key) and OpenAI-compatible engines, selected via TranslationService. - UI: MainWindow shows device selection, engine selection, live original and translated text. - Docs: mac/PROGRESS.md tracks progress and BlackHole setup for testing.
The Whisper recognition loop only guarded Task.Delay, so any exception from a transcription cycle terminated the background loop permanently, freezing captions (e.g. after pausing/resuming playback). - Wrap the whole loop body in try/catch: a single failed cycle is logged and skipped instead of killing the loop. - Skip pure-silence chunks to avoid Whisper hallucinations and wasted compute during paused audio. - Mark caption + translation pipeline as verified on real hardware in PROGRESS.md.
Add a borderless, always-on-top, semi-transparent overlay window that mirrors the caption/translation from the main window. - Draggable (move) and resizable via a manual bottom-right grip (BeginResizeDrag is unreliable on Avalonia macOS borderless windows). - Toggle button in MainWindow; original + translation forwarded live. - MacWindowInterop: pure-C# Objective-C runtime P/Invoke sets the NSWindow collectionBehavior (canJoinAllSpaces | fullScreenAuxiliary) and level, re-applied periodically so it stays above content across Spaces and the app's own fullscreen. Known limitation: a plain NSWindow cannot float over another app's native fullscreen Space (that requires an NSPanel, which Avalonia does not expose). Workaround documented in PROGRESS.md: use "webpage fullscreen" instead of true fullscreen.
- AppSettings + SettingsStore: persist Whisper model, preferred audio device, translation engine/target language/OpenAI config to ~/Library/Application Support/LiveCaptionsTranslator/settings.json. - SettingsWindow: edit model, engine, target language, and OpenAI (URL/key/model) with a masked key field. - MainWindow: add a Settings button; load persisted settings on startup, remember the selected audio device and chosen model across launches. - Add mac/README.md documenting setup (BlackHole), build/run, usage, features, known limitations, and data locations.
README now covers progress and status, so the separate PROGRESS.md is no longer needed. Also remove the dangling link to it from README.
Convert the Avalonia NSWindow to an NSPanel subclass at runtime so the overlay can float above other apps' fullscreen Space (e.g. Chrome on Bilibili), which a plain NSWindow cannot do regardless of level or collectionBehavior. - MacWindowInterop dynamically creates an _AvnFloatingPanel class that inherits from NSPanel and copies over all custom methods from Avalonia's original window class (getExtendedTitleBarHeight etc.). Without this, object_setClass to NSPanel crashes when AppKit calls Avalonia-specific selectors during resize. - After class swap, sets nonactivatingPanel style bit, isFloatingPanel, hidesOnDeactivate=false, becomesKeyOnlyIfNeeded=true, canJoinAllSpaces | fullScreenAuxiliary, and NSStatusWindowLevel. - OverlayWindow: convert once on Opened plus a delayed re-apply to guard against Avalonia post-init resets. Verified on Chrome/Bilibili real fullscreen; resize no longer crashes. - README: overlay now supports true-fullscreen overlay; removed the matching known-limitation entry.
Store every successful translation in a local SQLite database at ~/Library/Application Support/LiveCaptionsTranslator/history.db. - HistoryStore: insert/query/search/clear plus CSV export. Since Whisper emits progressive results for one utterance, a new entry whose source text extends the previous one overwrites it instead of appending, so the history holds complete sentences rather than every partial. CSV is written directly (UTF-8 BOM, quoted fields) to avoid pulling in a CSV dependency. - HistoryWindow: paged list of original + translation, debounced search, CSV export via the save file picker, and clear with a confirm step. - MainWindow: add a History button; log each translation, skipping engine failures (results prefixed with [ERROR]). - README: document the history feature and history.db location.
Quitting the app aborted with: NSRangeException: Cannot remove an observer for the key path "_windowLayerContext" from <_AvnFloatingPanel> because it is not registered as an observer. KVO tracks observers via the object's isa, so replacing the overlay's class with _AvnFloatingPanel invalidated the registrations AppKit had already made. Tearing the window down then failed inside -[NSWindow dealloc] and aborted the process. - MacWindowInterop.ConvertToFloatingPanel now returns the previous class, and RestoreClass puts it back; OverlayWindow restores on Closing so the window reaches dealloc with the class KVO registered against. Restore is skipped if the isa is no longer our class, to avoid clobbering a later KVO swizzle. - Skip NSKVONotifying_* levels when copying Avalonia's custom methods into the dynamic class; those are KVO's own notification wrappers. - MainWindow now hides the overlay instead of closing it, so no window is destroyed inside the main window's native windowWillClose: teardown. App uses ShutdownMode.OnMainWindowClose so a hidden overlay cannot keep the process alive (verified: the process still exits on its own).
LiveCaptionsTranslator.csproj sits in the repository root, so its implicit SDK globs pick up every .cs file below it, including the 23 sources of the macOS project (and generated files under mac/obj). The Windows build then tried to compile Avalonia code against net8.0-windows and failed, which is why CI broke on the macOS PR. Removing mac/** from Compile and None leaves the Windows project with the same 31 sources under src/ as before.
A Multi-Output Device disables the menu-bar volume control (a macOS limitation, not an app issue). Document adjusting volume via the speakers' slider in Audio MIDI Setup.
The project targets net10.0, so the stated ".NET SDK 8 or higher" requirement was wrong: SDK 8 cannot build it. State SDK 10, list what to install with the exact brew commands, and add a version check. Also document self-contained publishing for handing the app to someone without .NET, including the Gatekeeper workaround, and note that only Apple Silicon was actually tested.
…dels Downloading the Whisper model from Hugging Face is slow or unreachable on some networks, and the smallest option was still 141 MB. Download changes: - Try ModelScope first, then fall back to Hugging Face. ModelScope serves from a China-hosted CDN (cdn-lfs-cn-1) rather than redirecting abroad, measured ~3x faster here; both files verified byte-identical (same length, same SHA-256 over the leading 2 MB). - Resume partial downloads via Range, validate the final length, and delete the temp file on mismatch so a truncated file is never reused. - Send a User-Agent: ModelScope answers 403 without one, and HttpClient omits it by default (curl does send one, which masked this at first). - On failure, report every source tried plus where to drop the file manually; a manually placed model is used as-is without network access. - Selectable download source (auto/ModelScope/Hugging Face) and an optional custom base URL in Settings. Model changes: - Add quantized variants, cutting the smallest download to 31 MB: Tiny-Q5 31 MB, Base-Q5 57 MB, Small-Q5 181 MB, Medium-Q5 514 MB. Note medium only ships q5_0 upstream, the others use q5_1. - The model dropdown now shows size and trade-off instead of a bare enum name, and maps by index so display text is never parsed back. - Wrap the status text so multi-line download errors stay readable. Verified: fallback, cached-model short-circuit, resume (resumed file hashes identical to a full download), and the failure message; all 8 model file names resolve on both sources; existing settings.json still loads after the enum gained values.
The README was written for developers: it led with library choices and publish commands, which is not what someone installing this for the first time needs. - Restructure around the actual install order: what to install and why BlackHole is unavoidable, then four numbered steps (with the reboot), then how to use it, then troubleshooting. - Fix the stated requirement: the project targets net10.0, so SDK 8 cannot build it. Also cover `command not found: dotnet` needing a fresh terminal, and add `git clone` so the build steps stand alone. - Add a troubleshooting section built from problems actually hit: wrong system output device producing no captions, BlackHole missing until a reboot, model download failures (mirror, smaller models, manual placement), and stopping before changing the model. - Note that captions lag 1-3 seconds by design, and that audio stays local while recognised text does go to the translation service. - Move developer material behind a divider and drop the packaging section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
在
mac/下加了一个 macOS 版本，33 个新文件，2400 行左右。与windows版本代码分离，不影响原有功能。