SongDecrypter

A macOS (Apple Silicon) app for AI-assisted music transcription: turn audio into MIDI, then refine it — the AI does the first pass, and the editor gives you the visual tools to correct and steer the result.

Current milestone: the piano-first vertical slice, plus stems — load audio → optionally split it into stems (Demucs htdemucs_6s: vocals, drums, bass, guitar, piano, other) → transcribe the mix or any stem with the ByteDance high-resolution piano model (notes, velocities, sustain pedal) → edit in a piano roll with a CQT spectrogram underlay, confidence-colored notes, and synced audio/MIDI playback → export a Standard MIDI File.

Layout

• app/ — native SwiftUI app (piano-roll editor, playback, MIDI export)
• worker/ — Python inference worker (FastAPI). The app spawns it locally, but the same HTTP protocol works against a remote machine: point Settings → Worker URL at another host running the worker to offload processing.

Requirements

• Apple Silicon Mac, macOS 14+
• Xcode 15+ (project generated with XcodeGen)
• uv (brew install uv xcodegen)

Setup & run

# 1. Python worker (one-time; also happens automatically on first launch)
cd worker
uv sync

# 2. App
cd ../app
xcodegen generate
xcodebuild -project SongDecrypter.xcodeproj -scheme SongDecrypter -configuration Debug -derivedDataPath build build
open build/Build/Products/Debug/SongDecrypter.app

Or open app/SongDecrypter.xcodeproj in Xcode and hit Run.

On first Transcribe, the app auto-starts the worker (configure the worker directory in Settings if you move the repo) and the worker downloads the piano transcription checkpoint (~165 MB) — later runs are fast and fully offline. Inference runs on the GPU via MPS.

Worker API

GET  /health                      → {status, version, device}
POST /jobs/transcribe             {audio_path} → job state
POST /jobs/structure              {audio_path} → repeated-section groups
GET  /jobs/{id}                   → {status, stage, progress, error}
GET  /jobs/{id}/result            → notes/pedals/beats/spectrogram JSON
GET  /jobs/{id}/artifact/{name}   → cqt.png, transcription.mid

Run it standalone (e.g. on a remote GPU box):

cd worker && uv run uvicorn songdecrypter_worker.main:app --host 0.0.0.0 --port 51735

Editor cheatsheet

• Separate stems (grid button or File menu): the popover picks the Demucs model (Standard/Fine-tuned 4-stem, or 6-stem with guitar & piano) and which instruments are actually in the mix — anything unchecked is kept together as one coherent Rest stem instead of being smeared into wrong buckets. For synth demos: check only Drums (+ Vocals/Bass if present); all synths land in Rest. A stems bar appears — click a chip to switch the editor to that stem (each source keeps its own transcription; green dot = transcribed). The loop region survives switching. Save Stem… exports the active stem's audio.
  • Synth refinement (shown when Synths is selected): the synth stem is the residual (mix minus everything lifted out), so its worst contaminants are vocal and drum bleed. Cleaner vocal removal subtracts a BS-RoFormer vocal estimate (several dB cleaner than demucs) instead of demucs' own; Remove percussive bleed keeps only the harmonic part via HPSS, dropping the transients a polyphonic transcriber reads as phantom notes.
• Per-source transcriber (picker next to Transcribe, auto-picked per stem): Piano (high-res) — piano stem & mix, with pedal + ghosts; Polyphonic (Basic Pitch) — synths, guitar, Rest; Melody (mono) — bass and vocal lines, one clean line; Drums (neural) — ADTOF-pytorch CRNN (kick/snare/hat/tom/cymbal, trained on 359h of real music; onset heuristic as fallback), exported on MIDI channel 10. Candidate comparison currently supports the piano transcriber.
• Drum-lane editor: drum sources swap the piano roll for named kit lanes (Crash → Kick, plus a Perc catch-all). Hits are dots sized by velocity; drag between lanes to reassign a hit, double-click a lane to add one. The spectrogram underlay shows transient spikes for eyeballing timing.
• Space play/pause · ⇧Space play from selection · ⌘⏎ return to start · click the time ruler (or ⌥-click anywhere) to seek/scrub
• Audio / MIDI toggles A/B the original recording against the transcription
• Loop: ⇧-drag the time ruler to set a region, or select notes and hit ⌘L; loop + Audio/MIDI toggles = rapid A/B of one section. ⌘L again to disable.
• Re-transcribe region (⇧⌘R or the circular-arrows button): run the model again on just the loop/selection span — with current noise-reduction settings — and splice the new notes in (undoable)
• Compare candidates (⌥⇧⌘R or the stacked-rectangles button): transcribe the region three ways — Standard, Sensitive (lower thresholds, catches quiet notes), Denoised (second pass on noise-reduced audio). The region auto-loops; switch takes in the bar to audition each against your current notes (shown as faint dashed outlines), click notes to exclude them, then Accept Take to splice it in (undoable) or Cancel to keep everything
• Consensus across repeats (⌃⌘R or the layered-cards button): for music that reuses sections — synth loops especially — the worker finds the repeated passages (beat-synchronous chroma self-similarity) and votes the transcription across every occurrence. A note most repeats agree on but the model dropped in one gets filled in; a one-off detection inside otherwise-identical audio gets flagged and left out; surviving notes' timing and velocity are median-averaged to shed jitter. It opens the same take bar with a Consensus take (the vote) and, when they differ, a Best Repeat take (the cleanest occurrence copied over — the "transcribe once, replicate" path for sequencer-exact loops). Set a loop to target one section; otherwise the widest repeat is used. Works on any transcriber, not just piano — repeats are the point for synth and guitar parts.
• Learned sensitivity: the editor quietly adapts the model's decode threshold to your corrections, per source. Confirming ghost notes nudges it to surface more detections; deleting low-confidence notes nudges it to keep fewer. The next Transcribe (or region re-transcribe) of that source starts from what you've been keeping — the status bar shows the learned percentage when it has drifted from neutral. It's saved with the project. No model training involved; it just re-decodes at a threshold tuned to your edits.
• Trackpad: two-finger pan in any direction · pinch to zoom time · ⌥-pinch to zoom pitch height (both anchored at the pointer)
• Undo/redo every note edit with ⌘Z / ⇧⌘Z
• Drag notes to move (pitch/time), drag a note's right edge to resize, double-click empty space to add a note, Delete to remove selection, shift-click for multi-select, ⇧⌘A select all, ⌘D deselect
• Arrow keys nudge selected notes (10 ms / semitone; hold ⇧ for 100 ms / octave)
• Velocity: ⌘-drag a note vertically (live vel N readout + audible preview; applies to the whole selection), or ⌘↑ / ⌘↓ for ±5. Drum hit dots resize live. The status bar shows velocity when one note is selected.
• ⌘J merges selected same-pitch notes
• Click a key (or drum lane label) in the left gutter to select every note on that pitch/lane; ⇧-click extends
• Pitch bends (Polyphonic transcriber): glides and portamento come back as bend curves drawn on the notes, exported as MIDI pitch-wheel events (GM ±2 semitone range) and audible in playback
• Tempo map (metronome button): the detected beat grid becomes editable — drag beat handles in the ruler, double-click inserts a beat, ⌥-click deletes, ⌘-click sets the downbeat (amber handle; bars count from it). Exports build a real variable-tempo track from the map, so drifting live recordings land on the DAW's musical grid. Playback → Reset Beats to Detected starts over.
• Lock & refine: any note you touch (move, resize, nudge, velocity, add, ghost-promote, merge) is auto-locked — shown with an amber ring — and survives re-transcription: region re-runs, candidate accepts, and full re-transcribes keep locked notes and drop incoming notes that collide with them, so the model refines around your corrections. ⇧⌘L locks/unlocks the selection explicitly; Unlock All is in the Notes menu. During candidate previews, locked notes stay solid and conflicting take notes show as amber dashed outlines (they won't be accepted).
• Notes are colored by model confidence (red = uncertain → green = confident); the waveform button toggles the spectrogram underlay for visual pitch alignment
• Ghost notes (dashed cyan outlines): detections just under the model's threshold — click one to confirm it as a real note
• Split-note links (dashed orange): same-pitch notes with a tiny gap that are probably one sustained note — click the link to merge, or select notes and hit ⌘J
• The sliders popover has zoom controls plus spectrogram opacity, noise floor (uniform darkening), and spectral denoise (per-frequency noise-profile subtraction — try it on noisy live recordings); all display-only and instant. Noise reduction there is the audio-side cleanup applied on the next Transcribe.
• Advanced options (same popover) refine what the model hears on the next Transcribe: Channel picks which part of a stereo recording to transcribe — Both (mono), Left, Right, or Side ((L−R): centered vocals/bass/kick cancel, isolating wide synth pads); Skip volume normalization leaves your own levels alone; Note sensitivity slides the decode thresholds for the piano and polyphonic models (higher surfaces quieter notes). The Ensemble pass votes several takes and keeps notes ≥2 agree on — raw/denoised/dereverbed on the piano model, a spread of sensitivities on the polyphonic model.
• Export via the share menu: ⌘E exports the current source, ⇧⌘E exports a multi-track MIDI file — one named track per transcribed source (Mix and stems), with GM programs per instrument, drums on channel 10, and the mix's detected tempo

Roadmap

• Per-song threshold calibration by pitch range, not just a global sensitivity
• Optional Basic Pitch fine-tuning on locked (user-confirmed) notes — the editor already collects the labeled corrections; the open question is overfitting to a handful of songs
• Bend curve editing (bends are currently display/export-only)
• Quantize-strength export (snap notes toward the tempo-map grid)
• BS-RoFormer separation models as higher-quality alternatives to Demucs
• Candidate comparison with different models (Basic Pitch etc.), not just different decodes of one model
• Editable tempo map (tap tempo, draggable beat grid) and quantized export
• Tuning-offset detection for live recordings; pitch-bend capture for synth glides
• Remote worker discovery (Bonjour) and audio upload for true remote offload