AI-Powered Musical Learning with Real-Time Visual Feedback

An interactive music learning platform that combines pitch detection, chord recognition, and physical LED feedback to help musicians practice effectively—no MIDI required!

Posted on December 26, 2025 by Mikel Bahn
  1. Home
  2. Blog Overview
  3. Blog Post
Music Trainer Suite - Documentation

Music Trainer Suite

AI-Powered Musical Learning with Real-Time Visual Feedback

Project Overview

The Music Trainer Suite is a collection of interactive musical learning tools that combine AI-powered audio analysis with visual feedback systems. This project enables musicians to practice melodies, chords, and arpeggios with real-time pitch detection and optional LED strip visualization.

Key Innovation: By implementing both MIDI-based and audio-based detection, this suite is accessible to everyone—whether you have a MIDI keyboard or just an acoustic instrument and a microphone.
Audio Input
→
AI Analysis
→
Visual Feedback
→
Learning

Project Structure

Software Modules

  • Full Keyboard Trainer (88 keys)
  • Extended Trainer (single octave)
  • Chord Trainer
  • Arpeggio Trainer
  • Arduino-Integrated Version

Hardware Components

  • Arduino LED Controller
  • WS2811 LED Strip (60 LEDs)
  • Ultrasonic Motion Sensor
  • Touch Brightness Controls
  • Optional: MIDI Keyboard

Software Trainer Modules

Each trainer module is a standalone application with its own pedagogical focus. All modules share a common architecture but implement different detection strategies based on musical structure.

Full Keyboard Trainer

Location: music_trainer_full_keyboard/

Purpose: Learn melodies note-by-note with visual feedback across the entire 88-key piano range.

Features:

  • Visual piano keyboard spanning C1 to C8
  • Real-time pitch detection using Aubio
  • Highlighted target notes in red
  • Configurable tolerance and timeout settings
  • Lives system for practice management

Use Case: Perfect for practicing melodic sequences, songs, and scales.

Extended Trainer (Single Octave)

Location: music_trainer_extended/

Purpose: Focused practice on a single octave for beginners or specific exercises.

Features:

  • Simplified interface showing only one octave
  • Larger key visualization for easier viewing
  • Same pitch detection engine as full keyboard
  • Ideal for focused practice sessions

Use Case: Great for beginners or practicing specific patterns in a limited range.

Chord Trainer

Location: music_trainer_chords/

Purpose: Practice and recognize chord progressions through simultaneous note detection.

Features:

  • Chromagram-based chord recognition using Librosa
  • Template matching against known chord shapes
  • Support for major, minor, and custom chords
  • User-defined chord sequences at startup
  • Confidence scoring for chord matches

Use Case: Learn chord progressions for songs, practice transitions, build muscle memory.

Technical Note: Chord detection aggregates audio over 1-second windows and analyzes the frequency distribution across all 12 chromatic pitches using Short-Time Fourier Transform (STFT).

Arpeggio Trainer

Location: music_trainer_arpeggios/

Purpose: Practice arpeggios (broken chords) where notes are played sequentially.

Features:

  • Sequential note detection for arpeggio patterns
  • Configurable arpeggio sequences
  • Real-time pitch tracking with Aubio
  • Progress through complex patterns step-by-step

Use Case: Master arpeggio patterns, improve finger dexterity, practice scales in broken form.

Arduino-Integrated Full Keyboard

Location: music_trainer_full_keyboard_arduino/

Purpose: Full keyboard trainer with LED strip feedback for enhanced visual learning.

Features:

  • All features of the full keyboard trainer
  • Real-time LED feedback on physical strip
  • Green pulsing LED for target notes
  • Color-coded feedback (green=correct, red=incorrect)
  • Auto-detection of Arduino connection

Use Case: Immersive learning experience with synchronized on-screen and physical LED feedback.

AI & Signal Processing Technology

The Music Trainer Suite leverages state-of-the-art audio analysis libraries to convert sound waves into musical information in real-time.

Core Technologies

1

Audio Capture - SoundDevice

Raw audio is captured from the system microphone at 44.1kHz (melody/arpeggio) or 22.05kHz (chord) sample rate. The audio stream is processed in real-time using callback functions that analyze small chunks (buffers) of audio data.

2

Pitch Detection - Aubio YIN Algorithm

For single-note detection (melodies, arpeggios), the Aubio library implements the YIN algorithm—a sophisticated time-domain fundamental frequency estimator. YIN analyzes the autocorrelation of the audio signal to identify the fundamental frequency (pitch) with high accuracy, even in the presence of harmonics and noise.

pitch_o = aubio.pitch("default", buf_size*2, buf_size, samplerate)
pitch_o.set_unit("Hz")
pitch = pitch_o(audio_samples)[0]  # Returns frequency in Hz

Volume Gating: RMS (Root Mean Square) energy is calculated to ignore silence and background noise, ensuring only intentional notes are detected.

3

Chord Recognition - Librosa Chromagram

For chord detection, the system uses Librosa's chromagram feature extraction. This process involves:

  1. STFT (Short-Time Fourier Transform): Converts time-domain audio to frequency domain
  2. Chroma Feature Extraction: Maps all frequencies to 12 pitch classes (C, C#, D, ..., B)
  3. Temporal Averaging: Aggregates chroma values over a 1-second window
  4. Template Matching: Compares the chroma vector against predefined chord templates using dot product similarity
chroma = librosa.feature.chroma_stft(y=audio, sr=samplerate)
chroma_mean = np.mean(chroma, axis=1)  # 12-dimensional vector
score = np.dot(chroma_mean, chord_template)  # Match against template

This approach allows detection of multiple simultaneous notes, enabling chord recognition.

4

Frequency-to-Note Conversion

Detected frequencies are converted to musical note names using the equal temperament formula:

semitones = 12 × log₂(freq / 440Hz)
midi_number = 69 + round(semitones)
note = NOTE_NAMES[midi_number % 12] + octave

This maps continuous frequency values to discrete musical notes with octave information.

✨ Why This Matters: These AI-powered techniques enable accurate pitch detection from any sound source—acoustic instruments, voice, or electric instruments—without requiring MIDI connectivity. The system is instrument-agnostic and works in real-world conditions.

Detection Strategy Comparison

Module Algorithm Window Size Detection Type
Full Keyboard / Arpeggio Aubio YIN 1024 samples (~23ms) Single pitch, real-time
Chord Trainer Librosa Chromagram 1 second Multiple pitches, polyphonic
Extended Trainer Aubio YIN 1024 samples (~23ms) Single pitch, real-time

Arduino LED System

The LED controller is a standalone project that enhances the learning experience with physical, ambient feedback. It operates independently with three distinct modes and can be integrated with the Python trainer for synchronized visual feedback.

Independent Operation: The Arduino LED system works on its own without the Python trainers. It provides motion-reactive patterns, MIDI visualization (via separate MIDI script), and trainer integration when connected.

Hardware Setup

Components

  • Arduino Board: Any Arduino with sufficient I/O (Uno, Nano, Mega)
  • WS2811 LED Strip: 60 individually addressable LEDs (5 octaves coverage)
  • HC-SR04 Ultrasonic Sensor: Motion detection (trigger pin 7, echo pin 8)
  • Touch Sensors: Two capacitive sensors for brightness control (pins 4 & 5)
  • Power Supply: 5V adequate for LED current draw

Operating Modes

Mode 1: Motion-Activated Patterns

Activation: Ultrasonic sensor detects movement within 120cm

Behavior: Displays dynamic LED patterns (rainbow, confetti, sinelon, juggle, BPM sync)

Features:

  • Patterns auto-cycle every 10 seconds
  • Stays active for 60 seconds after last motion
  • Energy-efficient: LEDs off when no motion

Visual Indicator: White flash when entering mode

Mode 2: MIDI Keyboard Visualization

Activation: Serial commands from MIDI-to-LED Python script (not documented here)

Behavior: LEDs light up cyan-blue corresponding to pressed piano keys

Features:

  • Real-time key press visualization
  • Velocity-sensitive brightness
  • Smooth fade-out when keys released
  • 60 LEDs map to MIDI notes 36-96 (C2 to C7)

Visual Indicator: White flash when entering mode

⚠️ Note: This mode requires a separate Python script that reads MIDI input and sends LED commands to the Arduino. That script is not covered in this documentation but follows a similar architecture to the trainer integration.

Mode 3: Melody Trainer Integration

Activation: Automatic when running music_trainer_full_keyboard_arduino

Behavior: Synchronized feedback with on-screen piano trainer

Features:

  • Green pulsing LED: Indicates target note to play
  • Yellow-green flash: Correct note played ✓
  • Red flash: Incorrect note played ✗
  • Feedback persists for 500ms for clear visibility

Visual Indicator: ★ Blue flash when entering mode ★

Control Interface

Action Control Method Effect
Increase Brightness Tap touch sensor (pin 4) +3 brightness units
Decrease Brightness Tap touch sensor (pin 5) -3 brightness units
Cycle Modes Hold both sensors for 1 second Motion → MIDI → Trainer → Motion...

Serial Communication Protocol

The Arduino listens for commands over serial (115200 baud). This enables integration with Python scripts:

Command Format Description
Switch Mode M1\n, M2\n, M3\n Force mode switch (1=Motion, 2=MIDI, 3=Trainer)
Set Target Note T<index>\n Light up LED at index as target (green pulse)
Played Note Feedback P<index>,<correct>\n Show feedback at LED index (0=wrong/red, 1=correct/yellow-green)
MIDI LED Control L<index>,<brightness>\n Set LED brightness (0-255) for MIDI mode
Clear All C\n Turn off all LEDs

Full Integration: Software + Hardware

The music_trainer_full_keyboard_arduino module combines the software trainer with the LED hardware, creating a fully immersive learning environment.

Integration Architecture

Microphone Input
→
Aubio Pitch Detection
↓
Melody Trainer Logic
↓
Pygame Display
Arduino LED Strip

How It Works

1

Initialization

When the trainer starts with use_arduino=True, it auto-detects the Arduino via USB and sends command M3\n to switch to Trainer mode. The Arduino responds with a blue flash confirmation.

2

Target Note Display

For each note in the melody, the trainer:

  • Highlights the note on the on-screen piano (red)
  • Sends T<led_index>\n to Arduino
  • Arduino displays a green pulsing LED at the corresponding position
3

Real-Time Feedback

As you play your instrument:

  • Aubio detects the pitch and converts to note name
  • Trainer compares detected note to target (±0.5 semitone tolerance)
  • If correct: sends P<led_index>,1\n → yellow-green LED flash
  • If wrong: sends P<led_index>,0\n → red LED flash
4

Progression

When the correct note is played, the trainer advances to the next note automatically, clearing the previous LED and lighting up the new target.

LED Mapping Details

60 LEDs Cover 5 Octaves: The LED strip spans MIDI notes 36 to 96 (C2 to C7), with each note mapped linearly to LED indices 0-59. Middle C (C4, MIDI 60) corresponds to approximately LED index 24.

Advantages of Dual Feedback

Screen Display

  • Precise note identification
  • Musical context (full keyboard)
  • Lives counter and score
  • Portable (laptop only)

LED Strip

  • Ambient, peripheral awareness
  • No need to look at screen
  • Physical, tactile environment
  • Motivating color feedback
The Best of Both Worlds: Looking at a screen can distract from your instrument. The LED strip provides feedback in your peripheral vision, letting you focus on your technique while still receiving clear visual cues.

Setup Guide

Software Installation

1

Install Python Dependencies

pip install pygame sounddevice aubio numpy librosa pyserial
2

Choose Your Trainer Module

Navigate to the appropriate directory:

  • music_trainer_full_keyboard/ - Full 88-key piano
  • music_trainer_extended/ - Single octave
  • music_trainer_chords/ - Chord recognition
  • music_trainer_arpeggios/ - Arpeggio sequences
  • music_trainer_full_keyboard_arduino/ - With LED integration
3

Run the Trainer

python main.py

For Arduino-integrated version:

# In main.py, enable Arduino support:
trainer = MelodyTrainer(
notes,
use_arduino=True,
arduino_port=None  # Auto-detect
)

Arduino Setup (Optional)

1

Install Arduino IDE & FastLED Library

Download Arduino IDE from arduino.cc

Install FastLED library: Tools → Manage Libraries → Search "FastLED"

2

Wire Hardware Components

Component Arduino Pin Notes
LED Strip Data Pin 9 WS2811 data line
Ultrasonic Trigger Pin 7 Motion sensor
Ultrasonic Echo Pin 8 Motion sensor
Brightness Up Pin 4 Touch sensor
Brightness Down Pin 5 Touch sensor
5V & GND Power rails Adequate PSU for LEDs
3

Upload Arduino Code

Open TripleModeLEDController.ino in Arduino IDE and upload to your board.

4

Test Connection

Open Serial Monitor (115200 baud). You should see:

=================================
TRIPLE MODE LED CONTROLLER
Modes: 1=Motion, 2=MIDI, 3=Trainer
Current Mode: MOTION

Quick Start Examples

Example 1: Practice a Simple Melody

# main.py
from melody_game import MelodyTrainer
notes = ["C4", "D4", "E4", "F4", "G4", "A4", "B4", "C5"]
trainer = MelodyTrainer(notes)
trainer.play()

Example 2: Practice Chords

# For chord trainer
Run and input chords at startup: C, Am, G, D
python main.py
Enter: C,Am,G,D

Example 3: Arduino LED Integration

# main.py in arduino folder
from melody_game import MelodyTrainer
notes = ["E4", "D4", "C4", "D4", "E4", "E4", "E4"]
trainer = MelodyTrainer(
notes,
use_arduino=True  # Enable LED feedback
)
trainer.play()

Troubleshooting

Common Issues

  • No audio detected: Check microphone permissions, adjust volume_threshold in audio.py
  • Wrong notes detected: Reduce background noise, increase tolerance parameter
  • Arduino not found: Specify port manually: arduino_port='/dev/ttyUSB0'
  • LEDs not responding: Hold both touch sensors for 1s to cycle modes, check Serial Monitor for mode confirmation
  • Pygame window issues: Update pygame: pip install --upgrade pygame

⚙️ Customization Options

Adjust Detection Parameters

Parameter Location Purpose Default
tolerance MelodyTrainer init Semitone tolerance for correct note 0.5
timeout MelodyTrainer init Seconds to wait for each note 3
volume_threshold PitchDetector init Minimum RMS to process audio 0.01
lives MelodyTrainer init Number of mistakes allowed 1000

Add Custom Chords

Edit utils.py in the chords folder:

CHORD_NOTE_MAP = {
"C": ["C4", "E4", "G4"],
"Am": ["A3", "C4", "E4"],
"G": ["G3", "B3", "D4"],
"Cmaj7": ["C4", "E4", "G4", "B4"],  # Add custom chords
# ... more chords
}

Modify LED Colors

In Arduino code, adjust these constants:

#define TRAINER_TARGET_HUE    96   // Target note color (0-255)
#define TRAINER_CORRECT_HUE   64   // Correct feedback color
#define TRAINER_WRONG_HUE     0    // Wrong feedback color
// Hue values: 0=Red, 32=Orange, 64=Yellow, 96=Green, 160=Blue, 192=Purple

Future Enhancements

Potential Software Features

  • Real-Time integration of harmonic elements (e.g. Chords)
  • Rhythm detection and timing feedback
  • Difficulty progression algorithms
  • Song library with popular melodies
  • Progress tracking and statistics

Potential Hardware Features

  • Wireless Arduino connection (Bluetooth or WLAN)
  • Longer LED strips for full 88-key coverage
  • OLED display for standalone operation
  • Battery-powered portable version
  • Multi-color zones for hand position guidance