EmotionGesture AI

Real-time computer vision demos usually fail in the same two places, and neither one is about model accuracy.

First, flicker. Classifying each frame independently means a single bad frame flips the label, so the output jitters between answers even when the user's hand is perfectly still.

Second, stalling. Emotion models are far slower than hand tracking. Running everything in one loop means the whole pipeline runs at the speed of its slowest component, and the frame rate collapses.

The interesting work here was structural rather than a question of model selection — getting several components of very different speeds to share one camera feed and still feel instant.

1. Capture moved to its own thread

   VideoStream reads the camera on a background thread and the pipeline always pulls the most recent frame. Processing can never block capture, and a slow frame is dropped rather than queued into steadily growing latency.

2. A temporal stabiliser to kill flicker

   Gesture classification is smoothed across frames before it is displayed, so a single misread frame cannot flip the visible label. This is the difference between a demo that looks broken and one that looks solid.

3. A 1€ filter per hand

   Landmark coordinates are smoothed with a per-hand 1€ filter, chosen specifically because it suppresses jitter while a hand is still without adding perceptible lag when it moves quickly — the failure mode of a simple moving average.

4. Emotion detection on its own worker

   The slowest component runs threaded and off the critical path, so hand tracking keeps its frame rate regardless. DeepFace is the default backend; FER is supported behind an --emotion fer flag but left opt-in, since its PyPI packaging pins a broken ffmpeg==1.4.

5. One typed result object

   Every component writes into a single typed FrameResult, which the OpenCV overlay and the Streamlit dashboard both consume. Adding a new output surface means reading one struct, not rewiring the pipeline.

6. Built like a product, not a notebook

   SOLID module boundaries, typed configuration, structured logging and a pytest suite — plus a documented upgrade path from the current rule-based gesture engine to a trained YOLOv8 model, so the rules are a deliberate starting point rather than a ceiling.

1. Install

   pip install -r requirements.txt

2. Run the dashboard

   The Streamlit control panel shows FPS, current gesture, finger count, emotion and confidence breakdown, with live camera, hand and face status indicators. Camera selection and backend switching live in the sidebar.

   streamlit run app.py

3. Or run the annotated preview

   The OpenCV window draws landmarks, bounding boxes and labels straight onto the video feed.

   python main.py

4. Switch the emotion backend

   Optional — requires installing fer manually first.

   python main.py --emotion fer

5. Run the tests

   pytest

Read the source on GitHub ↗