Multimodal Emotion & Engagement
A multimodal engagement analytics platform that fuses real-time facial emotion convolutional networks with transformer-based natural language sentiment analysis using a 14-dimensional KNN late-fusion layer served over FastAPI.
Project Overview
A multimodal engagement analytics platform that fuses real-time facial emotion convolutional networks with transformer-based natural language sentiment analysis using a 14-dimensional KNN late-fusion layer served over FastAPI.
Problem Statement
Single-modality emotion and sentiment recognition systems suffer from high variance and deception susceptibility. Facial expression classifiers fail under variable illumination, head pose changes, and voluntary expression suppression. Conversely, text-only sentiment models miss sarcasm, stress cues, and emotional intensity. Combining heterogeneous modalities (visual facial embeddings and natural language text tokens) requires robust feature normalization and low-latency inference fusion.
Objectives & Scope
Project Objectives
- 01Train a robust convolutional network on FER2013 for 7-class facial emotion classification
- 02Integrate DistilRoBERTa for granular text sentiment and engagement score extraction
- 03Construct a 14-dimensional late-fusion layer combining multi-modal probability vectors
- 04Serve low-latency real-time inference using FastAPI asynchronous endpoints
- 05Evaluate cross-modality accuracy gains against single-modality baselines
Scope & Boundary
Dual-modality pipeline combining facial expression video frames with textual sentiment input. Serves real-time inference via REST APIs for engagement scoring and behavioral analytics. Audio speech prosody analysis is excluded from the current prototype.
Threat Model & Requirements
Technical Requirements & Integrity Bounds: Sub-150ms total inference budget for live interaction tracking; graceful degradation when one modality is occluded or absent; input validation preventing image buffer overflow attacks and prompt payload injection into the API service.
Architecture
Incoming requests provide concurrent frame data and text strings. Frame data passes through OpenCV Haar/DNN cascades for facial cropping, followed by CNN forward pass producing a 7D visual probability vector. Text passes through DistilRoBERTa tokenizer and transformer forward pass producing a 7D sentiment vector. The vectors concatenate into a 14D embedding evaluated by the KNN fusion model to output the final engagement state.
+-------------------------------------------------------------------+ | MULTIMODAL FUSION INFERENCE PIPELINE | +-------------------------------------------------------------------+ | | | Video Feed -> [OpenCV Face Crop] -> [Keras CNN] -> 7D Vector | | | | | v | | Text Input -> [DistilRoBERTa] -> [Transformer] -> 7D Vector | | | | | v | | +-----------+ | | | 14D KNN | | | | Late | | | | Fusion | | | +-----------+ | | | | | v | | FastAPI Service <--------------------------- [Engagement State] | | | +-------------------------------------------------------------------+
Multimodal Emotion & Engagement Architecture Decomposition
Cross-modal affective computing pipeline fusing facial micro-expression analysis and acoustic prosody to evaluate real-time participant engagement and synthetic deepfake anomaly detection.
Multimodal Analytics Pipeline
Computer Vision Stream
OpenCV & MediaPipe
ResNet-50 Feature Extractor
Acoustic Prosody Stream
Librosa Mel-Spectrogram
Wav2Vec 2.0 Encoder
Cross-Modal Attention Fusion
Cross-Attention Transformer
Deepfake Desync Detector
Serving & Telemetry
ONNX Runtime Engine
FastAPI Streamer
Implementation
Developed a multimodal engagement analytics platform fusing real-time computer vision with transformer-based NLP. The visual pipeline employs an OpenCV face detection stage feeding a convolutional neural network (CNN) trained on the FER2013 dataset. The text pipeline processes linguistic input using a fine-tuned Hugging Face DistilRoBERTa model. Extracted feature embeddings from both models are normalized into a unified 14-dimensional feature vector and fused using a late-fusion K-Nearest Neighbors (KNN) classifier. The complete inference pipeline is exposed via high-throughput FastAPI endpoints.
Technologies & Rationale
Asynchronous Python web framework enabling concurrent video frame and text payload ingestion with minimal serialization latency.
Distilled transformer retaining 95% of RoBERTa's contextual language representation capabilities while executing inference twice as fast with a 40% smaller memory footprint.
Lightweight multi-layer convolutional architecture optimized for rapid frame-by-frame emotion probability distribution output.
Late-fusion architecture decouples modality retraining, enabling dynamic re-weighting of visual versus textual confidence scores under noisy conditions.
Testing Methodology
CNN evaluated on FER2013 test dataset achieving baseline validation accuracy consistent with compact CNN architectures. DistilRoBERTa evaluated on sentiment benchmarks. Fusion layer tested across synthetic scenarios containing conflicting modality signals (e.g., smiling facial expression with negative text sentiment) to assess late-fusion arbitration behavior.
Results & Evidence
Multimodal fusion demonstrated higher classification confidence on ambiguous test samples than single-modality models. FastAPI asynchronous pipeline maintains average inference turnaround of ~110ms on GPU-assisted hardware.
Limitations & Trade-offs
FER2013 dataset contains class imbalances and human label noise. KNN late-fusion is heuristic-based and lacks cross-attention mechanism weights. Real-time webcam streaming requires stable client-side encoding.
Future Improvements
Incorporate attention-based cross-modal transformers (e.g., Multimodal Transformers) replacing KNN fusion. Add vocal prosody acoustic analysis. Explore cybersecurity applications in social engineering detection and interview fraud prevention.
Resources & Code
Repository URL: https://github.com/prajwalk-cyber/EmotionDetection