Lumora

Lumora is a highly advanced multi-modal AI platform that goes beyond simple transcription. It understands the entire context of a lecture—both visual and auditory—to generate structured, enriched, and accurate markdown notes.

🧠 Multi-Modal AI Pipeline

Lumora processes educational content through a complex 9-step intelligent pipeline:

1. Dynamic Ingestion: Accepts direct audio/video uploads or YouTube URLs.
2. Audio & Vision Extraction: Isolates pristine audio tracks while sampling keyframes from the video to capture slides and blackboard writings.
3. AI Transcription & OCR: Converts speech to text and passes visual frames through advanced vision models to capture on- screen context.
4. Data Fusion & Segmentation: Intelligently merges transcripts with visual data into a unified timeline, breaking it down into coherent chapters and topics.
5. Enrichment Engine: Synthesizes data into formatted markdown with adjustable depth:
   • CLASS_ONLY: Strictly retains the professor's words.
   • MODERATE/DEEP: Uses the Tavily API and OpenAI to actively web-search claims, add citations, and intelligently flag any contradictions between the lecture and external authoritative sources.

🏗️ Robust Enterprise Architecture

The platform operates on a resilient, distributed, three-tier architecture:

• Frontend (Next.js & TailwindCSS): A sleek, highly responsive UI featuring sophisticated dark mode aesthetics, glassmorphism, grainy gradients, and real-time AI job tracking (EXTRACTING → TRANSCRIBING → VISION → FUSING → ENRICHING).
• Orchestrator Backend (Spring Boot & Java 21): The core API gateway backed by PostgreSQL 16. It handles Google OAuth, JWT authentication, and acts as the central database orchestrator.
• Resilient AI Worker (Python & FastAPI): A distributed processing layer connected via AWS SQS for asynchronous job queueing. It features intelligent checkpointing—if a job fails, the worker automatically resumes from the last successful pipeline stage without losing progress.
• Cloud Infrastructure: Deeply integrated with AWS S3 for secure artifact storage and AWS SQS for decoupled message queueing (mocked locally via Localstack and Docker).