Skip to main content
JA jansher.aquib
Open Source 2024–2025

MySecondBrain: Open Source Knowledge Management System

Dockerized personal and organizational knowledge platform with Weaviate vector search, multi-format ingestion pipelines, and grounded OpenAI query resolution.

Weaviate OpenAI Python Docker PostgreSQL Next.js

Context

As personal and organizational document archives grow across notes, technical documentation, research papers, and meeting records, finding specific information using traditional keyword matching becomes tedious. Users need an intelligent search interface capable of conceptual understanding and direct source citations.

Problem

Building an effective knowledge assistant requires:

  1. Handling messy, heterogeneous file formats (TXT, DOCX, complex multi-column PDFs) with reliable text extraction and chunk boundary awareness.
  2. Generating high-quality vector embeddings and indexing them for fast semantic retrieval.
  3. Grounding model answers strictly in retrieved passages while preventing hallucinations and exposing verifiable source references.

What I Built

  • Robust Ingestion Pipeline: Built an ingestion service that parses raw TXT, DOCX, and PDF documents, extracts clean text, splits content using semantic chunking strategies, and extracts document-level metadata.
  • Weaviate Semantic Search: Integrated Weaviate vector database with hybrid search (combining dense vector embeddings with sparse lexical search) to maximize retrieval precision and recall.
  • Grounded Query Resolution: Designed an orchestration layer that takes user queries, performs semantic similarity search across user collections, and generates grounded contextual answers with direct document citations via OpenAI models.
  • Dockerized Architecture: Packaged the entire backend, vector store, and dependencies into reproducible Docker Compose environments for local self-hosting and cloud deployments.
  • Commercial SaaS Launch & Sunset: Deployed and operated as a commercial SaaS offering before sunsetting the hosted platform and open-sourcing the backend codebase for the community.

Architecture

  1. Document Upload & Parsing: Files uploaded via REST API are queued for background processing, converted to normalized text, and chunked with sliding window overlaps.
  2. Vector Indexing: Chunks are vectorized and indexed into Weaviate collections isolated by workspace or tenant ID.
  3. Retrieval & Reranking: Incoming natural language queries retrieve the top-k most relevant semantic passages.
  4. Context Synthesis: Contextual excerpts are passed to the LLM with prompt templates requiring attribution back to the specific source document and section.

Stack

  • Vector Store & Search: Weaviate (hybrid search), Local Embeddings
  • AI & Inference: OpenAI (Embeddings & GPT models)
  • Backend Services: Python, FastAPI / Flask, PostgreSQL, Redis
  • DevOps & Containers: Docker, Docker Compose, Vercel

Outcome

  • Successfully launched as a live SaaS product serving self-hosted and cloud users.
  • Open-sourced the backend architecture on GitHub with full documentation for reproducible local deployment.
  • Proved out end-to-end RAG architecture patterns subsequently applied to enterprise-scale document systems.