PRD-30: Modular Architecture Refactoring

Complete Codebase Restructuring for Standalone, Sellable Modules

Version: 3.0.0 - FINAL Status: ✅ COMPLETE Priority: CRITICAL Completed: 2024-12-04 Author: Automatos AI Platform Team

🎉 MIGRATION COMPLETE - December 4, 2024

What Was Achieved

Before

After

8+ scattered directories

3 clean layers: modules/, consumers/, core/

Code duplication everywhere

Single source of truth

Tight coupling

11 decoupled, sellable modules

~280+ service files scattered

263 organized files

Imports from 15+ locations

Clean from modules.X import Y

Directories DELETED (Fully Removed)

❌ services/ → modules/*/services/, core/services/
❌ context_engineering/ → modules/search/, modules/rag/
❌ memory/ → modules/memory/
❌ multi_agent/ → modules/agents/multi_agent/
❌ utils/ → core/utils/
❌ reasoning/ → modules/reasoning/
❌ credentials/ → core/credentials/
❌ database/ → core/database/
❌ models/ → core/models/
❌ seeds/ → core/seeds/

Final Architecture (ACTUAL)

orchestrator/
├── modules/           # 136 files - SELLABLE MODULES
│   ├── agents/        # Agent lifecycle + multi-agent systems
│   │   ├── factory/           # AgentFactory, AgentBuilder
│   │   ├── execution/         # AgentExecutionManager
│   │   ├── registry/          # AgentRegistry, AgentCapabilities
│   │   ├── communication/     # Inter-agent messaging
│   │   ├── multi_agent/       # Coordination, behavior monitoring
│   │   ├── services/          # Skill loader, platform tools
│   │   └── tests/
│   │
│   ├── codegraph/     # Code analysis & search
│   │   ├── analysis/          # AST parsing (structure ready)
│   │   ├── graph/             # Graph operations (structure ready)
│   │   ├── search/            # Semantic code search (structure ready)
│   │   └── tests/
│   │
│   ├── evaluation/    # AI evaluation (structure ready)
│   │
│   ├── learning/      # Self-improvement system
│   │   ├── engine/            # LearningSystemUpdater
│   │   ├── playbooks/         # PlaybookMiner
│   │   ├── patterns/          # Pattern detection (structure ready)
│   │   ├── feedback/          # Feedback learning (structure ready)
│   │   └── tests/
│   │
│   ├── memory/        # Multi-type memory system
│   │   ├── types/             # MemoryType, MemoryLevel, MemoryItem
│   │   ├── storage/           # AdvancedMemoryManager, KnowledgeSystem
│   │   ├── operations/        # Augmentation, consolidation, access
│   │   └── tests/
│   │
│   ├── nl2sql/        # Natural language to SQL
│   │   ├── query/             # NLToSQLService, SQLValidator
│   │   ├── schema/            # Introspection, SchemaProvider
│   │   └── tests/
│   │
│   ├── orchestrator/  # 9-Stage Workflow Pipeline
│   │   ├── stages/            # All 9 pipeline stages
│   │   ├── llm/               # Master orchestrator, context strategy
│   │   └── tests/
│   │
│   ├── rag/           # Retrieval Augmented Generation
│   │   ├── chunking/          # SemanticChunker
│   │   ├── ingestion/         # DocumentManager, processors
│   │   │   ├── handlers/      # Text handler
│   │   │   └── multimodal/    # Table, image, formula processors
│   │   ├── services/          # Multimodal knowledge
│   │   └── tests/
│   │
│   ├── reasoning/     # Collaborative reasoning
│   │   └── collaborative_reasoning.py
│   │
│   ├── search/        # Core search engine (foundation)
│   │   ├── vector_store/      # EnhancedVectorStore
│   │   ├── embeddings/        # Embedding management
│   │   ├── retrieval/         # ContextRetrievalEngine
│   │   ├── optimization/      # ContextOptimizer (knapsack, MMR)
│   │   ├── services/          # Context engineering services
│   │   └── tests/
│   │
│   └── tools/         # Tool registry & execution
│       ├── registry/          # ToolRegistry
│       ├── execution/         # UnifiedToolExecutor, MCPExecutor
│       ├── formatting/        # ResultFormatter
│       ├── services/          # MCP auto-activation, capability mapper
│       └── tests/
│
├── consumers/         # 11 files - BUSINESS LOGIC (thin wrappers)
│   ├── chatbot/       # Chat interface
│   │   ├── service.py         # ChatService
│   │   ├── streaming.py       # StreamingHandler
│   │   ├── prompt_analyzer.py # PromptAnalyzer
│   │   └── tool_router.py     # ToolRouter
│   ├── external/      # Third-party integrations
│   └── workflows/     # Workflow management
│       ├── analytics.py       # WorkflowAnalytics
│       ├── streaming.py       # WorkflowStreaming
│       └── usage_tracker.py   # UsageTracker
│
├── core/              # 64 files - PLATFORM FOUNDATION
│   ├── credentials/   # Credential management (encryption, resolver)
│   ├── database/      # Database connections, migrations, seeds
│   ├── llm/           # LLM providers & model registry
│   │   ├── clients/           # OpenAI, Anthropic, HuggingFace, etc.
│   │   ├── manager.py         # LLM manager
│   │   ├── embedding_manager.py
│   │   └── model_registry.py
│   ├── math/          # Mathematical foundations
│   │   ├── information_theory.py
│   │   ├── vector_operations.py
│   │   ├── optimization_algorithms.py
│   │   ├── distance_metrics.py
│   │   ├── graph_theory.py
│   │   ├── probability_theory.py
│   │   └── statistical_analysis.py
│   ├── models/        # SQLAlchemy ORM models
│   ├── redis/         # Redis client (pub/sub, caching)
│   ├── seeds/         # Database seed data
│   ├── services/      # Core platform services (analytics, audit, monitoring)
│   └── utils/         # Logging & utilities
│
└── api/               # 52 files - HTTP ENDPOINTS (thin routes)

Remaining Polish (Non-Critical)

field_theory/ - DELETED (not used, can re-add later) ✅
security/ - DELETED (not imported anywhere) ✅
Git tags for each phase
Module-level test coverage

1. Executive Summary

1.1 Purpose

Transform the Automatos AI Platform from a monolithic, scattered codebase into a modular architecture where each core capability (RAG, Memory, Agents, Tools) is:

Self-contained - Can run independently
Sellable - Can be packaged as a standalone product
Shared - Used by all platform consumers (Chatbot, Workflows, Agents, Third-party)
Testable - Has clear boundaries and interfaces
Maintainable - Single source of truth, no duplication

1.2 Business Value

Benefit

Impact

Reduced Maintenance

Fix bugs once, not in 5 places

Faster Development

Clear module boundaries

Revenue Streams

Sell RAG, Memory, Agents as products

Third-party Integration

Clean APIs for external use

Team Scaling

Different teams own different modules

1.3 Success Metrics

Zero duplicate implementations ✅
Each module has < 5 external dependencies ✅
Module can be imported in 1 line ✅ (from modules.rag import RAGService)
90%+ test coverage per module (future work)
API response time unchanged or improved ✅

2. Problem Statement

2.1 Current Issues

Issue 1: Massive Code Duplication

CHUNKING: 4 different implementations (1,706 lines duplicated)
VECTOR STORE: 4 different implementations (2,402 lines duplicated)
MEMORY: 10 scattered files (4,000+ lines scattered)
RAG RETRIEVAL: 6 different implementations (3,000+ lines duplicated)

Issue 2: Scattered Functionality

"Where is RAG code?"
- services/rag_service.py
- services/semantic_chunker.py
- context_engineering/chunking.py
- context_engineering/chunking/semantic_chunker.py
- context_engineering/vector_store.py
- context_engineering/retrieval/vector_store_enhanced.py
- context_engineering/context_retriever.py
- context_engineering/retrieval/context_retrieval_engine.py
- context_engineering/context_optimizer.py
- utils/document_manager.py
- api/documents.py (embedded)

Answer: EVERYWHERE. This is the problem.

Issue 3: Tight Coupling

# Current: Everything imports everything
from services.rag_service import get_rag_service
from context_engineering.chunking.semantic_chunker import SemanticChunker
from context_engineering.context_optimizer import ContextOptimizer
from utils.document_manager import DocumentManager
from context_engineering.retrieval.vector_store_enhanced import EnhancedVectorStore
# 5 different imports for one RAG operation!

Issue 4: No Clear Ownership

Who owns RAG? services/? context_engineering/? utils/?
Who owns Memory? services/? memory/? core/?
Who owns Agents? services/? multi_agent/? core/?

3. Current State Analysis

3.1 Directory Structure (Current)

orchestrator/                          # 100+ files, no clear organization
├── api/                               # 25+ endpoints, mixed concerns
├── context_engineering/               # RAG + Context (scattered)
│   ├── chunking/                      # Chunking (good)
│   ├── mathematical_foundations/      # Math (good, keep here)
│   ├── retrieval/                     # Vector stores (duplicated)
│   ├── chunking.py                    # DUPLICATE of chunking/
│   ├── context_optimizer.py           # Good, keep
│   ├── context_retriever.py           # DUPLICATE of retrieval/
│   ├── embeddings.py                  # DUPLICATE of llm_provider/
│   ├── vector_store.py                # DUPLICATE of retrieval/
│   └── ... 10 more files
├── core/                              # Orchestration + helpers
│   ├── llm/                           # LLM orchestration
│   ├── _vector_store_helper.py        # DUPLICATE
│   ├── memory_prompt_injector.py      # Memory (scattered)
│   └── ... 15 more files
├── credentials/                       # Good, isolated
├── database/                          # Good, isolated
├── evaluation/                        # Good structure
├── memory/                            # Memory types (partial)
├── models/                            # Good, isolated
├── multi_agent/                       # Good structure
├── reasoning/                         # Good structure
├── seeds/                             # Good, isolated
├── services/                          # DUMPING GROUND - 50+ files!
│   ├── chat/                          # Chat services
│   ├── llm_provider/                  # LLM clients (good)
│   ├── rag_service.py                 # RAG (wrapper)
│   ├── semantic_chunker.py            # DUPLICATE
│   ├── memory_knowledge_system.py     # Memory (1362 lines!)
│   ├── agent_factory.py               # Agents (2142 lines!)
│   └── ... 40 more files
└── utils/                             # Utilities (mixed)
    ├── document_manager.py            # RAG (embedded chunking)
    └── ... 9 more files

3.2 Duplication Analysis

CHUNKING - 4 Implementations

File

Lines

Has Math?

Status

services/semantic_chunker.py

400

❌ No

DELETE

context_engineering/chunking.py

429

❌ No

DELETE

context_engineering/chunking/semantic_chunker.py

477

✅ Yes

KEEP

utils/document_manager.py (embedded)

~150

❌ No

EXTRACT

Total duplicated lines: 979

VECTOR STORE - 4 Implementations

File

Lines

Features

Status

context_engineering/vector_store.py

529

Basic pgvector

MERGE

context_engineering/retrieval/vector_store_enhanced.py

672

Hybrid, ranking, math

KEEP

core/_vector_store_helper.py

~100

Helper functions

DELETE

api/documents.py (embedded)

~200

Search logic

EXTRACT

Total duplicated lines: 829

CONTEXT RETRIEVAL - 3 Implementations

File

Lines

Features

Status

context_engineering/context_retriever.py

585

Basic retrieval

MERGE

context_engineering/retrieval/context_retrieval_engine.py

655

Advanced, multi-strategy

KEEP

context_engineering/context_optimizer.py

928

Knapsack, MMR, entropy

KEEP

EMBEDDINGS - 2 Implementations

File

Lines

Features

Status

context_engineering/embeddings.py

364

SentenceTransformer, OpenAI

DELETE

services/llm_provider/embedding_manager.py

217

Centralized, multi-provider

KEEP

MEMORY - 10 Scattered Files

File

Lines

Location

Status

memory/manager.py

~400

memory/

KEEP

memory/augmentation.py

~300

memory/

KEEP

memory/consolidation.py

~250

memory/

KEEP

memory/access_patterns.py

~200

memory/

KEEP

memory/memory_types.py

~150

memory/

KEEP

services/memory_knowledge_system.py

1362

services/

MERGE

core/memory_prompt_injector.py

~200

core/

MERGE

core/workflow_memory_integrator.py

~300

core/

MERGE

services/chat/memory_injector.py

~150

chat/

MERGE

Total memory-related lines: ~3,312 across 10 files

4. Target Architecture

4.1 New Directory Structure

orchestrator/
│
├── modules/                           # 🎯 STANDALONE PRODUCTS
│   │
│   ├── search/                       # 🔥 CORE SEARCH ENGINE (Phase 1a)
│   │   ├── __init__.py               # Public API
│   │   ├── service.py                # SearchService class
│   │   ├── config.py                 # Search configuration
│   │   │
│   │   ├── vector_store/             # pgvector operations
│   │   │   ├── __init__.py
│   │   │   ├── store.py              # PgVectorStore
│   │   │   ├── indexing.py           # Index management
│   │   │   └── queries.py            # Query builders
│   │   │
│   │   ├── embeddings/               # Embedding generation
│   │   │   ├── __init__.py
│   │   │   ├── manager.py            # EmbeddingManager
│   │   │   └── providers.py          # OpenAI, HuggingFace, etc.
│   │   │
│   │   ├── retrieval/                # Search algorithms
│   │   │   ├── __init__.py
│   │   │   ├── vector_search.py      # Cosine similarity
│   │   │   ├── hybrid_search.py      # Vector + BM25
│   │   │   └── reranking.py          # Cross-encoder reranking
│   │   │
│   │   ├── optimization/             # Mathematical optimization
│   │   │   ├── __init__.py
│   │   │   ├── context_optimizer.py  # Main optimizer
│   │   │   ├── knapsack.py           # Token budget
│   │   │   ├── mmr.py                # Diversity (MMR)
│   │   │   └── entropy.py            # Information theory
│   │   │
│   │   └── tests/
│   │
│   ├── rag/                          # RAG MODULE (Phase 1b) - uses search/
│   │   ├── __init__.py               # Public API
│   │   ├── service.py                # RAGService class
│   │   ├── config.py                 # RAG configuration
│   │   │
│   │   ├── chunking/                 # Document chunking
│   │   │   ├── __init__.py
│   │   │   ├── semantic.py           # SemanticChunker
│   │   │   ├── strategies.py         # ChunkingStrategy enum
│   │   │   └── metadata.py           # ChunkMetadata
│   │   │
│   │   ├── ingestion/                # Document ingestion
│   │   │   ├── __init__.py
│   │   │   ├── processor.py          # DocumentProcessor
│   │   │   ├── handlers/             # File type handlers
│   │   │   │   ├── pdf.py
│   │   │   │   ├── markdown.py
│   │   │   │   ├── docx.py
│   │   │   │   └── text.py
│   │   │   └── pipeline.py           # Ingestion pipeline
│   │   │
│   │   └── tests/
│   │
│   ├── knowledge/                    # KNOWLEDGE BASE MODULE (Phase 1c) - uses search/
│   │   ├── __init__.py               # Public API
│   │   ├── service.py                # KnowledgeService class
│   │   ├── config.py
│   │   │
│   │   ├── graph/                    # Knowledge graph
│   │   │   ├── __init__.py
│   │   │   ├── builder.py            # Graph construction
│   │   │   ├── traversal.py          # Graph queries
│   │   │   └── entities.py           # Entity extraction
│   │   │
│   │   ├── storage/                  # Knowledge storage
│   │   │   ├── __init__.py
│   │   │   ├── postgres.py
│   │   │   └── neo4j.py              # Future: graph DB
│   │   │
│   │   └── tests/
│   │
│   ├── nl_to_sql/                    # NL-TO-SQL MODULE (Phase 1d) - uses search/
│   │   ├── __init__.py               # Public API
│   │   ├── service.py                # NLToSQLService class
│   │   ├── config.py
│   │   │
│   │   ├── schema/                   # Schema awareness
│   │   │   ├── __init__.py
│   │   │   ├── introspection.py      # DB schema discovery
│   │   │   ├── embeddings.py         # Schema embeddings
│   │   │   └── provider.py           # SchemaProvider
│   │   │
│   │   ├── query/                    # Query generation
│   │   │   ├── __init__.py
│   │   │   ├── builder.py            # SQL builder
│   │   │   ├── validator.py          # SQL validation
│   │   │   └── executor.py           # Safe execution
│   │   │
│   │   └── tests/
│   │
│   ├── codegraph/                    # CODEGRAPH MODULE (Phase 1e) - uses search/
│   │   ├── __init__.py               # Public API
│   │   ├── service.py                # CodeGraphService class
│   │   ├── config.py
│   │   │
│   │   ├── analysis/                 # Code analysis
│   │   │   ├── __init__.py
│   │   │   ├── parser.py             # AST parsing
│   │   │   ├── dependencies.py       # Dependency analysis
│   │   │   └── metrics.py            # Code metrics
│   │   │
│   │   ├── graph/                    # Code graph
│   │   │   ├── __init__.py
│   │   │   ├── builder.py            # Graph construction
│   │   │   ├── queries.py            # Graph queries
│   │   │   └── visualizer.py         # Graph visualization
│   │   │
│   │   ├── search/                   # Code search
│   │   │   ├── __init__.py
│   │   │   ├── semantic.py           # Semantic code search
│   │   │   └── structural.py         # Structural search
│   │   │
│   │   └── tests/
│   │
│   ├── learning/                     # LEARNING MODULE (Phase 5.5) - cross-cutting
│   │   ├── __init__.py               # Public API
│   │   ├── service.py                # LearningService class
│   │   ├── config.py
│   │   │
│   │   ├── patterns/                 # Pattern recognition
│   │   │   ├── __init__.py
│   │   │   ├── detector.py           # Pattern detection
│   │   │   ├── extractor.py          # Pattern extraction
│   │   │   └── storage.py            # Pattern storage
│   │   │
│   │   ├── playbooks/                # Playbook mining
│   │   │   ├── __init__.py
│   │   │   ├── miner.py              # Workflow pattern mining
│   │   │   ├── templates.py          # Playbook templates
│   │   │   └── executor.py           # Playbook execution
│   │   │
│   │   ├── feedback/                 # Feedback learning
│   │   │   ├── __init__.py
│   │   │   ├── collector.py          # Feedback collection
│   │   │   ├── analyzer.py           # Feedback analysis
│   │   │   └── adapter.py            # Model adaptation
│   │   │
│   │   ├── engine/                   # Learning engine
│   │   │   ├── __init__.py
│   │   │   ├── reinforcement.py      # RL-based learning
│   │   │   └── continuous.py         # Continuous improvement
│   │   │
│   │   └── tests/
│   │
│   ├── memory/                       # MEMORY MODULE (Phase 2)
│   │   ├── __init__.py
│   │   ├── service.py                # MemoryService class
│   │   ├── config.py
│   │   │
│   │   ├── types/                    # Memory types
│   │   │   ├── __init__.py
│   │   │   ├── episodic.py           # Event-based
│   │   │   ├── semantic.py           # Factual knowledge
│   │   │   ├── procedural.py         # How-to
│   │   │   └── working.py            # Short-term
│   │   │
│   │   ├── storage/                  # Storage backends
│   │   │   ├── __init__.py
│   │   │   ├── postgres.py
│   │   │   ├── knowledge_graph.py
│   │   │   └── cache.py
│   │   │
│   │   ├── operations/               # Memory operations
│   │   │   ├── __init__.py
│   │   │   ├── retrieval.py
│   │   │   ├── consolidation.py
│   │   │   ├── augmentation.py
│   │   │   └── injection.py          # Prompt injection
│   │   │
│   │   └── tests/
│   │
│   ├── agents/                       # AGENTS MODULE (Phase 3)
│   │   ├── __init__.py
│   │   ├── service.py                # AgentService class
│   │   ├── config.py
│   │   │
│   │   ├── factory/                  # Agent creation
│   │   │   ├── __init__.py
│   │   │   ├── builder.py            # AgentBuilder
│   │   │   ├── registry.py           # Agent registry
│   │   │   └── templates.py          # Agent templates
│   │   │
│   │   ├── execution/                # Agent execution
│   │   │   ├── __init__.py
│   │   │   ├── executor.py           # AgentExecutor
│   │   │   ├── runtime.py            # AgentRuntime
│   │   │   └── context.py            # Execution context
│   │   │
│   │   ├── skills/                   # Skill management
│   │   │   ├── __init__.py
│   │   │   ├── loader.py
│   │   │   ├── registry.py
│   │   │   └── mapper.py
│   │   │
│   │   ├── communication/            # Inter-agent
│   │   │   ├── __init__.py
│   │   │   ├── messaging.py
│   │   │   ├── coordination.py
│   │   │   └── protocols.py
│   │   │
│   │   ├── selection/                # Agent selection
│   │   │   ├── __init__.py
│   │   │   ├── intelligent.py
│   │   │   ├── llm_based.py
│   │   │   └── scoring.py
│   │   │
│   │   └── tests/
│   │
│   ├── tools/                        # TOOLS MODULE (Phase 4)
│   │   ├── __init__.py
│   │   ├── service.py                # ToolService class
│   │   │
│   │   ├── registry/                 # Tool registration
│   │   │   ├── __init__.py
│   │   │   ├── registry.py
│   │   │   ├── discovery.py
│   │   │   └── validation.py
│   │   │
│   │   ├── execution/                # Tool execution
│   │   │   ├── __init__.py
│   │   │   ├── executor.py
│   │   │   ├── formatter.py          # Result formatting
│   │   │   └── sandbox.py
│   │   │
│   │   ├── mcp/                      # MCP integration
│   │   │   ├── __init__.py
│   │   │   ├── client.py
│   │   │   ├── executor.py
│   │   │   └── discovery.py
│   │   │
│   │   └── tests/
│   │
│   ├── reasoning/                    # REASONING MODULE (Phase 5)
│   │   └── (restructure existing)
│   │
│   └── evaluation/                   # EVALUATION MODULE (Phase 6)
│       └── (restructure existing)
│
├── shared/                           # SHARED INFRASTRUCTURE
│   │
│   ├── llm/                          # LLM providers
│   │   ├── __init__.py
│   │   ├── manager.py                # From services/llm_provider/
│   │   ├── embedding_manager.py
│   │   └── clients/
│   │       ├── openai.py
│   │       ├── anthropic.py
│   │       ├── huggingface.py
│   │       └── ...
│   │
│   ├── math/                         # Mathematical foundations
│   │   ├── __init__.py
│   │   ├── information_theory.py     # From context_engineering/
│   │   ├── vector_operations.py
│   │   ├── statistical_analysis.py
│   │   ├── graph_theory.py
│   │   ├── probability_theory.py
│   │   ├── distance_metrics.py
│   │   └── optimization_algorithms.py
│   │
│   ├── database/                     # Database utilities
│   │   ├── __init__.py
│   │   ├── connection.py
│   │   ├── models.py
│   │   └── migrations/
│   │
│   └── utils/                        # Common utilities
│       ├── __init__.py
│       ├── logging.py
│       ├── config.py
│       └── validation.py
│
├── core/                             # ORCHESTRATION ONLY (minimal)
│   ├── __init__.py
│   ├── orchestrator.py               # 9-stage workflow
│   ├── task_decomposer.py
│   ├── result_aggregator.py
│   └── workflow_executor.py
│
├── api/                              # API LAYER (thin)
│   ├── __init__.py
│   ├── rag.py                        # → modules.rag
│   ├── memory.py                     # → modules.memory
│   ├── agents.py                     # → modules.agents
│   ├── tools.py                      # → modules.tools
│   ├── workflows.py
│   └── system.py
│
├── consumers/                        # PLATFORM CONSUMERS
│   ├── chatbot/                      # Chatbot integration
│   │   ├── __init__.py
│   │   ├── service.py
│   │   └── streaming.py
│   │
│   ├── workflows/                    # Workflow integration
│   │   ├── __init__.py
│   │   └── integrator.py
│   │
│   └── external/                     # Third-party API
│       ├── __init__.py
│       └── client.py
│
└── deprecated/                       # TO DELETE AFTER MIGRATION
    └── (old files moved here first)

4.2 Module Dependency Graph

┌─────────────────────────────────────────────────────────────────────────┐
│                           CONSUMERS (THIN)                               │
│    ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐        │
│    │ Chatbot  │    │ Workflows│    │ Third    │    │  CLI     │        │
│    │          │    │          │    │ Party    │    │          │        │
│    └────┬─────┘    └────┬─────┘    └────┬─────┘    └────┬─────┘        │
└─────────┼───────────────┼───────────────┼───────────────┼───────────────┘
          │               │               │               │
          ▼               ▼               ▼               ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                        API LAYER (THIN - existing routes)                │
│    /api/documents  /api/chat  /api/agents  /api/workflows  /api/code    │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         PRODUCT MODULES                                  │
│                                                                          │
│   ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐          │
│   │    RAG     │ │ KNOWLEDGE  │ │  NL2SQL    │ │ CODEGRAPH  │          │
│   │  Document  │ │    Base    │ │  Database  │ │    Code    │          │
│   │  Retrieval │ │   Graph    │ │   Query    │ │  Analysis  │          │
│   └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘          │
│         │              │              │              │                  │
│         └──────────────┴──────────────┴──────────────┘                  │
│                                 │                                        │
│                                 ▼                                        │
│                    ┌────────────────────────┐                           │
│                    │    🔥 SEARCH (CORE)    │                           │
│                    │                        │                           │
│                    │  • Vector Store        │                           │
│                    │  • Embeddings          │                           │
│                    │  • Retrieval           │                           │
│                    │  • Optimization        │                           │
│                    │    (Knapsack/MMR)      │                           │
│                    └────────────────────────┘                           │
│                                                                          │
│   ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐          │
│   │   MEMORY   │ │   AGENTS   │ │   TOOLS    │ │  LEARNING  │          │
│   │            │ │            │ │            │ │            │          │
│   │ • Episodic │ │ • Factory  │ │ • Registry │ │ • Patterns │          │
│   │ • Semantic │ │ • Executor │ │ • Executor │ │ • Playbooks│          │
│   │ • Procedur │ │ • Skills   │ │ • MCP      │ │ • Feedback │          │
│   └────────────┘ └────────────┘ └────────────┘ └────────────┘          │
│         │              │              │              │                  │
│         └──────────────┴──────────────┴──────────────┘                  │
│                                 │                                        │
│                    (All can use SEARCH for retrieval)                   │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         SHARED INFRASTRUCTURE                            │
│                                                                          │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────┐ │
│   │     LLM      │  │     MATH     │  │   DATABASE   │  │   UTILS    │ │
│   │              │  │              │  │              │  │            │ │
│   │ • Manager    │  │ • Entropy    │  │ • Connection │  │ • Logging  │ │
│   │ • Embeddings │  │ • MMR        │  │ • Models     │  │ • Config   │ │
│   │ • Clients    │  │ • Knapsack   │  │ • Migrations │  │ • Validate │ │
│   └──────────────┘  └──────────────┘  └──────────────┘  └────────────┘ │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

4.3 Sellable Products

Module

Standalone Product

Dependencies

search/

automatos-search

shared/ only

rag/

automatos-rag

search/

knowledge/

automatos-knowledge

search/

nl_to_sql/

automatos-nl2sql

search/

codegraph/

automatos-codegraph

search/

memory/

automatos-memory

search/ (optional)

agents/

automatos-agents

memory/, tools/, search/

tools/

automatos-tools

shared/ only

learning/

automatos-learning

all modules (cross-cutting)

5. Module Specifications

5.1 RAG Module

5.1.1 Purpose

Provide complete Retrieval-Augmented Generation capabilities as a standalone, sellable product.

5.1.2 Public API

# modules/rag/__init__.py

from .service import RAGService
from .config import RAGConfig
from .chunking import SemanticChunker, ChunkingStrategy
from .retrieval import VectorStore, ContextRetriever
from .optimization import ContextOptimizer

__all__ = [
    'RAGService',
    'RAGConfig',
    'SemanticChunker',
    'ChunkingStrategy',
    'VectorStore',
    'ContextRetriever',
    'ContextOptimizer',
]

# Convenience function
async def retrieve(query: str, **config) -> RAGResult:
    """One-liner RAG retrieval"""
    service = RAGService()
    return await service.retrieve(query, **config)

5.1.3 Service Class

# modules/rag/service.py

from dataclasses import dataclass
from typing import List, Dict, Any, Optional

from .config import RAGConfig
from .chunking import SemanticChunker
from .retrieval import VectorStore, ContextRetriever
from .optimization import ContextOptimizer
from .ingestion import DocumentProcessor

@dataclass
class RAGResult:
    """Result from RAG retrieval"""
    chunks: List[Dict[str, Any]]
    formatted_context: str
    total_tokens: int
    sources: List[str]
    metrics: Dict[str, float]  # diversity_score, information_gain, etc.

class RAGService:
    """
    Unified RAG Service
    
    Usage:
        # Simple usage
        from modules.rag import RAGService
        rag = RAGService()
        result = await rag.retrieve("How do agents work?")
        
        # With configuration
        from modules.rag import RAGService, RAGConfig
        config = RAGConfig(max_chunks=10, diversity=0.4)
        rag = RAGService(config)
        result = await rag.retrieve("query", context_type="agent")
        
        # Document ingestion
        await rag.ingest(document_path="/path/to/doc.pdf")
    """
    
    def __init__(self, config: RAGConfig = None):
        self.config = config or RAGConfig()
        
        # Initialize components
        self._chunker = None
        self._vector_store = None
        self._retriever = None
        self._optimizer = None
        self._processor = None
    
    @property
    def chunker(self) -> SemanticChunker:
        """Lazy initialization of chunker"""
        if self._chunker is None:
            self._chunker = SemanticChunker(
                strategy=self.config.chunking_strategy,
                target_size=self.config.chunk_size,
                min_size=self.config.min_chunk_size,
                max_size=self.config.max_chunk_size
            )
        return self._chunker
    
    @property
    def vector_store(self) -> VectorStore:
        """Lazy initialization of vector store"""
        if self._vector_store is None:
            self._vector_store = VectorStore(
                connection_string=self.config.database_url,
                dimension=self.config.embedding_dimension
            )
        return self._vector_store
    
    @property
    def retriever(self) -> ContextRetriever:
        """Lazy initialization of retriever"""
        if self._retriever is None:
            self._retriever = ContextRetriever(
                vector_store=self.vector_store,
                config=self.config.retrieval_config
            )
        return self._retriever
    
    @property
    def optimizer(self) -> ContextOptimizer:
        """Lazy initialization of optimizer"""
        if self._optimizer is None:
            self._optimizer = ContextOptimizer()
        return self._optimizer
    
    async def retrieve(
        self,
        query: str,
        max_chunks: int = None,
        max_tokens: int = None,
        diversity: float = None,
        context_type: str = "default"
    ) -> RAGResult:
        """
        Retrieve relevant context for a query.
        
        Args:
            query: Search query
            max_chunks: Override max chunks
            max_tokens: Override max tokens
            diversity: Override diversity (0=relevance, 1=diversity)
            context_type: Type of context (chatbot, agent, workflow)
        
        Returns:
            RAGResult with chunks, formatted context, and metrics
        """
        # Get candidates
        candidates = await self.retriever.search(
            query=query,
            limit=self.config.candidate_multiplier * (max_chunks or self.config.max_chunks)
        )
        
        # Optimize selection using mathematical algorithms
        optimized = await self.optimizer.optimize_context(
            candidates=candidates,
            max_tokens=max_tokens or self.config.max_tokens,
            objective=self._get_objective(diversity or self.config.diversity)
        )
        
        # Format result
        return RAGResult(
            chunks=optimized.chunks,
            formatted_context=self._format_context(optimized.chunks, query),
            total_tokens=optimized.total_tokens,
            sources=list(set(c['source'] for c in optimized.chunks)),
            metrics={
                'diversity_score': optimized.diversity_score,
                'information_gain': optimized.information_gain,
                'retrieval_time_ms': optimized.retrieval_time_ms
            }
        )
    
    async def ingest(
        self,
        document_path: str = None,
        content: str = None,
        metadata: Dict[str, Any] = None
    ) -> Dict[str, Any]:
        """
        Ingest a document into the RAG system.
        
        Args:
            document_path: Path to document file
            content: Raw content (alternative to path)
            metadata: Document metadata
        
        Returns:
            Ingestion result with document_id, chunk_count, etc.
        """
        if self._processor is None:
            self._processor = DocumentProcessor(
                chunker=self.chunker,
                vector_store=self.vector_store
            )
        
        return await self._processor.process(
            path=document_path,
            content=content,
            metadata=metadata
        )
    
    def _format_context(self, chunks: List[Dict], query: str) -> str:
        """Format chunks into LLM-ready context"""
        parts = [f"## Retrieved Context for: {query}\n"]
        
        for i, chunk in enumerate(chunks, 1):
            source = chunk.get('source', 'unknown')
            relevance = chunk.get('relevance', 0)
            content = chunk.get('content', '')
            
            parts.append(f"\n### Source {i}: {source} (relevance: {relevance:.0%})")
            parts.append(content)
        
        return "\n".join(parts)
    
    def _get_objective(self, diversity: float) -> str:
        """Map diversity to optimization objective"""
        if diversity > 0.6:
            return "maximize_diversity"
        elif diversity < 0.3:
            return "maximize_information"
        return "balanced"

5.1.4 RAG Module Flow Diagram

┌─────────────────────────────────────────────────────────────────────────┐
│                         RAG MODULE FLOW                                  │
└─────────────────────────────────────────────────────────────────────────┘

                           ┌─────────────────┐
                           │   User Query    │
                           │ "How do agents  │
                           │     work?"      │
                           └────────┬────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                           RAG SERVICE                                    │
│                                                                          │
│   ┌──────────────┐                                                      │
│   │   retrieve() │                                                      │
│   └──────┬───────┘                                                      │
│          │                                                              │
│          ▼                                                              │
│   ┌──────────────────────────────────────────────────────────────┐     │
│   │                    1. QUERY PROCESSING                        │     │
│   │                                                               │     │
│   │   ┌─────────────┐      ┌─────────────┐      ┌─────────────┐ │     │
│   │   │   Expand    │ ──▶  │  Generate   │ ──▶  │  Normalize  │ │     │
│   │   │   Query     │      │  Embedding  │      │  Vector     │ │     │
│   │   └─────────────┘      └─────────────┘      └─────────────┘ │     │
│   │                                                               │     │
│   │   Input: "How do agents work?"                               │     │
│   │   Output: [0.12, -0.45, 0.78, ...] (1024d)                  │     │
│   └──────────────────────────────────────────────────────────────┘     │
│          │                                                              │
│          ▼                                                              │
│   ┌──────────────────────────────────────────────────────────────┐     │
│   │                    2. CANDIDATE RETRIEVAL                     │     │
│   │                                                               │     │
│   │   ┌─────────────┐      ┌─────────────┐      ┌─────────────┐ │     │
│   │   │   Vector    │ ──▶  │   Hybrid    │ ──▶  │    Get      │ │     │
│   │   │   Search    │      │   (BM25)    │      │ Top 30      │ │     │
│   │   │  (pgvector) │      │             │      │ Candidates  │ │     │
│   │   └─────────────┘      └─────────────┘      └─────────────┘ │     │
│   │                                                               │     │
│   │   SQL: SELECT ... ORDER BY embedding <=> query_vec LIMIT 30  │     │
│   └──────────────────────────────────────────────────────────────┘     │
│          │                                                              │
│          ▼                                                              │
│   ┌──────────────────────────────────────────────────────────────┐     │
│   │                    3. OPTIMIZATION (Mathematical)             │     │
│   │                                                               │     │
│   │   ┌─────────────┐      ┌─────────────┐      ┌─────────────┐ │     │
│   │   │  Calculate  │ ──▶  │  Knapsack   │ ──▶  │    MMR      │ │     │
│   │   │  Entropy    │      │  Selection  │      │  Diversity  │ │     │
│   │   └─────────────┘      └─────────────┘      └─────────────┘ │     │
│   │                                                               │     │
│   │   H(X) = -Σ p(x) * log2(p(x))                                │     │
│   │   Knapsack: max value within token budget                    │     │
│   │   MMR: λ*relevance - (1-λ)*max_sim_to_selected              │     │
│   └──────────────────────────────────────────────────────────────┘     │
│          │                                                              │
│          ▼                                                              │
│   ┌──────────────────────────────────────────────────────────────┐     │
│   │                    4. FORMAT RESULT                           │     │
│   │                                                               │     │
│   │   ┌─────────────────────────────────────────────────────────┐│     │
│   │   │  RAGResult:                                              ││     │
│   │   │    chunks: [chunk1, chunk2, ...]                        ││     │
│   │   │    formatted_context: "## Retrieved Context..."         ││     │
│   │   │    total_tokens: 1847                                   ││     │
│   │   │    sources: ["AGENT_GUIDE.md", "FACTORY.md"]           ││     │
│   │   │    metrics: {diversity: 0.82, info_gain: 0.76}         ││     │
│   │   └─────────────────────────────────────────────────────────┘│     │
│   └──────────────────────────────────────────────────────────────┘     │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
                           ┌─────────────────┐
                           │  Return to      │
                           │  Consumer       │
                           │  (Chatbot/      │
                           │   Workflow/     │
                           │   Agent)        │
                           └─────────────────┘

5.1.5 RAG Ingestion Flow

┌─────────────────────────────────────────────────────────────────────────┐
│                      RAG INGESTION FLOW                                  │
└─────────────────────────────────────────────────────────────────────────┘

         ┌──────────────┐
         │   Document   │
         │   Upload     │
         │  (PDF/MD/    │
         │   DOCX/TXT)  │
         └──────┬───────┘
                │
                ▼
┌──────────────────────────────────────┐
│      1. FILE HANDLER SELECTION       │
│                                      │
│  ┌─────────┐ ┌─────────┐ ┌────────┐ │
│  │   PDF   │ │Markdown │ │  Text  │ │
│  │ Handler │ │ Handler │ │Handler │ │
│  └────┬────┘ └────┬────┘ └───┬────┘ │
│       └──────────┼───────────┘      │
└──────────────────┼──────────────────┘
                   │
                   ▼ (raw text)
┌──────────────────────────────────────┐
│      2. SEMANTIC CHUNKING            │
│                                      │
│  Strategy: ADAPTIVE                  │
│                                      │
│  ┌────────────────────────────────┐ │
│  │ Input: "# Agent Factory\n\n    │ │
│  │ The AgentFactory class..."     │ │
│  └────────────────────────────────┘ │
│              │                       │
│              ▼                       │
│  ┌────────────────────────────────┐ │
│  │ Chunks:                        │ │
│  │ [                              │ │
│  │   {content: "# Agent Factory   │ │
│  │    \nThe AgentFactory...",     │ │
│  │    entropy: 4.2,               │ │
│  │    coherence: 0.89},           │ │
│  │   {...},                       │ │
│  │ ]                              │ │
│  └────────────────────────────────┘ │
└──────────────────┼──────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│      3. EMBEDDING GENERATION         │
│                                      │
│  Model: BAAI/bge-large-en-v1.5      │
│  Dimension: 1024                     │
│                                      │
│  For each chunk:                     │
│    embedding = model.encode(chunk)   │
│                                      │
└──────────────────┼──────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│      4. VECTOR STORE INSERT          │
│                                      │
│  INSERT INTO document_chunks (       │
│    document_id, content, embedding,  │
│    metadata, parent_content, headers │
│  ) VALUES (...)                      │
│                                      │
└──────────────────┼──────────────────┘
                   │
                   ▼
         ┌──────────────┐
         │   Document   │
         │   Ready for  │
         │   Retrieval  │
         └──────────────┘

5.1.6 Files Migration for RAG Module

Source File

Destination

Lines

Action

context_engineering/chunking/semantic_chunker.py

modules/rag/chunking/semantic.py

477

MOVE

context_engineering/chunking/__init__.py

modules/rag/chunking/__init__.py

MOVE

context_engineering/retrieval/vector_store_enhanced.py

modules/rag/retrieval/vector_store.py

672

MOVE

context_engineering/retrieval/context_retrieval_engine.py

modules/rag/retrieval/context_retriever.py

655

MOVE

context_engineering/context_optimizer.py

modules/rag/optimization/context_optimizer.py

928

MOVE

utils/document_manager.py (extract)

modules/rag/ingestion/processor.py

~300

EXTRACT

Files to DELETE after migration:

File

Lines

Reason

services/semantic_chunker.py

400

Duplicate

context_engineering/chunking.py

429

Duplicate

context_engineering/vector_store.py

529

Merged into enhanced

context_engineering/context_retriever.py

585

Merged into engine

context_engineering/embeddings.py

364

Use shared/llm

core/_vector_store_helper.py

~100

Duplicate

services/rag_service.py

370

Replaced by module

5.2 Memory Module

5.2.1 Purpose

Provide complete memory management (episodic, semantic, procedural, working) as a standalone product.

5.2.2 Public API

# modules/memory/__init__.py

from .service import MemoryService
from .types import EpisodicMemory, SemanticMemory, ProceduralMemory

__all__ = ['MemoryService', 'EpisodicMemory', 'SemanticMemory', 'ProceduralMemory']

async def store(content: str, memory_type: str = "episodic", **metadata):
    """Store a memory"""
    service = MemoryService()
    return await service.store(content, memory_type, **metadata)

async def recall(query: str, memory_types: List[str] = None, limit: int = 5):
    """Recall relevant memories"""
    service = MemoryService()
    return await service.recall(query, memory_types, limit)

5.2.3 Memory Module Flow Diagram

┌─────────────────────────────────────────────────────────────────────────┐
│                        MEMORY MODULE FLOW                                │
└─────────────────────────────────────────────────────────────────────────┘

                    ┌─────────────────────────────────────┐
                    │         MEMORY SERVICE              │
                    │                                     │
                    │   store() / recall() / consolidate()│
                    └──────────────────┬──────────────────┘
                                       │
              ┌────────────────────────┼────────────────────────┐
              │                        │                        │
              ▼                        ▼                        ▼
    ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
    │    EPISODIC      │    │    SEMANTIC      │    │   PROCEDURAL     │
    │    MEMORY        │    │    MEMORY        │    │    MEMORY        │
    │                  │    │                  │    │                  │
    │  • Events        │    │  • Facts         │    │  • How-to        │
    │  • Conversations │    │  • Concepts      │    │  • Procedures    │
    │  • Experiences   │    │  • Relations     │    │  • Skills        │
    │                  │    │                  │    │                  │
    │  Time-ordered    │    │  Graph-based     │    │  Step-ordered    │
    └────────┬─────────┘    └────────┬─────────┘    └────────┬─────────┘
             │                       │                       │
             └───────────────────────┼───────────────────────┘
                                     │
                                     ▼
                    ┌─────────────────────────────────────┐
                    │           STORAGE LAYER             │
                    │                                     │
                    │   ┌──────────┐    ┌──────────┐     │
                    │   │PostgreSQL│    │Knowledge │     │
                    │   │ (pgvector)│   │  Graph   │     │
                    │   └──────────┘    └──────────┘     │
                    └─────────────────────────────────────┘
                                     │
                                     ▼
                    ┌─────────────────────────────────────┐
                    │         CONSOLIDATION               │
                    │                                     │
                    │   • Merge similar memories          │
                    │   • Update importance scores        │
                    │   • Decay old, irrelevant memories  │
                    └─────────────────────────────────────┘

5.2.4 Files Migration for Memory Module

Source File

Destination

Action

memory/manager.py

modules/memory/service.py

MERGE

memory/memory_types.py

modules/memory/types/__init__.py

MOVE

memory/augmentation.py

modules/memory/operations/augmentation.py

MOVE

memory/consolidation.py

modules/memory/operations/consolidation.py

MOVE

memory/access_patterns.py

modules/memory/operations/retrieval.py

MERGE

services/memory_knowledge_system.py

modules/memory/ (split)

EXTRACT

core/memory_prompt_injector.py

modules/memory/operations/injection.py

MOVE

core/workflow_memory_integrator.py

modules/memory/integrations/workflow.py

MOVE

services/chat/memory_injector.py

modules/memory/integrations/chat.py

MOVE

5.3 Agents Module

5.3.1 Purpose

Provide complete agent lifecycle management as a standalone product.

5.3.2 Public API

# modules/agents/__init__.py

from .service import AgentService
from .factory import AgentFactory, AgentBuilder
from .execution import AgentExecutor, AgentRuntime

__all__ = ['AgentService', 'AgentFactory', 'AgentBuilder', 'AgentExecutor']

async def create_agent(name: str, agent_type: str, skills: List[str] = None):
    """Create an agent"""
    service = AgentService()
    return await service.create(name, agent_type, skills)

async def execute_task(agent_id: str, task: str, context: Dict = None):
    """Execute a task with an agent"""
    service = AgentService()
    return await service.execute(agent_id, task, context)

5.3.3 Agent Module Flow Diagram

┌─────────────────────────────────────────────────────────────────────────┐
│                        AGENTS MODULE FLOW                                │
└─────────────────────────────────────────────────────────────────────────┘

                         ┌─────────────────┐
                         │  Create Agent   │
                         │    Request      │
                         └────────┬────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                          AGENT FACTORY                                   │
│                                                                          │
│   ┌────────────────┐    ┌────────────────┐    ┌────────────────┐       │
│   │  Load Skills   │───▶│  Build Agent   │───▶│   Register     │       │
│   │  from Registry │    │  Configuration │    │   in DB        │       │
│   └────────────────┘    └────────────────┘    └────────────────┘       │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
                         ┌─────────────────┐
                         │  Agent Ready    │
                         │  (AgentRuntime) │
                         └────────┬────────┘
                                  │
                         ┌────────┴────────┐
                         │  Execute Task   │
                         └────────┬────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         AGENT EXECUTOR                                   │
│                                                                          │
│   ┌────────────────┐    ┌────────────────┐    ┌────────────────┐       │
│   │ Get Context    │───▶│  Execute with  │───▶│  Process       │       │
│   │ (uses RAG)     │    │  LLM + Tools   │    │  Result        │       │
│   └────────────────┘    └────────────────┘    └────────────────┘       │
│          │                      │                      │                │
│          ▼                      ▼                      ▼                │
│   ┌────────────────┐    ┌────────────────┐    ┌────────────────┐       │
│   │ modules.rag    │    │ modules.tools  │    │ modules.memory │       │
│   │ .retrieve()    │    │ .execute()     │    │ .store()       │       │
│   └────────────────┘    └────────────────┘    └────────────────┘       │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

5.3.4 Files Migration for Agents Module

Source File

Destination

Lines

Action

services/agent_factory.py

modules/agents/factory/ (split)

2142

SPLIT

services/skill_loader.py

modules/agents/skills/loader.py

1212

MOVE

core/agent_execution_manager.py

modules/agents/execution/executor.py

1317

MOVE

core/intelligent_agent_selector.py

modules/agents/selection/intelligent.py

239

MOVE

core/llm/llm_agent_selector.py

modules/agents/selection/llm_based.py

~300

MOVE

services/inter_agent_communication.py

modules/agents/communication/messaging.py

1196

MOVE

multi_agent/coordination_manager.py

modules/agents/communication/coordination.py

878

MOVE

5.4 Tools Module

5.4.1 Public API

# modules/tools/__init__.py

from .service import ToolService
from .registry import ToolRegistry
from .execution import ToolExecutor

__all__ = ['ToolService', 'ToolRegistry', 'ToolExecutor']

async def execute_tool(tool_name: str, parameters: Dict = None):
    """Execute a tool by name"""
    service = ToolService()
    return await service.execute(tool_name, parameters)

5.4.2 Files Migration for Tools Module

Source File

Destination

Action

services/tool_registry.py

modules/tools/registry/registry.py

MOVE

services/unified_tool_executor.py

modules/tools/execution/executor.py

MOVE

services/tool_result_formatter.py

modules/tools/execution/formatter.py

MOVE

services/mcp_tool_executor.py

modules/tools/mcp/executor.py

MOVE

services/mcp_auto_activation.py

modules/tools/mcp/discovery.py

MOVE

services/tool_capability_mapper.py

modules/tools/registry/mapper.py

MOVE

6. Shared Infrastructure

6.1 LLM Providers (Keep from services/llm_provider)

shared/llm/
├── __init__.py
├── manager.py                    # From services/llm_provider/manager.py
├── embedding_manager.py          # From services/llm_provider/embedding_manager.py
└── clients/
    ├── __init__.py
    ├── base.py
    ├── openai.py
    ├── anthropic.py
    ├── huggingface.py
    └── ...

6.2 Mathematical Foundations (Keep from context_engineering)

shared/math/
├── __init__.py
├── information_theory.py         # Entropy, mutual information
├── vector_operations.py          # Cosine similarity, etc.
├── statistical_analysis.py       # Trends, confidence intervals
├── graph_theory.py               # Dependency graphs
├── probability_theory.py         # Bayesian, confidence
├── distance_metrics.py           # Distance calculations
└── optimization_algorithms.py    # Knapsack, etc.

6.3 Database (Keep from database/)

shared/database/
├── __init__.py
├── connection.py                 # From database/database.py
├── models.py                     # SQLAlchemy models
└── migrations/

7. Migration Plan

7.1 Phase Overview

Phase

Module

Duration

Dependencies

Sellable As

Preparation

2 days

None

Search (Core)

3 days

Shared infra

automatos-search

RAG Module

3 days

Phase 1a

automatos-rag

Knowledge Module

2 days

Phase 1a

automatos-knowledge

NL-to-SQL Module

2 days

Phase 1a

automatos-nl2sql

CodeGraph Module

2 days

Phase 1a

automatos-codegraph

Memory Module

1 week

Phase 1a

automatos-memory

Agents Module

1 week

Phase 1a, 2

automatos-agents

Tools Module

3 days

Shared infra

automatos-tools

Reasoning Module

3 days

Phase 3

5.5

Learning Module

3 days

All modules

automatos-learning

Evaluation Module

2 days

Phase 3

Cleanup

3 days

All

Total Estimated Duration: 5-6 weeks

7.2 Phase 0: Preparation (2 days)

Tasks:
□ Create modules/ directory structure
□ Create shared/ directory structure
□ Move mathematical_foundations to shared/math/
□ Move llm_provider to shared/llm/
□ Update imports in affected files
□ Run tests to verify no breakage

7.3 Phase 1: RAG Module (1 week)

Day 1-2: Chunking
□ Create modules/rag/chunking/
□ Move semantic_chunker.py from context_engineering/chunking/
□ Delete services/semantic_chunker.py
□ Delete context_engineering/chunking.py
□ Update all imports
□ Test chunking

Day 3-4: Retrieval
□ Create modules/rag/retrieval/
□ Move vector_store_enhanced.py
□ Move context_retrieval_engine.py
□ Delete context_engineering/vector_store.py
□ Delete context_engineering/context_retriever.py
□ Delete core/_vector_store_helper.py
□ Update all imports
□ Test retrieval

Day 5: Optimization
□ Create modules/rag/optimization/
□ Move context_optimizer.py
□ Split into knapsack.py, mmr.py, entropy.py
□ Update all imports
□ Test optimization

Day 6: Ingestion
□ Create modules/rag/ingestion/
□ Extract document processing from document_manager.py
□ Create file handlers (pdf.py, markdown.py, etc.)
□ Update all imports
□ Test ingestion

Day 7: Integration
□ Create modules/rag/service.py
□ Create modules/rag/__init__.py
□ Update api/documents.py to use module
□ Update all consumers (chatbot, workflows)
□ Delete services/rag_service.py
□ Run full integration tests

7.4 Phase 2: Memory Module (1 week)

Day 1-2: Memory Types
□ Create modules/memory/types/
□ Move memory_types.py
□ Create episodic.py, semantic.py, procedural.py
□ Test types

Day 3-4: Storage
□ Create modules/memory/storage/
□ Extract from memory_knowledge_system.py
□ Create postgres.py, knowledge_graph.py
□ Test storage

Day 5-6: Operations
□ Create modules/memory/operations/
□ Move augmentation.py, consolidation.py
□ Move memory_prompt_injector.py → injection.py
□ Test operations

Day 7: Integration
□ Create modules/memory/service.py
□ Delete services/memory_knowledge_system.py
□ Delete scattered memory files
□ Update all consumers
□ Run full integration tests

7.5 Phase 3-6: (Similar detailed breakdown)

8. Files to Delete

8.1 After Phase 1 (RAG)

File

Lines

Reason

services/semantic_chunker.py

400

Duplicate

context_engineering/chunking.py

429

Duplicate

context_engineering/vector_store.py

529

Merged

context_engineering/context_retriever.py

585

Merged

context_engineering/embeddings.py

364

Use shared

core/_vector_store_helper.py

100

Duplicate

services/rag_service.py

370

Replaced

Total deleted: 2,777 lines

8.2 After Phase 2 (Memory)

File

Lines

Reason

core/memory_prompt_injector.py

200

Moved

core/workflow_memory_integrator.py

300

Moved

services/chat/memory_injector.py

150

Moved

Total deleted: 650 lines

8.3 After All Phases

Estimated total lines deleted: 5,000+ Estimated duplicate code eliminated: 15,000+ lines

9. Testing Strategy

9.1 Unit Tests per Module

# modules/rag/tests/test_chunking.py
async def test_semantic_chunker_splits_by_headers():
    chunker = SemanticChunker(strategy=ChunkingStrategy.HIERARCHICAL)
    chunks = chunker.chunk("# Header\nContent...")
    assert len(chunks) > 0
    assert chunks[0].metadata.entropy > 0

# modules/rag/tests/test_retrieval.py
async def test_vector_store_search():
    store = VectorStore()
    results = await store.search("test query", limit=5)
    assert len(results) <= 5

# modules/rag/tests/test_optimization.py
async def test_knapsack_respects_budget():
    optimizer = ContextOptimizer()
    result = await optimizer.optimize(chunks, max_tokens=1000)
    assert result.total_tokens <= 1000

9.2 Integration Tests

# tests/integration/test_rag_module.py
async def test_rag_end_to_end():
    from modules.rag import RAGService
    
    rag = RAGService()
    
    # Ingest document
    await rag.ingest(content="# Test\nTest content...")
    
    # Retrieve
    result = await rag.retrieve("test")
    
    assert result.chunks is not None
    assert len(result.sources) > 0

9.3 Consumer Tests

# tests/integration/test_chatbot_uses_rag.py
async def test_chatbot_retrieves_context():
    from modules.rag import RAGService
    from consumers.chatbot import ChatService
    
    chat = ChatService()
    response = await chat.process("How do agents work?")
    
    # Verify RAG was called
    assert response.context_used is True

# tests/integration/test_workflow_uses_rag.py
async def test_workflow_has_context():
    from core.orchestrator import WorkflowOrchestrator
    
    result = await WorkflowOrchestrator().execute("Analyze agent code")
    assert result.rag_context is not None

9.4 Performance Tests

# tests/performance/test_rag_performance.py
import time

async def test_retrieval_latency():
    from modules.rag import RAGService
    
    rag = RAGService()
    
    start = time.time()
    result = await rag.retrieve("complex technical query")
    duration = time.time() - start
    
    assert duration < 2.0  # Max 2 seconds
    assert result.metrics['retrieval_time_ms'] < 2000

10. Rollback Plan

10.1 Backup Strategy

Before each phase:

Git tag current state: git tag pre-phase-{N}
Document working state
Ensure all tests pass

10.2 Rollback Procedures

If Phase 1 fails:

# Restore RAG from backup
git checkout pre-phase-1 -- services/rag_service.py
git checkout pre-phase-1 -- services/semantic_chunker.py
git checkout pre-phase-1 -- context_engineering/

# Remove new module
rm -rf orchestrator/modules/rag/

# Restart services
docker-compose restart backend

If specific component fails:

# Fallback in code
try:
    from modules.rag import RAGService
except ImportError:
    from services.rag_service import RAGService  # Legacy fallback

10.3 Feature Flags

# config.py
FEATURE_FLAGS = {
    "use_rag_module": False,      # Set True when ready
    "use_memory_module": False,
    "use_agents_module": False,
}

# Usage
from config import FEATURE_FLAGS

if FEATURE_FLAGS["use_rag_module"]:
    from modules.rag import RAGService
else:
    from services.rag_service import RAGService

11. Task Checklist

Phase 0: Directory Structure

Note: Shared infrastructure (mathematical_foundations, llm_provider) will be moved AFTER modules are working to avoid breaking imports.

Pre-work Fixes (DONE)

PRE-001: Fix services/memory_knowledge_system.py SQLAlchemy model to use 1024 dimensions ✅
PRE-002: Fix services/chat/memory_injector.py to use dynamic embedding dimension ✅
PRE-003: Fix memory/augmentation.py to use dynamic embedding dimension ✅
PRE-004: Run DB migration for memory_items.embedding to vector(1024) ✅
PRE-005: Fix rag_service.py retrieve method structure ✅
PRE-006: Add retrieve_context() backward-compatible alias ✅
PRE-007: Fix agent_platform_tools.py to handle RAGResult.chunks ✅
PRE-008: RAG returns results (quality tuning needed later) ✅

Phase 1a: Search (Core) Module - COMPLETE ✅

Phase 2: Memory Module - COMPLETE ✅

Phase 3: Agents Module - COMPLETE ✅

Phase 4: Tools Module - COMPLETE ✅

Phase 1b: RAG Module - COMPLETE ✅

Chunking ✅

P1b-001: Create modules/rag/chunking/__init__.py ✅
P1b-002: Move context_engineering/chunking/semantic_chunker.py → modules/rag/chunking/semantic_chunker.py ✅
P1b-003: Update imports to use modules.search ✅
P1b-004: Chunking strategies included in SemanticChunker ✅
P1b-005: ChunkMetadata included in semantic_chunker ✅
P1b-006: Server test passed ✅ (2024-12-04) - delete services/semantic_chunker.py (deferred)
P1b-007: Server test passed ✅ (2024-12-04) - delete context_engineering/chunking.py (deferred)

Ingestion ✅

P1b-008: Create modules/rag/ingestion/__init__.py ✅
P1b-009: Create modules/rag/ingestion/processor.py ✅
P1b-010: Create modules/rag/ingestion/handlers/markdown.py (deferred)
P1b-011: Create modules/rag/ingestion/handlers/pdf.py (deferred)
P1b-012: Create modules/rag/ingestion/handlers/text.py ✅ (2024-12-04)
P1b-013: Create modules/rag/ingestion/pipeline.py ✅ (2024-12-04)

Integration ✅

P1b-014: Create modules/rag/service.py (RAGService - wraps search/) ✅ (2024-12-04)
P1b-015: Create modules/rag/config.py ✅ (2024-12-04)
P1b-016: Create modules/rag/__init__.py ✅ (2024-12-04)
P1b-017: Update api/documents.py to use modules.rag ✅ (2024-12-04)
P1b-018: Delete services/rag_service.py (370 lines) (deferred)
P1b-019: Write tests (deferred)
P1b-020: Create git tag post-phase-1b

Phase 1c: Knowledge Module - SKIPPED ❌

Reason: Knowledge Graph functionality is not actively used. The services/database_knowledge_service.py belongs to NL-to-SQL (Phase 1d), not Knowledge Graph. Entity extraction can be added later if needed.

P1c-xxx: SKIPPED - Knowledge Graph not in use

Phase 1d: NL-to-SQL Module - COMPLETE ✅

P1d-001: Create modules/nl_to_sql/__init__.py ✅ (2024-12-04)
P1d-002: Move services/database_knowledge_service.py → modules/nl_to_sql/service.py ✅ (2024-12-04)
P1d-003: Create modules/nl_to_sql/config.py (deferred - not critical)

Schema ✅

P1d-004: Create modules/nl_to_sql/schema/__init__.py ✅ (2024-12-04)
P1d-005: Move services/database_introspection.py → modules/nl_to_sql/schema/introspection.py ✅ (2024-12-04)
P1d-006: Move services/schema_provider.py → modules/nl_to_sql/schema/provider.py ✅ (2024-12-04)
P1d-007: Create modules/nl_to_sql/schema/embeddings.py (deferred - not critical)

Query ✅

P1d-008: Create modules/nl_to_sql/query/__init__.py ✅ (2024-12-04)
P1d-009: Move services/nl_to_sql_service.py → modules/nl_to_sql/query/nl_to_sql_service.py ✅ (previously done)
P1d-010: Move services/sql_validator.py → modules/nl_to_sql/query/validator.py ✅ (2024-12-04)
P1d-011: Create modules/nl_to_sql/query/executor.py (deferred - not critical)

Integration ✅

P1d-012: Update consumers to use modules.nl_to_sql ✅ (2024-12-04)
P1d-013: Delete old service files ✅ (2024-12-04) - 5 files deleted (~1,500 lines)
P1d-014: Write tests (deferred)
P1d-015: Create git tag post-phase-1d

Phase 1e: CodeGraph Module - COMPLETE ✅

P1e-001: Create modules/codegraph/__init__.py ✅
P1e-002: Move services/codegraph_service.py → modules/codegraph/codegraph_service.py ✅
P1e-003: Create modules/codegraph/config.py (deferred - not critical)

Analysis (deferred - structure exists)

P1e-004: Create modules/codegraph/analysis/__init__.py ✅
P1e-005: Split into modules/codegraph/analysis/parser.py (deferred)
P1e-006: Create modules/codegraph/analysis/dependencies.py (deferred)
P1e-007: Create modules/codegraph/analysis/metrics.py (deferred)

Graph (deferred - structure exists)

P1e-008: Create modules/codegraph/graph/__init__.py ✅
P1e-009: Split into modules/codegraph/graph/builder.py (deferred)
P1e-010: Create modules/codegraph/graph/queries.py (deferred)

Search (deferred - structure exists)

P1e-011: Create modules/codegraph/search/__init__.py ✅
P1e-012: Create modules/codegraph/search/semantic.py (deferred)
P1e-013: Create modules/codegraph/search/structural.py (deferred)

Integration ✅

P1e-014: Update consumers to use modules.codegraph ✅ (2024-12-04)
P1e-015: Delete services/codegraph_service.py (1382 lines) ✅ (2024-12-04)
P1e-016: Write tests (deferred)
P1e-017: Create git tag post-phase-1e

Phase 2: Memory Module - COMPLETE ✅ (Duplicate Section - See Above)

All memory module tasks completed - see Phase 2 checklist above

Phase 3: Agents Module - COMPLETE ✅ (Duplicate Section - See Above)

All agents module tasks completed - see Phase 3 checklist above

Phase 4: Tools Module - COMPLETE ✅ (Duplicate Section - See Above)

All tools module tasks completed - see Phase 4 checklist above

Phase 5: Reasoning Module - COMPLETE ✅

P5-001: Create modules/reasoning/__init__.py ✅ (2024-12-04)
P5-002: Move multi_agent/collaborative_reasoning.py → modules/reasoning/collaborative_reasoning.py ✅
P5-003: Update multi_agent/__init__.py to import from modules.reasoning ✅
P5-004: Old file converted to backward-compat shim ✅
P5-005: Write tests (deferred)
P5-006: Create git tag post-phase-5

Note: Original PRD referenced reasoning/ directory that didn't exist. Actual reasoning code was in multi_agent/collaborative_reasoning.py.

Phase 5.5: Learning Module - COMPLETE ✅

P5.5-001: Create modules/learning/__init__.py ✅ (2024-12-04)
P5.5-002: Create modules/learning/engine/core.py (LearningSystemUpdater) ✅
P5.5-003: Create modules/learning/playbooks/miner.py (PlaybookMiner) ✅
P5.5-004: Create directory structure (patterns/, feedback/, engine/, playbooks/) ✅
P5.5-005: Update all consumers to use from modules.learning import ... ✅
P5.5-006: Delete old shims (services/playbook_miner.py, core/learning_system_updater.py) ✅ (2024-12-04)
P5.5-007: Write tests (deferred)
P5.5-008: Create git tag post-phase-5.5

Consumers Updated:

core/llm/master_orchestrator.py → from modules.learning import LearningSystemUpdater
api/api_playbooks.py → from modules.learning import PlaybookMiner
api/workflows.py → from modules.learning import LearningSystemUpdater

Phase 6: Evaluation Module - STRUCTURE READY ✅

P6-001: Create modules/evaluation/__init__.py ✅ (2024-12-04)
P6-002: Implement EvaluationEngine (no source files to migrate)
P6-003: Implement BenchmarkRunner (no source files to migrate)
P6-004: Write tests

Note: Original PRD referenced evaluation/ directory that didn't exist. Module structure created - implementation to be done fresh when needed.

Phase 7: Consumers & Cleanup - COMPLETE ✅

Consumers (Created & Populated)

P7-001: Create consumers/ directory ✅
P7-002: Create consumers/chatbot/service.py ✅
P7-003: Create consumers/chatbot/streaming.py ✅
P7-004: Create consumers/chatbot/prompt_analyzer.py ✅
P7-005: Create consumers/chatbot/tool_router.py ✅
P7-006: Create consumers/workflows/analytics.py ✅
P7-007: Create consumers/workflows/streaming.py ✅
P7-008: Create consumers/workflows/usage_tracker.py ✅
P7-009: Create consumers/external/ directory ✅

Old Directories Deleted

P7-010: Delete services/ directory ✅
P7-011: Delete context_engineering/ directory ✅
P7-012: Delete memory/ directory ✅
P7-013: Delete multi_agent/ directory ✅
P7-014: Delete utils/ directory ✅
P7-015: Delete reasoning/ directory ✅

Final Cleanup (Completed 2024-12-04)

P7-016: Delete field_theory/ - not used, can re-add later ✅
P7-017: Delete security/ - not imported anywhere ✅
P7-018: Git tags for each phase
P7-019: Module-level test coverage
P7-020: Create final git tag v3.0-modular

12. Summary Metrics

Before Refactoring

Metric

Value

Duplicate code

~15,000 lines

Files with RAG/Search logic

12+ scattered

Files with Memory logic

10 scattered

Files with Agent logic

8+ scattered

Files in services/

50+ (dumping ground)

Module import depth

3-5 imports per feature

Sellable products

After Refactoring (ACTUAL)

Metric

Actual

Duplicate code

0 lines ✅

modules/ files

136 files ✅

core/ files

64 files ✅

consumers/ files

11 files ✅

api/ files

52 files ✅

Total organized

263 files ✅

services/ directory

DELETED ✅

context_engineering/

DELETED ✅

multi_agent/

DELETED ✅

Module import depth

1 import per feature ✅

Sellable products

11 standalone modules ✅

Module Summary (ACTUAL)

Module

Purpose

Files

Sellable As

search/

Core vector search engine

automatos-search

rag/

Document RAG (chunking, ingestion)

automatos-rag

nl2sql/

Natural language to SQL

automatos-nl2sql

codegraph/

Code analysis & search

automatos-codegraph

memory/

Multi-type memory system

automatos-memory

agents/

Agent lifecycle + multi-agent

automatos-agents

tools/

Tool registry & execution

automatos-tools

orchestrator/

9-stage workflow pipeline

automatos-orchestrator

learning/

Self-improvement system

automatos-learning

reasoning/

Collaborative reasoning

automatos-reasoning

evaluation/

Evaluation (structure ready)

automatos-evaluation

Code Health Goals - ALL ACHIEVED ✅

✅ Single Source of Truth per feature
✅ One module = One concern
✅ Clear public APIs (1-line imports)
✅ Testable boundaries
✅ Sellable as standalone products
✅ Third-party consumable
✅ Thin consumers (chatbot, workflows)
✅ Thin API layer (routes only)
✅ No shared/ directory (merged into core/)
✅ modules/orchestrator/ added (9-stage workflow)
✅ modules/*/services/ for module-specific services

Key Improvements Over Original Plan

Original Plan

Actual Implementation

Why Better

shared/ directory

Merged into core/

Less indirection, cleaner imports

Orchestration in core/

modules/orchestrator/

9-stage workflow is sellable

multi_agent/ standalone

modules/agents/multi_agent/

Agents own multi-agent logic

Flat module structure

modules/*/services/

Module-specific services organized

modules/knowledge/

SKIPPED

Not actively used

9 modules planned

11 modules created

Added orchestrator, reasoning

Document History

Version

Date

Author

Changes

1.0.0

2024-12-03

Initial comprehensive PRD

Approval

Role

Name

Date

Signature

Product Owner

Tech Lead

DevOps

END OF PRD-30

PreviousPRD-29: Future AGI Observability & Evaluation Platform Integration NextPRD-31: Chatbot UX Upgrade (Vercel Chat SDK UX + Automatos Explorer)

Last updated 23 days ago

Good afternoon

hashtagComplete Codebase Restructuring for Standalone, Sellable Modules

hashtag🎉 MIGRATION COMPLETE - December 4, 2024

hashtagWhat Was Achieved

hashtagDirectories DELETED (Fully Removed)

hashtagFinal Architecture (ACTUAL)

hashtagRemaining Polish (Non-Critical)

hashtagTable of Contents

hashtag1. Executive Summary

hashtag1.1 Purpose

hashtag1.2 Business Value

hashtag1.3 Success Metrics

hashtag2. Problem Statement

hashtag2.1 Current Issues

hashtagIssue 1: Massive Code Duplication

hashtagIssue 2: Scattered Functionality

hashtagIssue 3: Tight Coupling

hashtagIssue 4: No Clear Ownership

hashtag3. Current State Analysis

hashtag3.1 Directory Structure (Current)

hashtag3.2 Duplication Analysis

hashtagCHUNKING - 4 Implementations

hashtagVECTOR STORE - 4 Implementations

hashtagCONTEXT RETRIEVAL - 3 Implementations

hashtagEMBEDDINGS - 2 Implementations

hashtagMEMORY - 10 Scattered Files

hashtag4. Target Architecture

hashtag4.1 New Directory Structure

hashtag4.2 Module Dependency Graph

hashtag4.3 Sellable Products

hashtag5. Module Specifications

hashtag5.1 RAG Module

hashtag5.1.1 Purpose

hashtag5.1.2 Public API

hashtag5.1.3 Service Class

hashtag5.1.4 RAG Module Flow Diagram

hashtag5.1.5 RAG Ingestion Flow

hashtag5.1.6 Files Migration for RAG Module

hashtag5.2 Memory Module

hashtag5.2.1 Purpose

hashtag5.2.2 Public API

hashtag5.2.3 Memory Module Flow Diagram

hashtag5.2.4 Files Migration for Memory Module

hashtag5.3 Agents Module

hashtag5.3.1 Purpose

hashtag5.3.2 Public API

hashtag5.3.3 Agent Module Flow Diagram

hashtag5.3.4 Files Migration for Agents Module

hashtag5.4 Tools Module

hashtag5.4.1 Public API

hashtag5.4.2 Files Migration for Tools Module

hashtag6. Shared Infrastructure

hashtag6.1 LLM Providers (Keep from services/llm_provider)

hashtag6.2 Mathematical Foundations (Keep from context_engineering)

hashtag6.3 Database (Keep from database/)

hashtag7. Migration Plan

hashtag7.1 Phase Overview

hashtag7.2 Phase 0: Preparation (2 days)

hashtag7.3 Phase 1: RAG Module (1 week)

hashtag7.4 Phase 2: Memory Module (1 week)

hashtag7.5 Phase 3-6: (Similar detailed breakdown)

hashtag8. Files to Delete

hashtag8.1 After Phase 1 (RAG)

hashtag8.2 After Phase 2 (Memory)

hashtag8.3 After All Phases

hashtag9. Testing Strategy

hashtag9.1 Unit Tests per Module

hashtag9.2 Integration Tests

hashtag9.3 Consumer Tests

hashtag9.4 Performance Tests

hashtag10. Rollback Plan

hashtag10.1 Backup Strategy

hashtag10.2 Rollback Procedures

hashtag10.3 Feature Flags

hashtag11. Task Checklist

hashtagPhase 0: Directory Structure

hashtagPre-work Fixes (DONE)

hashtagPhase 1a: Search (Core) Module - COMPLETE ✅

hashtagPhase 2: Memory Module - COMPLETE ✅

hashtagPhase 3: Agents Module - COMPLETE ✅

Complete Codebase Restructuring for Standalone, Sellable Modules

🎉 MIGRATION COMPLETE - December 4, 2024

What Was Achieved

Directories DELETED (Fully Removed)

Final Architecture (ACTUAL)

Remaining Polish (Non-Critical)

Table of Contents

1. Executive Summary

1.1 Purpose

1.2 Business Value

1.3 Success Metrics

2. Problem Statement

2.1 Current Issues

Issue 1: Massive Code Duplication

Issue 2: Scattered Functionality

Issue 3: Tight Coupling

Issue 4: No Clear Ownership

3. Current State Analysis

3.1 Directory Structure (Current)

3.2 Duplication Analysis

CHUNKING - 4 Implementations

VECTOR STORE - 4 Implementations

CONTEXT RETRIEVAL - 3 Implementations

EMBEDDINGS - 2 Implementations

MEMORY - 10 Scattered Files

4. Target Architecture

4.1 New Directory Structure

4.2 Module Dependency Graph

4.3 Sellable Products

5. Module Specifications

5.1 RAG Module

5.1.1 Purpose

5.1.2 Public API

5.1.3 Service Class

5.1.4 RAG Module Flow Diagram

5.1.5 RAG Ingestion Flow

5.1.6 Files Migration for RAG Module

5.2 Memory Module

5.2.1 Purpose

5.2.2 Public API

5.2.3 Memory Module Flow Diagram

5.2.4 Files Migration for Memory Module

5.3 Agents Module

5.3.1 Purpose

5.3.2 Public API

5.3.3 Agent Module Flow Diagram

5.3.4 Files Migration for Agents Module

5.4 Tools Module

5.4.1 Public API

5.4.2 Files Migration for Tools Module

6. Shared Infrastructure

6.1 LLM Providers (Keep from services/llm_provider)

6.2 Mathematical Foundations (Keep from context_engineering)

6.3 Database (Keep from database/)

7. Migration Plan

7.1 Phase Overview

7.2 Phase 0: Preparation (2 days)

7.3 Phase 1: RAG Module (1 week)

7.4 Phase 2: Memory Module (1 week)

7.5 Phase 3-6: (Similar detailed breakdown)

8. Files to Delete

8.1 After Phase 1 (RAG)

8.2 After Phase 2 (Memory)

8.3 After All Phases

9. Testing Strategy

9.1 Unit Tests per Module

9.2 Integration Tests

9.3 Consumer Tests

9.4 Performance Tests

10. Rollback Plan

10.1 Backup Strategy

10.2 Rollback Procedures

10.3 Feature Flags

11. Task Checklist

Phase 0: Directory Structure

Pre-work Fixes (DONE)

Phase 1a: Search (Core) Module - COMPLETE ✅

Phase 2: Memory Module - COMPLETE ✅

Phase 3: Agents Module - COMPLETE ✅

Phase 4: Tools Module - COMPLETE ✅