re-organise repo structure

Author: anhnh2002

README.md

@@ -1,15 +1,239 @@
-install Node
+# CodeWiki: Automated Repository-Level Documentation Generation
+
+<div align="center">
+
+![CodeWiki Architecture](img/framework-overview.png)
+
+<!-- [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) -->
+[![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+<!-- [![Paper](https://img.shields.io/badge/paper-arXiv-red.svg)](https://arxiv.org/abs/XXXXX) -->
+
+**The first open-source framework for holistic, structured repository-level documentation across multilingual codebases**
+
+[Features](#features) • [Installation](#installation) • [Quick Start](#quick-start) • [Benchmark](#benchmark) • [Documentation](#documentation-structure) • [Citation](#citation)
+
+</div>
+
+---
+
+## 🎯 Overview
+
+Developers spend **58% of their working time** understanding codebases, yet maintaining comprehensive documentation remains a persistent challenge. CodeWiki addresses this by providing automated, scalable repository-level documentation generation that captures:
+
+- 🏗️ **System Architecture** - High-level design patterns and module relationships
+- 🔄 **Data Flow Visualizations** - How information moves through your system
+- 📊 **Cross-Module Dependencies** - Interactive dependency graphs and sequence diagrams
+- 📝 **Comprehensive API Documentation** - From architectural overviews to implementation details
+
+### Key Innovations
+
+| Feature | Description | Impact |
+|---------|-------------|--------|
+| **Hierarchical Decomposition** | Dynamic programming-inspired strategy that partitions repositories into coherent modules | Handles codebases of arbitrary size (86K-1.4M LOC tested) |
+| **Recursive Agentic System** | Adaptive processing with dynamic delegation for complex modules | Maintains quality while scaling |
+| **Multi-Format Synthesis** | Generates textual documentation, architecture diagrams, data flows, and sequence diagrams | Comprehensive understanding from multiple perspectives |
+| **Multilingual Support** | Supports 7 languages: Python, Java, JavaScript, TypeScript, C, C++, C# | Universal applicability |
+
+---
+
+## Features
+
+### Documentation Generation
+
+- ✅ **Repository-Level Documentation** - First framework to generate complete repo-level docs at scale
+- ✅ **Visual Artifacts** - Automatic generation of architecture diagrams and data flow visualizations
+- ✅ **Cross-Module References** - Intelligent reference management prevents redundancy
+- ✅ **Hierarchical Structure** - Multi-level documentation from high-level overviews to detailed APIs
+
+### Benchmark
+
+- ✅ **[CodeWikiBench](https://github.com/FSoft-AI4Code/CodeWikiBench.git)** - First benchmark specifically designed for repository-level documentation
+
+#### Performance Results
+
+CodeWiki has been evaluated on the **CodeWikiBench** dataset, the first benchmark specifically designed for repository-level documentation quality assessment.
+
+| Language Category | CodeWiki Avg | Improvement over Baseline |
+|-------------------|--------------|---------------------------|
+| High-Level (Python, JS, TS) | **79.14%** | +10.47% |
+| Managed (C#, Java) | **68.84%** | +4.04% |
+| Systems (C, C++) | 53.24% | -3.15% |
+| **Overall Average** | **68.79%** | **+4.73%** |
+
+CodeWiki demonstrates significant improvements in high-level and managed languages, with an overall 4.73% improvement over baseline approaches.
+
+---
+
+## Installation
+
+### Prerequisites
+
+- Python 3.12+
+- Node.js (for mermaid validation)
+- Docker (optional, for containerized deployment)
+
+### Standard Installation
+
 ```bash
-# macos
+# Clone the repository
+git clone https://github.com/yourusername/codewiki.git
+cd codewiki
+
+# Create and activate virtual environment
+python3.12 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Install Node.js (if not already installed)
+# macOS
 brew install node
 
-# linux
+# Linux
 sudo apt update && apt install -y nodejs npm
 ```
 
+### Docker Installation
+
 ```bash
-python3.12 -m venv .venv
+# Copy environment configuration
+cp env.example .env
+# Edit .env with your API keys
+
+# Create network
+docker network create codewiki-network
+
+# Start services
+docker-compose up -d
+```
+
+---
+
+## Quick Start
+
+### 1. Configure API Keys
+
+Create a `.env` file from the template:
+
+```bash
+cp env.example .env
+```
+
+Edit `.env` with your configuration:
+
+```bash
+# LLM Configuration
+MAIN_MODEL=claude-sonnet-4
+FALLBACK_MODEL_1=glm-4p5
+LLM_BASE_URL=http://litellm:4000/
+LLM_API_KEY=your-api-key-here
+
+# Application
+APP_PORT=8000
+
+# Optional: Logfire for monitoring
+LOGFIRE_TOKEN=your-logfire-token
+```
+
+### 2. Run the Web Application
+
+```bash
+# Activate virtual environment
 source .venv/bin/activate
-pip install -r requirements.txt
+
+# Start the web application
 python run_web_app.py
 ```
+
+Access the application at `http://localhost:8000` to generate documentation by github url and commit id (optional)
+
+---
+
+## Workflow
+
+```mermaid
+graph TB
+    A[Repository Input] --> B[Dependency Graph Construction]
+    B --> C[Hierarchical Decomposition]
+    C --> D[Module Tree]
+    D --> E[Recursive Agent Processing]
+    E --> F{Complexity Check}
+    F -->|Complex| G[Dynamic Delegation]
+    F -->|Simple| H[Generate Documentation]
+    G --> E
+    H --> I[Cross-Module References]
+    I --> J[Hierarchical Assembly]
+    J --> K[Comprehensive Documentation]
+    K --> L[Architecture Diagrams]
+    K --> M[Data Flow Visualizations]
+    K --> N[API Documentation]
+```
+
+### Processing Pipeline
+
+1. **Repository Analysis** - AST parsing and dependency graph construction
+2. **Hierarchical Decomposition** - Feature-based module partitioning
+3. **Recursive Documentation** - Agent-based processing with dynamic delegation
+4. **Hierarchical Assembly** - Bottom-up synthesis of comprehensive docs
+
+---
+
+## Documentation Structure
+
+Generated documentation includes:
+
+### 📄 Textual Documentation
+- **README Overview** - High-level project introduction
+- **Architecture Guide** - System design and component relationships
+- **API Reference** - Detailed interface specifications
+- **Usage Examples** - Practical code samples and patterns
+
+### 📊 Visual Artifacts
+- **System Architecture Diagrams** - Component relationships and hierarchies
+- **Data Flow Visualizations** - Information flow through the system
+- **Sequence Diagrams** - Inter-component communication patterns
+- **Dependency Graphs** - Module and function dependencies
+
+
+---
+
+## Citation
+
+If you use CodeWiki in your research, please cite:
+
+```bibtex
+@article{codewiki2025,
+  title={CodeWiki: Automated Repository-Level Documentation Generation with Hierarchical Decomposition and Agentic Processing},
+  author={Your Name},
+  journal={arXiv preprint arXiv:XXXXX},
+  year={2025}
+}
+```
+
+<!-- ---
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+--- -->
+
+
+<!-- ---
+
+## 📧 Contact
+
+- **Issues**: [GitHub Issues](https://github.com/yourusername/codewiki/issues)
+- **Discussions**: [GitHub Discussions](https://github.com/yourusername/codewiki/discussions)
+- **Email**: your.email@domain.com -->
+
+---
+
+<div align="center">
+
+**Made with ❤️ by the CodeWiki Team**
+
+[⬆ Back to Top](#codewiki-automated-repository-level-documentation-generation)
+
+</div>

img/framework-overview.png

@@ -35,24 +35,23 @@
     logger.warning(f"Failed to configure logfire: {e}")
 
 # Local imports
-from agent_tools.deps import CodeWikiDeps
-from agent_tools.read_code_components import read_code_components_tool
-from agent_tools.str_replace_editor import str_replace_editor_tool
-from agent_tools.generate_sub_module_documentations import generate_sub_module_documentation_tool
-from llm_services import fallback_models
-from prompt_template import (
+from .agent_tools.deps import CodeWikiDeps
+from .agent_tools.read_code_components import read_code_components_tool
+from .agent_tools.str_replace_editor import str_replace_editor_tool
+from .agent_tools.generate_sub_module_documentations import generate_sub_module_documentation_tool
+from .llm_services import fallback_models
+from .prompt_template import (
     SYSTEM_PROMPT,
     LEAF_SYSTEM_PROMPT,
     format_user_prompt,
 )
-from utils import is_complex_module
-from cluster_modules import cluster_modules
+from .utils import is_complex_module
 from config import (
     Config,
     MODULE_TREE_FILENAME,
 )
 from utils import file_manager
-from dependency_analyzer.models.core import Node
+from .dependency_analyzer.models.core import Node
 
 
 class AgentOrchestrator:

src/agent_orchestrator.py src/be/agent_orchestrator.pysrc/agent_orchestrator.py renamed to src/be/agent_orchestrator.py

@@ -1,5 +1,5 @@
 from dataclasses import dataclass
-from dependency_analyzer.models.core import Node
+from ..dependency_analyzer.models.core import Node
 
 @dataclass
 class CodeWikiDeps:

src/agent_tools/deps.py src/be/agent_tools/deps.pysrc/agent_tools/deps.py renamed to src/be/agent_tools/deps.py

@@ -1,13 +1,12 @@
 from pydantic_ai import RunContext, Tool, Agent
-import json
 
 from .deps import CodeWikiDeps
 from .read_code_components import read_code_components_tool
 from .str_replace_editor import str_replace_editor_tool
-from llm_services import fallback_models
-from prompt_template import SYSTEM_PROMPT, LEAF_SYSTEM_PROMPT, format_user_prompt
-from utils import is_complex_module, count_tokens
-from cluster_modules import format_potential_core_components
+from ..llm_services import fallback_models
+from ..prompt_template import SYSTEM_PROMPT, LEAF_SYSTEM_PROMPT, format_user_prompt
+from ..utils import is_complex_module, count_tokens
+from ..cluster_modules import format_potential_core_components
 from config import MAX_TOKEN_PER_LEAF_MODULE
 
 

…ls/generate_sub_module_documentations.py …ls/generate_sub_module_documentations.pysrc/agent_tools/generate_sub_module_documentations.py renamed to src/be/agent_tools/generate_sub_module_documentations.py src/agent_tools/generate_sub_module_documentations.py renamed to src/be/agent_tools/generate_sub_module_documentations.py

@@ -22,7 +22,7 @@
 from pydantic_ai import RunContext, Tool
 
 from .deps import CodeWikiDeps
-from utils import validate_mermaid_diagrams
+from ..utils import validate_mermaid_diagrams
 
 
 # There are some super strange "ascii can't decode x" errors,

src/agent_tools/read_code_components.py …c/be/agent_tools/read_code_components.pysrc/agent_tools/read_code_components.py renamed to src/be/agent_tools/read_code_components.py src/agent_tools/read_code_components.py renamed to src/be/agent_tools/read_code_components.py

@@ -3,11 +3,11 @@
 import logging
 logger = logging.getLogger(__name__)
 
-from dependency_analyzer.models.core import Node
-from llm_services import call_llm
-from utils import count_tokens
+from .dependency_analyzer.models.core import Node
+from .llm_services import call_llm
+from .utils import count_tokens
 from config import MAX_TOKEN_PER_MODULE, CLUSTER_MODEL
-from prompt_template import format_cluster_prompt
+from .prompt_template import format_cluster_prompt
 
 
 def format_potential_core_components(leaf_nodes: List[str], components: Dict[str, Node]) -> tuple[str, str]: