|
| 1 | +# Development Guide |
| 2 | + |
| 3 | +This guide provides information for developers who want to contribute to or extend CodeWiki. |
| 4 | + |
| 5 | +## Project Structure |
| 6 | + |
| 7 | +``` |
| 8 | +codewiki/ |
| 9 | +├── codewiki/ # Main package |
| 10 | +│ ├── cli/ # CLI implementation |
| 11 | +│ │ ├── commands/ # CLI commands (config, generate) |
| 12 | +│ │ ├── models/ # Data models |
| 13 | +│ │ ├── utils/ # Utilities |
| 14 | +│ │ └── adapters/ # External integrations |
| 15 | +│ ├── src/ # Web application |
| 16 | +│ │ ├── be/ # Backend (dependency analysis, agents) |
| 17 | +│ │ │ ├── agent_orchestrator.py |
| 18 | +│ │ │ ├── agent_tools/ |
| 19 | +│ │ │ ├── cluster_modules.py |
| 20 | +│ │ │ ├── dependency_analyzer/ |
| 21 | +│ │ │ ├── documentation_generator.py |
| 22 | +│ │ │ └── llm_services.py |
| 23 | +│ │ └── fe/ # Frontend (web interface) |
| 24 | +│ │ ├── web_app.py |
| 25 | +│ │ ├── routes.py |
| 26 | +│ │ ├── github_processor.py |
| 27 | +│ │ └── visualise_docs.py |
| 28 | +│ ├── templates/ # HTML templates |
| 29 | +│ └── run_web_app.py # Web app entry point |
| 30 | +├── docker/ # Docker configuration |
| 31 | +│ ├── Dockerfile |
| 32 | +│ ├── docker-compose.yml |
| 33 | +│ └── env.example |
| 34 | +├── img/ # Images and assets |
| 35 | +├── paper/ # Research paper source |
| 36 | +├── tests/ # Test suite |
| 37 | +├── output/ # Generated documentation output |
| 38 | +├── pyproject.toml # Project metadata |
| 39 | +├── requirements.txt # Python dependencies |
| 40 | +└── README.md # Main documentation |
| 41 | +``` |
| 42 | + |
| 43 | +## Development Setup |
| 44 | + |
| 45 | +### Prerequisites |
| 46 | + |
| 47 | +- Python 3.12+ |
| 48 | +- Node.js (for mermaid validation) |
| 49 | +- Git |
| 50 | +- Tree-sitter language parsers |
| 51 | + |
| 52 | +### Installation |
| 53 | + |
| 54 | +```bash |
| 55 | +# Clone the repository |
| 56 | +git clone https://github.com/FSoft-AI4Code/CodeWiki.git |
| 57 | +cd CodeWiki |
| 58 | + |
| 59 | +# Create virtual environment |
| 60 | +python3.12 -m venv .venv |
| 61 | +source .venv/bin/activate # On Windows: .venv\Scripts\activate |
| 62 | + |
| 63 | +# Install in development mode |
| 64 | +pip install -e . |
| 65 | + |
| 66 | +# Install development dependencies |
| 67 | +pip install -r requirements.txt |
| 68 | +``` |
| 69 | + |
| 70 | +## Core Components |
| 71 | + |
| 72 | +### Backend Architecture |
| 73 | + |
| 74 | +#### 1. Dependency Analysis (`src/be/dependency_analyzer/`) |
| 75 | + |
| 76 | +- **AST Parser**: Tree-sitter based parsing for 7 languages |
| 77 | +- **Dependency Graph Builder**: Constructs call graphs and dependency relationships |
| 78 | +- **Analyzers**: Language-specific analyzers (Python, Java, JavaScript, TypeScript, C, C++, C#) |
| 79 | + |
| 80 | +#### 2. Module Clustering (`src/be/cluster_modules.py`) |
| 81 | + |
| 82 | +- Hierarchical decomposition of repository structure |
| 83 | +- Feature-oriented module partitioning |
| 84 | +- Topological sorting for dependency ordering |
| 85 | + |
| 86 | +#### 3. Agent System (`src/be/agent_orchestrator.py`) |
| 87 | + |
| 88 | +- Recursive agent-based documentation generation |
| 89 | +- Dynamic delegation for complex modules |
| 90 | +- Cross-module reference management |
| 91 | + |
| 92 | +#### 4. Agent Tools (`src/be/agent_tools/`) |
| 93 | + |
| 94 | +- `read_code_components.py`: Code reading utilities |
| 95 | +- `generate_sub_module_documentations.py`: Sub-module documentation generation |
| 96 | +- `str_replace_editor.py`: Documentation editing tools |
| 97 | +- `deps.py`: Dependency traversal tools |
| 98 | + |
| 99 | +### Frontend Architecture |
| 100 | + |
| 101 | +#### Web Application (`src/fe/`) |
| 102 | + |
| 103 | +- **FastAPI Backend**: `web_app.py`, `routes.py` |
| 104 | +- **GitHub Integration**: `github_processor.py` |
| 105 | +- **Documentation Viewer**: `visualise_docs.py` |
| 106 | +- **Background Processing**: `background_worker.py` |
| 107 | + |
| 108 | +### CLI Architecture |
| 109 | + |
| 110 | +#### Command Structure (`cli/commands/`) |
| 111 | + |
| 112 | +- `config.py`: Configuration management |
| 113 | +- `generate.py`: Documentation generation |
| 114 | + |
| 115 | +#### Utilities (`cli/utils/`) |
| 116 | + |
| 117 | +- `fs.py`: File system operations |
| 118 | +- `validation.py`: Input validation |
| 119 | +- `progress.py`: Progress tracking |
| 120 | +- `logging.py`: Logging configuration |
| 121 | + |
| 122 | +## Adding Support for New Languages |
| 123 | + |
| 124 | +To add support for a new programming language: |
| 125 | + |
| 126 | +1. **Add language analyzer** in `src/be/dependency_analyzer/analyzers/`: |
| 127 | + |
| 128 | +```python |
| 129 | +# new_language.py |
| 130 | +from .base import BaseAnalyzer |
| 131 | + |
| 132 | +class NewLanguageAnalyzer(BaseAnalyzer): |
| 133 | + def __init__(self): |
| 134 | + super().__init__("new_language") |
| 135 | + |
| 136 | + def extract_dependencies(self, ast_node): |
| 137 | + # Implement dependency extraction |
| 138 | + pass |
| 139 | + |
| 140 | + def extract_components(self, ast_node): |
| 141 | + # Implement component extraction |
| 142 | + pass |
| 143 | +``` |
| 144 | + |
| 145 | +2. **Register the analyzer** in `src/be/dependency_analyzer/ast_parser.py`: |
| 146 | + |
| 147 | +```python |
| 148 | +LANGUAGE_ANALYZERS = { |
| 149 | + # ... existing languages ... |
| 150 | + "new_language": NewLanguageAnalyzer, |
| 151 | +} |
| 152 | +``` |
| 153 | + |
| 154 | +3. **Add file extensions** in configuration |
| 155 | + |
| 156 | +4. **Add tests** for the new language |
| 157 | + |
| 158 | +## Testing |
| 159 | + |
| 160 | +```bash |
| 161 | +# Run all tests |
| 162 | +pytest |
| 163 | + |
| 164 | +# Run specific test file |
| 165 | +pytest tests/test_dependency_analyzer.py |
| 166 | + |
| 167 | +# Run with coverage |
| 168 | +pytest --cov=codewiki tests/ |
| 169 | +``` |
| 170 | + |
| 171 | +## Code Style |
| 172 | + |
| 173 | +- Follow PEP 8 for Python code |
| 174 | +- Use type hints where applicable |
| 175 | +- Write docstrings for public functions and classes |
| 176 | +- Keep functions focused and modular |
| 177 | + |
| 178 | +## Contributing |
| 179 | + |
| 180 | +1. **Fork the repository** |
| 181 | +2. **Create a feature branch**: `git checkout -b feature/your-feature` |
| 182 | +3. **Make your changes** |
| 183 | +4. **Write/update tests** |
| 184 | +5. **Ensure tests pass**: `pytest` |
| 185 | +6. **Commit your changes**: `git commit -am 'Add new feature'` |
| 186 | +7. **Push to the branch**: `git push origin feature/your-feature` |
| 187 | +8. **Submit a pull request** |
| 188 | + |
| 189 | +## Documentation Generation Workflow |
| 190 | + |
| 191 | +```mermaid |
| 192 | +graph TB |
| 193 | + A[Repository Input] --> B[Dependency Graph Construction] |
| 194 | + B --> C[Hierarchical Decomposition] |
| 195 | + C --> D[Module Tree] |
| 196 | + D --> E[Recursive Agent Processing] |
| 197 | + E --> F{Complexity Check} |
| 198 | + F -->|Complex| G[Dynamic Delegation] |
| 199 | + F -->|Simple| H[Generate Documentation] |
| 200 | + G --> E |
| 201 | + H --> I[Cross-Module References] |
| 202 | + I --> J[Hierarchical Assembly] |
| 203 | + J --> K[Comprehensive Documentation] |
| 204 | +``` |
| 205 | + |
| 206 | +## Debugging |
| 207 | + |
| 208 | +### Enable Verbose Logging |
| 209 | + |
| 210 | +```bash |
| 211 | +# CLI |
| 212 | +codewiki generate --verbose |
| 213 | + |
| 214 | +# Environment variable |
| 215 | +export CODEWIKI_LOG_LEVEL=DEBUG |
| 216 | +``` |
| 217 | + |
| 218 | +### Common Issues |
| 219 | + |
| 220 | +**Tree-sitter parser errors**: |
| 221 | +- Ensure language parsers are properly installed |
| 222 | +- Check file encoding (UTF-8 expected) |
| 223 | + |
| 224 | +**LLM API errors**: |
| 225 | +- Verify API keys and endpoints |
| 226 | +- Check rate limits |
| 227 | +- Enable retry logic |
| 228 | + |
| 229 | +**Memory issues with large repositories**: |
| 230 | +- Adjust module decomposition threshold |
| 231 | +- Increase delegation depth limit |
| 232 | + |
| 233 | +## Performance Optimization |
| 234 | + |
| 235 | +- **Caching**: Results are cached to avoid redundant processing |
| 236 | +- **Parallel Processing**: Multiple modules can be processed concurrently |
| 237 | +- **Incremental Updates**: Only process changed modules (future work) |
| 238 | + |
| 239 | +## Support |
| 240 | + |
| 241 | +For development questions: |
| 242 | +- GitHub Issues: https://github.com/FSoft-AI4Code/CodeWiki/issues |
| 243 | +- Main Documentation: [README.md](README.md) |
| 244 | + |
0 commit comments