Skip to content

Commit cf81e1d

Browse files
committed
update readme
1 parent 5c5d884 commit cf81e1d

File tree

4 files changed

+338
-322
lines changed

4 files changed

+338
-322
lines changed

.DS_Store

0 Bytes
Binary file not shown.

DEVELOPMENT.md

Lines changed: 244 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
# Development Guide
2+
3+
This guide provides information for developers who want to contribute to or extend CodeWiki.
4+
5+
## Project Structure
6+
7+
```
8+
codewiki/
9+
├── codewiki/ # Main package
10+
│ ├── cli/ # CLI implementation
11+
│ │ ├── commands/ # CLI commands (config, generate)
12+
│ │ ├── models/ # Data models
13+
│ │ ├── utils/ # Utilities
14+
│ │ └── adapters/ # External integrations
15+
│ ├── src/ # Web application
16+
│ │ ├── be/ # Backend (dependency analysis, agents)
17+
│ │ │ ├── agent_orchestrator.py
18+
│ │ │ ├── agent_tools/
19+
│ │ │ ├── cluster_modules.py
20+
│ │ │ ├── dependency_analyzer/
21+
│ │ │ ├── documentation_generator.py
22+
│ │ │ └── llm_services.py
23+
│ │ └── fe/ # Frontend (web interface)
24+
│ │ ├── web_app.py
25+
│ │ ├── routes.py
26+
│ │ ├── github_processor.py
27+
│ │ └── visualise_docs.py
28+
│ ├── templates/ # HTML templates
29+
│ └── run_web_app.py # Web app entry point
30+
├── docker/ # Docker configuration
31+
│ ├── Dockerfile
32+
│ ├── docker-compose.yml
33+
│ └── env.example
34+
├── img/ # Images and assets
35+
├── paper/ # Research paper source
36+
├── tests/ # Test suite
37+
├── output/ # Generated documentation output
38+
├── pyproject.toml # Project metadata
39+
├── requirements.txt # Python dependencies
40+
└── README.md # Main documentation
41+
```
42+
43+
## Development Setup
44+
45+
### Prerequisites
46+
47+
- Python 3.12+
48+
- Node.js (for mermaid validation)
49+
- Git
50+
- Tree-sitter language parsers
51+
52+
### Installation
53+
54+
```bash
55+
# Clone the repository
56+
git clone https://github.com/FSoft-AI4Code/CodeWiki.git
57+
cd CodeWiki
58+
59+
# Create virtual environment
60+
python3.12 -m venv .venv
61+
source .venv/bin/activate # On Windows: .venv\Scripts\activate
62+
63+
# Install in development mode
64+
pip install -e .
65+
66+
# Install development dependencies
67+
pip install -r requirements.txt
68+
```
69+
70+
## Core Components
71+
72+
### Backend Architecture
73+
74+
#### 1. Dependency Analysis (`src/be/dependency_analyzer/`)
75+
76+
- **AST Parser**: Tree-sitter based parsing for 7 languages
77+
- **Dependency Graph Builder**: Constructs call graphs and dependency relationships
78+
- **Analyzers**: Language-specific analyzers (Python, Java, JavaScript, TypeScript, C, C++, C#)
79+
80+
#### 2. Module Clustering (`src/be/cluster_modules.py`)
81+
82+
- Hierarchical decomposition of repository structure
83+
- Feature-oriented module partitioning
84+
- Topological sorting for dependency ordering
85+
86+
#### 3. Agent System (`src/be/agent_orchestrator.py`)
87+
88+
- Recursive agent-based documentation generation
89+
- Dynamic delegation for complex modules
90+
- Cross-module reference management
91+
92+
#### 4. Agent Tools (`src/be/agent_tools/`)
93+
94+
- `read_code_components.py`: Code reading utilities
95+
- `generate_sub_module_documentations.py`: Sub-module documentation generation
96+
- `str_replace_editor.py`: Documentation editing tools
97+
- `deps.py`: Dependency traversal tools
98+
99+
### Frontend Architecture
100+
101+
#### Web Application (`src/fe/`)
102+
103+
- **FastAPI Backend**: `web_app.py`, `routes.py`
104+
- **GitHub Integration**: `github_processor.py`
105+
- **Documentation Viewer**: `visualise_docs.py`
106+
- **Background Processing**: `background_worker.py`
107+
108+
### CLI Architecture
109+
110+
#### Command Structure (`cli/commands/`)
111+
112+
- `config.py`: Configuration management
113+
- `generate.py`: Documentation generation
114+
115+
#### Utilities (`cli/utils/`)
116+
117+
- `fs.py`: File system operations
118+
- `validation.py`: Input validation
119+
- `progress.py`: Progress tracking
120+
- `logging.py`: Logging configuration
121+
122+
## Adding Support for New Languages
123+
124+
To add support for a new programming language:
125+
126+
1. **Add language analyzer** in `src/be/dependency_analyzer/analyzers/`:
127+
128+
```python
129+
# new_language.py
130+
from .base import BaseAnalyzer
131+
132+
class NewLanguageAnalyzer(BaseAnalyzer):
133+
def __init__(self):
134+
super().__init__("new_language")
135+
136+
def extract_dependencies(self, ast_node):
137+
# Implement dependency extraction
138+
pass
139+
140+
def extract_components(self, ast_node):
141+
# Implement component extraction
142+
pass
143+
```
144+
145+
2. **Register the analyzer** in `src/be/dependency_analyzer/ast_parser.py`:
146+
147+
```python
148+
LANGUAGE_ANALYZERS = {
149+
# ... existing languages ...
150+
"new_language": NewLanguageAnalyzer,
151+
}
152+
```
153+
154+
3. **Add file extensions** in configuration
155+
156+
4. **Add tests** for the new language
157+
158+
## Testing
159+
160+
```bash
161+
# Run all tests
162+
pytest
163+
164+
# Run specific test file
165+
pytest tests/test_dependency_analyzer.py
166+
167+
# Run with coverage
168+
pytest --cov=codewiki tests/
169+
```
170+
171+
## Code Style
172+
173+
- Follow PEP 8 for Python code
174+
- Use type hints where applicable
175+
- Write docstrings for public functions and classes
176+
- Keep functions focused and modular
177+
178+
## Contributing
179+
180+
1. **Fork the repository**
181+
2. **Create a feature branch**: `git checkout -b feature/your-feature`
182+
3. **Make your changes**
183+
4. **Write/update tests**
184+
5. **Ensure tests pass**: `pytest`
185+
6. **Commit your changes**: `git commit -am 'Add new feature'`
186+
7. **Push to the branch**: `git push origin feature/your-feature`
187+
8. **Submit a pull request**
188+
189+
## Documentation Generation Workflow
190+
191+
```mermaid
192+
graph TB
193+
A[Repository Input] --> B[Dependency Graph Construction]
194+
B --> C[Hierarchical Decomposition]
195+
C --> D[Module Tree]
196+
D --> E[Recursive Agent Processing]
197+
E --> F{Complexity Check}
198+
F -->|Complex| G[Dynamic Delegation]
199+
F -->|Simple| H[Generate Documentation]
200+
G --> E
201+
H --> I[Cross-Module References]
202+
I --> J[Hierarchical Assembly]
203+
J --> K[Comprehensive Documentation]
204+
```
205+
206+
## Debugging
207+
208+
### Enable Verbose Logging
209+
210+
```bash
211+
# CLI
212+
codewiki generate --verbose
213+
214+
# Environment variable
215+
export CODEWIKI_LOG_LEVEL=DEBUG
216+
```
217+
218+
### Common Issues
219+
220+
**Tree-sitter parser errors**:
221+
- Ensure language parsers are properly installed
222+
- Check file encoding (UTF-8 expected)
223+
224+
**LLM API errors**:
225+
- Verify API keys and endpoints
226+
- Check rate limits
227+
- Enable retry logic
228+
229+
**Memory issues with large repositories**:
230+
- Adjust module decomposition threshold
231+
- Increase delegation depth limit
232+
233+
## Performance Optimization
234+
235+
- **Caching**: Results are cached to avoid redundant processing
236+
- **Parallel Processing**: Multiple modules can be processed concurrently
237+
- **Incremental Updates**: Only process changed modules (future work)
238+
239+
## Support
240+
241+
For development questions:
242+
- GitHub Issues: https://github.com/FSoft-AI4Code/CodeWiki/issues
243+
- Main Documentation: [README.md](README.md)
244+

0 commit comments

Comments
 (0)