Contributors
Meet the team behind Site Crawler MCP
A powerful Model Context Protocol server for comprehensive website analysis with 12 different extraction modes. Multi-mode crawling in a single pass with advanced pattern matching and rate limiting.
Choose from comprehensive extraction modes tailored for different use cases
Extract all images with metadata, file sizes, formats, and alt-text analysis
Complete SEO analysis including title tags, meta descriptions, and structured data
Extract brand assets, logos, company information, and brand guidelines
Analyze loading times, resource sizes, and performance optimization opportunities
Comprehensive security analysis including SSL, HTTPS, and security headers
Detect privacy policies, GDPR compliance, terms of service, and legal pages
Technical infrastructure details, hosting information, and technology stack
Extract job postings, career pages, and hiring information
Find contact information, addresses, phone numbers, and social media links
Extract client testimonials, case studies, and business references
Product images, pricing information, and e-commerce platform analysis
Text analysis, keyword density, content structure, and readability metrics
See how to use Site Crawler MCP in different scenarios
{
"modes": ["images"],
"depth": 2
}Extract all images from a website up to 2 levels deep with metadata
{
"modes": ["images", "meta"],
"depth": 1
}Get both image assets and SEO metadata in a single crawl session
{
"modes": ["images", "meta", "security", "brand"],
"depth": 3
}Comprehensive analysis including security headers, brand info, and compliance
Meet the team behind Site Crawler MCP
Built with modern Python technologies for optimal performance
Modern Python with async/await support for concurrent operations
Built with aiohttp for high-performance asynchronous web crawling
Advanced HTML parsing and extraction with CSS selectors
1-2 second delays with maximum 5 concurrent requests
Automatic retry with exponential backoff for failed requests
Native Model Context Protocol integration for AI assistants
Multiple installation options to suit your development environment
pip install site-crawler-mcpSimple installation from Python Package Index when available
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp
uv pip install -e .Development installation using the modern uv package manager
python -m venv venv
source venv/bin/activate # or venv\Scripts\activate on Windows
pip install -e .Recommended setup with isolated Python environment