NanmiCoder/MediaCrawler

MediaCrawler is a Python-based web scraping tool designed for extracting public data from major Chinese social media and content platforms. The repository supports crawling from Xiaohongshu (Little Red Book), Douyin (TikTok), Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu. According to the repository's feature matrix, all seven platforms support keyword search, specified post ID extraction, secondary comments, creator homepage access, login state caching, IP proxy pools, and comment word cloud generation.

The technical foundation relies on Playwright browser automation framework combined with login state preservation, eliminating the need for JavaScript reverse engineering. The approach leverages saved browser contexts to obtain signature parameters through JavaScript expressions, significantly reducing technical complexity compared to traditional methods that require decrypting complex algorithms. The project offers both a command-line interface and a web-based UI for visual operation, with the WebUI providing real-time status monitoring, log viewing, and data export capabilities.

Data persistence is handled through multiple storage formats including CSV, JSON, JSONL, Excel, SQLite, and MySQL databases. The project includes a WebUI built with Vite that can run in development mode or be compiled for production deployment. The default configuration uses Chrome DevTools Protocol (CDP) mode to connect to existing Chrome browser instances, preserving login states, cookies, and extensions to reduce platform detection risks. Users can optionally switch to standard Playwright mode through configuration settings.

The most active labels are question (136 items), bug (109 items), and enhancement (14 items).

The project emphasizes educational use with explicit disclaimers against commercial application or illegal activity. A premium version called MediaCrawlerPro is available separately, offering features like breakpoint resume functionality, multi-account support, improved architecture design, desktop video downloader, homepage feed recommendations, and AI Agent skill integration for platforms like OpenClaw and Claude Code. The repository includes comprehensive setup documentation covering uv package manager installation, Node.js requirements (version 16.0.0 or higher), Chrome browser configuration with remote debugging enabled, and environment variable setup. Community engagement is facilitated through WeChat groups and Bilibili channel presence.