Weibo Crawler is a web scraper that extracts data from Sina Weibo user profiles and posts.
The tool addresses the need to systematically collect Weibo user and post data at scale. It works by crawling one or multiple Weibo user accounts and extracting comprehensive information including user metadata such as follower counts and post counts, along with detailed post data like text content, timestamps, and engagement metrics. The scraper can optionally download images and videos from both original posts and retweets, including Live Photo videos, and can retrieve first-level comments and retweets associated with posts. Results can be written to CSV files by default, or alternatively to JSON, MySQL, MongoDB, or SQLite databases. The tool supports incremental updates, allowing users to periodically re-run the scraper to capture newly published posts from target accounts without re-downloading existing data.
Adoption makes sense for researchers, data analysts, or developers building applications that require historical or ongoing Weibo data collection. The tool suits projects needing comprehensive post metadata and associated media files from specific user accounts. Users can optionally configure database storage instead of CSV output, add authentication cookies to access additional data or bypass rate limiting, and set up automated periodic crawling to keep datasets current. The README provides guidance on obtaining user IDs, managing cookies, and validating cookie validity.
Development activity shows consistent maintenance with regular updates addressing functionality and reliability. The project includes Docker support for simplified deployment across environments. Documentation is thorough, covering setup procedures, configuration options, and optional features like database integration and scheduled execution. The codebase remains actively refined to handle Weibo's platform changes and user requirements.
