HeadlessX
Self-hosted scraping and agent platform
- Status
- Live
- Role
- Creator and maintainer
- Source
- Open source
Overview
HeadlessX is a scraping and browser-automation platform you run on your own box. It ships a web dashboard, a protected HTTP API, queue-backed workflows, and a remote MCP endpoint, so the same infrastructure serves a human clicking through the UI and an agent calling tools.
Operators are the unit of work. Website (scrape, crawl, map, screenshots), Google AI Search, Tavily, Exa, and YouTube each expose the same job contract, which means a workflow can mix sources without special-casing any of them.
The problem
Hosted scraping APIs price per request, rate-limit you at the worst possible moment, and give you no say in how the browser presents itself. The moment a target site tightens its detection, your pipeline stops and there is nothing you can do from the outside.
Self-hosting normally trades that for a different problem: you inherit browser fleet management, queueing, retries, proxy rotation, and the fingerprinting arms race. Most teams give up somewhere around the third headless-Chrome detection bug.
What was built
- A custom-patched Firefox engine (Camoufox) carrying 40+ anti-detection mechanisms, so pages render as an ordinary browser session rather than an obvious automation client.
- Queue-backed job execution on Redis with retries and backpressure, so a burst of crawl requests degrades in throughput instead of failing.
- A remote MCP endpoint that exposes every operator as an agent tool, plus agent skill support for coding assistants.
- A web dashboard for run history, job inspection, and credential management, over a token-protected API for programmatic access.
- CLI bootstrap that provisions a working instance, and built-in proxy pool management with per-job assignment.
Tech stack
Node.js
Next.js
TypeScript
Playwright
PostgreSQL
Redis
MCPOutcomes
- 1.5k+ GitHub stars and an active issue tracker from self-hosting users.
- 0% bot-detection rate against the detection suites tracked in the repository.
- Runs as the scraping layer behind several of the automations listed on this site.