Hello Urx 👋🏼
Hi everyone! I’m excited to share my latest project, Urx, which I recently introduced on X (check out the announcement here)! 🎉🚀 Urx is a command-line tool designed to collect URLs from OSINT1 archives like the Wayback Machine and Common Crawl. Built in Rust for speed and efficiency, it uses asynchronous processing to query multiple sources quickly, making it a handy tool for gathering URL datasets for security testing, analysis, and more.
Why Rust?
I’ve been intrigued by Rust for a while. Years ago, when I started learning Go, Rust was the other language I nearly chose to explore. Its performance, safety features, and growing ecosystem always stood out to me. Recently, I finally dove in, and to my surprise, it clicked faster than I expected—despite its reputation for a steep learning curve. Urx is my first official Rust project, and I’m thrilled with how it turned out. I suspect Rust might become my go-to language, though I’ll still use Go, Ruby, and Crystal where they shine best.
Installation
Here’s how you can get started with Urx:
From Cargo (Recommended)
cargo install urx
From Homebrew (Tap)
brew tap hahwul/urx
brew install urx
Note: This is currently a custom tap, but I plan to transition it to an official Homebrew formula when conditions allow.
From Source
git clone https://github.com/hahwul/urx.git
cd urx
cargo build --release
The compiled binary will be available at target/release/urx.
From Docker
Key Features
Urx comes packed with useful features:
- Fetch URLs from multiple sources (Wayback Machine, Common Crawl, OTX)
- Process multiple domains concurrently
- Filter results by file extensions or patterns
- Use presets (predefined filter sets) for convenience (like “no-image” to exclude all image-related extensions)
- Multiple output formats (plain, JSON, CSV)
- Output to console or file
- Support for reading domains from stdin (pipeline integration)
- URL testing capabilities (status checking, link extraction)
Basic Usage
Urx is easy to use for basic tasks, yet powerful for advanced needs. Here are the essentials:
# Scan a single domain
urx example.com
# Scan multiple domains
urx example.com example.org
# Scan domains from a file
cat domains.txt | urx
Advanced Options
Urx offers extensive configuration options:
Output Options
# Save results to a file
urx example.com -o results.txt
# Output in JSON format
urx example.com -f json -o results.json
# Merge endpoints with the same path and URL parameters
urx example.com --merge-endpoint
Filtering Options
# Filter for JavaScript files only
urx example.com -e js
# Exclude HTML and text files
urx example.com --exclude-extensions html,txt
# Filter for API endpoints
urx example.com --patterns api,v1,graphql
# Exclude specific patterns
urx example.com --exclude-patterns static,images
# Use Filter Preset (similar to --exclude-extensions=png,jpg,.....)
urx example.com -p no-images
Provider and Network Options
# Use specific providers
urx example.com --providers wayback,otx
# Include subdomains
urx example.com --subs
# Network configuration
urx example.com --proxy http://localhost:8080 --timeout 60 --parallel 10
Testing Options
# Check status of collected URLs
urx example.com --check-status
# Extract additional links from collected URLs
urx example.com --extract-links
Integration with Other Tools
Urx works well in pipelines with other security and reconnaissance tools:
# Find domains, then discover URLs
echo "example.com" | urx | grep "login" > potential_targets.txt
# Combine with other tools
cat domains.txt | urx --patterns api | other-tool
Inspiration
Urx was inspired by gau (GetAllUrls), a tool that fetches known URLs from AlienVault’s Open Threat Exchange, the Wayback Machine, and Common Crawl. While sharing similar core functionality, Urx was built from the ground up in Rust with a focus on performance, concurrency, and expanded filtering capabilities.
Final Remarks
Urx has been a blast to build, and it’s deepened my love for Rust. I’m excited to keep improving it and exploring more Rust projects. If you find Urx helpful or have ideas to make it better, feel free to drop by the GitHub page to open an issue or submit a pull request. Happy URL hunting, and thanks for checking it out!
-
OSINT(Open Source Intelligence) ↩