Our crawler: cryptovssbot

Who we are

cryptovss.com is an independent comparison and transparency site for crypto services (crypto cards, proxies, phone numbers and more), operated by Connect Privacy LLC (75 E 3rd St, Sheridan, WY 82801, United States). Each category has its own site, for example cryptocardss.com. Every value we show (a fee, a limit, a country, a status) carries a mark with the host and date of the official page it comes from. cryptovssbot is the fetcher that keeps these values correct.

What it fetches

• Public pages of the providers we list or evaluate for listing: product, pricing and fee pages, terms, help-center and status pages, and the PDFs they link (cardholder agreements, fee schedules).
• Public listings that describe a provider: app-store pages (App Store, Google Play, F-Droid), aggregate ratings on public review platforms, domain registration data (RDAP) and archived copies in the Internet Archive.
• Never: checkout pages, forms, search boxes, personal data, or anything a robots.txt disallows for us.

What happens with the content

We extract single facts (for example "card issuance fee: 10 USD") and link them to the page they come from. We do not republish your pages, do not resell their content and do not use it to train AI models. A copy of each fetched page is kept internally as evidence for the value it supports, so we can show when and where a value was published.

How often and how fast

• Each monitored page is checked about once a day or every two days.
• Per host at most one request every 10 seconds, never in parallel. Hosts behind Cloudflare additionally share one common slot, so many monitored sites never add up to a burst on the network.
• A larger Crawl-delay in your robots.txt slows us down further.
• Pages that need JavaScript are opened in a headless Chromium whose user agent carries the same cryptovssbot token. It loads the page like a browser, including its scripts and images.

Rules we follow

• robots.txt (RFC 9309) is read by our fetcher before any page of a host and cached for at most 24 hours. We match the product token cryptovssbot; without a group for it, the * group applies. If robots.txt cannot be read (network error or HTTP 5xx), we fetch nothing from that host.
• Bot protection is never circumvented. A challenge, captcha or HTTP 429 counts as blocked and is recorded as such: we do not solve challenges, do not rotate IP addresses and do not disguise the client.
• One identity. All requests come from our own servers and carry the cryptovssbot token; we use no residential or rotating proxies.
• Agreements. Only where a provider has agreed in writing to other access (for example an allowlisting token for its own bot protection) does that agreement apply to its pages instead.

How to verify it is us

Requests of our HTTP fetcher are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). Each request carries three headers:

Signature-Agent: "https://cryptovss.com/.well-known/http-message-signatures-directory"
Signature-Input: sig1=("@authority" "signature-agent");created=…;expires=…;keyid="…";alg="ed25519";nonce="…";tag="web-bot-auth"
Signature: sig1=:…:

The public key is published in our key directory (JWKS, the response itself is signed). A signature is valid for 60 seconds and covers the target host, so it cannot be replayed against another site. Cloudflare's open-source library web-bot-auth verifies it. Our outgoing IP address can change, so we do not publish an IP list; please verify by signature or by the user agent.

Blocking or slowing it

Block everything:

User-agent: cryptovssbot
Disallow: /

Block a part of your site or slow us down:

User-agent: cryptovssbot
Disallow: /account/
Crawl-delay: 60

Changes to robots.txt take effect within 24 hours. If you block us, the values from your pages are shown as "unknown" or with the last date we could check them.

Contact

Need a different rate, an exception, or found a wrong value? Write to [email protected]. Providers can also give us an allowlisting token so our checks pass their own bot protection on purpose, and every page of our sites has a "Report an error" button.