VINE Crawler
You're seeing this page because VINE's web crawler visited your site.
You may have seen VineAI/1.0 in your server logs. Here's
what it does, what it doesn't do, and why it works differently from
anything else that crawls the web.
What VINE records from your site
Your web address. That's it. VINE stores your URL and places it in a geometric map of the web. Your site's position in that map tells the AI what your site is about. No text is extracted for training. No content is copied into a dataset. No embeddings are computed from your words.
Your URL joins a topological representation of the internet — a continuous geometric surface where related sites naturally cluster together. A local bakery sits near other food businesses. A physics journal sits near other science resources. The placement emerges from the structure of the web itself, not from scraping your content.
What VINE does NOT do
- Does not scrape your text into a training dataset
- Does not compute embeddings from your content
- Does not store your page text, images, or media
- Does not feed your content to a language model
- Does not sell your data or ranking position
- Does not inject advertising into search results
- Does not bypass robots.txt or crawl-delay directives
How this differs from traditional search engines
Traditional crawlers
- Copy your full page text
- Compute word embeddings
- Store content in massive indexes
- Rank by backlinks and ad spend
- Feed content to LLMs for training
- Your content becomes their product
VINE crawler
- Records your URL only
- No embeddings — geometric position instead
- Your site's position in the topology IS the index
- Rank by content coherence, not popularity
- No LLM training from your content
- The web is VINE's live context, not a database
How the topology works
VINE navigates a geometric surface where every website has a natural position. Sites about similar topics settle near each other — not because VINE read their content, but because the link structure of the web already encodes what sites are about. VINE's crawler follows these connections, and the topology forms through the geometry of how sites reference each other.
When someone searches on vineai.net/search, the query settles to a position on the same surface. The nearest sites to that position are the results. No keyword matching. No ad auction. Just geometric proximity.
About the image observer
If VINE's crawler encounters images on your site, it may briefly perceive them — extracting only visual properties like brightness, warmth, and complexity. This is geometric observation, not image storage. The image bytes are released immediately after perception. No images from your site are stored, cached, or used for any training purpose.
VINE's agents — who knocked on your door
VINE talks to the web through several specialised agents, each with a distinct User-Agent so you can tell them apart in your logs and opt them out individually if you wish.
VINEGeneralForager — broad web index
Looks like: VINEGeneralForager/1.0 (+https://vineai.net/crawler#general; Broad Web Index; NEXICOG Ltd)
What it does: Crawls broadly across the web building VINE's general topic index. Slow, polite, respects robots.txt with crawl-delay. One request per 2 seconds per domain, max.
Opt-out: User-agent: VINEGeneralForager / Disallow: /
VINELocalForager — geographic community index
Looks like: VINELocalForager/1.0 (+https://vineai.net/about/local-crawler; Geometric Community Index; NEXICOG Ltd)
What it does: Indexes pages tied to specific places — community sites, local guides, regional news. Fires when a visitor from a new geographic region arrives, so VINE can talk meaningfully about their part of the world.
Opt-out: User-agent: VINELocalForager / Disallow: /
Full explainer: vineai.net/about/local-crawler
VINEBardTendril — music + audio index
Looks like: VINEBardTendril/1.0 (+https://vineai.net/crawler#bard; Music + Audio Index; NEXICOG Ltd)
What it does: Crawls music-adjacent pages (lyrics sites, music wikis, audio databases) for VINE's Bard subsystem. Same politeness pacing as the general crawler.
Opt-out: User-agent: VINEBardTendril / Disallow: /
VINELiveFetch — on-demand citations
Looks like: VINELiveFetch/1.0 (+https://vineai.net/crawler#live; On-demand Sensorium + Citation Fetcher; NEXICOG Ltd)
What it does: Single, on-demand fetches when a visitor asks VINE something that requires a fresh look. Not crawl-paced — one URL at a time, in response to a real human question. Used by the chat citation pipeline and the sensorium feed.
Opt-out: User-agent: VINELiveFetch / Disallow: /
VINESearch — search-engine query client
Looks like: VINESearch/1.0 (+https://vineai.net/crawler#search; Search-Engine Query Client; NEXICOG Ltd)
What it does: Talks to search-engine APIs (DuckDuckGo, SearXNG, Wikipedia, etc.) to find URLs to investigate. Queries only — never fetches your site directly.
(Search engines are the ones VINESearch contacts, not your site.)
Technical details (all agents)
- Crawl rate: Maximum 1 request per 2 seconds per domain (per agent)
- Respects: robots.txt, crawl-delay, noindex, nofollow
- Data retained: URL + geometric position only
- Built by: NEXICOG Ltd, Hampshire, England
- Research: vineai.net/research
How to opt out — all agents at once
To block every VINE agent from your site, add this to your robots.txt:
User-agent: VINEGeneralForager User-agent: VINELocalForager User-agent: VINEBardTendril User-agent: VINELiveFetch User-agent: VineAI Disallow: /
Each agent's specific opt-out lives next to its description above.
Or email [email protected] and we'll remove your site immediately across every agent.
About VINE
VINE is a geometric AI built by NEXICOG Ltd in Hampshire, England. It replaces the entire traditional AI stack — no neural networks, no training loops, no loss functions, no embeddings. Every decision is made by geometric settling on a continuous surface. The web is not a training set. It's a live context window that VINE navigates the same way it navigates every other part of its world.