TECH Signal 305
Marlin open-source search engine indexes 560,000 homepages for $10 using a small language model
↗
Developer Alex Morley-Finch built Marlin, an open-source personal search engine that crawls homepages and uses a small OpenAI-compatible language model to summarize and tag them, indexing roughly 560,000 pages for about $10 in cloud GPU time and under a gigabyte of storage.
Marlin shows a viable, cheap path to a personalized search index by weighting crawls toward specific content types and offloading summarization to a small LLM. Its main failure mode is taxonomy: letting the model freely invent categories and tags produced 671 categories and over 121,000 tags, more than half used only once, requiring a manual merge tool that the author says won't scale beyond hobbyist use.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Marlin runs four processes: a fetcher, a worker calling an OpenAI-compatible LLM, a steward that filters bad pages, and a search API with a web UI.
A weighting system pushes prioritized pages to the top of the crawl queue after the first attempt indexed mostly corporate sites and documentation.
The code is open source with instructions for building custom weighting lists and optionally renting a cloud GPU.
THE CLUSTER