AI Signal 468
These startups are chasing the next big thing in LLMs
↗
Startups are developing new attention mechanisms to replace the dense attention in transformers that power today’s LLMs.
Dense attention makes transformer inference and training extremely compute-intensive, driving billions in annual AI spend and rising data-center electricity use. New sparse-attention and power-retention approaches promise to cut those costs and expand context windows, which directly affects the scalability and operating expense of LLM services. Engineers will need to evaluate whether the newer mechanisms deliver comparable accuracy before they can replace existing models in production.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Dense attention scales quadratically with token count, causing huge compute and power demands for long documents.
Subquadratic claims a sparse-attention model that matches mainstream LLM performance on tasks like search and coding, while Manifest AI proposes a "power retention" method to keep only task-relevant information.
Both approaches aim to reduce computation and enable larger context windows, but their real-world effectiveness and integration complexity remain unproven.
THE READ
What the cluster adds up to.
The transformer architecture introduced in 2017 relies on dense attention, which compares every token with every other token. This design yields high accuracy but its computational cost grows rapidly with text length, leading to multi-billion-dollar annual compute budgets. Engineers building LLM pipelines now face a trade-off between model capability and resource consumption, prompting interest in alternative attention schemes.
Subquadratic, a Miami-based startup, says it has created a sparse-attention mechanism that dynamically selects which token pairs to process, reducing the number of multiplications required. If the claim holds, developers could run comparable models with lower GPU usage and electricity draw, potentially shrinking inference latency and cloud spend. Adoption would involve swapping the existing transformer backbone for SubQ, which may require retraining or fine-tuning on proprietary data.
Manifest AI takes a different route by replacing attention entirely with a "power retention" system that stores only the most relevant information for a given task. This could further shrink the data footprint that an LLM must manage, especially for applications needing very large context windows such as code bases or document libraries. However, the description stops short of detailing how the mechanism preserves semantic fidelity, leaving open the risk of degraded performance on nuanced language tasks.
Both startups are still early in their rollout; Subquadratic reports a waitlist and plans to make its model widely available soon, while Manifest AI's approach is described only in conceptual terms. Engineers must therefore treat these solutions as experimental and allocate resources for testing, integration, and fallback to existing dense-attention models. The potential cost savings are attractive, but the lack of extensive benchmarks means the new methods may not yet be suitable for production-critical workloads.
If sparse attention or power retention prove viable, the broader AI ecosystem could see a shift away from the current transformer-centric stack toward more compute-efficient architectures. This would impact everything from model training pipelines to inference serving infrastructure, possibly lowering barriers for smaller firms to deploy large-scale LLMs. Until the claims are independently validated, the practical impact remains limited to pilot projects and research prototypes.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER