Painting with Gaussians

The author previously built an edge-aware pixelation system that deformed a grid to follow image edges, preserving detail while simplifying flat regions. Building on that, the new tool treats each brush stroke as a 2-D Gaussian whose mean, covariance, color, and opacity encode the stroke’s location, direction, and softness. By extracting edge maps from the source image, the system can decide where strokes belong and how they should be oriented, eliminating the need for a separate stroke-placement algorithm.

A Gaussian splat naturally mimics a paint mark: its central intensity fades toward the edges, and its covariance matrix captures elongation and rotation, matching the visual characteristics of a brush drag. Rendering many splats with standard over-operator compositing ensures each mark occludes what lies behind it according to its alpha, preventing the runaway brightness seen with additive blending. This compositing choice also cleanly separates sampled color from opacity, simplifying the shader logic.

Earlier open-source attempts used random seeding followed by gradient descent to nudge splat parameters until the rendered field resembled the target image, a process described as slow and opaque. Those methods produced a lossy reconstruction rather than a painterly effect because the optimization focused on pixel-wise error instead of artistic stroke placement. By contrast, the new approach skips the optimization loop entirely, using deterministic edge cues to place strokes, which speeds up rendering and yields a more stylized output.

The implementation includes interactive sliders that let users adjust parameters such as stroke density and size, with the painting updating in real time as the sliders move. Because the pipeline reuses the existing edge extraction code, the main engineering effort lies in integrating the Gaussian rasterizer and the over-operator blending stage. This makes the tool relatively lightweight to add to existing image-processing workflows, though developers must still handle the math for covariance-based orientation.

The prototype operates on low-resolution inputs (e.g., a 64 × 64 image) and demonstrates that additive blending can cause bright white blobs when many splats overlap, a problem solved by the over-operator approach. However, the technique does not replicate the full fidelity of real paint and may not scale cleanly to high-resolution images without further optimization. Engineers should be aware that while the method improves speed and control, it remains a stylized approximation suited for artistic rendering rather than photorealistic reproduction.