Open-source pipeline trains coding model to generate watercolour paintings via reinforcement learning

The project clones and open-sources a previously closed workflow that maps reinforcement learning to subjective artistic taste. Instead of training on math problems or unit tests, the model learns from pairwise human ratings of watercolour paintings. This shift requires a scorer model that approximates human judgement, an RL environment that renders model-generated JavaScript into images, and a reference pool of hand-curated examples. The entire stack is now available as a single command that spins up a 48-hour training job on Hugging Face Jobs, making the approach reproducible without custom infrastructure.

Costs are concentrated in compute and dataset curation. The reference article specifies a 35B-parameter base model, LoRA fine-tuning, and a 48-hour run on an H200 instance. While the base model and LoRA adapter are open, the compute budget is non-trivial for individual engineers. Dataset curation is equally expensive: the reference pool consists of hand-rated images, and the reward model itself is trained on pairwise human preferences. These steps cannot be automated without degrading the subjective quality the pipeline aims to capture.

The pipeline stops working when the artistic criteria become too abstract or the reference pool too small. The current implementation restricts the model to ten methods from the p5.brush library, which keeps the output space tractable but limits stylistic diversity. If the reference pool shrinks below a critical size, the reward model overfits and the generated paintings collapse into repetitive patterns. Similarly, if the artistic criteria shift from watercolours to, say, oil paintings, the entire reference pool and reward model would need to be rebuilt from scratch.

The open implementation reveals trade-offs between reproducibility and artistic control. The original project was driven by an artist who iterated on the reward design and reference pool in private. The open version publishes every piece, including the hand-rated dataset and three reward mixes, but this transparency also exposes the brittleness of the approach. Small changes in the reward mix or reference pool can produce large changes in the output, making the pipeline sensitive to hyperparameters and dataset curation choices.

For engineers, the pipeline is a template for operationalising subjective criteria in reinforcement learning. The key insight is that the RL environment does not need to be a game or a simulator; it can be a rendering engine that turns code into images. This opens the door to training models on other subjective tasks, such as generating music, poetry, or even UI designs, provided a reference pool and a scorer model can be built. The trade-off is that these tasks require human judgement, which is expensive to collect and difficult to scale.