Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms

Modern vision models are trained on very large noisy datasets. While thesemodels acquire strong capabilities, they may not follow the user's intent tooutput the desired results in certain aspects, e.g., visual aesthetic,preferred style, and responsibility. In this paper, we target the realm ofvisual aesthetics and aim to align vision models with human aesthetic standardsin a retrieval system. Advanced retrieval systems usually adopt a cascade ofaesthetic models as re-rankers or filters, which are limited to low-levelfeatures like saturation and perform poorly when stylistic, cultural orknowledge contexts are involved. We find that utilizing the reasoning abilityof large language models (LLMs) to rephrase the search query and extend theaesthetic expectations can make up for this shortcoming. Based on the abovefindings, we propose a preference-based reinforcement learning method thatfine-tunes the vision models to distill the knowledge from both LLMs reasoningand the aesthetic models to better align the vision models with humanaesthetics. Meanwhile, with rare benchmarks designed for evaluating retrievalsystems, we leverage large multi-modality model (LMM) to evaluate the aestheticperformance with their strong abilities. As aesthetic assessment is one of themost subjective tasks, to validate the robustness of LMM, we further propose anovel dataset named HPIR to benchmark the alignment with human aesthetics.Experiments demonstrate that our method significantly enhances the aestheticbehaviors of the vision models, under several metrics. We believe the proposedalgorithm can be a general practice for aligning vision models with humanvalues.