Most of my academic friends know me for my work in generative and multimodal AI and, more recently, the behavior of models in complex settings — from compositional instruction following and generalization to theory of mind, multi-agent cooperation, and safety. But I don’t often talk about my passion for AI applications in fashion — which shaped my PhD thesis, and a good portion of my industry career.

In the past decade, I’ve applied core concepts of AI and computer vision to practical problems in the fashion and retail domain in both big tech and startups — like the computational analysis of style, outfit creation, clothing compatibility, personalized styling, trend analysis, capsule wardrobe generation, 3D body shape estimation from a 2D image, body-type-aware clothing recommendation, fashion image and video generation, or modelling the physics of clothing and its movement in 3D.

I’ve been fascinated by fashion and styling since I was a little girl cutting out excerpts of editorial photoshoots from my mom’s magazines. When I entered my PhD program and decided to focus on AI, I knew I had to pick a problem that would keep me interested for several years.

At the time, I didn’t think anyone would take me seriously if I said, “I want to study fashion and the way people shop for, select and put together clothes computationally.” My PhD advisor suggested I work on developing algorithms for autonomous taxiing of airplanes. I was bored.

I used to take breaks from research by going online and browsing fashion, endlessly searching up emerging designers, concept stores, vintage retailers. On one of those rabbit hole breaks, I paused and realized … hold on a minute. Why do I always need to type in a search query?

Sometimes, I don’t have the right words to describe what I’m looking for. Today, we have aesthetics like “cottagecore,” “darkwave,” “coquette,” and many more that couldn’t possibly be known to everyone. So I walked into my PhD advisor’s office and presented to him this problem.

“We need to have better visual search tools for fashion. We need to be able to provide an inspiration photo to a search engine, and get back results to match it. And just because the photo may be of green pants, that doesn’t mean the search engine should return more green pants — maybe we’re more interested in how the pants go with the sweater and the jacket, rather than the color.”

If you’ve ever worked on visual search problems, you’ve probably immediately recognized that this was a hard demand to meet! From those questions, my first project was born.

Can we tell what makes clothing items compatible, and what visual relationships dictate how we can organize them together in outfits? Are there rules for creating outfits that we can learn? What are the dominant axes of clothing compatibility — color, silhouette, something else? And then, when we take into account the end customer, can we predict what would look good on them based on their body type, current wardrobe, physical features of themselves they want to hide or highlight, aspirational photos of styles and aesthetics they wish to emulate?

Taking it a step further, can we add a time component to these questions, and consider the lifecycles of trends, the recurrence of certain aesthetics, the large-scale findings of what people around the world wear, the subtle evolution of personal style and shifting preferences? Can we use those insights to drive fashion marketing? And, to make things even more appealing to a potential customer, can we generate “the unseen”? Can we create new aesthetics in multiple modalities — images, video, 3D fashion avatars of an “aspirational” version of ourselves?

These questions ended up taking me through a good portion of my PhD and, later, my industry career. But one of the earliest artifacts that came out of asking them was the Polyvore Outfits Dataset.

We originally built it to study a deceptively simple question: what makes two clothing items go together?

The dataset contains 68,306 outfits and 261,058 fashion items, with outfits assembled by real Polyvore users rather than generated algorithmically. We used those combinations as a source of implicit human knowledge about clothing compatibility — not merely whether two items look similar, but whether they complement one another as parts of an outfit.

Years later, I still occasionally come across papers using some version of this dataset. So I wanted to surface the original again — both because I think the questions that motivated it remain interesting, and because, if you use it, I’d love for you to cite the original work. :)

Dataset page: https://mariya.fyi/polyvore

Hugging Face: https://huggingface.co/datasets/mvasil/polyvore-outfits

Paper: https://arxiv.org/abs/1803.09196

Thesis, for what became of this line of research: http://hdl.handle.net/2142/108720

@inproceedings{vasileva2018learning,
                          title     = {Learning Type-Aware Embeddings for Fashion Compatibility},
                          author    = {Vasileva, Mariya I. and Plummer, Bryan A. and Dusad, Krishna and
                                      Rajpal, Shreya and Kumar, Ranjitha and Forsyth, David A.},
                          booktitle = {European Conference on Computer Vision (ECCV)},
                          year      = {2018}
                        }

                        @phdthesis{vasileva2020understanding,
                          title     = {Understanding the rich world of outfits: a study of fashion
                                      compatibility, latent style, and outfit behavior},
                          author    = {Vasileva, Mariya Ivanova},
                          school    = {University of Illinois at Urbana-Champaign},
                          type      = {Ph.D. dissertation},
                          year      = {2020},
                          url       = {http://hdl.handle.net/2142/108720}
                        }