Jeeyoon Hyun's blog

December 08, 2020

Doodlering

fall-2020itpmachine-learning-for-the-webmididiceml5jsweb-experiments
Images for Doodlering, an incomplete storybook where you can fill in the blanks with your own sketches using machine learning. Image credit: Juyeon Park

Demo

doodlering.mididice.works

Exhibition website (Korean)

The website is currently still in progress and works best at 360 * 760.

Doodlering is a mobile web storybook generator that I made in mididice, which is a media art team that I made and is based in Seoul. I used the resources I learned from Yining Shi’s Machine Learning for the Web class.

Ideation

2020 has been a year of online communication. I focused on the way how people try to interpret others’ intentions using technology.

While my intentions seem obvious to me, will it be still obvious to others when we are communicating online? If a machine learning model tries to interpret my intentions, how it will be different from the interpretation made by other humans?

Diagram of how Doodlering works(1): Our intentions are different from how other people or how technology perceives them.
Diagram of how Doodlering works(1): Our intentions are different from how other people or how technology perceives them.
Diagram of how the Doodleling website works(2) a SktechRNN model recognizes your sketch and lets it compare to other people's interpretations about your sketch.
Diagram of how the Doodleling website works(2): a SktechRNN model recognizes your sketch and lets it compare to other people’s interpretations about your sketch.

Doodlering is a storybook generator made of incomplete sentences. It lets you complete the stories by drawing your own pictures. I wanted to show that while you might think other people have understood your intentions clearly, your actions interpreted by the ML model and other people in the online world might turn out as a completely different story.

ML Models

Text generation

Generating Korean text via GPT-2 in Runway, based on a dataset of Korean fables and fairytales
Generating Korean text via GPT-2 in Runway, based on a dataset of Korean fables and fairytales
Editing the sentences manually to make it sound plausible
Editing the sentences manually to make it sound plausible

I generated the storybook sentences in GPT-2 via Runway, based on a dataset of Korean fables and fairytales where I could find from Korean Learner’s Corpus Search Engine and the National Institute of Korean Language‘s open database.

I deliberately chose fairytales as dataset to set the story’s background as recalling a dream. The most primary reason is also because I knew combining different sentences won’t produce a solid storyline, and I needed a background that would still be plausible even if the sentences looked strange.

Sketch recognition

Doodlering uses DoodleNet’s CNN image classifier and ml5.js to recognize the user’s sketches.

Credits

mididice

  • Interaction design: Jeeyoon Hyun, Juyeon Park
  • Software engineering: Hyunki Min, Changmin Kim