Carnegie Mellon College’s Robotics Institute has a brand new artist-in-residence.
FRIDA, a robotic arm with a paintbrush taped to it, makes use of synthetic intelligence to collaborate with people on artistic endeavors. Ask FRIDA to color an image, and it will get to work placing brush to canvas.
“There’s this one portray of a frog ballerina that I believe turned out actually properly,” stated Peter Schaldenbrand, a Faculty of Laptop Science Ph.D. pupil within the Robotics Institute working with FRIDA and exploring AI and creativity. “It’s actually foolish and enjoyable, and I believe the shock of what FRIDA generated primarily based on my enter was actually enjoyable to see.”
FRIDA, named after Frida Kahlo, stands for Framework and Robotics Initiative for Growing Arts. The challenge is led by Schaldenbrand with RI school members Jean Oh and Jim McCann, and has attracted college students and researchers throughout CMU.
Customers can direct FRIDA by inputting a textual content description, submitting different artistic endeavors to encourage its model, or importing {a photograph} and asking it to color a illustration of it. The staff is experimenting with different inputs as effectively, together with audio. They performed ABBA’s “Dancing Queen” and requested FRIDA to color it.
“FRIDA is a robotic portray system, however FRIDA is just not an artist,” Schaldenbrand stated. “FRIDA is just not producing the concepts to speak. FRIDA is a system that an artist might collaborate with. The artist can specify high-level targets for FRIDA after which FRIDA can execute them.”
The robotic makes use of AI fashions just like these powering instruments like OpenAI’s ChatGPT and DALL-E 2, which generate textual content or a picture, respectively, in response to a immediate. FRIDA simulates how it might paint a picture with brush strokes and makes use of machine studying to guage its progress as it really works.
FRIDA’s closing merchandise are impressionistic and eccentric. The brushstrokes are daring. They lack the precision sought so typically in robotic endeavors. If FRIDA makes a mistake, it riffs on it, incorporating the errant splotch of paint into the tip outcome.
“FRIDA is a challenge exploring the intersection of human and robotic creativity,” McCann stated. “FRIDA is utilizing the type of AI fashions which were developed to do issues like caption photos and perceive scene content material and making use of it to this inventive generative downside.”
FRIDA faucets into AI and machine studying a number of instances throughout its inventive course of. First, it spends an hour or extra studying how one can use its paintbrush. Then, it makes use of giant vision-language fashions skilled on huge datasets that pair textual content and pictures scraped from the web, similar to OpenAI’s Contrastive Language-Picture Pre-Coaching (CLIP), to grasp the enter. AI methods use these fashions to generate new textual content or photos primarily based on a immediate.
Different image-generating instruments similar to OpenAI’s DALL-E 2, use giant vision-language fashions to provide digital photos. FRIDA takes {that a} step additional and makes use of its embodied robotic system to provide bodily work. One of many greatest technical challenges in producing a bodily picture is decreasing the simulation-to-real hole, the distinction between what FRIDA composes in simulation and what it paints on the canvas. FRIDA makes use of an thought referred to as real2sim2real. The robotic’s precise brush strokes are used to coach the simulator to replicate and mimic the bodily capabilities of the robotic and portray supplies.
FRIDA’s staff additionally seeks to handle among the limitations in present giant vision-language fashions by frequently refining those they use. The staff fed the fashions the headlines from information articles to offer it a way of what was occurring on this planet and additional skilled them on photos and textual content extra consultant of various cultures to keep away from an American or Western bias. This multicultural collaboration effort is led by Zhixuan Liu and Beverley-Claire Okogwu, first-year RI grasp’s college students, and Youeun Shin and Youngsik Yun, visiting grasp’s college students from Dongguk College in Korea. Their efforts embrace coaching information contributions from China, Japan, Korea, Mexico, Nigeria, Norway, Vietnam and different nations.
As soon as FRIDA’s human person has specified a high-level idea of the portray they need to create, the robotic makes use of machine studying to create its simulation and develop a plan to make a portray to attain the person’s targets. FRIDA shows a shade pallet on a pc display for a human to combine and supply to the robotic. Automated paint mixing is at present being developed, led by Jiaying Wei, a grasp’s pupil within the Faculty of Structure, with Eunsu Kang, school within the Machine Studying Division.
Armed with a brush and paint, FRIDA will make its first strokes. Every now and then, the robotic makes use of an overhead digicam to seize a picture of the portray. The picture helps FRIDA consider its progress and refine its plan, if wanted. The entire course of takes hours.
“Individuals marvel if FRIDA goes to take artists’ jobs, however the primary aim of the FRIDA challenge is kind of the alternative. We need to actually promote human creativity by way of FRIDA,” Oh stated. “As an illustration, I personally wished to be an artist. Now, I can truly collaborate with FRIDA to precise my concepts in portray.”
Extra details about FRIDA is offered on its web site. The staff will current its newest analysis from the challenge, “FRIDA: A Collaborative Robotic Painter With a Differentiable, Real2Sim2Real Planning Setting” on the 2023 IEEE Worldwide Convention on Robotics and Automation this Could in London. FRIDA resides within the RI’s Bot Intelligence Group (BIG) lab within the Squirrel Hill neighborhood of Pittsburgh.
