Creative Generation of 3D Objects with Deep Learning and Innovation Engines

Lehman J
Risi S
Clune J

Advances in supervised learning with deep neural net- works have enabled robust classification in many real world domains. An interesting question is if such advances can also be leveraged effectively for computational creativity. One insight is that because evolutionary algorithms are free from strict requirements of mathematical smoothness, they can exploit powerful deep learning representations through arbitrary computational pipelines. In this way, deep networks trained on typical supervised tasks can be used as an ingredient in an evolutionary algorithm driven towards creativity. To highlight such potential, this paper creates novel 3D objects by leveraging feedback from a deep network trained only to recognize 2D images. This idea is tested by extending previous work with Innovation Engines, i.e. a principled combination of deep learning and evolutionary algorithms for computational creativity. The results of this automated process are interesting and recognizable 3D-printable objects, demonstrating the creative potential for combining evolutionary computation and deep learning in this way.

Figure 1: Gallery of automatically generated high-confidence objects. A curated selection of high-confidence champions from many experiments. Representing multiple copies of the same object (e.g. Banana, Ice Lolly, and Matchstick) helps maximize DNN confidence. The system often evolves roughly rotationally-symmetric objects (e.g. Goblet, Joystick, Bubble), both because many classes of real-world objects are symmetric in such a way and because it is the easiest way to maximize DNN confidence from all rendered perspectives. However, objects such as Conch, Mask, and Dalmatian show that asymmetric and more complex geometries can also evolve when necessary to maximize DNN confidence. Overall, the results show the promise of Innovation Engines for cross-modal creativity. Best viewed in color.

Figure 2: Gallery of 3D printed objects. Shown are photographs of 3D-printed objects oriented similarly to one of their simulated renders (which is inlaid). Note that the evolved Hammerhead consisted of two similar objects; one was chosen arbitrarily for 3D printing. In all cases, a clear resemblance is seen between each 3D-printed object and its render, demonstrating the feasibility of automatically generating real-world objects using the approach.

Pub. Info: 
Proceedings of the International Conference on Computational Creativity