Sketchbook → Procreate → Code → Production
I made my Xcode project available on github, go grab it if you are curious to learn more about ARKit!
This experiment with iOS’s ARKit made me realize there’s so much potential for the technology as a medium of artistic expression. The art card idea spawned about four or five additional ideas I’d like to pursue. If only I could find four or five additional days in the week to work on those ideas.
Some of my earliest memories are me drawing on things. My dad was a welder and our yard was full of hulking, rusty pieces of metal and I took chalk rocks from our driveway to doodle murals all over their surfaces. I wanted to be a comic book artist when I was a kid. I’m also the person whose meeting notes are 90% doodle and 10% words. One day a sketched the image below.
Over the days and weeks my sketchbooks and notebooks filled up with other doodles, I worked on other art projects, and I did my product design thing. But my mind kept returning to the sketch. I thought that maybe a refined illustration would help me stop fixating on it, so I recreated the drawing in Procreate.
But, nope. It actually just made matters worse. Once I saw the sketch in color, I wanted to do more with it. But I didn’t know what that “more” was.
Into the Real-ish World
I’ve done augmented reality-based work for previous jobs, but in the capacity of product design. Then one day those experiences collided with the memory of the sketch, and everything clicked into place: I could see all the individual elements of the illustration floating in different layers of physical space. That’s when I knew why my brain was so fixated…my drawing wanted to live in the world of AR.
Off to XCode and iOS’s ARKit!
I quick search on Google yields countless tutorials for ARKit and SceneKit, so if you are familiar with iOS development the barrier to entry is low. Apple does such a great job with their documentation that you can do something pretty close to this even if you are starting with zero experience.
In my vision, all of the individual elements animated independently but it took a lot of trial and error to get the timing and z-depth just right. I also used keyframe animations to give each element a slight bounced as they animated into view. The blinking skulls were a total afterthought, but their charm is the icing on the cake imo.
I spent most of my time producing the art card. ARKit uses machine learning to train for image recognition, and it requires a 1:1 relationship between what is referenced in the code verses what is expected in reality. The physical dimensions must be exactly specified, and the image content must be of sufficient color contrast and pattern variance to work. It took some trial-and-error to design something that was minimal, interesting, and functional.
I put the image on cardboard backing to help with stability…a bend in the surface from holding it the wrong way will cause the tracking to fail. Pro-tip: don’t laminate the paper (the glossy art card in the top left of the image) as any glare will render the image unrecognizable to the ML algorithm.
I think the results are pretty dope!