Cuby

Cuby is an augmented reality application for mobile devices, designed for offering users a visual snapshot of their environment. The project combines augmented reality, generative arts and device input processing. The application generates a structure from parameters specific to the user’s environment. The structure’s base is a cube, textured with a height and satellite map according to the users location. Surrounding particles change their shape according to the ambient noise, and their color based on the colors captured by the camera. Using and adjusting these individual parameters, each user is able to generate a unique and personal Cuby. The project has been realized with Unity and Unity’s AR Foundation package to cover both iOS and Android based devices.

  • Collaborators: Marlene Mayr, Anna Moser
  • Software: Unity, AR Foundation, Statecharts
  • Hardware: AR Core and AR Kit compatible device (Xiaomi Mi Mix 2s)
  • My Task Areas: Concept, Feature Extraction, State Charts, Serialization, UI Integration

Concept

My main focus in this project was to realize an AR application for mobile devices, working with the device’s data, such as time and location. Additionally, camera and microphone input was analyzed, processed into usable data and then applied to the generated structure. Thereby I gained experience in accessing and processing various data sources in mobile applications.

Here you can see a demo .gif and some screenshots of the application.

demo gif screenshot colors screenshot sound screenshot save

Implementation

I want to start of with an overview of what has been implemented. I developed the application flow and integrated the UI design from my team mate as menus. Most of my time, however, has gone into processing the device data (see more in section Features).

  • Serialization and sharing are based on the awesome file browser and native sharing implementations by yasirkula.

  • The multitude of menus and the user’s navigation through the application have been realized with clean OOP and Statecharts. The .NET compatible Statecharts have been used to easily manage the application flow as reliably as possible.

  • An info card references all the information currently applied to the cuby. It’s design resembles a museum’s art description.

For detailed information, take a look at the project report: show .pdf

Features

The following paragraphs briefly describe some of the lessons learned. This demo .gif images show the main features: location, sound, colors, timestamp, age.

location gif sound gif colors gif sun gif age gif

Location

The Mapbox API delivers a heightmap and satellite image from the user’s current location. Unfortunately, Unity’s URP does not provide a height map functionality that distorts geometry at the time of working on the project. This is the reason why the current Cuby is limited to using the height map as a basic normal map.

satellite map terrain map

Sound

For the sound recording I considered various factors: maximum amplitude, absolute vs. relative loudness, overall vs. temporal changes, sampling and smoothing rates. We decided to save a recorded clip in our data that is then replayed in a loop. The recorded audio clip and the smoothed loudness data are played simultaneously, however, the actual recording can be muted. The loudness continuously influences the spreading of the particles around the Cuby.

Colors

AR Foundation offers a way to get the latest camera image on the CPU. I then save this raw data to a temporary texture and process it with a custom computer vision algorithm. There are numerous of extracting the primary colors of an image. However, the “right” algorithm depends on the source image and the goal colors. By combining various algorithms I came up with an implementation that creates a histogram from HSV colors. This means that the algorithm fills bins in the cylindrical HSV color space representation with the pixels’ values and counts the most populated bins. This way the hues are retained. The 7 most dominant colors of the resulting histogram can selectively be applied to Cuby’s particles.

hsv cylinder histogram value

Timestamp

The timestamp feature adds a sun or a moon. It is placed in the Cuby’s orbit depending on the current time when adding the feature. The sun and moon optionally leave a golden glow or blue shine on the cube’s surface. Deciding whether a time of day is during daylight or in the night time was simply done by a threshold for morning and evening. Twelve hours of sunlight map to 180 degrees in a hemisphere around Cuby.

Age

Cuby ages. By opening the application for the first time, Cuby’s life starts and its birthday is saved. This value, however, is fixed and cannot be changed by the user. With the days passing by, the spawn rate of the particles increases. The maximum age is set to 100 days.

Search