Echoes - Audio Game with A Purpose
Share your sounds in Twitch streaming!
Project Duration: 16 weeks
Clients: Bosch, Philips
Role: Gameplay Programmer, Game Designer, Co-Producer
Teammates: Jingya Chen, Yikai Han, Ann Maria Jose, Ruochen Li
Platform: Windows (Rolling Rhapsody Game), Android, iOS (Companion App)
Team Echoes is doing an audio game with a purpose to collect home audio data for training the AI for the home health care system. This project is a series of Twitch experience, that enables extra interactions between streamers and viewers.
Streamers can set up events for collecting specific categories of sounds, like bathroom, kitchen, etc. Viewers can join the events from the companion app, record, tag, and submit the sounds they collected. Then streamers can view the statistics from the website and congratulate the viewers with most sounds collected. Finally, the streamer will play a game that all sounds submitted for this event will be randomly spawned into the game map, the streamer can go collect them as many as possible, give real-time recognition to viewers and listen to the rhapsody these sounds composed.
Project Website: https://www.etc.cmu.edu/projects/echoes/
Original GitHub repo is private.
The new repo doesn't include any earlier commit history, but you can find my contribution mainly in this folder:
I coded the entire Rolling Rhapsody game, including sound object random generation, dynamic task assignment, real-time recognition, and physics-based movement and interactions for the player character.
I also implemented a room-decoration game that players can decorate the room using stickers with sounds, and play the sounds in the room in certain patterns. However, this game is not used for the final product.
For the rolling rhapsody, I designed the dynamic task because when we are doing internal playtesting, we found the streamer doesn't have a clear goal - so I thought about some unique tasks that streamers can accomplish during the game. I also designed the real-time recognition - this can print the name of the uploader (viewer) on the screen when their sounds are collected by the streamer. I watched some Twitch recognition to get inspired, and it worked pretty well.
The room-decoration game is almost entirely designed by me, but sadly it was not used later.
I wrote some dev blog posts, edited the playtest video and project trailer. I managed some meetings with real streamers with the team, and during the playtests with real streamers, I watched the entire process and was doing Q&A along the time.