|
New ways to move in Virtual and Extended Reality. Our CHI 2024 paper and video proposed a novel way to visualize and guide dancing in VR. In our CHI PLAY 2023 paper (honorable mention) and video, we explored new types of VR locomotion where the user's feet are not used for balancing, which enables dynamic and expressive movement in a large virtual space, without moving in the real (and typically very limited) space. At CHI PLAY 2020, we proposed 3PP-R, a VR locomotion and visualization technique that allows natural movement in a 3rd person view, which enables large-scale dynamic locomotion without motion sickness (video).
|
|
AI-assisted user research. Our CHI 2023 best paper explored and evaluated LLMs in generating synthetic open-ended question answers about game experiences.
As part of the paper, we also developed a LLM-based qualitative data coding approach that has evolved into LLMCode, a toolkit for AI-assisted qualitative research (e.g., thematic analysis & qualitative content analysis).
We've also explored how LLMs can generate human-like self-reports of emotion and work as counterfactual "What if?" engines for HCI research.
|
|
Player modeling with AI agents. We collaborated with Rovio for our CHI PLAY 2020 and CHI PLAY 2021 papers on predicting game level difficulty and churn based on RL game agent gameplay. At CHI 2020, we provided a first demonstration of combining RL, physics simulation, and a biomechanical fatigue model to simulation-based user testing of gestural interaction. At SIGGRAPH ASIA 2023, we also demonstrated how such fatigue model can allow a full-body simulated humanoid to discover realistic fatigued movements. At UIST 2022, we published User-in-the-Box, a toolkit for such biomechanical intelligent user simulation. We also showed how Factorization Machines can be used for personalized game level difficulty prediction. At UIST 2024, we presented Sim2VR, extending User-in-the-Box to Virtual Reality.
|
|
Mixed reality trampoline games (2012-2019). One of the main themes of my research has been "mixed reality empowerment", i.e., boosting the player's abilities both digitally in the game world and in the real world using devices such as trampolines, in order to motivate exercise. When I started at Aalto, I purchased a trampoline and we published a CHI 2013 WIP video, CHI 2013 WIP paper, CHI 2014 HCI in sport workshop paper, and a IJCSS 2014 continuation paper. After that, the work went on a hiatus due to lack of funding, but Valo Motion, a spin-off from my group eventually picked it up, creating the Valo Jump trampoline game platform and Super Stomp, the world's first multiplayer trampoline game, also a CHI PLAY 2019 paper.
|
|
Exaggerating Avatar Flexibility in Virtual Reality (2018), a CHI PLAY 2018 paper that extends the movement empowerment theme above from jumping to flexibility. Paper, Video.
|
|
Proximal Policy Optimization with Covariance Matrix Adaptation (2018), a novel deep reinforcement learning method that combines ideas from Proximal Policy Optimization (PPO) and Covariance Matrix Adaptation Evolution Strategy (CMA-ES).
|
|
Simulating humanoid climbing (2017-2019). Climbing is a complex motor skill that requires both full-body coordination and planning long movement sequences. We solve the problem using a combination of search, optimization, and learning approaches, controlling a physically based simulated climber without any reference movement data. SIGGRAPH 2017 paper & video, SCA 2018 paper & video, CoG 2019 paper (to appear).
|
|
Combining online tree search and machine learning for continuous control (2017-2018). Our 3D characters learn stable locomotion gaits in less than a minute of wall-clock time on a modest 4-core computer. SCA 2017 paper & video, TVCG 2018 continuation paper.
|
|
Predictive Physics Simulation in Game Mechanics (2017). This CHI PLAY 2017 paper explores the novel design space of game mechanics based on reproducible forward simulation of the game physics. We propose and evaluate a number of novel game prototypes, including both turn-based and real-time gameplay and simple and complex simulations. Paper, Video.
|
|
Augmented Climbing Wall (2013-2016), a research project into digitally augmented sports, and now a spinoff startup. Viral video success with dozes of millions of views, CHI video showcase 2014, CHI 2016 paper with honorary mention, featured at wired.com, TechCrunch, New Scientist, Gizmodo... My role: supervising prof., computer vision for the first pilot, parts of the design and user studies, writing the CHI paper.
|
|
Come As You Are (2016) An award-winning Finnish Game Jam game with the theme "ritual". It's a short audio puzzle game that plays with the notions of 1) circular motion as oscillation, 2) movement and sound synchrony and harmony. There's also some procedural sound and animation. My role: game mechanics design, programming, procedural audio, shaders. Video, Code.
|
|
Gravity as a design resource (2015). Together with colleagues from Melbourne and Nottingham, we analyzed how gravity could be used in novel ways for digital and physical game design. The paper got a best paper honorary mention at CHI PLAY 2015, and also gave us a reason to try indoor skydiving =)
|
|
Online Control of Simulated Humanoids Using Particle Belief Propagation (2015). A novel online trajectory optimization / tree search approach for real-time continuous control of 3D characters. Extends the work below to more diverse movements such as controllable locomotion, juggling, and climbing over obstacles. SIGGRAPH 2015 Paper, Video 1, Video 2, Windows demo.
|
|
Online Motion Synthesis Using Sequential Monte Carlo (2014). We present the the world's first "active ragdolls" that can get back up when pushed over, in real-time, without breaking the physics simulation or using unrealistic forces, and without using any animation data or a time-consuming offline learning phase. SIGGRAPH 2014 Paper, Video, Windows demo.
|
|
Benefits of 3D topos for information sharing and planning in rock climbing. We created an interactive photogrammetry model of the popular Koivusaari boulder in Helsinki, with various information overlays for climbers, and conducted a user study. Sports Technology 2014 paper, Windows executable
|
|
Unpublished Kinect game prototypes from Virtual Air Guitar Company (2011), developed solo: 3d melee combat, exploring physics-based gameplay.
|
|
Kung-Fu Live (2010), and Kung-Fu High Impact (2011).
KFL is a PlayStation 3 combat game played by punching and kicking in front of the PS3 Eye camera, implemented using custom computer vision based body tracking. Kung-Fu High Impact is a sequel for Xbox 360 Kinect. My role: head of the computer vision team (wrote about 50k lines of the code myself), design and implementation of player's movement, controls & physics.
|
|
Kick Ass Kung-Fu (2004-2007), an immersive game installation that transforms computer gaming into an immersive physical performance. The game lets you experience Kung-Fu movie action and aesthetics by merging the real and the virtual with a perceptive user interface.
My role: Lead design & original idea, technology, music. Additional music by eBottle. Trailer , CHI 2005 paper.
Here's also a full-page ad from WIRED magazine that invited the game to NextFest three times.
|
|
Preceding Kick Ass Kung-Fu, I experimented with video "magic" mirrors that provide novel visual feedback for sports training. The simplest example is to point a camera at yourself and watch a delayed camera feed on screen while training. For example, when you practice spin kicks, you can't keep your eyes on a mirror when spinning around, so it's far more convenient to do the kick and then stop to watch the delayed view. Check out the video for more examples.
|
|
Musical interface research (2004-2006). I've played various musical instruments, and during my doctoral studies I participated in the ALMA project. Together with Teemu Mäki-Patola, we had a NIME 2006 paper about an augmented djembe drum (my part: original idea, computer vision, sound synthesis), and an ICMC 2004 paper about how the perception of interface latency varies with musical content.
|
|
Animaatiokone (2002), an installation for learning about stop-motion animation. The installation is built on an animation software and user interface I designed and implemented. Trailer, CHI 2004 paper.
|
|
Kukakumma Muumaassa (2001-ca. 2004), the first game where one could express yourself as the avatar using both body movements and voice. This was my M.A. thesis project and also part of my doctoral research. My responsibilities were concept and interaction design (with Johanna Höysniemi), sound design and technology. Video, Game design case paper from 2002. During the process, Höysniemi and I also developed some game user research methodology utilizing peer tutoring and Wizard of Oz prototyping. I proposed the methods and developed the software needed for the studies.
|
|
I've also done some audio DSP development, such as this lookahead limiter algorithm presented at DAFx 2002. I think this is quite an elegant solution to the problem. Paper, Matlab code.
|
|
Software that turns a laptop keyboard into a velocity sensitive midi keyboard or drumpad, from around 2002. The sound input of the built-in microphone is analyzed and the loudness of keystrokes is mapped to loudness of synthesized sounds. Video, Windows executable.
|
|
Design and programming of the sound engine of the Soittopeli (PlaySingMusic) musical edutainment game
(in collaboration with Tero Tolonen) by Elmorex Ltd.
Released in 2000, the game was the first to let you control an avatar by singing and learn to sing
with help of immediate visual feedback. I also developed some other singing game concepts and later wrote a retrospective paper about singing game design.
In 2003, the project was resurrected as Staraoke that won an interactive Emmy.
|
|
MediaObjects, a modular audio/video processing tool. It's the computer vision and audio processing engine in Animaatiokone, Kukakumma, and Kick Ass Kung-Fu (back in early 2000's).
I was going to release it, but never had the time, and now there's no point as pd has become the de facto tool in that space. As a curiosity, here's a gallery of self portraits generated when testing various video processing algorithms with a webcam.
|
|
Design of soundscapes and audio technology for the Semana Santa exhibition at Helinä Rautavaara's museum, summer 2000. I designed and implemented a 20-channel 3D interactive sound animation system for behavioural modelling of large acoustic spaces. Homepage:
http://mlab.uiah.fi/semanasanta.
|
|
Electronics for Jalmari Helander's (of Rare Exports fame) early short film Maximillian Tarzan (1999), using a 8-bit Atmel microcontroller.
|
|
Retordion (1998?), a distortion plugin for vst-compatible audio software such as Steinberg Cubase. It was conceived accidentally when I was experimenting
with nonlinearities inside feedback loops in a Griesinger style allpass reverb algorithm. The result is quite out of control weird distortion.
To use, download the dll to the vst plugin folder of your computer. Sound example:
original and distorted drums.
|
Research and Design Awards & Shortlistings
Best paper honorable mention at CHI 2024, with Solip Park and Annakaisa Kultima.
Best paper award at CHI 2023, with Mikke Tavast and Anton Kunnari
Best paper honorable mention at CHI PLAY 2023, with Reetu Kontio, Markus Laattala, and Robin Welsch
Best paper award at Academic Mindtrek 2021, with Tuure Saloheimo, Maximus Kaos, and Pia Fricker
Best paper award at ACM CHI 2019, with Maximus Kaos, Nick Graham and Ryan Rhodes
Finnish Applied Game of the Year (“vuoden hyötypeli”) award for Augmented Climbing Wall, 2017, with Valo Motion (my role: computer vision and parts of the design of the first version, supervising professor in commercialization phase)
Finnish Game Jam 2016 award for the jam game Come as you are. My role: game mechanics design, programming, procedural audio.
Augmented Climbing Wall shortlisted for Unity Awards 2016
Best paper honorable mention at ACM CHI 2016. With Raine Kajastila and Leo Holsti.
Best paper honorable mention at CHI PLAY 2015. With Raine Kajastila, Joe Marshall, Rich Byrne, Floyd Mueller.
Best Low-Cost, Minimally-Intrusive Solution award in 3DUI contest, Orlando, U.S., 2013. In collaboration with Tuukka Takala and Meeri Mäkäräinen
Kung-Fu High Impact (Virtual Air Guitar Company) won the “Control Design, 2D or Limited 3D” category in the 11th annual National Academy of Video Game Testers and Reviewers awards, 2011.
Kick Ass Kung-Fu (Perttu Hämäläinen, Ari Nykänen, Mikko Lindholm) won the Games Platforms category in Europrix Top Talent multimedia innovation competition in Vienna, Austria, 2004
Tampere Mindtrek Grand Prix for Kick Ass Kung-Fu, Tampere, Finland, 2004
Korjaamo Young Design Award for Animaatiokone (Mikko Lindholm, Perttu Hämäläinen, Ari Nykänen), Helsinki, Finland, 2004
Animaatiokone won the Pikku Kakkonen (best multimedia for children) and non-commercial categories at Tampere Mindtrek competition, Finland, 2003
Prix Spécial du Jury for Animaatiokone in the international Prix Möbius multimedia competition in Athens, Greece, 2003
Kukakumma Muumaassa (Perttu Hämäläinen, Johanna Höysniemi, Teppo Rouvi, Laura Turkki) was shortlisted for the Milia New Talent Competition in Cannes, France, February 2002.
Kukakumma Muumaassa won the Pikku Kakkonen category in Tampere Mindtrek competition (best multimedia for children), 2001
Exhibitions
Kick Ass Kung-Fu at Wired NEXTFEST, Los Angeles, US, 2007
Kukakumma Muumaassa at Ars Electronica Center, Linz, Austria, 2007
Kick Ass Kung-Fu at Karate World Championships, Tampere, Finland, 2006
Animaatiokone and Kukakumma Muumaassa at Ars Electronica campus exhibition, Linz, Austria,2006
Kick Ass Kung-Fu at Wired NEXTFEST, New York, US, 2006
Animaatiokone at Red Sticks international animation festival, Baton Rouge, Louisiana, US, 2006
Kick Ass Kung-Fu at Ultrasound festival, Huddersfield, UK, 2005
Animaatiokone at Japan Pop, Tennis Palace Art Museum, Helsinki, Finland, 2005
Kick Ass Kung-Fu at Wired NEXTFEST, Chicago, US, 2005
Kick Ass Kung-Fu at Tennispalatsi, Helsinki, Finland, 2005
Kick Ass Kung-Fu at pixelACHE festival, Helsinki, Finland, 2005
Kick Ass Kung-Fu at Korjaamo gallery, Helsinki, Finland, 2005
Kick Ass Kung-Fu at Nordic Exceptional Trendshop, Copenhagen, Denmark, 2004
Kick Ass Kung-Fu at Europrix Top Talent, Vienna, Austria, 2004
Kick Ass Kung-Fu at MindTrek, Tampere, Finland, 2004
Kick Ass Kung-Fu at Kiasma Theater, pixelACHE, Helsinki, Finland, 2004
Animaatiokone at Korjaamo gallery, Helsinki, Finland, 2004
Animaatiokone at Animex animation festival, Middlesbrough, UK, 2004
Kukakumma Muumaassa at Musta tuntuu exhibition, Rauma Art Museum, Rauma, Finland, 2004
Kukakumma Muumaassa at Game On exhibition, Tennis Palace Art Museum, Helsinki, Finland, 2003
Animaatiokone at Nordic Exceptional Trendshop, Copenhagen, Denmark, 2003
Animaatiokone at MindTrek, Tampere, Finland, 2003
Animaatiokone at Prix Möbius International, Athens, Greece, 2003
Animaatiokone at Annecy animation festival, Annecy, France, 2003
Animaatiokone at Tough Eye animation festival, Turku, Finland, 2003
Animaatiokone at Kunsthalle Lophem, Belgium, 2003
Animaatiokone at pixelACHE festival: Kiasma in Helsinki, Finland, Gerschwin Hotel in New York, USA, and SAT in Montreal, Canada, 2003
Animaatiokone at FF media festival, Rovaniemi, Finland, 2003
Animaatiokone at Tampere Film Festival, Tampere, Finland, 2003
Animaatiokone at Prix Möbius Preview Exhibition, Helsinki, Finland, 2003
Kukakumma Muumaassa in Design Museo, Helsinki, Finland, 2002
Animaatiokone at Leffakansio, Helsinki, Finland, 2002
Animaatiokone at Kettupäivät, Helsinki, Finland, 2002
Animaatiokone at Animatricks animation festival, Helsinki, Finland, 2002
Animaatiokone at Mainonnan Viikko, Helsinki, Finland, 2002
Animaatiokone at Love & Anarchy festival, Helsinki, Finland, 2002
Design of interactive soundscapes for Semana Santa exhibition at Helinä Rautavaara's museum, Espoo, Finland, 2000
Invited talks
Summer School on Experimental Play, RMIT University, Australia, 2018
Keynote at GamiFIN 2018, Pori, Finland
Keynote at Next Wave Conference, Seoul, South Korea, 2018
Opening Keynote of ITK conference, Hämeenlinna, Finland, 2013
“Teknologian ja käyttöliittymien mahdollisuudet opetuspeleissä”. Peliakatemia, Helsinki, Finland, 2013
“Designing (player motion in) motion games”, ManseDanse Festival, Tampere, Finland, 2011
“Exertion interfaces and digital sports”, University of Art and Design Helsinki Media Lab, Finland, 2006
“Exertion interfaces and digital sports”, University of Pori, Finland, 2006
“Designing Kick Ass Kung-Fu”, Flimmer international film festival, Norrköping, Sweden, 2005
“Mixed reality martial arts”, International symposium on Mixed Reality, RIXC Media Space, Riga, Latvia, 2005
“Interface driven edutainment”, University of Turku, Pori, Finland, 2004
“Computer vision based interfaces”, Experimental interaction workshop, University of Art and Design, Helsinki, Finland, 2004
“Experimental interaction in games”, Game On exhibition, Tennispalatsi art museum, Helsinki, Finland, 2003
“Computer vision based interfaces”, Experimental interaction workshop, University of Art and Design, Helsinki, Finland, 2003
“Experimental Interaction in games”, Pelin kosketus seminar day, YLE, Finland, 2002
“Designing QuiQui’s Giant Bounce”, Jyväskylän kesä festival, Jyväskylä, Finland, 2002
Audiovisual
DISCLAIMER: the work below is more than 20 years old now, from around 1995-2001.
I've done audiovisual stuff for the interactive projects above, such as the
Kick Ass Kung-Fu
trailer (music & editing),
Animaatiokone info movie (sound & music) and
Kukakumma Muumaassa intro animation soundtrack.
Here's also some samples of my non-interactive audiovisual work.
|
Sound and music for Animatricks 2001 trailer (mpeg1).
|
|
Music for Golfpiste.fi TV commercial.
|
|
Make-up, sound design, music and cinematography for the J5 surreal soap by the OUBS campus television of the Helsinki University of Technology. The series follows the life of students and mythical figures in a student community.
Here's a piece of music, most of which has been lost.
|
|
Cinematography, lights & music for the OUBS 1996 jingle. The jingle was based
on my idea, but having just watched Greenaway's Pillow Book, it was maybe not that original...
|
|
Music for the Speksi student theater orchestra, years 1996-1997. Samples: The
speksi 1997 theme (synthesizer version used for rehearsing, I don't have a live
recording) and a
live recording of my arrangement of "You Don't Have To Say You Love Me".
|
|
Make-up effects & part of the music of the short film Let's Fucking Die (1997). Fun fact: LFD was a first year film-school project of award-winning cinematographer Mika Orasmaa (Iron Sky, Rare Exports etc.).
The opening scene features two pieces of my music. Here's also a
climactic hymn where the hero meets the villain (the start overlaps preceding guitar score),
trailer music (guitar melody played & composed by Henri Penttinen,
noisy due to a damaged DAT tape) and
the ending credits music.
|
