Multi-Sensor Fusion for Robotic Arms

As an undergraduate researcher in the Machine Learning and Control Systems (MLCS) Lab at Yonsei University (advisor: Prof. Jongeun Choi), I worked on two projects.

For my graduation thesis I developed a Multi-Modal Variational Autoencoder (MVAE) that fuses RGB-D images, joint states, and pressure readings into a shared latent space, to be used as the state representation for reinforcement learning of a robotic arm. Data was collected in a CoppeliaSim (V-REP) simulation.

Simulation environment for robotic data collection.

An example of the collected multi-modal data.

Generated data when traversing each latent dimension.

Developed an obstacle avoidance algorithm robust to dynamic environments for a mecanum-wheel mobile robot, combining an Extended Kalman Filter, obstacle clustering, and the Dynamic Window Approach (DWA).

Multi-modal VAE sensor fusion (graduation thesis)

Mecanum-wheel mobile robot navigation