VR Teleoperation
Disclosure of Funding Source:
This research was conducted at the Georgia Institute of Technology
under the supervision of Dr. Jun Ueda, completing in June 2023,
resulting in the following publication: Encrypted Coordinate Transformation via Parallelized Somewhat Homomorphic Encryption for Robotic Teleoperation.
This study was supported in part by National Science Foundation Grant Nos. 2112793.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do
not necessarily reflect the views of the National Science Foundation.
Research Objective
The goal of this research was to create and maintain an encrypted coordinate space which couples a VR headset to a remote robotic arm via teleoperation. The system will have a user wear a VR headset and input wand which will map to a remote robotic manipulator. The headset will allow the user to visualize their environment and the wand will allow the user to command the manipulator.
The system works by constructing a virtual coordinate system which links the VR wand’s movement to the movement of the arm. This research seeks to increase the security of such a configuration by encrypting this virtual coordinate space and preforming all control calculations with respect to this encrypted space.
The high-level idea is illustrated below:
To system revolves around this virtual coordinate system which is constructed as follows. On system startup a global origin is established in the virtual coordinate system and the robot manipulator and VR wand’s initial position is mapped into this virtual coordinate space. In practice it is often fine to map the arm and wand’s initial position to the virtual origin.
An encrypted teleoperation controller was built to facilitate leader/follower behavior, where the controller will take action to ensure that the wand and arm maintain the same position in the virtual coordinate space.
Control Procedure
The control procedure is characterized by an initialization handshake between the remote and local machines. During this handshake cryptographic information is exchanged and a consistent cipher context is constructed allowing encrypted data from one machine to be added/multiplied with encrypted data from the other. After the initialization there is a control loop that runs indefinitely.
The whole control procedure is outlined in the sequence diagram below:
Using this procedure our encrypted controller was able to overcome the computation burden and achieve satisfactory control objectives. This is captured by the below reference/response plots. In these plots $x_r$, $y_r$, $z_r$ are the computed control commands and $x_c$, $y_c$, $z_c$ is the tracking achieved by the robot arm.
The tracking procedure is decent but there is an obvious phase lag. This is likely do with the limitations of the specific robot manufacture’s programming interface which was not designed for real-time control.
Experimental Conclusion
Setup
The experimental procedure is complicated by the need to provide a repeatable yet sufficiently complex trajectory to test the tracking behavior. A human operator is not able to provide a consistent enough movement for repeated experiments and constructing a mechanical rig would not provide complex enough movement. To resolve this trajectory issue a the wand was attached to a Universal Robot 5 (UR5) which was programmed to produce a consistent experimentation path.
The experimental setup is outlined by the following figure:
Homomorphically encrypted calculations are always plagued by painfully slow computation, which if not resolved make any real-time control impossible. Additionally, it is well observed that stronger cipher security parameters lead to slower computation, thus if we want our encrypted controller to be secure we must overcome this additional burden.
Parallelization
Thankfully there are many ways to parallelize homomorphic calculations such as Karatsuba Multiplication or the Number Theoretic Transform. With parallelization we can use larger keys and mitigate the increased computation cost of using larger more secure keys.
Since refresh rate is one of the main hurdles to realizing encrypted controllers this research analyzed the refresh rate achievable for increasingly larger cipher keys, the results of which are in the histograms below.
In the above figure you will see that without parallelization the reported refresh frequency make distinct jumps with each key, however with parallelization the difference in refresh rates is not as distinct. This indicates that parallelization is an effective measure to alleviate computation burden imposed by large keys.
The parallelization of this research realize heavily on maintain a large thread pool to distribute the workload. However their is an overhead to managing a large collection of threads and thus for small enough keys there is a crossover point where threading is no longer helpful. This will be system dependant but the behavior should look similar to the research result below:
Conclusion
Overall this research was able to successfully couple a VR headset to a remote robotic manipulator via an encrypted teleoperation controller. The controller was able to achieve real-time performance without leaking intermediate calculations or results.