Benefits of Having a VNC Server in a Car Head-Unit

Benefits of Having a VNC Server in a Car Head-Unit Agustin Almansi Technology Consultant 19 October 2017

Agenda About RealVNC & VNC Automotive VNC technology overview Remote Framebuffer protocol Use-cases for VNC Server in head-unit Integration on AGL

About &

VNC Automotive is a pioneer in delivering connected car solutions VNC & RFB invented in Olivetti / AT&T Labs Research Project RealVNC founded by inventors RFB3.8 published as an open internet standard, RFC 6143 VNC Automotive launched VNC MirrorLink released RFB4.1 released Available to License RealVNC achieves Core Member Status of CCC RealVNC delivers world s first MirrorLink 1.1 server certification First MirrorLink 1.1 in production VNC available for Raspberry Pi VNC Connect launched Corporate Rebranding 1994 2002 2007 2009 2010 2011 2012 2013 2014 2015 2016 2017 VNC Automotive, 2017 15 Auto OEMs 50+ Vehicle Models 10+ Million Cars 100+ Million Phones First Automotive partners signed VNC release support for ipod Out Contract with First contract with major Tier 1 First contract direct with Auto OEM First contract with First MirrorLink in cars (customer option) membership Telematics solution Major head unit contract for VW Group Major Chinese OEM Major contract for head units Sony phone contract membership First volume line-fit MirrorLink Handsets OEM Telematics solution VNC Automotive supports First MirrorLink 1.1 in production Volume line-fit production Partnerships Partnerships Member

VNC Technology Overview

VNC is a graphical screen mirroring system Device Under Control RFB Protocol Controlling Device VNC Server over network connection Screen Image Input Event VNC Viewer Virtual Network Computing (VNC) uses the Remote Framebuffer protocol (RFB) to remotely control another computer Device being controlled has a VNC Server Device directly operated by the user has a VNC Viewer Commonly used to provide remote access to one PC from another Operates over a reliable transport

Architecture overview Screen Image ( Framebuffer ) Device Under Control VNC Server Capture Comparison RFB Secure encrypted network connection USB, Wi-Fi, BT, LTE, etc. VNC Viewer Drawing Controlling Device Screen VNC Server takes a screen image from the device under control and, after comparing and compressing it, sends it via the RFB protocol to the controlling device VNC Viewer decompresses and renders the image for display on the controlling device Device Input Event Queue Applications / Services Compression Event Injection Data Handling Screen Image Input Event Data Communication De-compression Event Handling Data Handling Control Events (Keyboard, Touch, Mouse etc) Applications / Services Control events from the controlling device are transmitted via RFB to the VNC Server which translates and injects them into the device under control Any binary data can be exchanged across the Data Communication

VNC is highly efficient Operating system screen memory ( framebuffer ) Capture VNC Server device under control Comparison Viewer maintains a copy of the Server s framebuffer and the Server only sends updates when its screen contents change The framebuffer data can be encoded using different encoding methods to maximise performance depending on the bandwidth available Choice of lossless or lossy encodings, balance between image quality and framerate Can leverage hardware encoders/decoders like H.264 Compression RFB Protocol Operating system event queue Event injection Input Event 010010101110110 Screen image data to VNC Viewer Input events from VNC Viewer

Transport agnostic Wi-Fi (WLAN, Wi-Fi P2P) Ethernet LTE USB protocols Bluetooth Custom transports

Security Encryption End-to-end Industry standards like AES Authentication Different options available From passwords to more complex schemes like smartcards

Remote Framebuffer Protocol

RFB Protocol has three message phases Handshaking Phase Exchange of handshaking messages. After the handshaking phase, the VNC connection parameters are negotiated and the connection is established. Initialisation Phase Exchange of initialization messages. After this phase, both ends have agreed on all needed parameters for the following operational phase. Framebuffer Update Phase Client to server and server to client messages are used to reflect changes of the framebuffer content on the local endpoint and user interaction on the remote endpoint.

RFB Handshaking Phase

RFB Initialization Phase

RFB Framebuffer Update Phase VNC Server VNC Client Framebuffer Update Request Framebuffer Update

RFB is part of open Internet standards (RFC 6143) and publically available RFB 3.8 used widely in many 3 rd party VNC Viewers and Servers specification published on IETF (Internet Engineering Task Force) website: https://tools.ietf.org/html/rfc6143 RFB 4.1 used in our own solution products and services proprietary and available to licence

VNC has been widely adopted Linux distributions (including Raspbian for Raspberry Pi) Qt VNC plugin Embedded in Intel silicon (Intel AMT) In-car smartphone connectivity (MirrorLink) See previous presentation from Automotive Linux Summit 2017

VNC Server in Car Head-unit

Using VNC for Device Testing Remotely perform manual or automated testing tasks on a car head-unit VNC Viewer integrated into a testing tool or operated by a test engineer Can be used to simulate how a driver interacts with the HMI Including hardware buttons, like steering wheel controls VNC Viewer VNC Server

VNC can decouple the head unit and display unit IVI Unit using VNC - Passengers can interact with the HMI remotely to change songs, temperature controls, share navigation POIs, adjust seats, etc. - Multiple devices could be connected simultaneously Conventional IVI Unit As-Is - Display Unit and Board are coupled - Hard-wired using HDMI / DVI etc Screencasting Display IVI Board VNC Viewer VNC Server Embedded Unit / Black Box Module / VM Display Unit / Smartphone / Tablet Touch, Button Control please note the images shown is for illustration purposes only

Using VNC, the head-unit can remotely use apps running on Stick PC / Black-box Module Digital Dashboard / IVI Head Unit VNC Server on Module sends apps screen & audio to VNC Viewer on Head Unit VNC Viewer on head-unit sends touch & button events, mic audio to VNC Server on Module via Wi-Fi / USB / Bluetooth Simple Head-unit only requires basic features + VNC Viewer Flexible Module is easily upgradable, apps can be region-specific New Telematics/Headless AGL profile added in Electric Eel Stick PC / Black-box Module + GPS + Wi-Fi + Bluetooth + LTE + ODB II please note the images shown is for illustration purposes only

Using VNC, the head-unit can be monitored & controlled remotely over the Cloud Use Cases via 3G / 4G / 5G / Mobile Tethering / WiFi hotspot Roadside Assistance Audio / Video Audio / Video Screen Touch, Button Control Concierge Services Picture Overlay Screencasting Control Command Telemetry Data Remote Support for First Responders please note the images shown is for illustration purposes only

Integration on AGL

VNC Server on AGL Wayland Wayland Security Module proposal to safely expose privileged capabilities like screen capture Weston screen-share module for full-screen capture and event injection New Vehicle to Cloud Expert Group explore Remote Vehicle Interactions

VNC Automotive is a division of RealVNC. Other trademarks and logos are the property of their respective owners. Copyright 2002-2017 RealVNC Limited. All rights reserved.

Please visit our website for more information https://automotive.realvnc.com/ https://www.realvnc.com/