Multi-Robot Systems Monitoring and Controlling Large-Scale Phenomena Crop#Watering#Robots# An&'fungus#Spraying#UAVs# Oil#Vacuuming#Autonomous#Boats# (a) Oil spill clean up (b) Crop irrigation Mac Schwager Assistant Professor Aeronautics and Astronautics Stanford University (c) Forest fungus control April 19, 2016 Platform Lab Seminar 1
Multi-Robot Systems Large scale environment Actuation Communication Sensing Distributed Computation April 19, 2016 Platform Lab Seminar 2
Nature is Distributed Morphogenesis Herding Flocking Swarming Human group dynamics Cities and human institutions April 19, 2016 Platform Lab Seminar 3
Prototypical Application Deployment and coverage Adaptive sampling Self maintenance Environmental estimation Model learning/ system ID Environmental control April 19, 2016 Platform Lab Seminar 4
Other Application Areas Crop#Watering#Robots# An&'fungus#Spraying#UAVs# Oil#Vacuuming#Autonomous#Boats# Chemical (a) Oil spill clean up up Agriculture (b) Crop irrigation Infestation (c) Forest funguscontrol control Disaster relief Urban surveillance Transportation systems April 19, 2016 Platform Lab Seminar 5
Multi-robot Systems Lab MSL$ April 19, 2016 Platform Lab Seminar 6
Multi-robot Systems Lab MSL$ Motion capture system Flying Arena, 10m x 5m x 3m Floor projection system Robot fleet April 19, 2016 Platform Lab Seminar 7
April 19, 2016 Platform Lab Seminar 8
Multi-robot Systems Research Deployment and coverage Coordinated agile maneuvers Model learning/ System ID Active estimation with guarantees Neighbor trust adversaries Multi- Robot Systems Active 3D vision and control Multi-robot manipulation Multi-robot herding Persistent monitoring April 19, 2016 Platform Lab Seminar 9
Multi-robot Systems Research Deployment and coverage Coordinated agile maneuvers Model learning/ System ID Active estimation with guarantees Neighbor trust adversaries Multi- Robot Systems Active 3D vision and control Multi-robot manipulation Multi-robot herding Persistent monitoring April 19, 2016 Platform Lab Seminar 10
Deployment and Coverage April 19, 2016 Platform Lab Seminar 11
Existing Methods Geometric: J. Cortes, S. Martinez, T. Karatas, F. Bullo. Coverage control for mobile sensing networks. IEEE TRA, 20(2):243-255, 2004. A. Arsie, E. Frazzoli. Efficient routing of multiple vehicles with no explicit communications. IJRNC, 18(2):154-164. L. C. A. Pimenta, V. Kumar, R. C. Mesquita, G. A. S. Pereira. Sensing and coverage for a network of heterogeneous robots, CDC 08. M. Schwager, J.-J. Slotine, D. Rus. Decentralized, Adaptive Coverage Control for Networked ROBots. IJRR, 28(3):355-375, 2009. Probabilistic: W. Li, C. G. Cassandras. Distributed cooperative coverage control of sensor networks, CDC 05. M. L. Hernandez, T. Kirubarajan y. Bar-Shalom. Multisensor resource deployment using posterior Cramer-Raobounds. IEEE TAES 40(2):399-416, 2004. M. Pavone, S. L. Smith, F. Bullo, E. Frazzoli. Dynamics multi-vehicle routing with multiple classes of demands, ACC 09. Potential Fields: D. H. Kim, H. Wang, and S. Shin. Decentralized control of autonomous swarm systems using artificial potential functions: analytical design guidelines, JIRS, 45(4):369-395, 2006. A. Howard, M. J. Mataric, G. S. Sukhatme. Mobile sensor network deployment using potential fields, DARS 02. V. Gazi, K. M. Passino. A class of repulsion/atteraction forces for stable swarm aggregations, IJC, 77(18):1567-1579, 2004. April 19, 2016 Platform Lab Seminar 12
Deployment Optimization n robots in R 2 H(p 1,...,p n ) ṗ k H i = u i (t) ph i Gradient Theorem: Controller: Robots converge to a local minimum of the cost function. point in R 2n M. Schwager, D. Rus and J. J. Slotine, Unifying Geometric, Probabilistic, and Potential Field Approaches to Multi-Robot Deployment, International Journal of Robotics Research, March, 2011, vol. 30, no. 3. April 19, 2016 Platform Lab Seminar 13
Deployment of a Flying Camera Network M. Schwager, B. Julian, M. Angermann and D. Rus, Proceedings of the IEEE, 2011. M. Schwager, B. Julian and D. Rus, In Proceedings of the International Conference on Robotics and Automation (ICRA), 2009. April 19, 2016 Platform Lab Seminar 14
Sensor Cost from Optics camera one pixel height area/pixel From optics and geometry: field of view April 19, 2016 Platform Lab Seminar 15
Coverage Cost Function Combined cost for multiple cameras prior area/pixel Cost function: April 19, 2016 Platform Lab Seminar 16
Gradient Based Controller Controller: Components: Lateral Component: Move inside Q Move away from neighbors April 19, 2016 Platform Lab Seminar 17
Gradient Based Controller Vertical Component: Move up to see more of Q Move down away from neighbors Move down to see Q better April 19, 2016 Platform Lab Seminar 18
Convergence LaSalle s invariance principle Trajectories are bounded autonomous and locally Lipschitz Time derivative of non-increasing Therefore converges to largest invariant set where which implies April 19, 2016 Platform Lab Seminar 19
April 19, 2016 Platform Lab Seminar 20
April 19, 2016 Platform Lab Seminar 21
Harvard Forest April 19, 2016 Platform Lab Seminar 22
April 19, 2016 Platform Lab Seminar 23
Simulation Results 100 9 x 109 8 50 7 0 100 0 6 5 4 3 100 200 300 0 100 200 300 400 2 1 0 0 20 40 60 80 100 120 140 160 180 April 19, 2016 Platform Lab Seminar 24
April 19, 2016 Platform Lab Seminar 25
Five Quadrotors Landing Autonomously April 19, 2016 Platform Lab Seminar 26
Forest Reconstruction April 19, 2016 Platform Lab Seminar 27
Multi-robot Systems Research Deployment and coverage Coordinated agile maneuvers Model learning/ System ID Active estimation with guarantees Neighbor trust adversaries Multi- Robot Systems Active 3D vision and control Multi-robot manipulation Multi-robot herding Persistent monitoring April 19, 2016 Platform Lab Seminar 28
Multi-robot Systems Research Deployment and coverage Coordinated agile maneuvers Model learning/ System ID Active estimation with guarantees Neighbor trust adversaries Multi- Robot Systems Active 3D vision and control Multi-robot manipulation Multi-robot herding Persistent monitoring April 19, 2016 Platform Lab Seminar 29
Coordinated Agile Maneuvers Dingjiang Zhou April 19, 2016 Platform Lab Seminar 30
Dealing with Dynamics ẋ = f(x, u(t)) ṗ = u(t) April 19, 2016 Platform Lab Seminar 31
Quadrotor Dynamics Velocity Angular Velocity Position Orientation v = qe 3 + 1 m Rf ze 3 = J 1 J 1 J ṗ = v Ṙ = R x =(v,, p, R) 12 dimensional state space, 4 dimensional input space Nonlinear dynamics Non-Euclidean, evolve on SE(3) se(3) Potentially difficult control and planning problem April 19, 2016 Platform Lab Seminar 32
Trajectory Planning Desired trajectory x(t) How can we ensure that ẋ = f(x, u(t))? For what control input trajectory u(t) will this be true? April 19, 2016 Platform Lab Seminar 33
Differential Flatness A small miracle: quadrotor dynamics are differentially flat x = (,,..., (q) ) State trajectory x(t) planned flat output trajectory (t) (t) =(p(t), (t)) u = (,,..., (q) ) Input trajectory u(t) April 19, 2016 Platform Lab Seminar 34
Related Work Differential Flatness Theory Fliess, Levine, Martin, Rouchon, IJC 1995. Fliess, Levine, Martin, Rouchon, TCA 1999. Martin, Murray, Rouchon, CDC 2004. Martin, Murray, and Rouchon, Technical report, Caltech, CDS 2003. Richard M Murray. Optimization-based control. Caltech, 2009. Differential Flatness for Quadrotors Mellinger, Kumar, ICRA 2011 Mellinger, Michael, Kumar, IJRR 2012 Zhou, Schwager, ICRA 2014. Zhou, Schwager, ICRA 2015. Control Assuming Single Integrator Agents Olfati-Saber and Murray, TAC 2004. Jadbabaie, Lin, and Morse, TAC 2003. Cortes, Martinez, Karatas, and Bullo, TRO 2004. Zavlanos and Pappas, TRO 2007. Van den Berg, Lin, and Manocha, ICRA 2008. Schwager, Julian, Angermann, and Rus, Proc. IEEE 2011. Reynolds, Computer Graphics, 1987. Dimarogonas and Johansson, ICRA 2008. Ayanian and Kumar, TRO 2010. Habets and van Schuppen, Automatica 2004. April 19, 2016 Platform Lab Seminar 35
Differential Flatness A small miracle: quadrotor dynamics are differentially flat x = (,,..., (q) ) State trajectory x(t) planned flat output trajectory (t) (t) =(p(t), (t)) u = (,,..., (q) ) Input trajectory u(t) April 19, 2016 Platform Lab Seminar 36
Trajectory Following Control Flat output trajectory u ff (t) p(t), (t) u fb (t) ẋ = f(x, u) Differential Flatness Pipeline SE(3) controller x d (t) x(t) April 19, 2016 Platform Lab Seminar 37
April 19, 2016 Platform Lab Seminar 38
Vector Field Following We would like to deal with kinematic points in a vector field. We have: We want: ẋ = f(x, u) u = (p,...,... x d = (p,...,... p ) p ) u(p) = (p,...,... p (p)) x d (p) = (p,...,... p (p)) ṗ = v(p) p = v(p) T ṗ = v(p) T v(p). p (k) (k 1) = v(p)p Zhou, Schwager, ICRA 2014. April 19, 2016 Platform Lab Seminar 39
Multi-Robot Control n copies Vector Field Differential Derivatives Flatness u(p) = (p,...,... p (p)) u ff (t) Differential Flatness Pipeline x d (t) Vector Field SE(3) ṗ controller = v(p). u fb (t) p (k) = v(p)p (k 1) x(t) ẋ = f(x, u) ẋ = f(x, u) ṗ i = v(p) 1,...,p n ) Back to our kinematic multi-robot system. Zhou, Schwager, ICRA 2014. April 19, 2016 Platform Lab Seminar 40
April 19, 2016 Platform Lab Seminar 41
Virtual Rigid Body (VRB) r 1 Plan 6DoF trajectory for VRB p vrb (t),r vrb (t) r 3 VRB r 2 Multi-robot Formation = {r 1,...,r n } Flat output trajectories World p vrb (t),r vrb (t) p(t) i = p vrb (t)+r vrb (t)r i i(t) arbitrary Zhou, Schwager, ICRA 2015. April 19, 2016 Platform Lab Seminar 42
Formations and Transformations Formation A A Transformation B A VRB trajectory Formation B B April 19, 2016 Platform Lab Seminar 43
April 19, 2016 Platform Lab Seminar 44
VRB for Human Swarm Interface user input ẋ = f(x, u) chosen dynamics VRB user driven trajectory ABC scalable swarm trajectory April 19, 2016 Platform Lab Seminar 45
Nov 6, 2015 Grad Happy Hour 46
Multi-robot Systems Research Deployment and coverage Coordinated agile maneuvers Model learning/ System ID Active estimation with guarantees Neighbor trust adversaries Multi- Robot Systems Active 3D vision and control Multi-robot manipulation Multi-robot herding Persistent monitoring April 19, 2016 Platform Lab Seminar 47
My Approach Provable distributed control algorithms Multi-robot experiments Theoretical tools Control theory Estimation, filtering, machine learning Optimization April 19, 2016 Platform Lab Seminar 48
Current Work Cloud Robotics 50 45 40 35 30 25 20 15 10 5 0 0 10 20 30 40 50 Local low level control Centralized computation in the cloud Using cell phone and WiFi networks April 19, 2016 Platform Lab Seminar 49
Current Work Active 3D Forest Modeling UAV$Surveillance$Mission$ Nov 6, 2015 Grad Happy Hour 50
Current Work Intelligent Controlled Vision Eric Cristofalo 3D Printed Forest Model Structure from Motion Reconstruction Nov 6, 2015 Grad Happy Hour 51
Summary Multi-robot Systems Coverage with cameras Virtual Rigid Bodies for agile coordination Multi-robot systems will fundamentally change the way we interact with the world at large scales April 19, 2016 Platform Lab Seminar 52
Thanks! My Collaborators: Prof. Daniela Rus, MIT Prof. Vijay Kumar, UPenn Prof. Mark Friedl, BU Dr. Josh Gray, BU Dr. Brian Julian, Google My Funding Sources: April 19, 2016 Platform Lab Seminar 53
Summary Multi-robot Systems Coverage with cameras Virtual Rigid Bodies for agile coordination Multi-robot systems will fundamentally change the way we interact with the world at large scales April 19, 2016 Platform Lab Seminar 54