Multi-camera ARSession: Part I
JULY 18, 2019 github contact
AR Face Tracking in an ARWorldTrackingConfiguration Session
At WWDC 2019 Apple announced support for multi-camera sessions for iOS devices
(see here and here)
Included in this is support for multi-camera ARKit
sessions. While this was briefly demoed on stage, no sample code
was released to show how to create what was demoed. After finding this StackOverflow question, I thought I'd write up how to setup an ARSession
with multi-camera support, starting with an ARWorldTrackingConfiguration
that also has ARFaceTracking. Later, we'll come back and look at how to do the reverse: an ARFaceTrackingConfiguration
that has ARWorldTracking.
For this you'll need a copy of Xcode 11 (which is presently in beta), and a device running iOS 13 (also in beta), both of which can be downloaded from developer.apple.com. The device also needs a TrueDepth™ camera. With that out of the way, we can start coding.
Start by opening Xcode and creating a new iOS project. Choose an augmented reality, SceneKit
, app from the template picker.
First we'll need to add some properties to our ViewController
class.
@IBOutlet var sceneView: ARSCNView!
lazy var faceGeometry: ARSCNFaceGeometry = {
let device = sceneView.device!
let maskGeometry = ARSCNFaceGeometry(device: device)!
maskGeometry.firstMaterial?.lightingModel = .physicallyBased
maskGeometry.firstMaterial?.diffuse.contents = UIColor.lightGray
maskGeometry.firstMaterial?.metalness.contents = UIColor.white
maskGeometry.firstMaterial?.roughness.contents = UIColor.black
return maskGeometry
}()
lazy var tapGesture: UITapGestureRecognizer = {
let gesture = UITapGestureRecognizer(target: self, action: #selector(didTap(_:)))
return gesture
}()
Here, just after our sceneView
is declared, we init
the geometry we'll use for our face node using the metal device of the sceneView
. We do some minor setup like setting some properties on the material of the geometry to give it a metallic look. After that, we'll also instantiate a tap gesture recognizer that we'll use to place our face node into the scene.
Next, we'll setup our SCNScene
in viewDidLoad
.
override func viewDidLoad() {
super.viewDidLoad()
// Set the view's delegate
sceneView.delegate = self
// Set the scene to the view
sceneView.scene = SCNScene()
sceneView.addGestureRecognizer(tapGesture)
sceneView.automaticallyUpdatesLighting = true
sceneView.autoenablesDefaultLighting = true
}
First we set ViewController
as our view's delegate
. This will allow ViewController
to react to important events registered by the ARSession
. Next, we create an empty SCNScene
and add it as our view's scene. We also attach the UITapGestureRecognizer
we created as a property to our view and set the scene's lighting conditions.
Next, we'll look at viewWillAppear
where we'll configure our ARSession
.
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
// Create a session configuration
let configuration = ARWorldTrackingConfiguration()
// Check if device supports face tracking
if ARFaceTrackingConfiguration.isSupported {
configuration.userFaceTrackingEnabled = true
} else {
// Fall back to world tracking only experience
}
configuration.isLightEstimationEnabled = true
// Enable plane detection to allow face node to be anchored to plane
configuration.planeDetection = [.horizontal]
// Run the view's session
sceneView.session.run(configuration)
}
Now that our session is configured and running, we can begin to respond to events. The first event we want to respond to is tapping on the screen. For this, we'll fill in our didTap(_:)
selector we hooked up to our UITapGestureRecognizer
.
@objc func didTap(_ recognizer: UITapGestureRecognizer) {
// Get tap location
let tapLocation = recognizer.location(in: sceneView)
// Perform hit test with detected planes
let hitTestResults = sceneView.hitTest(tapLocation, types: .existingPlaneUsingExtent)
// Guard that a result exits
guard let hitTestResult = hitTestResults.first else { return }
// Create anchor from result
let newAnchor = ARAnchor(transform: hitTestResult.worldTransform)
// Add to session and wait for callback
sceneView.session.add(anchor: newAnchor)
}
The general flow of ARKit
is Event → Anchor → Content. The tap is the event that triggered an anchor, so now we need to respond to the creation of said anchor in nodeForAnchor
with our face content.
// Override to create and configure nodes for anchors added to the view's session.
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
// If ARPlaneAnchor create node with plane geometry
if let planeAnchor = anchor as? ARPlaneAnchor {
let node = SCNNode()
let planeGeometry = ARSCNPlaneGeometry(device: sceneView.device!)
planeGeometry?.update(from: planeAnchor.geometry)
// For debugging make planes visible by adding color
planeGeometry?.firstMaterial?.diffuse.contents = UIColor.blue.withAlphaComponent(0.5)
node.geometry = planeGeometry
return node
} else {
// Create empty node
let node = SCNNode()
// Add the stored face geometry as the node's geometry
node.geometry = faceGeometry
// Move node up to just above plane
node.position = SCNVector3(0.0, 0.15, 0.0)
// Create node to contain face
let parentNode = SCNNode()
parentNode.addChildNode(node)
// Return parent node
return parentNode
}
}
Now that we can get the face mask into the scene, it is only a matter of responding to updates. For that, we'll use didUpdateNode
.
func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
// Check if face anchor
if let faceAnchor = anchor as? ARFaceAnchor {
// Update node geometry using anchor geometry
faceGeometry.update(from: faceAnchor.geometry)
}
// Check if plane anchor
else if let anchor = anchor as? ARPlaneAnchor,
let plane = node.geometry as? ARSCNPlaneGeometry {
// Update node geometry using anchor geometry
plane.update(from: anchor.geometry)
}
}
If you do all of that, you should get something like the video above. When running, you'll want to make sure that the device has a clear view of your face. If the mask stops responding or node doesn't appear on tap, try to hold device in a different position, typically either closer or further from your face.
You can find the compeleted project on github. Feel free to send me feedback.