Multi-camera ARSession: Part I

JULY 18, 2019   github   contact

AR Face Tracking in an ARWorldTrackingConfiguration Session

At WWDC 2019 Apple announced support for multi-camera sessions for iOS devices (see here and here) Included in this is support for multi-camera ARKit sessions. While this was briefly demoed on stage, no sample code was released to show how to create what was demoed. After finding this StackOverflow question, I thought I'd write up how to setup an ARSession with multi-camera support, starting with an ARWorldTrackingConfiguration that also has ARFaceTracking. Later, we'll come back and look at how to do the reverse: an ARFaceTrackingConfiguration that has ARWorldTracking.

For this you'll need a copy of Xcode 11 (which is presently in beta), and a device running iOS 13 (also in beta), both of which can be downloaded from developer.apple.com. The device also needs a TrueDepth™ camera. With that out of the way, we can start coding.

Start by opening Xcode and creating a new iOS project. Choose an augmented reality, SceneKit, app from the template picker.

First we'll need to add some properties to our ViewController class.

            
    @IBOutlet var sceneView: ARSCNView!

    lazy var faceGeometry: ARSCNFaceGeometry = {
        let device = sceneView.device!
        let maskGeometry = ARSCNFaceGeometry(device: device)!
        maskGeometry.firstMaterial?.lightingModel = .physicallyBased
        maskGeometry.firstMaterial?.diffuse.contents = UIColor.lightGray
        maskGeometry.firstMaterial?.metalness.contents = UIColor.white
        maskGeometry.firstMaterial?.roughness.contents = UIColor.black
        return maskGeometry
    }()

    lazy var tapGesture: UITapGestureRecognizer = {
        let gesture = UITapGestureRecognizer(target: self, action: #selector(didTap(_:)))
        return gesture
    }()
            
        

Here, just after our sceneView is declared, we init the geometry we'll use for our face node using the metal device of the sceneView. We do some minor setup like setting some properties on the material of the geometry to give it a metallic look. After that, we'll also instantiate a tap gesture recognizer that we'll use to place our face node into the scene.

Next, we'll setup our SCNScene in viewDidLoad.

            
    override func viewDidLoad() {
        super.viewDidLoad()
        
        // Set the view's delegate
        sceneView.delegate = self
        
        // Set the scene to the view
        sceneView.scene = SCNScene()
        sceneView.addGestureRecognizer(tapGesture)
        sceneView.automaticallyUpdatesLighting = true
        sceneView.autoenablesDefaultLighting = true
    }
            
        

First we set ViewController as our view's delegate. This will allow ViewController to react to important events registered by the ARSession. Next, we create an empty SCNScene and add it as our view's scene. We also attach the UITapGestureRecognizer we created as a property to our view and set the scene's lighting conditions.

Next, we'll look at viewWillAppear where we'll configure our ARSession.

			
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        
        // Create a session configuration
        let configuration = ARWorldTrackingConfiguration()
        
        // Check if device supports face tracking
        if ARFaceTrackingConfiguration.isSupported {
            configuration.userFaceTrackingEnabled = true
        } else {
            // Fall back to world tracking only experience
        }
        configuration.isLightEstimationEnabled = true

        // Enable plane detection to allow face node to be anchored to plane
        configuration.planeDetection = [.horizontal]

        // Run the view's session
        sceneView.session.run(configuration)
    }
			
        

Now that our session is configured and running, we can begin to respond to events. The first event we want to respond to is tapping on the screen. For this, we'll fill in our didTap(_:) selector we hooked up to our UITapGestureRecognizer.

            
    @objc func didTap(_ recognizer: UITapGestureRecognizer) {
        // Get tap location
        let tapLocation = recognizer.location(in: sceneView)
        
        // Perform hit test with detected planes
        let hitTestResults = sceneView.hitTest(tapLocation, types: .existingPlaneUsingExtent)

        // Guard that a result exits
        guard let hitTestResult = hitTestResults.first else { return }
        
        // Create anchor from result
        let newAnchor = ARAnchor(transform: hitTestResult.worldTransform)
        
        // Add to session and wait for callback
        sceneView.session.add(anchor: newAnchor)
    }
            
        

The general flow of ARKit is Event → Anchor → Content. The tap is the event that triggered an anchor, so now we need to respond to the creation of said anchor in nodeForAnchor with our face content.

            
    // Override to create and configure nodes for anchors added to the view's session.
    func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
        // If ARPlaneAnchor create node with plane geometry
        if let planeAnchor = anchor as? ARPlaneAnchor {
            let node = SCNNode()
            let planeGeometry = ARSCNPlaneGeometry(device: sceneView.device!)
            planeGeometry?.update(from: planeAnchor.geometry)

            // For debugging make planes visible by adding color
            planeGeometry?.firstMaterial?.diffuse.contents = UIColor.blue.withAlphaComponent(0.5)
            node.geometry = planeGeometry
            return node
        } else {

            // Create empty node
            let node = SCNNode()

            // Add the stored face geometry as the node's geometry
            node.geometry = faceGeometry

            // Move node up to just above plane
            node.position = SCNVector3(0.0, 0.15, 0.0)

            // Create node to contain face
            let parentNode = SCNNode()
            parentNode.addChildNode(node)

            // Return parent node
            return parentNode
        }
    }
            
        

Now that we can get the face mask into the scene, it is only a matter of responding to updates. For that, we'll use didUpdateNode.

            
    func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
        // Check if face anchor
        if let faceAnchor = anchor as? ARFaceAnchor {
            
            // Update node geometry using anchor geometry
            faceGeometry.update(from: faceAnchor.geometry)
            
        }
        // Check if plane anchor
        else if let anchor = anchor as? ARPlaneAnchor,
            let plane = node.geometry as? ARSCNPlaneGeometry {
            
            // Update node geometry using anchor geometry
            plane.update(from: anchor.geometry)
        }
    }
            
        

If you do all of that, you should get something like the video above. When running, you'll want to make sure that the device has a clear view of your face. If the mask stops responding or node doesn't appear on tap, try to hold device in a different position, typically either closer or further from your face.

You can find the compeleted project on github. Feel free to send me feedback.

←Back