Multi-camera ARSession: Part II

JULY 23, 2019   github   contact



AR World Tracking in an ARFaceTrackingConfiguration Session

This is part two of our a look at multi-camera support for ARKit sessions. If you haven't already, I suggest you check out part one. Whereas in part one we looked at how to create an ARWorldTrackingConfiguration that also had face tracking, now we'll look at how to setup an ARFaceTrackingConfiguration that also has world tracking. That is, in part one we were showing the feed from the back camera and only using the front camera to monitor a face, now we're going to show the feed for the front camera but use the back camera to monitor the world around the device. In the process of learning how to set up our ARSession to support this, we'll create an app that changes the diffuse color of the facial mask provided by the ARFaceTrackingConfiguration as the distance between the camera's current location and the global origin of the ARSession changes.

The final product.

As with part one, you'll need a copy of Xcode 11 and a device that is running iOS 13 with a TrueDepth™ camera and an A12 chip. As of this writing both Xcode 11 and iOS 13 are still and beta and can be downloaded at developer.apple.com. Now that that is done, we can get started.

One thing you'll notice right away that is different from showing the back camera feed like in part one is that configuring the session of the secondary camera is somewhat limited. Whereas you'll use ARFaceTrackingConfiguration right out of the box, you typically configure ARWorldTrackingConfiguration further, almost always adding plane detection. However, because no ARWorldTrackingConfiguration object is exposed after turning on world tracking in an ARFaceTrackingConfiguration that is simply not possible. So, the amount of information you get about the world around the device might be lacking if you have elaborate plans. However, as briefly mentioned in this year's WWDC session, Introducing ARKit 3, you can get accurate device transformation data, which is exactly what we need for the app we'll build today.

Diagram of ARKit

Global orign is determined by the location where the device is when the ARSession begins.



Before we do that, I want to give a brief refresher about how ARKit works, as it will be partly relavent to the app we build. ARKit uses what known as visual-inertial odometry to create a model for the environment around the device. From Apple's developer documentation:

ARKit combines device motion tracking, camera scene capture, advanced scene processing, and display conveniences to simplify the task of building an AR experience.


Essentially, ARKit uses machine learning to track "visually interesting" points from the camera feed and track how they move over time. This data is then correlated with data from the gyroscope and compass of the device to model those points in 3D space. All of these points are placed in relation to a global origin. This global origin is the point (0, 0, 0) in the global space of the scene and is the location of the device when the ARSession begins. In the app we'll build, we'll look at the distance between the devices current location and this origin to update our face mask's material.

Start by opening Xcode and creating a new iOS project. Choose an augmented reality, SceneKit, app from the template picker.

Just like in part one, we're going to need geometry for our face mask. So we'll lazily instantiate a property, faceGeometry, to hold it in our ViewController class. We'll assign our faceGeometry a diffuse color using hue, saturation, and brightness. We do this because as the device moves around we'll change the hue giving the effect of moving through the continuous color spectrum. For this, we'll need a property that will determine essentially how long our color spectrum is.

            
    @IBOutlet var sceneView: ARSCNView!

    lazy var faceGeometry: ARSCNFaceGeometry = {
        let device = sceneView.device!
        let maskGeometry = ARSCNFaceGeometry(device: device)!
        maskGeometry.firstMaterial?.lightingModel = .physicallyBased

        // Using hue, saturation, and brightness because as the device moves we'll change the hue to give the continuous color spectrum
        maskGeometry.firstMaterial?.diffuse.contents = UIColor(hue: 0.0, saturation: 1.0, brightness: 1.0, alpha: 1.0)
        maskGeometry.firstMaterial?.roughness.contents = UIColor.black
        return maskGeometry
    }()

    // If the device moves twenty feet it will be at the end of the color spectrum.
    let maxFeet = Measurement(value: 20.0, unit: UnitLength.feet)
            
        



Essentially what we're trying to achieve can be seen in the following figure.

Diagram of face mask color relative to distance from origin

Face mask color as it relates to distance from origin



Next, in viewDidLoad, we'll do the typical setup, but we also need to set our ViewController as the session delegate. This will allow us to receive the func session(_ session: ARSession, didUpdate frame: ARFrame) callback.

            
    override func viewDidLoad() {
        super.viewDidLoad()
        
        // Set the view's delegate
        sceneView.delegate = self
        
        // Set session's delegate so we receive frame didUpdate callback
        sceneView.session.delegate = self
        
        // Create a new scene
        let scene = SCNScene()
        
        // Set the scene to the view
        sceneView.scene = scene
        
        // Light the scene
        sceneView.automaticallyUpdatesLighting = true
        sceneView.autoenablesDefaultLighting = true
    }
            
        



We have an ARSCNView with a scene, so now we need to setup and start our ARSession which will be done in viewWillAppear. Just like with part one, it's important to check if the configuration you want is supported on the user's device, for us that means checking if the ARFaceTrackingConfiguration supports world tracking. If it is, it is a simple matter of enabling world tracking.

            
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        
        guard ARFaceTrackingConfiguration.isSupported, 
              ARFaceTrackingConfiguration.supportsWorldTracking else {
            // In reality you would want to fall back to another AR experience or present an error
            fatalError("I can't do face things with world things which is kind of the point")
        }
        
        // Create a session configuration
        let configuration = ARFaceTrackingConfiguration()
        
        // Enable back camera to do world tracking
        configuration.isWorldTrackingEnabled = true

        // Yasss estimate light for me
        configuration.isLightEstimationEnabled = true

        // Run the view's session
        sceneView.session.run(configuration)
    }
            
        



Our session is now running, let's quickly look at our implementation of ARSCNViewDelegate as it's largely taken straight from part one.

            
    // MARK: - ARSCNViewDelegate -
    func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
        // Check if face anchor
        if let faceAnchor = anchor as? ARFaceAnchor {
            
            // Create empty node
            let node = SCNNode()
            
            // Update our `faceGeometry` based on the geometry of the anchor
            faceGeometry.update(from: faceAnchor.geometry)
            
            // Attach `faceGeometry to created node
            node.geometry = faceGeometry
            
            // Return node
            return node
        }
        
        // Don't know and don't care what kind of anchor this is so return empty node
        return SCNNode()
    }
    
    func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
        
        // If it's a face anchor with a node that has face geometry update the geometry otherwise do nothing
        if let faceAnchor = anchor as? ARFaceAnchor,
            let faceGeometry = node.geometry as? ARSCNFaceGeometry {
            faceGeometry.update(from: faceAnchor.geometry)
        } else {
            print(anchor)
        }
    }
            
        



Finally, let's fill in our implementation of ARSessionDelegate for this we're going to need two helper functions:

            
    // Length of a 3D vector
    func length(_ xDist: Float, _ yDist: Float, _ zDist: Float) -> CGFloat {
        return CGFloat(sqrt((xDist * xDist)+(yDist * yDist)+(zDist * zDist)))
    }
    
    // Distance between two vectors
    func distance(between v1:SCNVector3, and v2:SCNVector3) -> CGFloat {
        
        let xDist = v1.x - v2.x
        let yDist = v1.y - v2.y
        let zDist = v1.z - v2.z
        
        return length(xDist, yDist, zDist)
    }
            
        



Here, we're simply taking the difference between two vectors and using the generalized Pythagorean theorem to find the length of the result. Lastly, we'll make use of this by getting the translation of the camera from the origin, determine the distance travelled, converting that to a percentage of the maxFeet, and updating our faceGeometry's hue.

            
    func session(_ session: ARSession, didUpdate frame: ARFrame) {

        // Get translation of the camera
        let translation = frame.camera.transform.columns.3
        
        // Find distance between translation and origin
        let distance = self.distance(between: SCNVector3(translation[0], translation[1], translation[2]), and: SCNVector3(0, 0, 0))
        
        // Convert max feet to meter as in ARKit one unit is one meter not foot
        let maxMeters = maxFeet.converted(to: UnitLength.meters)
        
        // Normalize distance to our max
        let percentToMax = distance / CGFloat(maxMeters.value)
        
        // Clamp color so once you go past maxFeet in distance mask just stays red
        let hue = min(percentToMax, 1.0)
        
        // Update mask color
        faceGeometry.firstMaterial?.diffuse.contents = UIColor(hue: hue, saturation: 1.0, brightness: 1.0, alpha: 1.0)
        updateLabel(distance: Measurement(value: Double(distance), unit: UnitLength.meters))
    }
            
        



With that you should be able to build and run and get a face mask that changes color depending on the distance you travelled from the origin. Of course, you can also do it with different color spectrums, for example gray scale. Also, I could easily image a Simon Says style game where a user must travel in a certain direction from the origin with a face mask updating hotter and colder as they move.

Moving through grayscale spectrum

In conclusion, world tracking with an ARFaceTrackingConfiguration, while limited by not allowing for plane detection, can still be used to create compelling user experiences. It's easy to enable and works as expected even in beta form.

You can find the compeleted project on github. Feel free to send me feedback.


←Back