The US Patent and Trademark Office published an Apple patent application titled "Method and System for Determining at least one property that is at least part of a real environment" on Thursday. The application relates specifically to computer vision, namely the use of cameras to analyze a scene and to obtain useful data from what is seen.
The submission identifies a problem with existing computer vision analysis techniques in dynamic environments, in the sense that many algorithms are constructed with a static environment in mind, but that the camera setting is the only moving object, one that is also the only one where pose can be permanently tracked.
If there are multiple moving objects in a scene, the accuracy of tracking the camera posture is reduced or even disabled, depending on the movements of items in a scene. The movement of objects can also cause problems with recognition as a result of the occlusion of the view by the camera.
Although a 2D camera view can be translated to points in 3D space, the movement of objects in this space can also cause problems with the 3D positioning of the camera.
One of the answers of the application is to use a system of two cameras of different types, consisting of a primary thermal imager and another normal camera. Although both could effectively provide the same view, the thermal imager could provide heat memory data to the algorithm to accompany the imaging data provided by the second camera.
The algorithm could analyze the thermal image and combine it with the image data to determine which items in a scene are separated from others, adding heat in addition to color, contrast and other characteristics typically used for analysis.
For exle, if there are people playing football in a field with green uniforms, the thermal image would show heat where the players are, on a cooler background of the grass. The algorithm would determine if there are objects in the heated sections of the thermal map and use that to select the players in the normal image.
This can also be used to determine the occlusion of a flat image filmed by a similar camera setting. This system could use thermal imaging to identify a warm hand-covering part of a 2D image, with the data that might be used to change the screen, such as by hiding the element that is blocked by the hand.
In cases where a building is analyzed, such a system can be used to identify pedestrians in the neighborhood, which can then be automatically removed from a final composition of the structure when they are assembled together with other images from the same view.
Although the patent application could make use of image processing and video editing of consumers, but also of added reality, it could perhaps be used more in self-driving cars. Apple researches the concept with Project Titan, an operation that tests self-propelled vehicle systems and sensors, and one in which the addition of thermal cameras is useful and relatively simple.
Apple has other patents and applications in the field of self-driving, including some that affect what a self-driving car might see. For exle, a headline from early September describes special areas to mark areas of the road to alert the driver, but such a system can also strategically illuminate the road to better visualize vehicle visibility systems.
As always, Apple regularly registers a large number of patent applications at the USPTO and relatively little finds its way to consumer products. Its existence in a USPTO application is no guarantee that Apple will actually take over the described concept in the future, but it does indicate which areas it currently explores.