iOS17 and the evolution of physical-digital interaction

iOS 17 expands interaction methods through gestures, spatial audio, and vision-based search to enable future AR/VR interfaces.

Heath Gibson · Executive Manager – Industry Intelligence and Insights at NBN Australia15 September 2023Period: September 20233 min read

Last updated 8 June 2026

iOS 17→

New Apple operating system version released with iPhone 15

15→

iPhone lineup version showcased at the 'Wanderlust 2023' event

Locked

Companies covered:AppleGoogle

Abstract

This report examines the release of Apple's iOS 17 and its role in the evolution of physical-digital interaction. While the update offers incremental improvements, it introduces critical technological enablers for future digital realms, including gesture-based screen effects, 'Nudge to wake' sensors, and advanced 'LookUp' video search capabilities. These features represent a strategic shift away from physical touch toward hands-free, intuitive interfaces necessary for the eventual mass adoption of Augmented and Virtual Reality (AR/VR) technologies.

Key Takeaways

Apple's iOS 17 update introduces incremental but important steps in the evolution of physical-digital interaction.
New gesture-based controls allow users to trigger screen effects like fireworks or balloons directly into the camera feed.
The 'LookUp' feature now supports searching for information based on objects within a video and live camera translations.

stakeholder implications · PDF download

Frequently Asked Questions

The iOS 17 update introduces critical technological enablers for future digital realms, moving interaction away from physical touch toward hands-free interfaces. These advancements, including gesture-based screen effects and 'LookUp' video search, are essential precursors for the eventual mass adoption of AR and VR.

Showcased at the 'Wanderlust 2023' event, the iPhone 15 lineup utilizes advanced sensors for 'Nudge to wake' and environmental learning. New AirPod Pro improvements automatically adjust noise cancellation and volume based on user preferences, significantly reducing the need for manual hardware adjustments.

The system now allows users to record their own voice so the device can speak typed text live during calls or in-person sessions. This feature, combined with live camera translations and object-based video search, expands the utility of the device as a real-time interaction hub.

iOS17 and the evolution of physical-digital interaction

Abstract

Key Takeaways

Frequently Asked Questions

Unlock the full report

Free

Lite

Team