Highlights
The Dawn of Spatial Computing
Oliver Weidlich (https://www.contxtu.al) introduces us to the dawn of spatial computing, a paradigm shift that moves us from interacting with screens to experiencing digital content within our physical space.
Spatial computing, described by Simon Greenwald as early as 2003, is now entering a new era with the launch of Apple Vision Pro in 2024.
As Tim Cook puts it, it’s
the first computer you look through, not at.
Defining Spatial Computing
Cronin & Scoble (2020, 2025 2nd edition) offer a comprehensive definition:
Spatial Computing is a series of technologies that can 'see' and map the real world, understand it, and predict the next moves of objects within it, while doing the same for digital worlds... all the while helping humans and/or machines navigate either.
Design Principles for Spatial Experiences
Apple emphasizes key principles for spatial design:
- Familiarity: Start with familiar elements like windows within spaces and glass textures to ease the transition.
- Comfort: Prioritize user comfort by placing content directly in front and favoring horizontal over vertical movements.
- Immersion: Design experiences that leverage the immersive capabilities of the platform, but use immersion intentionally.
Developing for visionOS
Transitioning from iOS to visionOS requires developers to consider platform consistency, content size adjustments (incl tap sizes), and leveraging new interaction models, as Devin Davies from Crouton advises: “Avoid using opaque colors in a window. Areas of opacity can block people’s view, making them feel constricted and reducing their awareness of the virtual and physical objects around them.”
Accessibility from the Start
Apple emphasizes the importance of prioritizing accessibility in spatial computing. With Vision Pro being touted as the “most accessible v1 product at launch,” it boasts an array of features catering to diverse needs. “Almost everyone will temporarily or permanently experience disability at some point in their life,” making accessibility a crucial consideration for all developers.
Applications Across Industries
The potential applications of spatial computing span various sectors:
- Enterprise: Telstra Purple’s “Immersive Disaster Recovery Orchestration” (POC) project with Energy Queensland exemplifies how spatial computing can revolutionize disaster response, offering real-time data visualization and collaboration for field intelligence officers.
- Healthcare: Vision Pro holds immense promise in healthcare, from simulating real-world environments for research to enhancing communication for the hearing impaired, as showcased by NALscribe AR iOS app.
LabFlow highlights its potential for pathology labs, hospital environments, and medical imaging with an impressive demo. - Retail: [Graviton] with EcoOutdoor utilizes Vision Pro to create virtual showrooms, showcasing the potential for immersive customer experiences in retail, using CAD files to create 3D home models.
Machine Learning on visionOS
James Dale (Friday Technologies) emphasizes the importance of thoughtfully designing ML experiences on visionOS, considering factors like user privacy, data transparency, and providing alternative input options.
Key Takeaways
- Spatial computing represents a fundamental shift in human-computer interaction.
- Designing for spatial experiences demands a focus on familiarity, comfort, and intentional immersion.
- Accessibility needs to be a primary consideration from the outset.
- Applications across industries are vast and still being explored.
- Thoughtful integration of machine learning and AI will shape the future of spatial computing.
Key design principles for creating visionOS applications:
- Familiarity: Design using familiar elements like windows within spaces and glass textures.
- Comfortable experience: Position content right in front of the user, and design for left-to-right interactions over up-and-down interactions. Add padding to tappable items.
- Immersion: Start with a window for familiarity. Use a volume. Augment existing space.
- Space and Scale: Start small, designing for a specific space and scale. Keep interfaces in 2D, reserving 3D for content. Use immersion carefully.
- Platform Consistency: Adhere to platform conventions for content size, navigation, system gestures, and element style. Tap size should be 60pt in visionOS, compared to 44pt in iOS. Avoid using opaque colors in windows, as they can block the user’s view of their surroundings.
- Accessibility: Accessibility should be incorporated from the beginning of the design process. visionOS is designed to be accessible, and apps should follow suit. Consider inbuild features like: Hearing Device Support, Spoken Content, Color Filters, Audio Descriptions, Braille Support, Per-App Settings, Voice Control, Bold Text, Background Sounds, Dwell Control, Pointer Control, VoiceOver, Reduce White Point, Button Shapes, Accessibility Shortcut, AssistiveTouch, Zoom, Large Text, Increase Contrast, Subtitles and Closed Captions, Switch Control, Full Keyboard Access, Image Descriptions, Guided Access, Reduce Motion, Left/Right Balance, Mono Audio, Type to Siri
- Thoughtful Machine Learning: When designing apps that use ML, prioritize privacy, set clear user expectations, provide transparent attributions, and offer multimodal feedback mechanisms.
- Consider user appeal, delight, and the use of good datasets.
- Utilize Apple’s Neural Engine for on-device ML processing (Enterprise API only).
- Leverage camera sensors, including passthrough, main camera access, object tracking, and barcode/QR code scanning (Enterprise API only).
- Access external devices via UVC with optional developer strap (Enterprise API only).
- Prioritize privacy with thoughtful UI, clear expectations, mistake correction, transparency, confidence indicators, feedback mechanisms, and - alternative input options.
Additional considerations:
- Collaborative experiences: visionOS features like SharePlay, spatial personas (example ing Freeform), and communal device experiences offer opportunities for collaborative app design.
- Transitioning from iOS: Developers transitioning from iOS to visionOS should ensure their apps compile, including any third-party libraries. Interactions may need to be adapted for the spatial environment, such as multiplying card interaction distances.
- Leveraging Apple Technologies: Developers are encouraged to leverage Apple APIs, frameworks like SwiftUI, RealityKit, ARKit, and Metal, and tools like Reality Composer Pro.
- Conduct thorough user research: Understand user needs, behaviors, goals, and physical environments to ensure app content and design are appropriate.
- Consider the spectrum of immersion: Start with familiar window-based interfaces. Progress to volumes that occupy a defined space. Augment the existing environment by integrating digital content with the physical world. Use immersion intentionally and thoughtfully.
- Multiply card interaction distance: Adapt interaction distances for spatial environments.
- Maintain focus and avoid feature overload: With the flexibility of visionOS, it’s important to prioritize key features and maintain a clear focus for the app.
