New "Siri Recap" Feature for Apple Watches Can Create Summaries of Your Conversations Throughout the Day, Apple Says

Apple today announced Siri Recap, a new feature in their upcoming Apple Watches that will allow it to generate a summary of conversations you have throughout the day.

The feature was announced amidst three other new audio features involving the microphone: Live Rewind, Music Recognition, and Sound Recognition, although those are less interesting in terms of privacy.

Live Rewind lets you recall what happened in the last 15 seconds in a conversation, although this only works on a continuous 15-second buffer that's always being rewritten. Music Recognition is just Shazam but now it can happen in real time, with the only data processed being the audio signature of the song and not saved audio. It's a similar story with Sound Recognition: it listens continuously for a specific type of sound such as an alarm going off but no sound is ever recorded.

They explain in a newly-published whitepaper that for all of these features, the raw audio is processed in the Secure Exclave of the Apple Watch: "a hardware-isolated compartment built into the silicon to process sensor data separately from the rest of the system."

For iPhones 16 and later, the Secure Exclave in the phone can securely pair with the Secure Exclave in the watch so that the raw audio is never made available outside of the respective exclaves. The encryption keys are device-bound and rotate over time, so a lost device shouldn't be a security issue.

Siri Recap, the most interesting of the new features, stores a high-level summary of conversations throughout your day. The raw audio never leaves the Secure Exclave, but a condensed transcript of it is generated on-device and then sent encrypted to Apple's servers running Private Cloud Compute, the same technology that powers their cloud AI features.

The claim is that Apple can't access data sent to PCC, but the way the technology works at some point the data must be decrypted in order to be processed. This makes it distinctly less secure than purely on-device processing. Data is included about context like whether the speech was from a song, whether you were at an event based on your calendar data, and even some location information like your city, state, and country.

The summary is then sent back to your phone, encrypted, where it will be stored for 7 days unless you choose to save them.

The feature is opt-in, so you don't have to use it, but the amount of data it stores could be concerning. Even in condensed form, 7 days of summaries of all of your conversations could reveal a lot if an attacker gets access to your phone via malware or some other means.