By Ketki Karanam
Sync Project: What originally made you interested in music-related research?
Tristan Jehan: I’ve always played music and I listen to a lot of it, in every style. I found that I could merge both my interests in technology and music by getting into computer music and more specifically music analysis. My engineering background is in digital signal processing, and music as complex as it might be, is a signal: it happens to be perceived by the human ear and interpreted in a certain way by the brain. So I got interested in the A.I. side of music: could I build a machine that makes music like humans do. It turns out machines can listen to music hundreds of times faster than us and there is a lot of music to learn from out there. So that became my PhD research at MIT.
SP: You are the principal scientist at Spotify. What are you working on at the moment?
TJ: I’m working on a whole lot of things. In general, my work is about looking for innovative ways to describe, organize, and match musical content to users’ preferences. We use our understanding and analytics on music (what it sounds like, people’s reaction to it on the web) as well as our understanding of the listener (usage patterns and listening history) to find the best recommendations for a particular listening situation.
SP: The Echo Nest combines data on music and listeners to form recommendations. Can you tell us about the kind of data you have gathered?
TJ: We have loads and loads of data on every piece of music ever released online from very basic acoustic features like tempo, rhythm, pitch and timbre, to what has been written about the artist or the albums on the web, like on blog posts, in news, and reviews. And at Spotify we also collect billions of user patterns every day, from what they listen to, to when they skip or how they rate songs. We don’t minimize recommendations by single characteristics of music but rather as aggregates of information that best describe the probable music affinity for a given user in a particular listening situation.
SP: How much can you tell about a person based on their musical preferences?
TJ: With the amount data that we collect, we are able to make really interesting predictions about listeners simply from their musical taste. For instance, we have been able to predict with a certain reliability whether a listener votes Democrat or Republican or whether someone is interested in sports, in drinking or is a smoker or not. These experiments are fun and powerful, but correlation does not mean causation. So for instance, even though there might be a strong correlation between the two, listening to country music does not make you a republican. These are interesting examples to show the power of data analytics and segmentation at scale.
SP: What are some exciting questions you look forward to investigating?
TJ: Well we are going towards more and more personalization. In addition to understanding the user’s general preferences, we need to understand them in context. What are users doing while listening to certain type of music? What is their mental state? Are they having dinner with friends? Are they working? We are getting more and more granular with recommendation and more specific about our use cases. You don’t listen to music the same way whether you are relaxing on a rainy Sunday afternoon, or out for a run in the morning. Better context understanding allows us to build more specific and exciting experiences.
SP: You are an advisor to The Sync Project, what made you interested in it?
TJ: If Spotify is about personalizing the right music for the right moment, the Sync Project is doing it for health reasons and helping certain communities that could benefit from it the most. Music is so ubiquitous and so natural to people that we sometimes underestimate its power. The Sync Project is about understanding the ways and mechanisms through which music listening can help truly serious conditions like Parkinson’s, autism, or sleep disorders. With better knowledge, music could be utilized as an enjoyable, easy and really cost-effective way to support recovery and treatment.
SP: Who else do you think should be interested in the Sync Project?
TJ: In my mind the Sync Project touches every music listener out there. In order to understand and ultimately harness the effects of music listening on a given individual, we need to collect the right information at scale. Also, I think it might be interesting for listeners to know how music is affecting them, how various music in different contexts influences their physiological response. Ultimately it could help people better understand themselves and their relationship to music, who they are, what helps them and what doesn’t. For many, music is a form of therapy: it helps us go through the good and the hard times.
SP: The Sync Project is aiming at finding ways to utilize the influence that music has on physiology for health benefits. Do you personally use music for health benefits?
TJ: Well, probably not that deliberately. I mean, everyone uses music for something. Most typically what we seek from music is an emotional response – we listen to music because it is enjoyable, it makes us happy, it makes us feel better. The benefits of music listening probably stem from its emotional effects. If I do things that make me happy every day, like listening to music, it is bound to influence my overall wellbeing, for example preventing high stress levels. Of course, people have also discovered more specific health benefits like using music to help them sleep. Even this rather simple-sounding thing like sleep quality can have far-reaching positive effects on health, since good sleep is connected to so many other things like stress, obesity, attention, depression.
SP: As you said, many people intuitively use music to support wellbeing, if not deliberately then through the effects of good mood and positive emotion. In this case, why do you think we need to conduct scientific research on the effects of music listening?
TJ: Because if we wish to be able to help people with serious conditions like say Alzheimer’s, or use music as medicine in any circumstance, we need actual proof. We need to be able to measure and predict with a certain level of confidence, how music listening will have an effect on the listener. Of course an individual’s reaction to music will always be in part unique. But having reliable data to back up some hypotheses, and not just shared anecdotes, is the foundation for making an actual difference in creating therapies that actually work.
SP: What kinds of questions do you think that Sync Project will be able to answer?
TJ: There are many questions that we could answer with the kind of information we (The Sync Project) are interested in obtaining: ranging from how to support sports performance to applications in treatment of more severe conditions like dementia. Personally I’m also interested in how rhythm relates to treating movement disorders like Parkinson’s. There are interesting studies on the topic and I feel that this is one area where The Sync Project could really produce ground-breaking research. All in all, the collaboration between scientists, musicians, device manufacturers, music listeners and patients will be an exciting effort to be part of, unraveling the mysteries around the power of music, and finding scientific answers to questions about how music supports our wellbeing.
SP: How probable do you think it is that we will soon be collecting psycho-physiological information and use them in not only wellbeing related areas but also in making music recommendations for instance, on Spotify?
TJ: I believe it is highly probable. I mean, the technology is almost there, it’s just a matter of how convenient and comfortable measurement is for the listener. Already with all the sensors available out there we could obtain a whole lot of detailed information on the physiological effects of music listening. Data on heart rate, skin conductance, brain activity and such could be used to optimize instant music recommendation. Actually, I recently made this fun application at Spotify that uses the accelerometer data that comes with your phone to match music to your run: it chooses music that is not only appropriate for running and personalized to you, but is also in perfect sync with your running pace. There are many studies according to which running in sync with musical tempo increases performance and this mode within the Spotify app is a fun way to explore these effects.
So, in addition to understanding context, or the situation and environment of music listening, with sensor data we will be able to better understand the emotional state of listeners and their response to music and take it all into consideration. That is the next step: from understanding the music and context to also understanding the emotional state and individual responses of a listener. This is what the Sync Project rests on – understanding the individual responses to music listening and discovering something generalizable from it, something common to improve our health and wellbeing effectively.
SP: Your PhD was written on a very interesting topic: you created a system that analyzes a collection of music, models its musical content, and re-synthesizes new songs.
TJ: Yes, and here’s an idea: if we could find individual preferences, and physiological and emotional reactions to specific musical features, we could then use this to automatically create music that is highly individualized. We could have music tailored to match your specific context, needs, preferences, physiology and emotional reactions. Music would always sound good to you! Haha! In the context of the Sync Project, we could make music that “is” personalized medicine, based on the physiological reactions we know certain musical features have on you, very specifically.
SP: Interesting proposal! What difference do you think it makes, if any, if the music is composed by a machine or by humans? I mean, you are also a musician, does machine-created music mean you are obsolete?
TJ: Well, honestly, just thinking purely about the listening experience, I think that if you didn’t know that a machine composed the music, it would not necessarily bother you. We still have visceral reactions to the idea of machines dealing with what is considered art, for understandable reasons. But I think that machine-created, individually optimized music could become a new genre in itself alongside man-made music. Man-made vs. machine-made music doesn’t have such a clear distinction. There is already plenty of successful electronic music that, although composed by humans, is highly assisted by machines, and in which every sound is synthesized on a laptop. Plus in this scenario, the machine would actually learn from millions of man-made pieces ranging from Mozart to the Rolling Stones. So it would be highly humanized in every possible way, including how it generates sounds. That is how A.I. works really: it’s modeling aspects of us humans using huge amounts of data, like for example in the case of speech and language: the recognition, synthesis, and conversation we might have with the machine is all modeled after human-generated data. It also does not mean bands are no longer needed – the vinyl-collecting music buff that frequents concerts will not be satisfied with only machine-generated music. The situation is not either-or, it is both. It is not about replacing but complementing the man-made with the machine-made