In the engine room of Australia’s newest virtual band

^{Image: Sally Coleman in a motion capture suit used to animate Taal. Credit: Big Sand, Brecon James}

Welcome to the dusty bubblegum world of Big Sand, a band that doesn’t exist.

Taal’s road to music stardom was unconventional to say the least.

With her big ears and piercing purple eyes, the teenage sensation of sci-fi inspired band Big Sand looks as though she crawled out of an exiled, underground desert colony, destined to reimagine music as we know it. Which is exactly what she did.

Alongside her two alien bandmates, Taal is the lead singer of a new virtual science fiction band co-created by musician and radio host Sally Coleman, with a bevy of musical, visual and technical collaborators. Big Sand is an animated, narrative-driven musical adventure traversing virtual and physical worlds and inviting audiences along for the ride.

^{Image: Taal, the lead character in Big Sand. Credit: Big Sand, CDW Animation}

“While technology is a big focus of the project, the original drive was just that I love science fiction and fantasy storytelling,” Coleman said. “I grew up reading books that do amazing things with world building and I never grew out of it. There’s just not enough sci-fi in music.”

Whatever the tech, the story comes first

To connect audiences with the speculative world and characters of Big Sand, Sally knew that she was going to have to find a way to reach audiences on social media, using images and short form video. She developed a set of technicolour drawings that throwback to the classic hand-drawn sci-fi aesthetics that captivated her younger self, but quickly realised that current animation processes were going to require an investment of time and money at a scale well beyond her means as an independent artist.

“I realised I had to find a way to make this process faster,” she said. “So I started having a bunch of coffees with students in animation and game development and many of them said the same thing. Do it in a game engine.”

But Sally was hesitant.

“I thought, that’s the wrong look. I had this idea about the kind of aesthetic that comes out of game engines, like Fortnite, Roblox or Minecraft. They have a particular visual style, a specific 3D look.”

Her advisors begged to differ. They argued that the world of Big Sand could be translated into 3D using a game engine like the free software Unreal Engine, without compromising her aesthetic. They suggested developing a specific set of programming rules to dictate light and movement in such a way that would preserve that 2D, hand-drawn feel.

Soon enough, Taal emerged from the depths of Sally’s imagination into a harsh but beautiful desert landscape, with support from Create NSW. The next question was, how to make her move. She bought a motion capture suit that enabled her to embody her character and bring it to life in ways she wasn’t expecting.

“I thought it would be easier to learn motion capture than manually dragging a character around a game engine,” she said. “It turns out I was totally wrong. But the first time I got it to work, I walked and my character walked. I waved and my character waved. It was just this amazing moment of feeling that connection with the character.”

Motion capture gave Coleman rich visual content to share the process of developing Big Sand, enabling her to build an early audience for the project.

“It makes it really easy and visual for an audience to see what you’re doing,” she said. “It’s weird and funny to watch the process.”

Inviting audiences into the fictional world of Big Sand

Once the dusty, bubblegum world of Big Sand had been realised, Coleman began to wonder how to invite physical audiences in to explore it with her and perhaps, even shape it further.

After years of touring with her previous music project, Coda Conduct, she understood the power of live music. But the realities of touring had also sunk in.

“I get so much joy out of that social, physical experience of performing live, but I found that traditional touring, where you’re away from home and constantly travelling, could really drain me emotionally, not to mention financially,” she said. “So I began wondering how to create a live experience when your band is not physical and is it possible to tour without travelling somehow?”

There were also endless technical questions that, at first, she wasn’t sure how to tackle. But fast forward to December 2022, and she was in the midst of a complex, nail-biting technical experiment across multiple venues as part of Big Sand’s first prototype live show, supported by Australia Council/Creative Australia.

Hosted at The Lab in Adelaide, a venue that cocoons audiences in panoramic LED screens, Sally and her team placed six virtual cameras in the fictional world of Big Sand in Unreal Engine, mapping those to the venue walls. Using MIDI triggers in the music software Ableton, the sound activated automated changes to the visual material, a capability not yet fully exploited by the music industry yet, Sally believes.

Meanwhile in her motion capture suit, 20 minutes away at Flinders University, Sally embodied and controlled Taal’s performance whilst simultaneously seeing a live feed of the audience, which was critical in facilitating interaction.

“I really wanted it to be two-way, to grow a relationship between these fictional characters and the audience and luckily that’s what happened.”

The audience, who were each allocated individual character descriptions upon arrival at the venue, began to engage physically and verbally with Taal on her virtual stage.

“There’s no protocol about how they’re supposed to engage and react so I wanted to give them some sort of framework of how to relate to the characters,” she said. “So I’m experimenting with how to make that happen by giving them their own character and seeing how they participate in the world in their own way.”

^{Image: Audience engagement with Big Sand. Credit: Big Sand, Vipop}

“In science fiction, you’re literally building fictional worlds that don’t exist, which means other people can come in and add pieces to that world,” she said. “I’d really like to work out the mechanism through which people can contribute to that story and build it and feel as though they have some ownership over it.”

Futuring business models in new world of music and gaming

Sally now hopes to use Big Sand to experiment with what it means to create a home base for her audiences. Birthed from game development processes, she’s closely observing what business models underpin gaming that may be under-explored in the music industry.

“A glimmer for me, and where this project has room to explore, is licensing,” she said. “In the games industry, they’re licensing all kinds of IP to other creatives to make value through merchandise and lots of other fun things. The Big Sand business model could be building the world and the characters to the point where I can collaborate with other creatives and hopefully everyone wins.”

Sally Coleman is currently continuing her work developing the Big Sand live show, which was presented again at the Powerhouse Museum in Sydney in September 2023. You can find out more about Sally’s process and keep up to date with Big Sand on her YouTube channel.