Synthespian Escalation

|

36 minutes
Vtuber Escalation

This post is part of an ongoing scrapbook series. Exploring the emerging Dweb (Web3) ecosystem, the #supportnet, and the coming ‘Metaverse’.


2020 was a break out year for VTubers due to the convergence of falling technology price points and maturing software ecosystem.

This post is about recent developments in the English speaking VTuber scene. What it tells us about the future of entertainment, the Metaverse, and the type of content that may get made there.

During 2019 I started writing about Lil Miquela, NFT’s, DAO’s and the Metaverse. The Gdoc grew in size and scope to include observations on Post-Internet Art, Hyperpop, VTubers/VInfluncers.

With absolutely no end in sight, I’ve decided to pull some of that writing on to the blog. So I can start in the present as it were.

Drop me a line on Twitter or get in touch if you wanna discuss anything you may find of interest here. Happy to chat.

But first, I would like to introduce you to an Internet creator who I really admire, and work I’ve been following for a long time.

Nyanners

Nyannars Early Fan Art

Nyanners is a virtual YouTuber (VTuber), Twitch streamer, voice actress, and singer.

Currently represented by VShojo – the first US based, artist-first VTuber talent management agency.

I first came across Nyanners in 2012-ish after a video was linked in an IRC channel frequented by UK members of OpChanology. I have vague memories of the lyrics to one of Nyanners early songs becoming a meme in the chat some time during the aftermath of the HBgary/Lulzsec Lulz era.

Anyhoos here’s a whistle stop tour though Nyatasha Nyanners output as an online creator.

2014 – I Am UR Leader

There is not space here to fully place this cover Nicki Minaj’s track (and Nyanners) into its wider cultural context. I will however gesture towards the influence of ‘the form’ of Post-Internet Art and the subsequent emergence of the phenomenon thats been branded Hyperpop. Both these threads are of importance to how I personally understand emerging Metaverse artistic praxis. As an aside, also note the aesthetic and stylistic similarities between the I AM UR LEADER video and PC Music’s early foray in to music videos with GFOTY’s Bobby #2014.

Post-Internet As ‘Form

Post-Internet arrived into the Artworld contemporaneously with the Internet spawned music genre Witch House. Artists with ungooglable names like oOoOO , Gr†ll Gr†ll or ~▲†▲~ or  XXYYXX all spoke to an online performance of self and identity.

Post-Internet is best described at source by Artie Vierkant in their 2010 essay The Image Object – Post-Internet.

The Image Object - Post-Internet
The Image Object – Post-Internet – P1

Post-Internet art is art a ‘social object’. Not concerned with the nature of the art object, but instead the nature of its reception and social presence. It was a response to the alienation from production audiences felt (and still feel), and instead insisted on a participatory view of culture at large.

The Image Object - Post-Internet
The Image Object – Post-Internet – P7

This phenomenon of communities constructing archipelagos is present all over culture today. Both as part of the cultural logics of internet ‘Fandom’ and, one of the driving forces behind the inevitable creation of Permissive IPs.

Post-Internet’s concern of social presence is perhaps clearest in early multi authored essay Club Kids: The Social Life of Artists on Facebook published by dis mag in 2012.

It discuses in depth the performance of personal brands and the rise of ‘Artists Without Art’.

Rather than sharing or enforcing a certain aesthetic value structure through a set of objects, this same information is transmitted through a direct, consumable lifestyle projected by said peer through a sequence of posts. This could account for the condition of Artists Without Art: rather than artists producing identifiable aesthetic works, their disruptive and compositional energy is used towards relational exchange. Why go to such great lengths to make and photograph a painting that will net 5 Likes when a photo of you and your friends eating 50 McChickens could net hundreds?

Why bother doing anything else online in 2020 when you can play video games on Twitch and hundreds of thousands of people will not only watch you, but also pay you to play them?

Nyanners connection to Post-Internet art as VTuber / Vinfluencer should be seen in the context of their social presence. Nyanners started a Patreon in 2014 and became a Twitch partner in October 2016.

The 100k Subs celebration video below shows Nyanners hypostatising as an online personality. A great deal of who or rather what Nyanners is, could be described as the co-creation of an avatar between the artist and her community. The feedback loops of reception and social presence are evident in the evolution the characters look and personality. Entirely defined by the artist // audience relationship.

A similar sentiment can be found in Lizzy Szabo’s – curator of the Hyperpop playlist on Spotify – definition of “Hyperpop is a genre but it’s also an artist and listening community”

Nyanners – I Am UR Leader is a foundational work in Nyanners oeuvre as an artist. The words “I’m a brand bitch, I’m a brand” when translocated from the mouth of Nicki Minaj to Nyanners’ is an important moment considering who/what they are to become.

2017 – Voice Acting & Collaborations

Nyanners was working with lots of talented people in Anime Fandom around this time. The video above shows a super cut of Nyanners voice acting work for the year 2017. Making fan dubs, but also voicing dubs of visual novels and role-play games. Also of importance to me is her appearance in Episode 27 of Hope’s Peak High – Minecraft role play.

A Youtube series that probably deserves a dedicated post. Its relevance here will be picked up again later. Whilst the idea of Machinima is 20 years old, Hope’s Peak High Minecraft Roleplay was the first online content I saw that convinced me that large scale ambitious works of self produced culture made (and set) in virtual worlds was the future of entertainment.

Also around this time Nyanners also did a 300k Subscriber Karaoke live stream where she played instruments and sang live for over an hour. I think it was a significant moment in their career as an artist and creator and the relationship with the audience.

Nyanners continued to live stream, and last year played an absolutely awful NSFW visual novel they been creating for themselves since 2017

Nyanners had been building a ‘archipelago community’ for half a decade at this point.

Leveraging one of the key cultural logics of ‘fandom’. There are several fandom wikis dedicated to Nyanners fandom and of course an extensive Know Your Meme Page. The Nyan-Verse is community co-production, full of vibes, memes and in-jokes. Deep enough to be warm and welcoming but not alienating to outsiders. It is a testament to the artist as ‘community curator’ in developing and sustaining this thriving culture around the character.

I would also add that I cannot image what it must be like to be a serial shitposter, and a woman on the Internet operating right at the centre of the Anime fandom for a decade. I Am Ur Leader came out about the time G*mer G*te was kicking off.

2020 – Apotheosis

In July of this year Nyanners reached apotheosis.

It is important to remember that the character of Nyanners exists in the chat box on Twitch and in the comments on Youtube – as much the Artist performs Nyanners onscreen.

I really recommend you watch the first 3 mins or so of the video. It not only captures – in real time – the reaction of the community to Nyanners in their final form for the first time. But also the artist appearing before her audience puppeteering a Virtual Avatar for the first time. The culmination of a work that had been set in motion nearly a decade earlier – It’s a wonderful moment.

VShojo

VSojo Founding Line Up 2020
VSojo Founding Line Up 2020

In late November of 2020 Nyanners announced they had joined VShojo.

About VShojo

VShojo is a talent-first VTuber company, dedicated to the growth and success of its members.

Founded by fans of VTuber culture, we aim to create and foster content that pushes the boundaries of VTubing and talent freedom, while maintaining deep respect for the people and companies that helped paved the way.

Founded by @TheGunrun one of Twitch’s founding team from 2011-2018. For reference, @Thegunrun was responsible for the Twitch Bob Ross marathon back in 2015.

The company is co-founded by CEO Justin “TheGunrun” Ignacio, a streamer known in the competitive gaming community whose background also includes working at Twitch, and Phillip “MowtenDoo” Fortunat, who will serve as CEO and chief technology officer, respectively.

“VTubing experienced a massive upswing this year as more creators and viewers embraced avatar-rooted entertainers,” Ignacio said in a statement. “With the VTuber medium exploding and providing a large new audience for brands, it was the perfect time to launch VShojo as a conduit between both parties. Now if brands want to enter the VTuber space and tap into the top talent, we’ve streamlined the process for everyone.”

VShojo is an interesting development in the world of livestreaming and for english speaking VTubers. Hololive/cover-corp is a similar label/agency founded in Japan in 2016 that provides similar services. Hololive was the outcome of a pivoted project from Cover’s AR capture software, after merging with the label INoNaKa (INNK) Music. We’ll save more OG Avatar discussion on Hatsune Miku and Kizuna AI for another post.


To close out the thread on the connection of VTubers with Hyperpop I leave this quote from a 2014 article on A. G. Cook and the rise of PC Music.

PC Music’s artists are characters. They’re not living entities; they’re meticulously planned and considered long-running art pieces, kinda like living installations who put out music.

Projekt Melody

Projekt Melody also announced that they were a member of VShojo’s founding roster alongside Nyannersa and others.

If you aren’t aware, Projekt Melody is a self described Hentai AI who has been streaming SFW content on Twitch since 2019 and NSFW content on Chaturbate since early 2020.

Heres a screenshot from Projekt Melody’s about page (which I can’t find the URL for anymore), and a capture from a livestream I took from sometime in Feb 2020.

Projekt Melody About Page
VTuber Projekt Melody Gif

For a while this year, Projeckt Melody was the top earning performer on Chaturbate. Sparking a huge discussion amongst sex workers around authenticity and biopolitcs. At the time of writing Projekt Melody has 168,805 followers on Chaturbate and 321,923 followers on Twitch.

Given the coming Metaverse and advancements in AI – the roadmap is clearly laid out by a commentator in this recent WIRED Article.

Melody is promising technology for porn enthusiasts. “If ProjektMelody was a truly autonomous AI, then it would be a game changer in terms of sexual economics,” xhumanist says. “She could be ‘live’ 24 hours a day. She could be in private chats with a million different men at the same time … Her real-life cam model rivals are right to feel threatened.” Despite fan confidence, no one really knows what this timeline holds. Somehow it’s hard to imagine a majority of men defecting from real women to a swaying blue-haired cartoon with an adhesive bandage across her nose. Still, the benefit of virtual avatars is that they are easily changed and personalized. There are no real barriers to characters like Melody gaining as large a following as any actual human. A spokesperson for Chaturbates says the site required Melody’s operator to prove their age, but added that anyone can livestream using sexually explicit avatars or other interactive digital images as long as they don’t break any rules. So: Expect copycats.

Projekt Melody’s inclusion in the VShojo roster is a welcome one. Especially given the emergence of digisexuality as sexual identifier.

A term first introduced via Drs. Neil McArthur and Markie L.C. Twist’s 2017 Journal of Sexual and Relationship Therapy paper, “The Rise of Digisexuality: Therapeutic Challenges and Possibilities,” a digisexual is someone whose “primary sexual identity comes through the use of technology.” So if a big part of your sexual experiences and connections with another person have come via things such as sexting or Skype, you’re probably also a digisexual.

Honestly this whole article is fantastic:

Though there are a few real world examples of this second wave shift — such as the Japanese man who married hologram Hatsune Miku in 2018 — as Dr. McArthur says, “a lot of it is still quite underground” thanks to misconceptions about identifying as a digisexual. Being digisexual is far more than just marrying a sex robot. As he explains, though they’ve found a lot of people who do identify as digisexuals during their research, they’re “almost never willing to speak publicly” about the matter thanks to the stigma attached to the concept as a whole. But where exactly does he believe this stigma comes from?

Primarily, he attributes it to a mischaracterization of those who use immersive sex tech, particularly sex robots. Because while the many sci-fi stories sexually pairing humans with robots — à la Blade Runner or Westworld — may prove that there’s always been “fascination” with the concept, on the other hand, Dr. McArthur notes that this media fixation on sex robots is also rooted in a great amount of “repulsion.”


Becoming An Avatar – The Synthespian Tech Ecosystem

Lets take a step back and looking at the ecosystem as whole including the technologies/hardware/software involved in becoming a VTuber. It’s important to recognise upfront that performing and developing a virtual avatar requires squad production, and specialist skills.

Also this technology when used for anonymity is not without its downsides. In 2018, due to a glitch in facial tracking software anime VTuber Noracat was accidentally revealed.

Before we dive in to hardware. It’s worth noting Projekt Melody’s recent Twitch Ban – after bring cropystriked by the original creator/Modeller of the avatars body.

Many of you probably are wondering about the recent Twitch ban, so I wanted to share some context about my Twitch Ban since it relates to an issue I’ve been dealing with the last several months.

A 3D modeler (DigitrevX) filed DMCA takedowns on all of my VODs, claiming he owns the copyright to my body. He had also been posting about this on twitter for weeks as well, but since I try to avoid drama, I’ve ignored it (it’s been hard). Given that he is now trying to get me de-platformed, he has escalated the issue and now I have to make a response, publicly, which includes providing all of the relevant receipts.
At some point he asked if I would be interested in a partnership deal with his company, Giga Music Group. I was interested in hearing about the deal, but at that point I hadn’t agreed to anything, only that I’d like to hear what they had to offer. Over the next few months, he still continued to help me with things, but refused compensation for it. Even though I offered to pay him often, he said it wasn’t a big deal each time.
Here are some examples:
https://imgur.com/a/WFGh5dI
Around May, Giga Music Group finally sent me their proposed offer. They wanted $40,000 a month for their services.
I didn’t want to be involved in drama. I wanted to continue streaming and having fun with my friends and my community. I’m lucky enough to have a career that I absolutely love with supportive and redonkulous friends, and he is trying to take all that away, even though I paid him for my body. That’s what the DMCA claim is for, by the way. Not the T0Ms, not one of my CB toys, not music, not for being too explicit (I’ve been good on twitch)…. But no, because digi thinks he owns me.
He thinks this, even after I paid him for it, as you can see in those invoices, even though back on December 5th, 2019, at 3:42pm, he agreed, that I OWN THE IP, HE’S JUST THE CHARACTER DESIGNER/MODELER.
That’s what he’s copyright striking me on.

The Verge wrote the incident up under the headline “WHAT HAPPENS WHEN A VIRTUAL STREAMER DOESN’T OWN HER BODY?

Projekt Melody swears her body belongs to her — the purple hair, the cat-eared bow, and all the barely there clothing that strategically covers her up. She commissioned it from an artist for $5,000 and even kept the receipts as proof. And for her thousands of fans on Twitch, this is what they see when she streams herself playing Minecraft, watching movies, or just sitting around chatting in her room.

It wasn’t until this month that she ran into a problem: the artist, alleging that Melody owed him money, filed a copyright complaint claiming that she didn’t actually own her body — he did. Melody was banned from Twitch.

It’s a strange situation, but one that could become more common. That’s because Melody is part of a growing wave of virtual streamers who broadcast using a 3D model in place of their body and face. The setup offers anonymity for the streamer and huge branding potential around the literal cartoon character they’re inhabiting. The model speaks and moves in accord with the person behind it, but viewers of the stream have no idea what that person actually looks like. They just hear her voice and see her reactions through the model that represents her. In the case of Melody, that’s a skinny anime woman with huge blue eyes, a croptop sweatshirt, and not much else. You can buy clothes, stickers, and pillows featuring her image, though the most popular form of merch seems to be posters of her in explicit poses.

I’ll park a digression in to Brud’s Lil Miquela, and Headless Brands for now but I think this shows why the Dweb providing the technical and legal plumbing for PERMISSIVE IPs is so urgent and important.

Get New Posts via Email 📨

Join 4,514 other subscribers.

Are You Still With Me?

For people still with me. – especially people learning about all this for the first time. I understand your heads may spinning after learning of an imbroglio over the ownership of digital body assets involving a person who puppets a cartoon character online.

Andy Serkis gollum gif

But I would point out that Andy Serkis’ motion captured performance as Gollum in The Lord Of The Rings was 20 years ago. In the intervening two decades, Motion Capture has come a long way.

Iron Man Mocap

The progression of the technology fuel by both movie and video games industries means Robert Downey Jr doesn’t even wear a costume on set any more.

Pirates of the Caribbean: Dead Man's Chest mocap

This Image is from the making of Pirates of the Caribbean: Dead Man’s Chest. Acting in a CGI dominated movie mainly involves wearing grey pyjamas.

There has been an explosion in VTuber activity in 2020 partly due to the pandemic, and the ease of use / developments in video streaming. See mmhmm.app and my previous ‘Beyond Entertainment’ notes. Also as we will see the price points for motion capture technology have fallen to basically free if you own an iPhone S.

The idea of VTubers has been around for a long time. In The Moon is a Harsh Mistress, by Robert Heinlein (1966) a computer called Mike presents a virtual voice and image of themselves presenting as Adam Selene.

The first Synthespian was created by Jeff Kleiser and Diana Walczak created the first digital actor for their 1988 short film Nestor Sextone for President.

Becoming a budget VTuber is as easy as easy as activating a filter on Instagram. Tools like Facerig are very easy to use – Check the recent episode of Come Internet With Me I recorded with Cade Diehm where we take a deep dive into the economics and spending in the Furry Fandom.

It’s a little difficult to break down the technology chain in to discrete sections but I’m going to try.

If you want to be more fancy than a webcam face-rig you can just strap an Iphone X to your face.

Hardware/Tracking

DIY

IPhone True Depth Camera

Apples TrueDepth camera system built in to new phones has put high fidelity motion capture into anyones hands that can afford the phone

Apple Animoji

Data from the phone camera can easily be streamed realtime in to 3D tools like Unity, iClone or Maya.

You can also use face detection from a standard webcam (better camera is better) with the tools below, there are two tools currently available on Steam

Leap Motion

Luppet is a well known/standard software tool as it was the first to integrate directly into Ultra Leap’s Leap Motion Controller for hand tracking. 3tene followed suit recently, as Leap Motion continues to become more available, I assume it will become a standard integration.

As far as I can ascertain Luppet also supports iPhone TrueDepth tracking data.

At the time of writing the USB Leap Motion Controller is available from adafruit.com for $89.9.5

Hand tracking gives the puppeteer a much higher level of control and realism when performing their characters. I can imagine it being used for so much more than just this in a puppets rigging in future.

Pro Rigs

As with everything in this world you can spend money and move up into the workhorse level equipment used by the Video Games, Advertising and Film Industry.

I actually tried a pro motion control rig out last year at a VR/AR technology demonstration day. I was super cool and very enjoyable. I do however think the DIY aproach will be the market that brings this technology in to peoples homes.

Xsens

The lead suppler of this kind of tech is Xsens. The kit will run to about $5000. But it is very powerful and movie grade.

They also supply tracking software called Xsens MVN Animate Pro that integrates seamlessly with their Mocap hardware. This is another $1500 on top of the kit. However it plugs in to almost all major 3D environment programs like MAYA, Unity, Unreal, Blender, Fabric Engine etc

Xsens also currently is the most sophisticated mocap tool for creating 3D avatars the example on the right is the most recent example I’ve seen (Via @simondlr).

The rest of this article focuses on the creation of live2D models and rigs. However the mocap tech is still the same.

There are also AR/VR finger and hand capture hardware tools like the Manus Prime originally designed for technical VR work (one of their products provides haptic feedback to the fingers). It works with HTC Vive etc. But their products are increasingly being generalised into wider motion tracking pipelines.

There are all sorts of weird and wonderful gadgets for sale in this space. The commercial grade price points are coming down to consumer price points rapidly. Check out UK gear supplier Target3d.

I should point out that currently all the technologies above require a blazing fast PC with a powerful graphics card. As GPU prices fall and iPhones get refreshed it will open up opportunities to more and more folks.

Artware

Introducing the term Artware. As a field/market that has a lot of oppertunity as Metaverse like AR experiences continue to grow.

Artware is the production of virtual assets.

A VTubers body is known as a “Model” a set of assest designed either as an illustration or 3D model. As with all IRL puppets or Animation the model must then be ‘rigged’ for performance, movement etc.

VRoid Studio

The most popular emerging tool to build your Anime character is VRoid. It’s a one stop shop and does *everything*.

It’s honestly one of the most impressive pieces of illustration software I’ve ever seen.

VRoid Studio is an application for Windows and Mac to create 3D models of humanoid avatars (characters). VRoid Studio can be used for free even by people with no modeling knowledge; its intuitive controls make creating full-scale 3D characters easier than ever. 3D models created by users can be used for various purposes, commercial and non-commercial, including the adoption of such 3D characters as avatars in VR/AR contents.

VRoid does absolutely everything you need to create a fully rigged model / body for a VR/AR Avatar. It is a glimpse of the future of what Apples Animojis may eventually turn into.

Taking cues from the online gaming world Vroid are also building out an assets store and creator ecosystem where you can buy and sell items for Avatars. Indeed they have already partnered with both Levis and Chloma to create virtual collections/ranges.

With regard to the Dweb and the Metaverse. Assets made in the Unity game engine: clothes, outfits hats etc can be given the ‘properties’ of property using the Enjin SDK to assign NFT’s to items.

Enjin is a blockchain game development platform that allows game developers to include NFT assets through the Enjin SDK.

The Enjin SDK removes the complexity of the blockchain and provides a simple, streamlined API interface that game developers can use to build their own games.

With the Enjin Blockchain SDK extension, game developers that use the Unity engine can now create NFT assets and provide a marketplace directly from the “Unity Runtime API”.

Outfits, and items that don’t give game play advantage are how smash hit games like Fortnite and Roblox fund themselves. It won’t be long until Vinfluencers like Lil Miquella are selling the *ACTUAL CUSTOM JACKET* they wore in a photoshoot for Instagram etc. Or buying the actual Sword that someone used to win a world championship e-sports event.

Soon I think everything for sale on superrare.co and async.art will be portable to a game engine so you can enjoy the art on the wall of your house in the Metaverse.

Bespoke Creation

Whilst exploding as a form during 2020 the VTuber ecosystem is actually quite mature. Whilst Nyanners reached apotheosis this year as a fully rigged virtual avatar – they only got there by collaborating with a team of people. Many of the most established teams in the ecosystem are now connected to or in orbit of VShojo.

Nyanners was illustrated by @Nia_0703 a Chinese based artist who has forged a career producing Live 2D avatars. I don’t read Chinese and can’t for the life of me figure out the query parameters too embed a Bilibili video. If you want to see @Nin_0703 at work click the gif to the right.

@Nin_0703 works regularly with a Live2D creator guild Iron Vertex.

@Nin_0703 Gif

Iron Vertex Creator Guild

Iron Vertex was found by Brian in 2018, with the purpose of gathering and training Live2D designers. As the team grew, Brian has been working on training members and assisting them to grow their careers as a Live2D freelancer. As of late 2019, Live2D Ltd has reached out to Brian and IV, in a collaboration to work a series of tutorials to standardize Technical Live2D.Other than Brian, Iron Vertex currently consists of 5 active members and 4 instructors.

Illustrating a virtual avatar from scratch, then rigging the puppet takes a lot of skilled work.

Skills that are currently found in the commercial world and well remunerated. I have several friends that make their living modelling/ rigging/ animating consumer products in Unity as its easier to whip up adverts in a virtual environment than take new product photos all the time for the endless content mills of social media.

Iron Vertex are one of the most interesting elements of the VTuber ecosystem – and its pretty mind blowing.

They are a straight up Guild. They don’t call themselves a company, instead they describe themselves as a ‘North America based live2d guild community’.

To quote Mat Dryhurst’s 2019 piece DAO Guilds: Establishing Territory

Guild structures predate union structures by centuries, and can loosely be described as an association of decentralized artisans — a slight distinction from the role of the union as the standard-setting entity for workers employed by a particular industry or company. The Guild aspires to establish the dominion of a field, in contrast to the ambitions of unions that are largely limited to agreeing fair working conditions and compensation, under the dominion of employers. Guilds regulate access to a trade in accordance with collectively determined protocols of access and conduct. Guilds are the manifestation of the greatest fears of union critics. Guilds are cartels.

Iron Vertex provide tutorials, training programs and apprenticeships to people wanting to learn how to do the work. Though which they gain access the specialist knowledge (hard won no doubt) required to rig a Live2D virtual puppet. They also involved in setting industry standards around what to expect for various levels of work/rigging for Live2D avatars. Compare a Tier 1 Avatar to Tier 5 Avatar. Note: the Nero Bride character uses all the same tech as a Vuber but was created for a visual / interactive novel video game.

  • Head Tilts
  • Upper Body Tilts
  • Full Body Tilts
  • 1 Set of Hotkey Toggled Animations
  • Head Tilts
  • Head Turn (Left/Right)
  • Full Body Turn (Left/Right)
  • Full Body Tilts
  • Full Body Lean Forward
  • Game Sprite Usage
  • 6+ Animation Sets

Iron Vertex seemingly have a partnership with VShojo. Given the power but ‘locked in’ aesthetics of a tool like VRoid, there is huge room for bespoke services and teams wanting to work together to create and do new things. I wish the Iron Vertex community the best of luck in the future, operating as a guild and setting appropriate access, conduct, quality and pricing in this emerging space.

Software

Finally we have software. The industry standard is OBS or Open Broadcaster Software.

OBS Studio Screenshot

OBS is a free and open-source cross-platform streaming and recording program. It is an understatement to say that OBS is driving innovation in the live-streaming sphere everywhere. It so important that Canon’s recent Webcam Utility Software plugs right in to OBS also Discord.

Whilst most people viewers and creators in the live streaming as OBS provides real-time source and device capture, scene composition, encoding, recording, and broadcasting. I first used it back in 2015 helping the company I worked for livestream events with a multi-camera set.

The pandemic like video chat and live streaming in general has pushed OBS to a whole new user base – corporate users. Users looking for slicker presentations and quality of production in their video calls. OBS provides preset streaming settings for Skype, Youtube, Twitch etc.

OBS is literally a TV grade studio tool available to users for free. Hugely important and its worth following developments of the software. as their feature set will (I think) begin to dictate the kinds of entertainment and experiences that people can produce.

Note for musicians: VDMX also integrates seamlessly into OBS studio!

6 Growth Markets For The Scene

I’ve been playing with bits of this tech since Lockdown 1 started in the UK after a brief foray in to VDMX/OBS/Green Screen in 2018. If anyone would like to collaborate with me on building my robot alias J-TP3 drop me a line. I’d like to animate all my podcasts programmatically as a VTuber.

1. Non-Anime Aesthetics

The genre is currently dominated by anime aesthetics. I’d like to see it break out and explore new and different styles of animation and representation, ASAP.

Minecraft’s ‘Virtual Insanity’ skin pack is a good start but we need artists of all kind to be pushing the envelope of whats possible. Early 90’s digital art and DeepDream seem good touch points for the mood board.

As I said recently, Why can’t I be a swirling vortex of texture and colour and sounding trumpets when I hang out with my colleagues?

A move away from representation and realism would unlock so many possibilities of what Synthespians could be.

I think there is huge oppertunity for VTubers to experiment in this space. And explore ever increasing levels of weird abstraction, I think the most popular parts of the Metaverse are going to be a strange phantasmagoric places that operate more like evolving dreams than a simulation.

Minecraft Virtual Insanity skin pack

2. Adoption of Improv Techniques

I think there are a lot of performance techniques that could be adopted from improvisational theatre and integrated in to the Artist/Audience co-creation feedback loop. It seems to me that there is an obvious connection between Keith Johnstone’s Mask and mirror work and Mocap-ed animated live rigs.

The ability to ‘become’ something else and inhabit it completely is a huge skill that could be leveraged in the world of future Synthespians. Applied improvisation is already useful in many places outside of the stage. Multiple people onscreen at once riffing off one another as animated 2D characters could create something really special I feel. Something new/not seen before could emerge quickly.

Yet again the right wing media who have been hammering / trolling young people with headlines like ‘Is drama a ‘real’ subject’ or media studies being useless, and Theatre being dropped from school curricular etc is shown up. All those liberal arts degrees are worth 100x more than bullshit subjects like business studies.

3. Revivification of the ‘Drawing room play’

The Drawing Room Play is a type of play developed during the Victorian era, in which the actions take place in one room – designed to be reenacted in the drawing room of a home.

I like Drawing Room Plays for their focus on dialogue and dramatic monologue. With the destruction and neglect of creative industries, with seemingly no hope of rescue from the governments who distrust the arts. Moving improv and traditional theatre online, a move towards a ‘Zoom Room Play’ and live online theatre with specialist unity / game engine sets as backgrounds seems promising.

Whilst not a single room play, one of the best things I watched online during Lockdown 1 was a complete Zoom performance of Shakespeare’s As You Like It by Theatre Z. Involving many denizens of a forum I frequent. It was complete online theatre with costumes, custom commissioned Zoom backgrounds for each scene, music and sound effects. It was absolutely fantastic and very enjoyable.

As 2DLive technologies mature and the cost comes down, I can imagine a market for ‘theatre packs’ for plays and performances where all the tooling and models can be rented, bought or borrowed by dramatic troupes.

With virtual backgrounds being built in video game engines, there is absolutely no upper limit to the size and scope of animated drama. I wan’t to see plays set on space stations or historical dramas set in Ancient Rome, or even the paleolithic.

4. Online LARP & TTRPG adoption

Is I explored in Permissive Ips the trend is moving towards games, experiences and story worlds with audience agency. The deletion of performer spectator divide can become further accelerated with Avatar technologies. I’ve played a few Zoom RPGs with Roll20 and its been fun. We also used Tabletop Audio for flavour and immersion etc.

Currently the biggest TTRPG broadcasting squad is Critcal Role who are currently exploring an animated version of their show.

It seems to me that this is an expensive cross over period. Squad production of a Streamed TTRPG with Animated Avatars and Mocap rigs only seem like a matter of time. Far cheaper than traditional animation.

I can imagine outside of the players and the DM, there being an artist and live VJ who is working/reacting in realtime to the drama unfolding onscreen. a ‘World Master’ who works in concert with the DM/GM.

I think these technologies and workflows will develop alongside DIY theatre groups also working in this space. I also see no reason at all why these emerging avatar entertainments couldn’t also be performed live in front of an audience with a big screen behind them whilst simultaneously live-streaming to the web.

I also see no reasons why these sort of avatar/environment packs couldn’t be made for sale alongside rulesets / background books on DriveThruRPG.

5. Multi Person Rigs

The War Horse puppet created by the Handspring Puppet Company was the single most incredible thing I’ve ever seen onstage.

Right now, all 2DLive / Avatars I’ve seen have been operated by a single person. This is – I suppose – because of fact that the technology involved (and price point) to DIY this has only just converged in the last 18months.

But I can’t help but think about what could be possible with more complicated rigs and multiple operators.

Grand dragons, horses, aliens, could all puppeteered remotely on screen by multi-person rigs.

Animation is still one of the most expensive parts of making cartoons etc. With specialist puppeteers who could either pre-record or act out strange creatures or characters in realtime the potential for puppetry to be a major art form in the metaverse and 21C is a big one.

I would also like to see the backgrounds and virtual environments being made characters in their own right. I think there is a lot of potential for interactive set design, or even psychedelic sets that are generated from the movements/interactions of a puppeteer/actor. I can’t imagine what this would look like in practice but I certainly can imagine it, and encourage others to give it a go.

Perhaps prototyped in a 2 person coop video game? One person plays the main character, and the other manipulates the environment.

6. Squad Drama’s

Lastly combing this all together: I think very soon we are going to see season long animated shows, made by Synthespian troupe/squads etc. Consider the Hope’s Peak High Minecraft Roleplay I mentioned much further up this post. High quality long running TV Shows very soon will be made by people at home co-created by its fans.

I have been very impressed by the Warhammer 40,000 communities fan production of Audio Dramas over at Cold Open Stories for example. It is only a small jump in my mind to making super high quality web shows.

As we move in to the Metaverse and virtual environments become more commen place. Either in a game world or just behind a VTuber on a livestream. I think LARPing *to an audience* will also become a popular form of entertainment.

Facebook is currently experimenting with a new form of reality game show called Rival Peak:

Genvid Technologies and Pipeworks Studios are unveiling Rival Peak, a new kind of interactive experience that is partly a game and partly a reality show. The audience consists of real people, but the characters they’re watching are not. If you’re confused about that, bear with us, as it’s a pretty cool idea.

The quasi-reality show stars 12 artificial intelligence characters who are contestants in a Survivor-like competition set in an animated Pacific Northwest. The live audience can influence the outcome of the contest by grinding away at tasks and helping their favorite characters. The show will run 24 hours a day, seven days a week, and its host is Will Wheaton (Star Trek: The Next Generation, TableTop board game show on YouTube). It will be available to play, or watch, if you will, on Facebook Watch.

It is a cross between the TV show Big Brother, Ian Cheng’s Emissaries, The Sims and the X Factor. With the audience taking part in a characters activites in the screen similar to the way that twitch played Pokemon.

I wish Navok and the team the best of luck. It’s a really ambitious project and full of ground breaking technology

Rival Peak is built in Unity, a popular game engine, but Genvid’s technology also supports Epic’s Unreal Engine. Genvid has developed software to timecode gameplay and video streams and then synchronize them. Genvid also developed a web interface and a full cloud gaming backend supported by Amazon Web Services instances. It all allows video streams to become truly interactive (click the stream and things happen within the underlying content, be it a video game or a sports statistic.).

But I can’t help shake the feeling that the future looks like a mix of Avatar based story telling, perhaps embedded in generative worlds alongside audience co-creation.

I’m interested in how the next few years play out.

Thanks to: Alex from commonknowledge.coop, RIVAL STRATEGY and Ben vickers for help and feedback with this post.

Dimensino is an ongoing series subscribe if you want to follow along.
You’ll also get posts about my weekly 5min podcast, my webshow, and my weeknotes.

#SUPPORTNET

Get New Posts via Email 📨

Join 4,514 other subscribers.

I’m hoping to support recurring payments via ETH very soon. Once Dweb Memberships are wordpress compatible, I may consider exclusive content/keybase chat room etc.

Work With Me

If you would like to hire me to puzzle this sort of thing though for you in a professional capacity – Get In Touch!



Leave a Comment 💬

Click to Expand

6 responses to “Synthespian Escalation”

  1. […] no mercado produtos que rastreiam os utilizadores, de uso obrigatório nalgumas instituições. Synthespian Escalation: Uma longa leitura, mas que vale bem a pena. Um estado da arte, com pistas sociais, estéticas e […]

  2. […] important architectural elements to built environments. They will soon be powering adverting, TVshows, theme parks, theatres and more. Which is why these entertainment partners lined up to work with […]

  3. […] year I posted a longread about Vtubers on my blog that’s quickly closing in on 10k views. (Link in the show notes). So this is the […]

  4. […] had posted this just after my long deep dive on Vtubers so I had them on the brain. There’s lots here about NPC/AI driven characters, audience co-created […]

  5. […] Synthespian Escalation – An exploration of the current state of the VTuber industry. One of the key insights is the feedback loop of audience co-creation. I do think we may see the first VInfluencer emerge – built on a Permissive Intellectual Protocol of some kind – in 2021. But it won’t be anywhere near as advanced Liu’s vision. […]

Leave a Reply

Your email address will not be published. Required fields are marked *