Notes:
This 72x item list represents an extraction of all GitHub links from the Virtual Beings Facebook group in January 2021.
- audio sequence
- azure kinect
- different model
- face generation
- facial motion capture system
- full day workshop
- gesture generation
- good talking-head video generation
- implementation
- intelligent agent
- lip generation
- lip generation model
- lip reading
- lip reading model
- lip video data
- microsoft rocketbox avatar
- mixed sequence
- morphable model
- motion model
- multimodal learning
- multiple speaker
- music conditioned dance generation
- narrative-guided script generation
- official tensorflow implementation
- order model
- paired data
- person image generation
- possible speaker
- probabilistic controllable motion synthesis
- python implementation
- pytorch implementation
- real avatar
- reproducing video model
- sensemaking system
- shared data
- speech-driven generation
- speech generation
- state-of-the-art model
- style-controllable speech-driven gesture synthesis
- talking lip
- unlabeled data
- unlabeled lip
- unlabeled text
- user avatar
- virtual agent
See also:
Virtual Beings & Artificial Intelligence | Virtual Beings 2020 (141x) | Virtual Beings Meta Guide
[72x Jan 2021]
- 2nd workshop on nlg for hri .. full day workshop at inlg 2020december 18, 2020, (virtual)
- a workshop co-located with lrec 2020 – onion 2020 .. website for the onion 2020 workshop
- audio2gestures | taras kucherenko .. aud2repr2pose: analyzing input and output representations for speech-driven gesture generation
- custom skin compiler for minecraft .. web site created using create-react-app
- duallip: a system for joint lip reading and generation .. lip reading aims to recognize text from talking lip, while lip generation aims to synthesize talking lip according to text, which is a key component in talking face generation and is a dual task of lip reading. in this paper, we develop duallip, a system that jointly improves lip reading and generation by leveraging the task duality and using unlabeled text and lip video data. the key ideas of the duallip include: 1) generate lip video from unlabeled text with a lip generation model, and use the pseudo pairs to improve lip reading; 2) generate text from unlabeled lip video with a lip reading model, and use the pseudo pairs to improve lip generation. we further extend duallip to talking face generation with two additionally introduced components: lip to face generation and text to speech generation. experiments on grid and tcd-timit demonstrate the effectiveness of duallip on improving lip reading, lip generation, and talking face generation by utilizing unlabeled data. specifically, the lip generation model in our duallip system trained with only 10% paired data surpasses the performance of that trained with the whole paired data. and on the grid benchmark of lip reading, we achieve 1.16% character error rate and 2.71% word error rate, outperforming the state-of-the-art models using the same amount of paired data.
- embodied multimodal learning workshop | iclr 2021 .. embodied multimodal learning workshop | iclr 2021
- face-vid2vid .. one-shot free-view neural talking-head synthesis for video conferencing
- first order model .. first order motion model for image animation
- github – 3d-morphable-models/curated-list-of-awesome-3d-morphable-model-software-and-data .. the idea of this list is to collect shared data and algorithms around 3d morphable models. you are invited to
- github – ai4r/gesture-generation-from-trimodal-context .. speech gesture generation from the trimodal context of text, audio, and speaker identity (siggraph asia 2020)
- github – broekens/gamygdala .. emotion engine for games including phaser plugin, based on the gamygdala approach (with an eyewink to game and amygdala)
- github – danmcduff/avatarsim .. an avatar simulation for airsim (https://github.com/microsoft/airsim).
- github – daod/scriptwriter .. acl 2020: scriptwriter: narrative-guided script generation
- github – digitalphonetics/adviser .. adviser is a flexible framework to encourage task-oriented dialog system research & development
- github – ensemble-engine/ensemble .. a rules-based ai framework for social simulation.
- github – facebookresearch/pifuhd .. high-resolution 3d human digitization from a single image.
- github – facebookresearch/pytorch3d.. pytorch3d is fair’s library of reusable components for deep learning with 3d data
- github – facebookresearch/slowfast .. pyslowfast: video understanding codebase from fair for reproducing state-of-the-art video models.
- github – facebookresearch/svoice .. we provide a pytorch implementation of the paper voice separation with an unknown number of multiple speakers in which, we present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. the new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. a different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.
- github – gaips-inesc-id/fatima-toolkit .. fatima toolkit is a collection of tools/assets designed for the creation of characters with social and emotional intelligence.
- github – getnamo/tensorflow-ue4 .. tensorflow plugin for unreal engine 4.
- github – google/mediapipe .. cross-platform, customizable ml solutions for live and streaming media.
- github – ha0tang/xinggan .. [eccv 2020] xinggan for person image generation.
- github – happyocean/full_body_photogrammetry .. full body 3d scan, photogrammetry for real avatar
- github – ircss/procedural-painting .. procedural painting algorithms in unity 3d with compute shaders based on genetic evolution algorithms
- github – isguser/vaif .. the virtual agent interaction framework (vaif) is a unity package that allows users to create intelligent agents with minimal effort.
- github – kelaberetiv/tagui .. command-line tool for robotic process automation (rpa) by ai singapore
- github – lattas/avatarme .. public repository for the cvpr paper avatarme.
- github – lelechen63/talking-head-generation-survey .. official github repo for paper “what comprises a good talking-head video generation?: a survey and benchmark”
- github – lelechen63/talking-head-generation-with-rhythmic-head-motion .. pytorch implementation for audio driven talking-head video synthesize.
- github – makehumancommunity/makehuman-plugin-for-blender .. plugin for makehuman and related programs.
- github – michaelglu/smartedgepaper .. edge-based provisioning of holographic content for contextual and personalized augmented reality
- github – microsoft/botframework-composer .. dialog creation and management for microsoft bot framework applications
- github – microsoft/movebox-for-microsoft-rocketbox .. movebox is a toolbox to animate the microsoft rocketbox avatars using motion captured (mocap). motion capture is performed using a single depth sensor, such as azure kinect or windows kinect v2. our toolbox enables real-time animation of the user’s avatar by converting the transformations between systems that have different joints and hierarchies. additional features of the toolbox include recording, playback and looping animations, as well as basic audio lip sync, blinking and resizing of avatars. our main contribution is both in the creation of this open source tool as well as the integration of movebox with kinect v2 and azure kinect, as well as the interfacing with the microsoft rocketbox avatars.
- github – msubzero2000/jarvis-generative-chatbot .. jarvis generative chatbot
- github – nmchan/waidayo .. facial motion capture system for 3d vtuber that supports vrm
- github – nvidia/tacotron2 .. tacotron 2 – pytorch implementation with faster-than-realtime inference
- github – nvlabs/stylegan2 .. stylegan2 – official tensorflow implementation.
- github – oist-cnru/nrl .. neural robotics library
- github – okdalto/visualizemnist .. this project is real-time visualization of a network recognizing digits from user’s input. – okdalto/visualizemnist
- github – polakowo/gpt2bot .. your new telegram buddy powered by transformers.
- github – r9y9/deepvoice3_pytorch .. pytorch implementation of convolutional neural networks-based text-to-speech synthesis models
- github – royorel/lifespan_age_transformation_synthesis .. lifespan age transformation synthesis code.
- github – rudrabha/wav2lip .. this repository contains the codes of “a lip sync expert is all you need for speech to lip generation in the wild”, published at acm multimedia 2020.
- github – rustyoldrake/character-cartridges-embodied-identity .. poc for sensemaking systems with emotional & anthropomorphic traits – and synthetic identity
- github – rwsarmstrong/strongtrack .. a python tool with facial landmark annotation and coefficient finder
- github – scenemaker/visualscenemaker .. visual scene maker master version
- github – simonalexanderson/stylegestures .. this repository contains code for reproducing the papers “moglow: probabilistic and controllable motion synthesis using normalising flows” and “style-controllable speech-driven gesture synthesis using normalising flows”.
- github – snhwang/unity-watson-stt-assistant-tts .. chatbot on unity using ibm watson speech-to-text, assistant, and text-to-speech
- github – tangtan/istoryline.js .. for more information about istoryline.
- github – tensorflow/tfjs-models .. pretrained models for tensorflow.js.
- github – towneszhou/pysigma .. python implementation of the sigma cognitive and graphical architecture grounded on pytorch tensor processing.
- github – unity-technologies/ml-agents .. unity machine learning agents toolkit.
- github – uvacw/cart .. conversational agent research toolkit.
- github – voxml/voxsim .. unity3d simulator for voxml platform.
- github – watson-developer-cloud/unity-sdk .. video_game: unity sdk to use the ibm watson services.
- github – wuminye/nhr .. pytorch implementation of multi-view neural human rendering.
- github – yemount/pose-animator .. pose animator takes a 2d vector illustration and animates its containing curves in real-time based on the recognition result from posenet and facemesh.
- github – zhuhao-nju/facescape .. a large-scale high quality 3d face dataset and detailed riggable 3d face prediction (cvpr2020)
- home – lnxusr1/karen wiki – github .. a synthetic human written in python.
- learn to dance with aist++: music conditioned 3d dance generation .. project page for ai choreographer
- moglow .. probabilistic and controllable motion synthesis using normalising flows
- nanogenmo .. spend the month writing code that generates a 50k word novel, share the novel & the code at the end
- neural voice puppetry .. given an audio sequence of a source person or digital assistant, we generate a photo-realistic output video of a target person that is in sync with the audio of the source input.
- overview – corenlp overview | corenlp link search .. nlp processing in java
- prophet | forecasting at scale. .. prophet is a forecasting procedure implemented in r and python. it is fast and provides completely automated forecasts that can be tuned by hand by data scientists and analysts.
- rignet .. neural rigging for articulated characters
- special session on situated dialogue with virtual agents and robots | robodial .. part of sigdial 2020virtual
- stanford-cs221 .. artificial intelligence: principles and techniques
- synthetic expressions are better than real for learning to detect facial actions ..
- tfjs-models/facemesh at master – tensorflow/tfjs-models – github .. pretrained models for tensorflow.js.
- vmagicmirror .. mirror-like software to reflect yourself as vrm avatar, without any special devices.