1687-6180-2007-047970 1687-6180 Research Article <p>3D-Audio Matting, Postediting, and Rerendering from Field Recordings</p> GalloEmmanuelEmmanuel.Gallo@sophia.inria.fr TsingosNicolasNicolas.Tsingos@sophia.inria.fr LemaitreGuillaumeguillaume.lemaitre@sophia.inria.fr

Rendu & Environnements Virtuel Sonorisés, Institut National de Recherche en Informatique et en Automatique, Sophia-Antipolis Cedex 06902, France

Centre Scientifique et Technique du Bâtiment, Sophia-Antipolis Cedex 06904, France

EURASIP Journal on Advances in Signal Processing 1687-6180 2007 2007 1 047970 http://asp.eurasipjournals.com/content/2007/1/047970 10.1155/2007/47970
1520061192006241120062622007 2007Gallo et al.This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

We present a novel approach to real-time spatial rendering of realistic auditory environments and sound sources recorded live, in the field. Using a set of standard microphones distributed throughout a real-world environment, we record the sound field simultaneously from several locations. After spatial calibration, we segment from this set of recordings a number of auditory components, together with their location. We compare existing time delay of arrival estimation techniques between pairs of widely spaced microphones and introduce a novel efficient hierarchical localization algorithm. Using the high-level representation thus obtained, we can edit and rerender the acquired auditory scene over a variety of listening setups. In particular, we can move or alter the different sound sources and arbitrarily choose the listening position. We can also composite elements of different scenes together in a spatially consistent way. Our approach provides efficient rendering of complex soundscapes which would be challenging to model using discrete point sources and traditional virtual acoustics techniques. We demonstrate a wide range of possible applications for games, virtual and augmented reality, and audio visual post production.

Spatial Sound and Virtual Acoustics

[1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374]

<p>3-D sound spatialization using ambisonic techniques</p>MalhamDGMyattAComputer Music Journal1995194587010.2307/3680991Soundfield http://www.soundfield.com/ <p>Plenoptic stitching: a scalable method for reconstructing 3D interactive walkthroughs</p>AliagaDGCarlbomIProceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01), August 2001, Los Angeles, Calif, USA443450<p>Unstructured lumigraph rendering</p>BuehlerCBosseMMcMillanLGortlerSCohenMProceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01), August 2001, Los Angeles, Calif, USA425432<p>View interpolation for image synthesis</p>ChenSEWilliamsLProceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '93), August 1993, Anaheim, Calif, USA279288<p>Tour into the picture: using a spidery mesh interface to make animation from a single image</p>HorryYAnjyoK-IAraiKProceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97), August 1997, Los Angeles, Calif, USA225232<p>Compositing digital images</p>PorterTDuffTProceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '84), July 1984, Minneapolis, Minn, USA253259YewdallDLPractical Art of Motion Picture SoundFocal Press, Boston, Mass, USA22003<p>The decca tree - it's not just for stereo anymore</p>StreicherR http://www.wesdooley.com/pdf/Surround_Sound_Decca_Tree-urtext.pdf The New Stereo SoundbookAudio Engineering Associate, Pasadena, Calif, USAStreicher R, Everest FA21998<p>Ambisonics encoding of other audio formats for multiple listening conditions</p>DanielJRaultJ-BPolackJ-DProceedings of the 105th Convention of the Audio Engineering Society, September 1998, San Francisco, Calif, USApreprint 4795<p>Ambisonics in multichannel broadcasting and video</p>GerzonMAJournal of the Audio Engineering Society19853311859871<p>Ambisonic surround sound FAQ (version 2.8)</p>LeeseMJ1998 http://members.tripod.com/martin_leese/Ambisonic/ <p>Applications of a 3-D microphone array</p>MerimaaJ112th AES Convention, May 2002, Munich, Germanypreprint 5501<p>A new comprehensive approach of surround sound recording</p>LaborieABrunoRMontoyaSProceedings of the 114th Convention of the Audio Engineering Society, March 2003, Amsterdam, The Netherlandspreprint 5717<p>A comparative study of 3D audio encoding and rendering techniques</p>JotJ-MLarcherVPernauxJ-MProceedings of the AES 16th International Conference on Spatial Sound Reproduction, April 1999, Rovaniemi, Finland<p>Theory and design of high order sound field microphones using spherical microphone array</p>AbhayapalaTDWardDBProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '02), May 2002, Orlando, Fla, USA219491952<p>High spatial resolution multi-channel recording</p>LaborieABrunoRMontoyaSProceedings of the 116th Convention of the Audio Engineering Society, May 2004, Berlin, Germanypreprint 6116<p>Spherical microphone arrays for 3D sound recording</p>MeyerJElkoGAudio Signal Processing for Next-Generation Multimedia Communication SystemsKluwer Academic, Boston, Mass, USA(Arden) Huang Y, Benesty J2004chapter 2<p>Acoustic control by wave field synthesis</p>BerkhoutAJde VriesDVogelPJournal of the Acoustical Society of America19939352764277810.1121/1.405852<p>Spatial sound-field reproduction by wave-field synthesis</p>BooneMMVerheijenENGvan TolPFJournal of the Audio Engineering Society1995431210031012<p>The plenacoustic function and its sampling</p>AjdlerTVetterliMProceedings of the 1st IEEE Benelux Workshop on Model Based Processing and Coding of Audio (MPCA '02), November 2002, Leuven, Belgium<p>Toward sound-based synthesis: the far-field case</p>DoMNProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '04), May 2004, Montreal, Que, Canada2601604<p>The lumigraph</p>GortlerSJGrzeszczukRSzeliskiRCohenMFProceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96), August 1996, New Orleans, La, USA4354<p>Light field rendering</p>LevoyMHanrahan PProceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96), August 1996, New Orleans, La, USA3142<p>Design and applications of a data-based auralization system for surround sound</p>HorbachUKaramustafaogluAPellegriniRMackensenPTheileGProceedings of the 106th Convention of the Audio Engineering Society, May 1999, Munich, Germanypreprint 4976<p>Comparison of data and model-based simulation algorithms for auditory virtual environments</p>PellegriniRS106th Convention of the Audio Engineering Society, May 1999, Munich, Germanypreprint 4953BregmanASAuditory Scene Analysis, The Perceptual Organization of SoundMIT Press, Cambridge, Mass, USA1990<p>Binaural cue coding—part I: psychoacoustic fundamentals and design principles</p>BaumgarteFFallerCIEEE Transactions on Speech and Audio Processing200311650951910.1109/TSA.2003.818109<p>Binaural cue coding—part II: schemes and applications</p>FallerCBaumgarteFIEEE Transactions on Speech and Audio Processing200311652053110.1109/TSA.2003.818108<p>Spatial impulse response rendering</p>MerimaaJPulkkiVProceedings of the 7th International Conference on Digital Audio Effects (DAFx '04), October 2004, Naples, Italy139144<p>Directional audio coding in spatial sound reproduction and stereo upmixing</p>PulkkiVProceedings of the 28th AES International Conference, June 2006, Pitea, Sweden<p>Survey of sparse and non-sparse methods in source separation</p>O'GradyPDPearlmutterBARickardSTInternational Journal of Imaging Systems and Technology2005151183310.1002/ima.20035<p>A tentative typology of audio source separation tasks</p>VincentERodetXRöbelAProceedings of the 4th International Symposium on Independent Component Analysis and Blind Signal Separation (ICA '03), April 2003, Nara, Japan715720<p>Sparse sources are separated sources</p>RickardSProceedings of the 14th Annual European Signal Processing Conference, September 2006, Florence, Italy<p>Efficient coding of natural sounds</p>LewickiMSNature Neuroscience20025435636310.1038/nn83111896400<p>Independent component analysis. A new concept?</p>ComonPSignal Processing199436328731410.1016/0165-1684(94)90029-9<p>Blind extraction of dominant target sources using ICA and time-frequency masking</p>SawadaHArakiSMukaiRMakinoSIEEE Transactions on Audio, Speech and Language Processing200614621652173<p>Blind separation of disjoint orthogonal signals: demixing N sources from 2 mixtures</p>JourjineARickardSYilmazOProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '00), June 2000, Istanbul, Turkey529852988<p>Blind separation of speech mixtures via time-frequency masking</p>YilmazÖRickardSIEEE Transactions on Signal Processing20045271830184710.1109/TSP.2004.828896<p>Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppression and re-panning applications</p>AvendanoCProceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA '03), October 2003, New Paltz, NY, USA5558<p>Audio interpolation</p>RadkeRRickardSProceedings of the AES 22nd International Conference on Virtual, Synthetic and Entertainment Audio (AES22 '02), June 2002, Espoo, Finland5157<p>An auto-calibration method for unattended ground sensors</p>MosesRLKrishnamurthyDPattersonRProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '02), May 2002, Orlando, Fla, USA329412944FaugerasOThree-Dimensional Computer Vision: A Geometric ViewpointMIT Press, Cambridge, Mass, USA1993MooreBCJAn Introduction to the Psychology of HearingAcademic Press, New York, NY, USA4th1997<p>The fusion of distributed microphone arrays for sound localization</p>AarabiPEURASIP Journal on Applied Signal Processing20032003433834710.1155/S1110865703212014<p>Microphone arrays for video camera steering</p>(Arden) HuangYBenestyJElkoGWAcoustic Signal Processing for TelecommunicationKluwer Academic, Boston, Mass, USA2000239259chapter 11<p>The generalized correlation method for estimation of time delay</p>KnappCHCarterGCIEEE Transactions on Acoustics, Speech, and Signal Processing197624432032710.1109/TASSP.1976.1162830<p>Two decades of array signal processing research: the parametric approach</p>KrimHVibergMIEEE Signal Processing Magazine1996134679410.1109/79.526899<p>Multiple emitter location and signal parameter estimation</p>SchmidtROIEEE Transactions on Antennas and Propagation198634327628010.1109/TAP.1986.1143830<p>Acoustic source localization and beamforming: theory and practice</p>ChenJCYaoKHudsonREEURASIP Journal on Applied Signal Processing20032003435937010.1155/S1110865703212038DiBiaseJHSilvermanHFBransteinMSMicrophone Arrays, Signal Processing Techniques and ApplicationsSpringer, New York, NY, USA2001chapter 8<p>Enhanced sound localization</p>MungamuruBAarabiPIEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics20043431526154010.1109/TSMCB.2004.826398<p>Time delay estimation in room acoustic environments: an overview</p>ChenJBenestyJ(Arden) HuangYEURASIP Journal on Applied Signal Processing2006200619 pages<p>Estimation of wavefront arrival delay using the cross-power spectrum phase technique</p>RabinkinDVRenomeronRJFrenchJCFlanaganJL132nd Meeting of the Acoustical Society of America, December 1996, Honolulu, Hawaii, USA<p>Performance of GCC- and AMDF-based time-delay estimation in practical reverberant environments</p>ChenJBenestyJ(Arden) HuangYEURASIP Journal on Applied Signal Processing200520051253610.1155/ASP.2005.25<p>New direct approaches to robust sound source localization</p>RuiYFlorencioDProceedings of International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA1737740<p>Acoustic source localization in distributed sensor networks</p>AjdlerTKozintsevILienhartRVetterliMProceedings of the 38th Asilomar Conference on Signals, Systems and Computers, November 2004, Pacific Grove, Calif, USA213281332SametHThe Design and Analysis of Spatial Data StructuresAddison-Wesley, Reading, Mass, USA1990<p>Perceptual audio rendering of complex virtual environments</p>TsingosNGalloEDrettakisGACM Transactions on Graphics2004233249258Proceedings of SIGGRAPH 200410.1145/1015706.1015710<p>A new approach to linear filtering and prediction problems</p>KalmanRETransactions of the ASME - Journal of Basic Engineering196082354510.1115/1.3662552<p>Spherical harmonic coding of sound objects - the ambisonic 'O' format</p>MalhamDGProceedings of the 19th AES International Conference, Surround Sound—Techniques, Technology, and Perception, June 2001, Schloss Elmau, Germany5457<p>Fast rendering of sound occlusion and diffraction effects for virtual acoustic environments</p>TsingosNGascuelJ-DProceedings of the 104th Audio Engineering Society Convention, May 1998, Amsterdam, The Netherlandspreprint 4699<p>Methods for blind computational estimation of perceptual attributes of room acoustics</p>BaskindAWarusfelOProceedings of the AES 22nd International Conference on Virtual, Synthetic and Entertainment Audio, June 2002, Espoo, Finland402411<p>On the approximate W-disjoint orthogonality of speech</p>RickardSYilmazOProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '02), May 2002, Orlando, Fla, USA1529532<p>Learning overcomplete representations</p>LewickiMSSejnowskiTJNeural Computation200012233736510.1162/08997660030001582610636946<p>Matching pursuits with time-frequency dictionaries</p>MallatSGZhangZIEEE Transactions on Signal Processing199341123397341510.1109/78.258082<p>Automatic audio morphing</p>SlaneyMCovellMLassiterBProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '96), May 1996, Atlanta, Ga, USA210011004<p>Source localization in complex listening situations: selection of binaural cues based on interaural coherence</p>FallerCMerimaaJJournal of the Acoustical Society of America200411653075308910.1121/1.179187215603153<p>Speech enhancement using a minimum mean-square error short-time spectral amplitude estimator</p>EphraimYMalahDIEEE Transactions on Acoustics, Speech, and Signal Processing19843261109112110.1109/TASSP.1984.1164453<p>Multiple acoustic sources location based on blind source separation</p>HuangGYangLHeZProceedings of the 1st International Conference on Natural Computation (ICNC '05), August 2005, Changsha, China683687<p>Blind source separation combining independent component analysis and beamforming</p>SaruwatariHKuritaSTakedaKItakuraFNishikawaTShikanoKEURASIP Journal on Applied Signal Processing20032003111135114610.1155/S1110865703305104<p>Learning a precedence effect-like weighting function for the generalized cross-correlation framework</p>WilsonKWDarellTIEEE Transactions on Audio, Speech and Language Processing200614621562164<p>Audio textures: theory and applications</p>LuLWenyinLZhangH-JIEEE Transactions on Speech and Audio Processing200412215616710.1109/TSA.2003.819947