Pervasive Pixels Henning Schulzrinne (PI) Steven K. Feiner Gail Kaiser

Pervasive Pixels Henning Schulzrinne (PI) Steven K. Feiner Gail Kaiser John Kender Kathleen McKeown Department of Computer Science Columbia University

Proposed Research Goal: seamless mobile multimedia collaboration across distance Integrate advances across fields Collaborative work Graphical and visual interfaces Spoken language understanding and generation Vision sensing and understanding Networking and security

Contributions Contextual information management Harmonizing physical and virtual environments use workflow to determine display content multimedia summaries of past and present sessions map changing virtual information onto physical displays map layout of physical environment onto virtual space Network services clear, flexible interface to common services authentication and privacy support operating system infrastructure for large displays

Features of Research Infrastructure Large, instrumented multi-display workspaces Networked mobile devices of various capabilities Transparent and automatic adaptability to changes of place, platform or group Support for a wide range of hardware and software, from commercial to novel

Pervasive Pixels Teams Contextualized Information Management Workflow Kaiser Summarization Kender, McKeown Database Ross Project Leader Schulzrinne Harmonizing Physical and Virtual Environments Environment Management Feiner Capture and Display Nayar, Kender 3D Modeling Allen Network Services Enabling communications Schulzrinne Virtual Display Nieh Access and Privacy Keromytis

Pervasive Pixels Teams Contextualized Information Management Workflow Kaiser Summarization Kender, McKeown Database Ross Project Leader Schulzrinne Harmonizing Physical and Virtual Environments Environment Management Feiner Capture and Display Nayar, Kender 3D Modeling Allen Network Services Enabling communications Schulzrinne Virtual Display Nieh Access and Privacy Keromytis

Pervasive Pixels Teams Contextualized Information Management Workflow Kaiser Summarization Kender, McKeown Database Ross Project Leader Schulzrinne Harmonizing Physical and Virtual Environments Environment Management Feiner Capture and Display Nayar, Kender 3D Modeling Allen Network Services Enabling communications Schulzrinne Virtual Display Nieh Access and Privacy Keromytis

Pervasive Pixels Teams Contextualized Information Management Workflow Kaiser Summarization Kender, McKeown Database Ross Project Leader Schulzrinne Harmonizing Physical and Virtual Environments Environment Management Feiner Capture and Display Nayar, Kender 3D Modeling Allen Network Services Enabling communications Schulzrinne Virtual Display Nieh Access and Privacy Keromytis

Pervasive Pixels Teams Contextualized Information Management Workflow Kaiser Summarization Kender, McKeown Database Ross Project Leader Schulzrinne Harmonizing Physical and Virtual Environments Environment Management Feiner Capture and Display Nayar, Kender 3D Modeling Allen Network Services Enabling communications Schulzrinne Virtual Display Nieh Access and Privacy Keromytis

The Department of Computer Science A growing faculty A large student population 5 new assistant professors (pictures and areas here) 1 new full professor: Peter Belhumeur 1 new full professor 9/02: Julia Hirschberg 80 PhDs 155 MS 380 undergraduate majors Internal, fluid collaboration Interdisciplinary connections

Building Critical Mass Networking Vision Schulzrinne, Misra, Yemini links to EE Allen, Belhumeur, Kender, Nayar links to computer graphics Human Computer Interaction graphical and visual user interfaces: Feiner, Kender spoken and written language: Hirschberg, McKeown links to speech and multimedia in EE

Areas of Growth Security and systems Computer engineering Computer graphics Theory Sofware systems (e.g., programming languages)

Administrative Support Institutional cost share on this proposal: 1 million in total Support for growth: slots, space Financial support 100,000/year for 5 years (Provost) 100,000/year for 5 years (Dean) return in revenue (e.g., Columbia Video Network) Technical transfer support dedication of Columbia Innovative Enterprise services

Current Departmental Facilities Shared infrastructure of over 20 Sun and Pentium servers Two teaching laboratories 40 seat student lab (CLIC) of Intel Pentium PCs, dual projection capability, wireless microphone, video mixer and multiple cameras 30 seat Microsoft Research Lab Nortel Meridian telephone switch Research infrastructure of 500 nodes plus servers Staffed by 4 professional system administrators

Research Infrastructure User-based personalization: user location magnetic card readers active badge system triangulation on mobile devices visual tracking Stationary setups Seminar room 3 meeting rooms 12 faculty offices 5 public areas portable units

Seminar room High-resolution DV video camera, 2 pan-tilt speaker cameras, Omnicam omnidrectional audience camera, video switcher, 16 ceiling mounted microphones, electronic whiteboard, 3 XGA highbrightness video projectors

Meeting Room 2 remotecontrolled pan-tilt video cameras, Omnicam, conference table PZM boundary microphones, automatic audio mixer, ceiling speakers, 3 highlumen XGA video projectors

Faculty Office Mimio electronic whiteboard, XGA video projector, Ethernet speaker phone, wal-mounted pan-tilt video camera, PocketPCs

Public areas Walk-by stations and laboratories XGA video projector, steerable ceiling-mounted dome video camera, embedded ceiling mounted netowrked Pentium computer, celing speaker, Andrea DA-400 array microphone, Microtouch IBID 600 whiteboard. Picture here

Design for walk-by collaboration Figure 1 here

Functionalities Conferencing Interconnection with analog phone Internet conferencing server to mis IP and PSTN audio streams Digital hybrid connects digital or analog sound to existing telephone system in classroom Netowrk voice-over-IP interface attached to Nortel Meridian PBX for 20 simultaneous conversations Multi-processor servers and IA64 compute and database server, plus others File storage Face, speaker and fingerprint recognition 2 printers and tape library system in year 4 for backup

Remaining Agenda Research Overview Contextualized Information Management (Kaiser) Harmonizing Physical and Virtual (Feiner) Network Services (Schulzrinne) Genomics testbed (Noble) Responding to reviews (Kender) Telephony demo (Schulzrinne) Lunch Demos Visit with administrators Demos Small group meetings