A Roadmap for Generative Design of Visual Intelligence

Kushagra Tiwary, Tzofi Klinghoffer*, Aaron Young*, Siddharth Somasundaram, Nikhil Behari, Akshat Dave, Brian ​Cheung, Dan-Eric Nilsson, Tomaso Poggio, Ramesh Raskar (* Equal Contribution)

MIT Press (coming soon!), 2024

Accelerating Discovery: Using AI to accelerate Science, R&G, and Augment Engineering ​& Design

Co-Organizer and Speaker at Workshop in Media Lab Member’s week 2023

Talk on: Accelerating R&D and AI-based Vision Stack Design

This workshop explores how AI can accelerate science, R&D processes, and augment engineering with a special ​emphasis on Research and Development (R&D). We will introduce the "Scientist AI" that can vastly accelerate time-​consuming scientific and R&D design processes through case studies from the fields of Autonomous Vehicles, ​Computational Imaging, Drug Discovery, and Theraputics.

https://discovery.media.mit.edu/

DISeR: Designing Imaging Systems with Reinforcement Learning

Tzofi Klinghoffer*, Kushagra Tiwary*, Nikhil Behari, Bhavya Agarwalla, Ramesh Raskar, ICCV 2023

We propose a new way to co-design imaging systems and task-specific perception models. The camera ​designer selects imaging hardware candidates, which are used to capture observations in simulation. The ​perception model is then updated and computes the reward for the camera designer using the captured ​observations. In our work, we implement the camera designer with reinforcement learning and the ​perception model with a neural network.

[Project Page] [arXiv] [Video]

DecentNeRFs: Decentralized Neural Radiance Fields from Crowdsourced Images

Zaid Tasneem, Akshat Dave, Abhishek Singh, Kushagra Tiwary, Praneeth Vepakomma, Ashok Veeraraghavan, ​Ramesh Raskar

We transform personal images into decentralized Neural Radiance Fields (NeRFs) to create immersive 3D ​experiences. Our key insight is to decompose users' 3D views into personal and global NeRFs, aggregating only ​the latter to construct photorealistic scene representations with minimal server computation.

[Project Page] • [Paper] • [Video] • [Talk]

SUNDIAL: 3D Satellite Understanding through Direct, Ambient, and Complex Lighting Decomposition

Nikhil Behari, Akshat Dave, Kushagra Tiwary, William Yang, Ramesh Raskar

EARTHVISION CVPR Workshop, 2024

3D modeling from satellite imagery is essential in areas of environmental science, urban planning, agriculture, ​and disaster response. In this work, we introduce SUNDIAL, a comprehensive approach to 3D reconstruction of ​satellite imagery using neural radiance fields.

[arXiv]

Objects As Radiance Field Cameras

Kushagra Tiwary*, Akshat Dave*, Nikhil Behari, Tzofi Klinghoffer, Ashok Veeraghavan, Ramesh Raskar, CVPR ​2023

We convert objects with unknown geometry into radiance-field cameras to image the world from the object's ​perspective. Our key insight is to convert the object surface into a virtual sensor that captures cast reflections as ​a 2D projection of the 5D environment radiance field visible to the object.

[Project Page] • [Paper] • [Code] • [Talk] • [MIT News]

Towards Neural Representations through Shadows

Kushagra Tiwary*, Tzofi Klinghoffer*, Ramesh Raskar,

ECCV 2022

Neural Representations cannot use shadows to learn about hidden information in the scene. We present a ​method that learns neural shadow fields which are neural scene representations that are only learnt from the ​shadows present in the scene.

[Project Page] • [Paper]

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

Kushagra Tiwary*, Tzofi Klinghoffer*, Siddharth Somasundaram*, Ramesh Raskar,

ICCP 2022

Cameras were originally designed using physics-based heuristics to capture aesthetic images, but now with the advent of ​physics-based machine learning, we can design cameras to be task-specific i.e. directly for their intended application!

[ Paper ]