E39: Seeing is Believing with MIT’s Ziming Liu

Ziming Liu is a Physics PhD student at MIT and IAIFI, advised by Prof. Max Tegmark. Ziming’s research is at the intersection of AI and physics. Today’s discussion goes in-depth on Liu’s paper “Seeing is Believing” where he presents Brain-Inspired Modular Training (BIMT), a method for making neural networks more modular and interpretable. The ability to see modules visually can complement current mechanistic interpretability strategies.

RECOMMENDED PODCAST:

Founding a business is just the tip of the iceberg; the real complexity comes with scaling it. On 1 to 1000, hosts Jack Altman and Erik Torenberg dig deep into the inevitable twists and turns operators encounter along the journey of turning an idea into a business. Hear all about the tactical challenges of scaling from the people that built up the world’s leading companies like Stripe, Ramp, and Lattice. Our first episode with Eric Glyman of Ramp is out now: https://link.chtbl.com/1to1000

The Cognitive Revolution is a part of the Turpentine podcast network. To learn more: Turpentine.co

Watch this episode on YouTube to see research figures and graphics: https://www.youtube.com/watch?v=VKxwlxb2XWw

Links:

Seeing is Believing, authors: Ziming Liu, Eric Gan, and Max Tegmark https://arxiv.org/pdf/2305.08746.pdf

Ziming Liu: https://kindxiaoming.github.io/

TIMESTAMPS:

(00:00) Episode preview

(01:16) Nathan introduces Ziming Liu and his research

(04:50) What motivated Ziming's research into neural networks and modularity

(08:06) Key Visuals of "Seeing is Believing"

(08:44) Biology analogy & paper overview

(15:07) Sponsor: Omneky

(26:08) Research precedents

(30:00) Locality vs modularity

(37:40) Shoutout Tiny Stories : Episode 33

(39:00) Quantization Model of Neuro Scaling.

(1:00:00) How can this work impact mechanistic interpretability strategies

(1:27:00) Training

(1:46:30) Downscaling large-scale models

TWITTER:

@labenz (Nathan)

@ZimingLiu11 (Ziming)

@cogrev_podcast

SPONSOR:

Thank you Omneky (www.omneky.com) for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off.

MUSIC CREDIT:

MusicLM