Danny Hillis (Thinking Machines Corporation Co-founder) – UCLA Distinguished Lecturer Series (Nov 1992)


Chapters

00:00:57 Emergent Behavior and the Promise of Massive Parallelism
00:11:52 Common Misconceptions about Parallel Processing
00:15:42 Common Misconceptions About Parallel Processing
00:23:08 Parallel Processing: A Personal Journey Through the Minefield
00:26:24 Connection Machines: From Neural Nets to Physics Simulations
00:32:38 Parallel Processing: Uncovering Hidden Potential and Efficiency Gains
00:36:06 Data Parallel Processing: A Paradigm for Simple Parallel Programming
00:39:59 Parallel Processing: Grain Size, Amdahl's Law, and Rambo's
00:43:49 Massively Parallel Computing: Overcoming Scaling Challenges
00:49:43 The Evolution of SIMD and its Limitations
00:55:06 The CM5: A More General Massively Parallel Computer
00:58:34 Convergence and Limits of Parallel Machine Architectures

Abstract

Updated Article: Revolutionizing Computing: The Journey of Parallel Processing and Danny Hillis’ Connection Machine

In the world of computing, the evolution of parallel processing marks a pivotal shift, epitomized by the groundbreaking work of Danny Hillis and his Connection Machine. This article delves into Hillis’ journey, from his early inspirations to the development of the Connection Machine, and examines the profound implications of his work on the field of parallel processing. We explore the challenges and myths associated with parallel processing, the evolution of understanding in the field, and the technological advancements that have shaped its current state and future potential.

Early Influences and Collaboration:

Danny Hillis, influenced by his childhood experiences with limited building blocks and his father’s medical research in various countries, developed a keen interest in biology, neurobiology, and independent behavior. His educational background in biology, mathematics, and computer science provided a unique perspective. Hillis’ childhood fascination with juggling and his father’s research sparked his interest in emergent behavior. His collaboration with Marvin Minsky at the AI lab further solidified his focus on parallel processing, leading to the development of the Connection Machine. Marvin Minsky, a well-known AI researcher, recognized Hillis’ talent and welcomed him into his laboratory.

Early Perceptions of Semiconductor Technology:

In the early days of parallel processing, non-saturated bipolar technology (ECL transistors) was seen as the fastest but power-hungry, while CMOS technology was considered slow and suitable for simple applications. Hillis’ determination to explore parallel processing was driven by his personal interest in solving problems in artificial intelligence, supported by biological evidence like the fast processing capabilities of neurons.

Connection Machine and Emergent Behavior:

Hillis’ doctoral research culminated in the concept of the Connection Machine, a testament to his belief in massive parallelism. Despite initial rejections, Hillis persevered, securing funding from DARPA and founding Thinking Machines Corporation. The machine emphasized emergent behavior, demonstrating complex behaviors from simple components, a concept Hillis believed held immense potential across various fields. Hillis’ creative and unusual means of transportation during visits to Claude Shannon’s house, a prominent information theorist and juggler, gained media attention.

Arguments Against Parallel Processing:

Critics argued that slow components would result in slow performance, making parallel processing impractical. Hillis initially focused on developing algorithms expressed in parallel, anticipating the future availability of suitable machines. His work involved common sense reasoning problems and later expanded to vision-related issues.

Overcoming the Myths of Parallel Processing:

The journey of parallel processing involved debunking numerous myths. Contrary to the belief that problems were inherently sequential, Hillis’ work demonstrated that with strategic algorithm design, many problems could be parallelized. This shift in understanding was pivotal in advancing the field, challenging long-standing assumptions like Amdahl’s Law and the sequential nature of direct methods in numerical problems.

Generalization Challenges:

Initial skepticism centered around solving specific problems, not general-purpose computing. Concerns were raised about efficiency and limitations of single-bit processors for arithmetic operations. However, Richard Feynman’s challenge to solve quantum chromodynamics (QCD) revealed the machine’s potential for general numerical problems, opening up a larger market beyond neural networks.

Technological Advances and Hillis’ Motivation:

Hillis’ interest in artificial intelligence and the brain’s neural structure fueled his motivation for massive parallelism. His initial approach, influenced by exposure to CMOS technology, leaned towards using multiple CMOS components for parallel algorithms. Hillis’ vision was for a general-purpose machine, versatile enough to handle various computations. The rise of CMOS technology, initially used in consumer electronics, proved cost-effective and rapidly advanced, further enabling parallel computing.

Feynman’s Challenge and Unexpected Performance:

The Connection Machine’s potential for general numerical problems was highlighted when Richard Feynman, challenging Hillis’ initial notion, proposed using the machine for physics problems like quantum chromodynamics. The machine’s surprising performance in these areas, despite its lack of dedicated arithmetic units, underscored the power of massive parallelism.

Trivializing Parallel Processing:

The realization that many problems thought to be inherently sequential could be solved in parallel emerged gradually. The key lies in considering data as an active participant rather than just manipulating it. The Connection Machine’s architectural advantage allows distributed processing, and techniques like the pointer trick effectively halve the problem size, transforming it into a logarithmic problem.

Implications of Data Parallelism:

Data parallelism emerged as a key aspect, simplifying programming and scaling well with increasing processor counts. Contrary to previous beliefs, more processors in data parallelism actually simplified problems, leading to faster execution. This understanding influenced the evolution of parallel computing, as demonstrated by Rambo’s Law, emphasizing the desire to solve larger problems within the same timeframe.

Solving the I/O Problem and Technology Trends:

The initial I/O bottleneck in parallel processing was addressed through the data vault concept, utilizing parallel arrays of small, cheap disks. Additionally, the rise of CMOS technology, initially used in consumer electronics, proved cost-effective and rapidly advanced, further enabling parallel computing.

Parallelism’s Role and Architectural Advances:

Parallelism transformed high-volume technology into high-performance solutions. Hillis’ exploration of SIMD and MIMD architectures revealed the advantages of each, with SIMD excelling in simple parallel tasks and MIMD offering flexibility for complex algorithms. This led to the development of the CM5 Architecture, which addressed previous limitations and optimized for data parallel programming.

Transition from CO2 to SIMD Model and the Evolution of Parallel Processing:

– The transition from the CO2 to the CO2 model involved a shift from a parallel data model to a sequential data parallel model.

– The main argument for a data parallel model was to avoid confusion and deadlocks caused by multiple programs running in different processors at the same time.

– At the time, SIMD machines were simpler to build from a hardware standpoint due to technological limitations.

– The SIMD model over-synchronizes things, causing serialization steps where processors sit idle due to conditional statements.

– The data parallel model’s strength lies in its sequential semantics in terms of dependencies, not in every instruction being executed sequentially.

CM5: A More Practical and General Parallel Processing Machine:

– The CM5, like its predecessor, the CM2, is a massively parallel processing machine, but with significant advancements.

– It consists of numerous RISC processors, a data network, and a control network, interconnected in a manner resembling a bus structure.

– The CM5 can be programmed in a data parallel manner, similar to the CM2.

– The CM5 offers greater flexibility compared to the CM2, enabling the execution of a wider range of parallel algorithms.

– The CM5 has been successfully employed to emulate shared memory machines, demonstrating its versatility in supporting different programming paradigms.

Maturing of Computer Architectures and Limits of Parallel Machines:

– As computer architectures mature, there is a convergence of different approaches.

– Parallel machines are not useful for all problems.

– Some problems, such as simulating the motion of 10 planets over time, have limited data but require extensive computation.



Danny Hillis’ journey with parallel processing and the development of the Connection Machine revolutionized computing. His work challenged conventional wisdom, introduced new paradigms, and opened avenues for future advancements in computing. Hillis’ legacy is a testament to the power of curiosity, persistence, and innovation in pushing the boundaries of technology.


Notes by: MatrixKarma