Motivation:
Initial inspiration came from a desire to build thinking machines capable of reasoning. Previous attempts at parallel computing, like the ILLIAC 4, faced challenges in programming and coordination.

Single-Program Approach:
Adopted the idea of running only one program on the machine, simplifying programming and coordination. This concept formed the core of the first Connection Machine (CM1) architecture.

CM1 Architecture:
Consisted of an ordinary computer (host computer) connected to a network of simple bit-serial processors. Each processor had its own memory and could perform basic arithmetic and Boolean operations. Communication between processors was facilitated through broadcast and data networks.

Virtual Processors:
Introduced the concept of virtual processors to overcome the limitation of fixed processor count. Allowed programmers to imagine an infinite number of processors, with each physical processor simulating multiple virtual processors. This programming model provided flexibility and scalability.

Flaws and Limitations of CM1:
Insufficient memory posed a significant constraint, especially when running millions of virtual processors. Lack of floating-point capabilities limited the machine’s appeal to scientific and engineering applications. Input-output became a bottleneck, overshadowing the gains in computation speed.

CM2 Improvements:
Increased memory capacity with 256 kilobits of DRAM per processor. Introduced floating-point units clustered with groups of processors, enabling efficient floating-point operations. Integrated I-O directly into the processors, allowing parallel I-O operations. Developed I-O devices, such as frame buffers and disk drives, to match the machine’s capabilities.

Thinking Machine’s Data Vault Disk:
Developed a parallel disk system where 32-bit words were stored across 32 drives, with error correction codes on the remaining drives for fault tolerance. This system, known as a RAID system today, served as Thinking Machine’s data vault disk.

Multiple Hosts and Partitioning:
To accommodate multiple users, Thinking Machine introduced the capability of having multiple hosts on their systems. This allowed partitioning of the machine, enabling different users to utilize portions of the machine simultaneously or for a single user to occupy the entire machine during off-peak times.

Abstraction of Processors and Virtual Processors:
As compiler technology advanced and the concept of virtual processors became more refined, users began to disregard specific processor concerns and focused on writing their programs in a loop structure. The compiler handled distributing loop iterations across processors, simplifying the programming process.

Data Parallelism and Fortran 90:
Users started thinking about parallelism in general terms, considering algorithms’ inherent parallelism rather than specific processor assignments or communication steps. The focus shifted towards aligning data for efficient processing, allowing compilers to optimize data placement without explicitly specifying processor allocations. Fortran 90 introduced a syntax that enabled expressing parallel operations in a single line of code, abstracting away the details of synchronization and multithreading.

Data Parallelism and Scalability:
Data parallelism offered a significant advantage in scalability, as the potential for parallelism increased with the amount of data. Doubling the problem size typically allowed for solving it in the same amount of time with a correspondingly larger machine. This contrasted with control flow parallelism, where the potential for parallelism was limited by program structure rather than data size.

Machine Synchronization:
SIMD machines force synchronization on every instruction, causing inefficiencies when conditional statements are encountered. MIMD machines can execute different instructions simultaneously but are harder to program. The CM5 aimed to combine the advantages of both by allowing broadcast instructions and synchronization while enabling local instruction fetching.

Addressing and Memory:
The distinction between distributed and shared memory machines was blurring due to improved networks. Shared address space provides programming convenience, while distributed physical memory offers faster access. The CM5 employed a shared address space with a distributed physical implementation of memory, achieved through indirect addressing and a network for accessing other processors’ memory.

Time Sharing:
The CM5 was designed from the outset to support time sharing, unlike earlier systems that were not built for that purpose.

CM5 Architecture:
The CM5’s architecture consists of two networks: a control network and a data network. Standard RISC processors serve as the processing units, capable of receiving broadcast instructions, synchronizing, and communicating with the data network. These processors hold the central copy of the program and execute it locally. Some processors have more memory and can load parts of the program as needed.

Programming Model and Execution Capabilities:
The Connection Machine offers both SIMD and MIMD-like behavior. The compiler optimizes the execution by determining whether synchronization is necessary. The user retains the same programming model while gaining the execution capabilities of a MIMD machine.

I/O Integration:
The network acts as a system bus, allowing for direct I/O connection. This enables arbitrary data movement patterns, such as striping across multiple disks. The expandable network supports an arbitrary amount of I/O devices.

Timesharing:
The machine was designed with timesharing capabilities from the beginning. Partitions can be dedicated to specific jobs or used for timesharing. This allows for both batch processing and interactive use.

Future Software Developments:
The design of the Connection Machine has stimulated new software development. The machine’s capabilities enable new programming paradigms. Hillis anticipates further advancements in software for the machine.

Goals:
Achieve teraflop performance (trillions of floating-point operations per second) Utilize terabytes of dynamic memory Implement terabits per second of I/O Incorporate fault tolerance for high availability Maintain time-sharing and forward compatibility with CM2 software

Processing Node:
Employed a standard RISC processor (SPARC) as the core Added special features to optimize parallel processing Provided 32 megabytes of memory per node Implemented a network interface chip for communication

Memory Bandwidth:
Addressed the limited memory bandwidth issue by incorporating vector processing units Each vector unit had a dedicated 64-bit path to memory Achieved a memory bandwidth of half a gigabyte per second per node

Network Structure:
Designed a scalable network architecture to support teraflop performance Analyzed real-world problems to determine network requirements Implemented a three-dimensional torus network topology Utilized wormhole routing for efficient data transfer

Network Bandwidth Requirements:
The bandwidth required by massively parallel computers scales with the number of processors. Doubling the number of processors doubles the required bandwidth.

Two-Dimensional Grid Networks:
Two-dimensional grid networks are not suitable for massively parallel computers because the number of wires across the center of the network does not increase proportionally with the number of processors.

Butterfly-Structured Network:
The Connection Machine utilized a butterfly-structured network designed by Charles Leiserson. This network adds additional layers of wires and switches when more processors are added. The latency of the network grows logarithmically with the number of processors. The bandwidth across the network grows proportionally with the number of processors. The number of wires across the center of the network is proportional to the number of processors.

Adaptive Routing Algorithm:
The Connection Machine networks employed an adaptive routing algorithm.

Networks:
CM-5 uses adaptive routing algorithms that select free paths in the network, increasing fault tolerance and minimizing latency. Wormhole routing is implemented, enabling partial packet delivery while the transmission is still ongoing.

Fault Tolerance:
An independent diagnostic network detects faulty chips and isolates them from the network. Extensive error checking is performed at each step of message transmission, identifying the exact location of errors. Comprehensive error logging tracks every message and ensures that all messages are received. Special error-correcting codes are used to handle non-independent failure modes, such as simultaneous failure of multiple bits in a chip.

Checkpointing and Recovery:
Elaborate checkpointing facilities allow the state of computation to be saved at regular intervals or under program control. In case of an error, the failed part of the machine is electrically isolated, and the computation is remapped to spare processors. A page map in each processor helps maintain a contiguous address space despite the presence of holes due to switched-in spare processors.

Shared Mechanisms:
Timesharing and fault tolerance mechanisms are often shared, providing benefits for both aspects. The ability to quickly extract and store network messages facilitates timesharing by allowing messages to be swapped in and out during user context switches.

Packaging for Fault Tolerance:
The ability to power down parts of the machine allows failed components to be fixed offline while the rest of the machine continues to operate.

Clocking:
Maintaining a synchronous machine is crucial for reliable operation and deterministic program execution. Measured wires distribute clocks throughout the machine, with local phase-lock loops ensuring synchronization. This approach prevents any part of the system from gaining an extra clock cycle compared to other parts. Longer wires can be used across the machine without causing timing issues.

Machine’s Deterministic Delay System:
The Connection Machine-2 (CM-2) employs a novel deterministic delay system for transmitting data across wires. Bits are stored on wires based on specific time delays, such as 1 o’clock, 2 o’clock, or 3 o’clock periods. The system reliably determines the delay associated with each wire, enabling efficient data transmission.

Careful Electrical Design:
Differential signaling and conservative timing ensure the stability and reliability of the delay system. Precise wire cutting is not crucial as the timing mechanisms tolerate variations.

Packaging and Modular Design:
Four processing nodes are placed on each board, along with appropriate memory. Boards are arranged in a three-dimensional structure, with memory boards sandwiched between processor boards. SIMs (memory modules) extend vertically from the boards, creating a thick three-dimensional structure.

Field Repairability:
Connectors are designed to minimize mechanical stress on components during removal, preventing noise interference. Dedicated inserters with levers smoothly extract boards, facilitating field repairs.

Rack Configuration:
A pair of racks houses 256 processors, with an additional rack for communication components. Racks can be interconnected to form larger systems, with cables running across the top. Air cooling is carefully managed to keep chip temperatures below 80 degrees Celsius, ensuring system reliability.

Programming Model:
CM-2 utilizes a single-program model, eliminating the need for complex synchronization mechanisms. This simplifies programming and reduces the likelihood of errors.

Sources of Parallelism:
The primary source of parallelism in parallel computers lies in the parallelism across the data rather than pipelining or multiple program instances.

End of the SIMD-MIMD Debate:
The distinction between SIMD (Single Instruction Multiple Data) and MIMD (Multiple Instruction Multiple Data) architectures has blurred, leading to a convergence of ideas and a focus on scalability and programmability.

Redefining “Fastest Computer”:
The concept of the “fastest computer” has shifted from a singular focus on speed to a broader consideration of scalability, reliability, and programmability.

Impact on Programming Models and Algorithms:
The advent of parallel architectures, including the Connection Machine, is driving advancements in programming models, languages, applications, and algorithms, reshaping the landscape of computer science.

Fiber Optics and Latency Reduction:
While fiber optics offer potential advantages, concerns over reliability led to their initial exclusion in the Connection Machine design. However, future upgrades could incorporate fiber channels for improved bandwidth and reduced latency.

Data Vaults and I-O Control Processors:
The CM5 utilizes the same data vaults as the CM2, with additional projects exploring advanced disk units. The CM5 introduces a separate I-O control processor for orchestrating interactions.

Scalability of Data Parallel Software:
Data parallel software can adapt to different processor counts, allowing programs designed for 1,000 processors to run efficiently on 500 processors without modification.

Boot Time and File Loading:
Boot time and file loading scale favorably with the size of the machine due to parallel processing and the use of a broadcast network for memory loading.

Intellectual Property and Innovation:
The Connection Machine incorporates patented technologies, but the creators acknowledge the possibility of similar solutions being developed independently.

Riding Technology Curves:
One of the guiding principles behind the Connection Machine was the idea of riding technology curves, leveraging advancements to create powerful and scalable computing systems.

Benefits of Massively Parallel Processors:
Massively parallel processors have outperformed vector processors due to a significant advantage in engineering resources, including engineers working on RISC processors and CMOS process lines.

Decoupled Design:
To maximize independent improvements, the design of processor chips is decoupled from the design of switching circuits in the network.

Independent Clocking:
The clocking system is designed to run on phase-locked clocks with different frequencies for the processors and memory switches.

Bandwidth and Latency:
Increasing clock rates of memory switches will improve bandwidth but not latency. Reducing latency requires either compacting technology to fit more processors on a board or running more wires.

Future-Proof Packaging:
The packaging design allows for additional wires to be run between boards in the future to accommodate latency reduction strategies.