Alan Kay (VPRI Co-founder) – Is Software Engineering Still an Oxymoron? (Nov 2021)
Chapters
00:00:12 Engineering: Aspiring to Understand Hardness and Softness
Engineering Defined: Engineering involves designing, making, and repairing things in principled ways. Software engineering follows these principles, making it not an oxymoron but an aspiration.
Software Engineering Challenges: Soft aspects: Legacy systems’ resistance to change and buggy systems’ unpredictable behavior. Hard aspects: The complexity of large-scale software projects.
Case Study: Empire State Building vs. Software Projects: The Empire State Building was built by 3,000 people in less than a year 90 years ago. No modern software projects of comparable size have been completed within a similar timeframe.
Peruvian Rope Bridges and Craftsmanship: Engineering encompasses various disciplines, including traditional crafts like Peruvian rope bridges.
Akashi Straits Bridge as an Example of Engineering: Engineering marvels like the Akashi Straits Bridge demonstrate the magnitude of human ingenuity. The bridge’s pylons are as high as the Empire State Building, showcasing the scale of engineering achievements.
00:05:39 Software Engineering: Beyond Tinkering and Towards Comprehensive Design
The Five Types of Knowledge and Thinking: Tinkering: Experimental and playful exploration of ideas and materials. Aesthetics: The appreciation and creation of beauty and elegance. Engineering: The application of scientific knowledge to practical problems. Mathematics: The study of numbers, quantities, and structures. Science: The systematic study of the natural world and its phenomena.
Science and Technology as the Greatest Art Form of the 20th Century: The combination of tinkering, aesthetics, engineering, mathematics, and science has led to the creation of remarkable technologies that have transformed society.
The Current State of Software Engineering: Software engineering is still in its early stages of development compared to other fields like engineering and medicine. It lacks a comprehensive understanding of ancillary knowledge and is evolving to incorporate more of it.
The Importance of Designing for Failure: The internet’s success is attributed to its design, which prioritizes failure and ensures that it doesn’t hinder the system’s operation.
Facebook’s Outage and the Need for Comprehensive Internal Architecture: Facebook’s massive outage highlights the lack of internal architecture to handle small errors effectively. Configuration checking programs are insufficient, and a more comprehensive approach is needed.
The Absence of a Model for Understanding the System: Many enterprises, including Facebook, lack a model to understand the system as a whole, leading to potential vulnerabilities and failures.
The Importance of Ethical Considerations in System Design: Systems that deal with humans should incorporate ethical considerations, such as Asimov’s laws of robotics, to ensure responsible and beneficial operation.
Hacker Culture and the Need for a Mature Approach to System Building: The “Hacker Way” mentality, which emphasizes speed and breaking things, is not suitable for building large-scale, reliable systems. Rebranding alone cannot address the underlying issues; a more mature and responsible approach is required.
00:13:23 Dunning-Kruger Syndrome and Its Impact on Engineering and Management
Dunning-Kruger Syndrome: MacArthur Wheeler, a low-ability bank robber, believed he was invisible because he doused himself with lemon juice. Dunning and Kruger studied this case and found that people tend to overestimate their abilities, leading to the Dunning-Kruger syndrome.
Greek Tragedy and Hubris: The Dunning-Kruger syndrome is the basis of Greek tragedy, where superior people overreach and face consequences. Modesty is important, as debugging is harder than programming.
Einstein’s Insight: Einstein said that problems cannot be solved with the same thinking used to create them.
Dunning-Kruger in Software and the Boeing 737 Max: Many software systems, including Facebook’s, suffer from the Dunning-Kruger syndrome. The Boeing 737 Max autopilot is an example of a system that is put in charge of a task it is not capable of performing, leading to disastrous consequences.
Engineering and Management Implications: The Dunning-Kruger syndrome can lead to serious problems in engineering and management. It is important to study this phenomenon to understand and prevent its negative effects.
Caveman Wants Stories: We are like cavemen around a campfire, telling stories to understand complex ideas. Our brains haven’t evolved much, so we still crave stories. Systems, which are prevalent today, lack a clear start and finish, making them difficult to grasp as stories.
Wisdom vs Invention: Invention outpaces wisdom, leading to a mismatch between our capabilities and our ability to handle them wisely. We have given ourselves immense power without the wisdom to wield it responsibly.
The COVID Pandemic: The mishandling of the COVID pandemic highlights our lack of wisdom in dealing with complex issues. Despite being a familiar type of contagion, the pandemic exposed our inability to handle it effectively.
The Climate Problem: The climate problem is more complex and devastating than the COVID pandemic. Humanity is not prepared to address this challenge effectively due to our lack of wisdom.
00:21:00 Early Innovations in Human Intelligence Augmentation
Early Augmentation Efforts: Doug Engelbart, a prominent engineer, proposed the concept of augmenting human intellect in 1962 to address various problems. Engelbart’s report emphasized the need for tools and methods that align with human capabilities to solve problems. Engelbart envisioned a system that included a large display, pointing devices, and semi-intelligent assistants. Recognizing the potential dangers of unchecked power, Engelbart advocated for education and training to balance wisdom and power.
The Mother of All Demos: Four years after Engelbart’s proposal, a groundbreaking demonstration showcased many aspects of modern personal computing and networking. This demo included features such as multiple cursors, simultaneous information manipulation, and the use of the system in face-to-face meetings. The Engelbart system offered a broader range of capabilities than what is available in today’s computing systems.
Early Supercomputers and the Role of a Graduate Student: Engelbart’s work was inspired by a supercomputer from 60 years ago, which had limited processing power and memory. A graduate student was granted access to this supercomputer for three hours every morning. Using an oscilloscope display and a light pen, the student developed techniques to draw lines and shapes on the point-plotting display.
00:26:14 Early Computer Graphics: Alan Kay's Sketchpad System
Alan Kay’s Sketchpad: Introduced rubber band lines, enabling the creation of mutually perpendicular lines through desired goals. Demonstrated the first clipping window, allowing drawing on a vast canvas while zooming with knobs. Implemented constraint solving, enabling the creation of rough sketches that the system refines to find the closest solution.
Sketchpad’s Innovations: Introduced the concept of master objects, allowing the creation of instances of objects with independent sizes and properties. Demonstrated the first implementation of objects in a computer system, paving the way for object-oriented programming. Efficiently handled graphics rendering, with half of the computer’s memory dedicated to simulated graphic display.
Recreation of the Bridge Example: Reconstructed from the thesis due to the misplacement of the original movie. Demonstrates the use of Sketchpad to design a bridge, showcasing the system’s capabilities in creating complex structures.
00:30:19 Interactive Computer Graphics and Computer-Aided Design in the 1960
CAD with Sketchpad: Sketchpad allowed for the creation of computer-aided design (CAD) systems without requiring knowledge of the specific domain being modeled. It could handle various systems, including electric circuits and bridges, by propagating constraints between their parts.
General Problem Solving: Sketchpad introduced a new approach to programming where the focus was on specifying goals rather than detailed instructions. It utilized problem solvers to automatically find solutions based on the given requirements.
Interactive Computer Graphics and User Interface Concepts: Sketchpad pioneered interactive computer graphics, allowing users to manipulate objects in real time. It introduced concepts like clipping, windowing, pointing, and iconic user interfaces.
Object-Oriented Design: Sketchpad employed object-oriented design principles, utilizing masters and instances to represent relationships between objects.
Internal Innovations: Sketchpad featured internal innovations such as the inverted pointer, recursive garbage collection, and automatic dynamic simulations.
Impact on Alan Kay: Encountering Sketchpad in 1966 had a profound impact on Alan Kay, inspiring him to envision a future where all interactive computing could be goal-oriented and user-friendly.
Ivan Sutherland’s Dedication: Ivan Sutherland’s remarkable achievement of developing Sketchpad in machine code within a year highlighted his dedication and innovative spirit.
CAD’s Origins: Sketchpad laid the foundation for modern CAD systems, particularly in the electrical CAD domain.
00:34:05 Computer-Aided Design, Simulation, and Fabrication in Engineering
Unifying Engineering Disciplines: Alan Kay emphasizes the need for a comprehensive approach to engineering that includes CAD (Computer-Aided Design), SIM (Simulation), and FAB (Fabrication) across various disciplines. He highlights the importance of simulating systems before physical construction to ensure their proper functioning. Kay questions the lack of integration between science, math, and engineering in many organizations, leading to inefficiencies in software development.
Key Anecdotes and Insights: Consulting experiences with large organizations have provided Kay with a collection of anecdotes that offer insights into the relationship between computing and engineering. Organizations often know a subset of what computing knows, and there are important aspects that remain unknown or are yet to be invented. Joy arises when a project’s requirements align with an organization’s existing capabilities. Challenges arise when organizations try to reinvent known concepts or use inappropriate methods.
Real-World Examples: Kay cites the R1 system used by DEC for modeling product configurations as an example of a tool that could have prevented issues like the Facebook data scandal. IBM’s Autonomic Computing initiative is mentioned as a valuable resource for designing self-regulating systems. Nancy Levison’s book, “Engineering a Safer World,” is recommended for its insights into system safety and debunking common misconceptions.
Research and Innovation: Kay stresses the importance of ongoing research in computing to address the increasing complexity of systems and meet future requirements. He acknowledges the need for funding and support for research efforts.
Final Thoughts: Kay reiterates the need for a comprehensive CAD, SIM, FAB approach in engineering and proposes the development of a live, super high-level language to facilitate this integration.
00:40:18 Understanding Cognitive Biases and Their Impact on Decision-Making
Cognitive Biases: We have numerous cognitive biases that distort our thinking, often remaining unaware of them. Example: Loss aversion, where we strongly resist giving up things we believe are valuable, even when it’s detrimental. Wikipedia’s list of cognitive biases provides insights into these mental shortcuts.
Systems Thinking: Systems, not algorithms and data structures, are the real things to study in computing. Algorithms and data structures are only a starting point.
Perception and Reality: Our perception of reality is influenced by our beliefs and expectations, not just sensory input. We live in a waking dream, a delusion, shaped by our subjective interpretations. Science helps us recognize this dream-like state and reminds us to question our assumptions.
The Role of Stories: Stories are powerful tools for shaping our understanding of the world. We tend to prefer stories that confirm our existing beliefs, leading to confirmation bias. Stories can be effective in conveying ideas but should be treated as advertisements for further exploration.
Medicine as an Analogy: Medicine is a branch of engineering, offering valuable insights for understanding computing. Surgeries are complex procedures, not comparable to writing a few lines of Python code. Asking stack overflow for medical advice is analogous to seeking guidance on complex computing issues.
00:53:02 Engineering As A Process of Continuous Improvement
The Need for a Paradigm Shift in Software Engineering: Early PC and Mac OSs, as well as operating systems like VMS and Unix, were not built for modern computing needs. Current solutions, such as Band-Aids, are not sufficient for addressing complex issues. Better ideas often don’t scale due to the lack of continuity between different stages of development.
The Importance of Integration in Engineering: Engineering fields, including medicine, have integrated science, math, experimentation, and engineering into one discipline. This integration allows for the effective use of knowledge and the development of more robust solutions.
The Hippocratic Oath as a Model for Software Engineering: The Hippocratic Oath emphasizes the importance of avoiding harm, which is particularly relevant in the context of software engineering. Software is reaching everywhere and has the potential to cause significant harm if not developed responsibly.
The Engineering Pledge and the Need for Higher Standards: Engineering has a pledge similar to the Hippocratic Oath, emphasizing the importance of preventing collapse, failure, and harm. The engineering pledge should be of even higher quality for software engineering due to its pervasive impact and potential for harm.
Engineering as a Continuous Process of Improvement: Real civilization and engineering are not static destinations but ongoing processes of striving for improvement. Engineers should continuously strive to become better at their craft and contribute to the advancement of the field.
Abstract
Software Engineering: Aspiration, Learning, and Understanding
In the ever-evolving field of software engineering, the balance between aspiration, learning, and understanding defines the path of progress. This article explores the multifaceted nature of software engineering, highlighting the significance of comprehensive approaches, the integration of various disciplines, and the recognition of cognitive flaws in shaping the future of technology. Our brains crave stories to understand complex ideas. However, systems, which are prevalent today, lack a clear start and finish, making them difficult to grasp as stories. From the historical backdrop of engineering marvels like the Empire State Building to the modern challenges faced by platforms like Facebook, the journey of software engineering is marked by a constant quest for improvement, necessitating a blend of craftsmanship, science, and innovative thinking.
Main Ideas and Their Expansion
Engineering and Software Engineering
Engineering, at its core, involves designing, creating, and repairing objects and systems in principled ways. Software engineering adheres to these principles, not as a contradiction but as an aspiration. Historically, the construction of monumental structures like the Empire State Building and the Akashi Straits Bridge demonstrate the efficiency and grandeur achievable through engineering. These feats set a benchmark for meticulous design and execution, philosophies that are vitally important in the realm of software engineering.
The Evolution of Software Engineering
Software engineering encounters unique challenges, such as updating legacy systems and addressing bugs, underlining the necessity for careful planning and ongoing maintenance. The field today still largely depends on tinkering, mirroring the early days of engineering and medicine. This calls for a comprehensive approach that prioritizes resilience and robust failure handling mechanisms, as evidenced by incidents like the Facebook outage and other large-scale system failures.
Theoretical Foundations and Inspirations
The transition from mere tinkering and aesthetics to incorporating engineering, mathematics, and science has been critical in advancing technology. The contributions of visionaries like Alan Kay and Doug Engelbart, who emphasized the disconnect between our ancient brains and modern technology and the augmentation of human intellect, respectively, underline the need for a deeper comprehension of technology and its impact.
The Role of Computing and CAD Systems
The evolution of computing, particularly through innovations like Ivan Sutherland’s Sketchpad, brought about a revolution in interactive graphics and object-oriented design. The development of Sketchpad was instrumental in creating CAD systems that allowed design without domain-specific knowledge, employing constraints for dynamic adjustments and problem solvers for automated solutions. It introduced revolutionary user interfaces and interactive computer graphics. Alan Kay further accentuates the importance of a comprehensive approach combining CAD (Computer-Aided Design), SIM (Simulation), and FAB (Fabrication) in various engineering disciplines. This integration is crucial for the efficient functioning of systems, as seen in chip design and advanced modeling techniques.
Cognitive Aspects and System Thinking
Acknowledging and understanding our cognitive biases, such as loss aversion and learning aversion, are essential for informed decision-making in software engineering. Systems thinking requires a holistic perspective, transcending algorithms and data structures, much like the approach in medicine. Our subjective reality, shaped by beliefs and experiences, needs to be challenged through scientific inquiry. Stories, while powerful in conveying concepts, should be viewed as gateways for further exploration rather than conclusive explanations. The field of medicine offers valuable insights for computing, as it encompasses complex procedures beyond simple coding, paralleling the intricate issues in computing.
The Need for a Paradigm Shift in Software Engineering
Early operating systems like PC and Mac OS, as well as VMS and Unix, were not designed for modern computing demands. Simple fixes, or ‘Band-Aids’, are inadequate for complex issues, and innovative ideas often struggle to scale due to discontinuities in development phases.
The Importance of Integration in Engineering
Engineering disciplines, including medicine, have successfully integrated science, math, experimentation, and engineering. This amalgamation enables the effective utilization of knowledge and the development of robust solutions. The Hippocratic Oath in medicine, emphasizing harm avoidance, is particularly relevant in software engineering, given the widespread impact of software. Engineering, too, has a pledge focusing on preventing collapse, failure, and harm, which holds even greater significance in software engineering due to its pervasive nature and potential for damage.
Engineering as a Continuous Process of Improvement
Engineering is a continuous journey of striving for improvement, not a fixed destination. Engineers must persistently seek to enhance their skills and contribute to the advancement of their field, mirroring the dynamic nature of real civilization and engineering.
Conclusion
In conclusion, the field of software engineering stands at a crossroads where the integration of principles, the acknowledgment of cognitive limitations, and the aspiration for improvement define its future. Just like the great engineering feats of the past, software engineering must evolve, embracing a comprehensive approach that includes understanding the human element, system thinking, and the integration of diverse knowledge domains. This evolution is crucial for addressing the increasingly complex challenges of modern technology, ensuring safety, efficiency, and resilience in the systems that increasingly define our world.
Alan Kay's vision for personal workstations emphasized powerful tools for creativity and learning, leading to milestones like Smalltalk and the graphical user interface. His Dynabook concept aimed to provide information access anywhere, inspiring the development of modern portable devices....
Insights from pioneers like Alan Kay and Joe Armstrong reveal the evolution of computing, emphasizing the importance of learning, scaling, and addressing unsolved problems. Computer scientists should think like scientists, considering scaling aspects early and embracing failure to drive innovation....
Alan Kay, a Turing Award laureate, advocates for a practical, hands-on approach to computer science education, emphasizing simplicity, systems thinking, and a beginner's mindset. He calls for a shift from theoretical rigor to practical problem-solving, preparing students to contribute to the advancement of computing....
Alan Kay emphasized strategic thinking in software engineering with a focus on domain-specific languages, criticized current computing systems, and envisioned dynamic and efficient computing in the future. Kay highlighted the need for strategic thinking in software engineering with a focus on domain-specific languages, criticized current computing systems, and envisioned dynamic...
Computing's evolution reflects human creativity, perception, and adaptability, with promises of future transformation. Computers' impact on human experience will continue to shape the world in profound ways....
Alan Kay emphasized simplicity in computing, and advocated for a transformative approach to education that engages students with immersive and explorative experiences. Kay's insights highlight the need for a new mathematics to describe computing, and the importance of captivating children's interest in the field....
Alan Kay's revolutionary concepts, such as the Dynabook and Smalltalk, shaped modern computing and inspired future innovations, while his emphasis on open knowledge and transformative teaching fostered creativity and critical thinking in his students. His insights into early computers and programming languages laid the groundwork for modern object-oriented programming and...