Adam D’Angelo (Quora Co-founder) – Knowledge, Experimentation, and Quora | EconTalk (Aug 2016)
Chapters
00:00:07 Founding Quora: Knowledge Sharing and Signaling
Background: Adam D’Angelo, the founder of the website Quora, emphasizes its mission as a knowledge-sharing platform to connect individuals seeking knowledge with those who possess it. Quora presents its content in a question-and-answer format. Adam D’Angelo pursued computer science in college while also showing an interest in social science and economics.
Inspiration for Quora: D’Angelo’s interest in knowledge sharing led him to recognize the vast amount of untapped knowledge residing in people’s minds. He saw an opportunity to create a platform that would unlock this knowledge and provide long-lasting value.
Comparison with Existing Platforms: D’Angelo highlights the distinction between general Internet knowledge sharing and Quora’s focus on generating new knowledge. He acknowledges that platforms like Google excel at indexing and making existing information accessible. Quora, on the other hand, aims to expand the available knowledge on the Internet by allowing experts to contribute their specialized knowledge.
Challenges of Scalability: Previous attempts at creating question-and-answer platforms have faced challenges in maintaining quality at scale. D’Angelo acknowledges that ensuring consistency and preventing deterioration of content quality is a significant challenge in running such a system. The problem lies in preventing the platform from becoming chaotic and discouraging expert participation. D’Angelo suggests that Quora has found ways to address these challenges, ensuring experts’ involvement and delivering high-quality content.
00:05:26 Strategies for Building a Successful Online Platform
Real Names and Reputation: Quora requires users to use their real names, which establishes a sense of accountability and trustworthiness. This allows experts and individuals with real-world reputations to bring their credibility to the platform.
Personalization: Quora invests heavily in personalization to deliver the right answers to the right people. This involves matching users with questions they are interested in and providing relevant content based on their preferences.
Avoiding Duplicate Questions: Quora employs machine learning algorithms to identify and merge duplicate questions, preventing multiple versions of the same question from cluttering the platform. This helps concentrate answer energy into one place, leading to more comprehensive and definitive answers.
Additional Insights: A significant portion of Quora users (approximately half) actively seek answers to important questions, while the other half uses the platform for casual browsing and curiosity. Quora offers an anonymous question feature, but the vast majority of users prefer to ask questions using their real names. The volume of questions asked on Quora is substantial and continues to grow rapidly along with the platform’s user base.
00:12:07 Internet Personalization: Driving Growth and Reshaping Markets
Personalization and Market Growth: Adam D’Angelo, the CEO of Quora, reflects on the unexpected scale of growth in the online market, influenced by the adoption and personalization of digital products and services.
The Power of Personalization: D’Angelo emphasizes the pivotal role of personalization in the expansion of markets, primarily facilitated by digital platforms. In contrast to traditional media, such as magazines, where everyone views the same content, digital media provides tailored experiences for individual users.
Creating a Single Source for Diverse Needs: Personalization enables a small number of digital products to serve a vast audience effectively by showcasing content relevant to each user’s preferences.
Examples of Market Consolidation through Personalization: D’Angelo cites the example of Google as a single search engine catering to a global audience. He also mentions Netflix as a dominant player in streaming entertainment, eliminating the need for traditional cable providers and offering a wide range of content personalized for each user.
Incentive for Personalization in the Absence of Direct Competition: D’Angelo highlights the strong incentive for digital companies to continuously personalize their products, even in the absence of immediate competitors. They aim to maximize user engagement and, subsequently, their revenue streams.
Competition for Leisure Time: Different digital platforms, although operating in seemingly distinct markets, compete for users’ leisure time. Netflix and Facebook, for instance, vie for users’ attention, drawing them away from each other based on the quality of their respective offerings.
Quora’s Monetization Strategy: D’Angelo remains tight-lipped about Quora’s specific monetization plans, despite the company’s valuation exceeding a billion dollars.
00:15:31 Quora - Monetization, Growth, and Data-Driven Innovations
Quora’s Monetization Strategy: Adam D’Angelo confirmed Quora’s monetization through advertising, with promising results from initial small tests. Quora aims to strike a balance between generating revenue and maintaining a user experience free from excessive ads.
Valuation and Growth: D’Angelo refrains from commenting on specific valuations, noting that investor projections are based on usage and monetization assumptions.
Challenges of Exponential Growth: Quora’s rapid growth requires constant enhancements to infrastructure and technical systems. Scaling personalization becomes increasingly complex as the user base expands. The increasing volume of data enables enhanced machine learning and personalization capabilities.
Experimentation and Focus on Q&A: Quora has explored formats beyond questions and answers but found the Q&A model to be dominant. The company’s energy is dedicated to improving the Q&A product, with mobile compatibility and cost control being key considerations.
Personalization and User Experience: Adam D’Angelo uses Quora like a typical user to avoid distorting his experience. He generates theories and ideas from his usage but tests and validates them with controlled experiments and user data. Personalization on Quora involves various factors, including followed topics, followed individuals, past behavior, and question engagement. Personalization helps tailor the user experience to individual interests, enhancing relevance and engagement.
Quora’s Personalized Experience: Quora’s algorithm personalizes the user experience based on various factors such as location, interests, and past interactions. Different users have different experiences on Quora due to this personalization.
Personalizing Answer Recommendations: Quora uses signals to determine which answers to show each user. Factors like the user’s past answers and their relevance to the question are considered. Showing duplicate answers is avoided to focus on filling knowledge gaps.
Determining Expert Questions: Quora considers factors such as past answers and their relevance to the question when determining who gets expert questions.
Viewer Ratings and Signals: Viewer ratings and engagement metrics like time spent on an answer are valuable signals for Quora. Votes from authoritative users are given more weight.
The Importance of Community: Quora hosts an annual event where top writers can meet and interact with the team. Community building is a challenge for online platforms like Russ Roberts’ podcast. Virtual spaces are a scalable option for creating a sense of community among listeners or users.
00:30:45 Controlled Experimentation in Product Development
Mistakes and Regrets: Adam D’Angelo reflects on the excessive features added to Quora in its early days, leading to increased complexity and a heavy burden on the platform’s performance and development.
Controlled Experiments: Quora employs a systematic approach to product development through controlled experiments. The company’s in-house experiment framework facilitates the efficient execution of experiments. Quora has conducted approximately 2,000 experiments to date, with about 30 running concurrently.
Significance of Data Science: Quora allocates a significant portion of its workforce to data science, with 13 data scientists among its 130 employees. Data scientists analyze data, run and interpret experiments, and maintain the scientific integrity of the process. A separate team creates the changes that are tested in the experiments.
Continuous Improvement Mindset: Despite Quora’s success, the company remains committed to ongoing improvement. D’Angelo emphasizes the importance of constantly striving to enhance the user experience and address limitations.
00:35:26 Improving Product Quality Through Iterative Experimentation
Experimentation at Quora: Quora uses experiments to improve the user experience, with a focus on personalization (showing the best answers) and finding relevant questions. Experiments are conducted by comparing different versions of the product, with a hypothesis defined in advance. Most experiments (60-70%) fail to meet the expected outcome, but are crucial for learning and refinement.
Factors Driving Experimentation: User base growth: The growing user base provides more data and diverse user preferences. Advancing technology: New machine-learning techniques and platforms offer opportunities for improvement. Increasing engineering team: The larger team allows for faster progress and tackling more ambitious projects.
Unexpected Findings from Experiments: Importance of speed: Milliseconds of delay can significantly impact user engagement. Transaction costs: Any friction, however small, can greatly reduce usage. Perception of time: Even economists are surprised by the popularity of listening to podcasts at double or one and a half speed.
Feedback from Users: Quantitative feedback, such as user behavior and data, is more reliable than surveys in assessing product improvements. Qualitative feedback, such as surveys, is valuable for understanding user sentiment and reasons behind behavior.
Addressing Failed Experiments: Failures are common and can be depressing, especially for newcomers to experimentation. Overcoming bias: Wishful thinking can lead to misinterpreting results, and experiments ensure objectivity. Dealing with consequences: Organizations may have negative consequences (e.g., firing or reduced responsibility) for failed experiments, leading to irrational interpretations.
Running Quora: * Adam D’Angelo, the CEO of Quora, dedicates most of his time to internal company matters. * He regularly engages in management tasks, product direction discussions, recruiting, data analysis, and staying updated on industry developments. * To stay informed, D’Angelo reads books on management, follows machine learning advancements on Quora, and shares relevant information with his staff.
Keeping Up with Machine Learning: * Quora’s culture promotes learning and encourages employees to stay informed about the latest advancements in machine learning and other relevant fields. * Regular reading groups, external speakers, and mailing lists facilitate knowledge sharing within the company. * D’Angelo emphasizes the importance of conducting their own experiments and being cautious about relying solely on the results of other companies’ product changes.
Mission-Driven Company: * Quora’s mission, “share and grow the world’s knowledge,” serves as a guiding principle for the company and is frequently reiterated during candidate interviews and employee interactions. * Hiring employees who are genuinely interested in the mission is seen as more important than simply reciting the phrase. * Repetition and reinforcement help employees remember and embody the company’s mission.
Impact on Users’ Lives: * D’Angelo acknowledges the difference between creating a site people enjoy using and changing their lives through transformative knowledge. * He shares examples of users finding life-saving and life-changing answers on Quora, such as overcoming depression and conceiving a child. * Quora employees are encouraged to share these stories internally to foster a sense of purpose beyond individual work responsibilities.
00:56:48 Digital Diaries: Archiving Human Perception for the Future
Importance of Archiving: Long-term archiving is crucial to Quora, as most internet services lack concern for the future, focusing solely on the present. Quora’s focus on archiving aims to address this issue.
Historical Interpretation: History is often colored by the writer’s perspective, lacking a sense of people’s feelings and diverse viewpoints.
Quora as a Time Capsule: In 100 years, people can use Quora and other services to understand public sentiment towards historical figures such as Donald Trump, providing a more nuanced perspective.
Exploring Complex Issues: While understanding individual opinions from diaries is valuable, the internet offers a vast collection of perspectives and data. Complex issues like inequality are challenging to capture visually but could potentially be better understood using the wealth of information available online.
01:00:09 The Promise and Perils of Personalization in a Digital Economy
Adam D’Angelo’s Plans for Quora: Adam D’Angelo, CEO of Quora, plans to remain dedicated to the company’s mission for the long term and aims to eventually take Quora public.
Underrated Potential of Personalization: D’Angelo believes that the power of personalization is currently underappreciated and will revolutionize various industries. Through machine learning and AI, personalization enables tailored experiences for users, resulting in increased product quality.
Trade-offs of Personalization: D’Angelo acknowledges concerns that personalization may not always be welfare-improving, especially considering the profit-driven nature of companies. In the case of Amazon, personalization might increase purchases based on profit margins rather than genuine user interest.
Facebook’s Transparency and Regulation: D’Angelo suggests that Facebook’s disclosure of their news feed personalization might be a strategic move to avoid misconceptions and potential regulations. Regulation can be beneficial in protecting consumers, especially when companies depend on selling products rather than relying on advertising.
Price Discrimination and Competition: D’Angelo highlights the potential for personalization to enable price discrimination, as companies can adjust prices based on user preferences. While Amazon has been against this practice, other companies may not prioritize consumer value.
Conclusion: D’Angelo emphasizes the significance of competition in preventing companies from exploiting personalization solely for profit. The complex issue of balancing personalization and competition remains a challenge as the world advances in this direction.
Abstract
The Transformative Journey of Quora: Building a Knowledge-Sharing Ecosystem
In the rapidly evolving digital landscape, Quora emerges as a distinctive knowledge-sharing platform, fundamentally transforming how information is exchanged online. Founded by Adam D’Angelo, a visionary with a unique blend of computer science acumen and social science interest, Quora differentiates itself by focusing on generating and preserving high-quality knowledge rather than merely indexing existing information. This article delves into the platform’s unique features, such as the requirement of real names, personalization through advanced algorithms, and its commitment to maintaining a high-quality user base. Additionally, it examines Quora’s growth strategies, challenges in data management, and Adam D’Angelo’s philosophy and future aspirations for the platform, including the role of personalization in shaping internet markets and its long-term archiving vision.
Quora’s monetization strategy revolves around advertising, with promising results from initial small tests. The platform seeks to strike a balance between generating revenue and maintaining a user experience free from excessive ads. Despite its impressive valuation, D’Angelo refrains from commenting on specifics, noting that investor projections are based on usage and monetization assumptions.
1. Quora: A Knowledge-Sharing Revolution
Quora’s inception marked a significant shift in the online question-and-answer landscape. Its mission to build a comprehensive database of high-quality answers sets it apart. The platform’s user-centric approach, where questions are met with expert answers, has attracted a diverse user base, ranging from curious individuals to subject matter experts. Adam D’Angelo, the founder of Quora, emphasizes its mission as a knowledge-sharing platform to connect individuals seeking knowledge with those who possess it. Quora presents its content in a question-and-answer format.
2. The Genesis: Adam D’Angelo’s Vision
Adam D’Angelo’s journey from a Facebook executive to the founder of Quora is marked by his passion for merging computer science with social science. His experience at Facebook influenced his perspective on product design, leading him to create a platform that prioritizes knowledge sharing and quality content. D’Angelo pursued computer science in college while also showing an interest in social science and economics. His interest in knowledge sharing led him to recognize the vast amount of untapped knowledge residing in people’s minds. He saw an opportunity to create a platform that would unlock this knowledge and provide long-lasting value.
3. Unique Features of Quora
Quora’s success is attributed to several key features that distinguish it from other platforms. A significant aspect is its Real Names Policy, which requires users to use their real identities, fostering a credible and trustworthy environment. This policy is particularly effective in attracting expert contributions. Additionally, Quora invests heavily in personalization, tailoring content to match each user’s interests. This approach is enhanced through advanced machine learning techniques used for merging duplicate questions, streamlining the user experience. The platform caters to a diverse user base, balancing the needs of those seeking specific answers with those browsing for general knowledge. The use of real names establishes accountability, inviting experts to contribute while personalizing answer recommendations through various signals such as past answers and question relevance. This careful curation and presentation of content fill knowledge gaps while avoiding redundancy.
4. Growth and Challenges
Quora’s journey has been marked by rapid growth, reaching approximately 100 million monthly unique visitors. This expansion, however, brings its own set of challenges, particularly in infrastructure, data management, and maintaining a balance between monetization and user experience. D’Angelo’s primary focus is on enhancing the core question-and-answer feature of the platform while addressing these challenges. He acknowledges the difficulty in maintaining quality content at scale, emphasizing the importance of preventing the platform from becoming chaotic and discouraging expert participation. The growth of the platform necessitates continual enhancements to infrastructure and technical systems, making scaling personalization increasingly complex. However, this expansion also enables more sophisticated machine learning and personalization capabilities.
Quora uses experimentation as a tool for improvement, focusing on personalization and finding relevant questions. These experiments compare different versions of the product, with a predefined hypothesis. Interestingly, most experiments do not meet the expected outcomes, but they are essential for learning and refinement. The drivers of this experimentation include user base growth, advancing technology, and an increasing engineering team. Surprising findings from these experiments highlight the importance of speed, transaction costs, and the perception of time in user engagement. Quora values both quantitative and qualitative feedback, using it to assess product improvements and understand user sentiment. The process of addressing failed experiments involves overcoming bias and dealing with consequences, which can be challenging, especially for those new to experimentation.
5. Personalization’s Broader Impact
Adam D’Angelo recognizes the transformative effect of personalization in internet markets, citing examples like Netflix and Google, which have thrived by offering tailored experiences. This strategy has allowed Quora to consolidate its market presence by catering to diverse user needs. D’Angelo emphasizes the pivotal role of personalization in the expansion of markets, primarily facilitated by digital platforms. In contrast to traditional media, such as magazines, where everyone views the same content, digital media provides tailored experiences for individual users. Personalization enables a small number of digital products to serve a vast audience effectively by showcasing content relevant to each user’s preferences. D’Angelo cites the example of Google as a single search engine catering to a global audience. He also mentions Netflix as a dominant player in streaming entertainment, eliminating the need for traditional cable providers and offering a wide range of content personalized for each user.
Personalization on Quora involves various factors, including followed topics, followed individuals, past behavior, and question engagement. D’Angelo uses the platform like a typical user to generate theories and ideas, which he then tests and validates with controlled experiments and user data. This approach ensures that the user experience is tailored to individual interests, enhancing relevance and engagement.
6. Experimental Approach to Product Development
Quora’s product development is heavily reliant on data-driven experimentation. A significant proportion of experiments fail, but this trial-and-error approach has led to valuable insights and improvements. The platform continually seeks to leverage advancements in machine learning to enhance user experience. Reflecting on Quora’s early days, D’Angelo acknowledges that the addition of excessive features led to increased complexity and burdened the platform’s performance and development. To address this, Quora employs a systematic approach to product development through controlled experiments. The company’s in-house experiment framework facilitates the efficient execution of these experiments, with approximately 2,000 experiments conducted to date, and about 30 running concurrently.
7. Quora’s Cultural and Philosophical Aspects
Under D’Angelo’s leadership, Quora’s culture emphasizes continuous learning and knowledge sharing. The platform hosts events to foster community among top contributors and utilizes internal resources like research groups and external speakers to maintain an environment of intellectual growth. Quora hosts an annual event for top writers to interact with the team, addressing the challenge of community building in online platforms. As CEO, D’Angelo spends most of his time on internal company matters, including management, product direction, recruiting, and staying updated on industry developments. He keeps informed through books on management, following machine learning advancements on Quora, and sharing relevant information with his staff.
The company culture promotes learning about the latest advancements in machine learning and other relevant fields. Regular reading groups, external speakers, and mailing lists facilitate knowledge sharing within the company. D’Angelo emphasizes the importance of conducting their own experiments and being cautious about relying solely on the results of other companies’ product changes. The mission “share and grow the world’s knowledge” serves as a guiding principle for the company, with hiring focused on employees who are genuinely interested in this mission. D’Angelo shares stories of users finding life-saving and life-changing answers on Quora, encouraging employees to appreciate the impact of their work beyond individual responsibilities.
8. Archiving Digital Knowledge for the Future
Quora’s vast repository of information is not just a present-day resource but a potential archive for future generations. The platform offers unique insights into contemporary perceptions and responses to global events, making it a valuable historical record. The long-term archiving of digital knowledge is crucial, as most internet services focus solely on the present. History is often colored by the writer’s perspective, lacking a sense of people’s feelings and diverse viewpoints. Quora acts as a time capsule, providing future generations with a nuanced perspective on public sentiment towards historical figures like Donald Trump. While understanding individual opinions from diaries is valuable, the internet offers a vast collection of perspectives and data. Complex issues like inequality, challenging to capture visually, could potentially be better understood using the wealth of information available online.
9. Quora’s Future and Independence
Adam D’Angelo envisions keeping Quora independent and public. He acknowledges the challenges of personalization, such as ethical considerations and market exploitation, but remains committed to enhancing user experiences while maintaining transparency and ethical standards. D’Angelo remains tight-lipped about Quora’s specific monetization plans, despite the company’s valuation exceeding a billion dollars.
Adam D’Angelo on the Future of Quora and the Power of Personalization
D’Angelo plans to remain dedicated to Quora’s mission for the long term and eventually take the company public. He believes the power of personalization is currently underrated and will revolutionize various industries. Concerns exist that personalization may not always enhance welfare, especially if companies focus solely on profits. D’Angelo suggests Facebook’s transparency in disclosing news feed personalization might be strategic to avoid misconceptions and regulations. Price discrimination through personalization is a potential concern, as companies could adjust prices based on user preferences. However, competition is essential in preventing companies from exploiting personalization solely for profit.
Conclusion
Quora, under the leadership of Adam D’Angelo, has not only revolutionized the way people seek and share knowledge but also stands as a testament to the power of personalization and quality in the digital age. Its focus on creating a credible, high-quality knowledge base, combined with its innovative approach to user experience and content personalization, positions Quora as a unique entity in the internet landscape. As the platform continues to grow and evolve, it holds the promise of not just being a contemporary knowledge repository but also an invaluable archive for future generations.
Quora's commitment to quality and adaptability has enabled its evolution from a startup to a global knowledge-sharing hub, while Adam D'Angelo's dedication and technological advancements have driven its success....
Quora prioritizes quality expert-driven content while rejecting sensationalism and adapts its platform to local values and interests while maintaining a cohesive global platform. Quora's focus on quality, expert-driven content, adaptability to local cultures, and commitment to continual improvement sets it apart from competitors....
Machine learning revolutionizes knowledge sharing by addressing bottlenecks, enhancing content relevance, and combating fake news, promising a more informed and connected world. Quora exemplifies this transformation, leveraging machine learning to personalize recommendations, rank answers, and ensure answer credibility....
Quora's shift to a remote-first model during the pandemic led to increased productivity and employee satisfaction, while also expanding the talent pool and reducing housing costs in high-cost areas. Remote work offers societal benefits such as increased labor mobility, resolution of the two-body problem, and global access to opportunities....
Data labeling is crucial for training machine learning models in autonomous vehicles, and companies like ScaleAPI offer services to streamline this process. Sensor fusion annotation combines data from multiple sensors to create a holistic view of the car's surroundings, improving decision-making and safety....
Experimental design, a key component of the quality movement, helps manufacturers identify factors impacting quality and optimize them for better results. Factorial experiments, pioneered by Frank Yates, revolutionized experimental methodology by testing all combinations of factors simultaneously, leading to deeper understanding of complex systems....
Tech entrepreneurship involves recognizing good ideas, executing them effectively, and using metrics to guide growth and retention. Startups should prioritize retention, focus on execution over ideas, and use metrics to avoid the "ring of fire" scenario where user engagement declines over time....