FAO: @sqs
Last nights release to 1.20 on VSCode shone a light on some consistent issues that happen with each update, except this time, more dramatically.
The first, and more glaring issue is that the release of 1.20 happened on a Friday evening, just before most of the engineers would clock off to go have a weekend. This, in itself, it never usually a good idea, but when the release is one that’s missing key features such as the “Copy Code” “Enhanced Context (in the chat box, not after a reply)”, “Insert at Cursor” and “New File” buttons, it begs the question “What is even happening?”
Based on the above, combined with my love for both the product and the team at Sourcegraph, I have compiled a condensed report highlighting several of the “key issues” that have been seen.
The report follows:
Condensed Report on Key Issues Affecting Cody
1. Lack of Community Communication
Effective communication within a development community is crucial for fostering collaboration, gathering feedback, and ensuring transparency. Currently, there is a significant gap in communication between the development team and the community, leading to misunderstandings, unmet expectations, and a general feeling of disconnect among users and contributors.
Current State:
- Existing forums and Discord servers are managed primarily by a single Sourcegraph staff member. In a growing community, this can lead to significant issues such as staff burnout due to the overwhelming responsibility of managing large-scale interactions single-handedly. This overburdened staff member may struggle to provide timely and effective responses, leading to delayed issue resolution and diminished community engagement. Furthermore, the staff member may experience increased stress and reduced job satisfaction, potentially impacting their overall performance and well-being.
Recommendations:
- Staff Augmentation: Increase the number of staff members dedicated to community management to distribute the workload more evenly. This will prevent burnout and ensure a consistent presence in community interactions.
- Regular Updates: Establish a schedule for regular updates through newsletters, blog posts, or community meetings to keep everyone informed about progress, upcoming features, and important announcements.
- Dedicated Channels: Create specific channels for feedback, discussions, and support. Use platforms like forums, Slack, or Discord to facilitate easy interaction between the community and the development team.
- Community Liaison Role: Appoint a community liaison responsible for bridging the gap between users and developers, ensuring that concerns and suggestions are promptly addressed. This role can also help in gathering valuable feedback and relaying it to the development team.
2. Lack of Clarity in Project Direction
A clear and shared vision is essential for guiding development efforts and aligning the team’s objectives. Currently, there is ambiguity regarding the direction in which Cody is heading, causing confusion among stakeholders.
Recommendations:
- Roadmap Development: Create and publish a comprehensive roadmap that outlines short-term and long-term goals. This roadmap should include specific milestones, timelines, and detailed descriptions of planned features.
- Regular Reviews: Implement a process for regularly reviewing and updating the roadmap to reflect changes in priorities or new insights. This ensures the project remains adaptive and responsive to evolving needs.
- Town Hall Meetings: Hold periodic town hall meetings or Q&A sessions to discuss the project’s progress, address community concerns, and gather feedback. These sessions can foster a sense of inclusion and transparency.
3. Unplanned Product Changes
Introducing new features and changes without a clear, planned goal can disrupt the existing user experience and undermine product stability. This ad-hoc approach leads to confusion and frustration among users who rely on the software’s stability.
Recommendations:
- Structured Development Lifecycle: Adopt a structured product development lifecycle with defined phases for planning, development, testing, and deployment. This lifecycle should include detailed requirements gathering, design, implementation, and review stages.
- Community Engagement: Engage with the community early in the planning process to gather input on proposed changes. Prioritize features based on user feedback, impact, and alignment with the project’s goals.
- Feature Prioritization Framework: Implement a robust feature prioritization framework that balances innovation with the refinement and enhancement of existing functionalities. Use techniques such as the MoSCoW method (Must have, Should have, Could have, Won’t have) to prioritize features effectively.
4. Feature Pushes Causing Breaking Bugs
Although new features are pushed to the pre-release branch before becoming stable, they are not given adequate time for testing and do not seem to undergo sufficient Quality Assurance (QA). This results in breaking bugs when these features reach the stable branch.
Recommendations:
- Comprehensive Testing Strategy: Establish a comprehensive testing strategy that includes unit tests, integration tests, regression tests, and user acceptance testing (UAT). This multi-layered approach ensures thorough validation of new features.
- Extended Testing Periods: Ensure that features in the pre-release branch undergo rigorous and extended testing periods. This allows for the identification and resolution of subtle bugs and performance issues.
- Feature Flags: Utilize feature flags to enable gradual rollouts and allow for easy rollback if issues are detected. Feature flags provide a safety net for managing feature deployments.
- CI/CD Pipeline: Implement a robust continuous integration/continuous deployment (CI/CD) pipeline to ensure that new features are thoroughly tested in staging environments before reaching the stable branch. This pipeline should include automated tests and code reviews.
5. Rate Limit Issues
Users have been facing prolonged issues with rate limits on their accounts, leading to frequent complaints on forums and Discord. Even after their accounts are unblocked, the problem often reoccurs within hours, causing significant frustration.
Recommendations:
- Rate-Limiting Policy Review: Conduct a thorough review of the rate-limiting policies and adjust them to better balance resource usage and user experience. This review should focus on identifying and eliminating overly restrictive limits.
- Granular Rate-Limiting Strategies: Implement more granular rate-limiting strategies that consider user behavior and usage patterns to minimize unnecessary restrictions. Tailored rate limits can enhance user satisfaction while maintaining system stability.
- Enhanced Monitoring: Enhance monitoring and logging mechanisms to quickly identify and resolve rate-limiting issues, providing users with timely support and solutions. Proactive monitoring can help preempt potential issues before they impact users.
6. Outdated Documentation
Accurate and up-to-date documentation is essential for users and developers to effectively utilize and contribute to the project. Currently, the documentation is badly outdated, leading to confusion and hindering the adoption of new features and best practices.
Recommendations:
- Documentation Team: Establish a dedicated documentation team or task force responsible for regularly updating and maintaining the documentation. This team should include technical writers and subject matter experts.
- Community Contributions: Encourage community contributions to the documentation by providing clear guidelines and recognition for contributors. Community involvement can help keep the documentation comprehensive and current.
- Documentation Integration: Integrate documentation updates into the development process, ensuring that any changes or new features are accompanied by corresponding documentation updates. This integration ensures that documentation remains aligned with the software’s capabilities.
7. User Confusion Over Privacy Policy
Users have expressed confusion and concern over the current privacy policy. The documentation surrounding privacy is perceived as vague and not reassuring, leading to distrust and hesitance to fully engage with the product.
User Concerns:
- The phrase “Sourcegraph will not train on your data without your permission” lacks clarity, with users unsure of the opt-in/opt-out mechanisms.
- Uncertainty regarding whether files in .cody/ignore are ever sent to Sourcegraph servers.
- Lack of information on security measures to protect user source code, including who has access and in what form.
- The FAQ mentions strict permissions during search and code intelligence operations but does not clarify if these operations are performed locally or if the codebase is sent to Sourcegraph servers.
- Questions about whether relevant snippets are logged on servers or merely forwarded to the LLM API, and concerns over the term “read permission” which appears to be insufficiently detailed.
- Terms regarding the use of Cody under an Enterprise License before February 15, 2024, are irrelevant to most users. There is confusion over the definition of “individual user” which is not clearly defined in the Sourcegraph Terms.
Recommendations:
- Privacy Policy Revision: Revise the privacy policy and related documentation to provide clear, detailed explanations of how user data is handled, including specific opt-in or opt-out mechanisms.
- Documentation Clarification: Clarify whether files listed in .cody/ignore are strictly ignored and ensure this information is prominently featured in the documentation.
- Security Measures Outline: Outline the security measures in place to protect user source code, detailing who has access and under what conditions.
- Operation Specificity: Specify whether search and code intelligence operations are performed locally or if any data is transmitted to Sourcegraph servers. Clearly state how and where data is processed and stored.
- Permissions Details: Address how permissions are enforced, what “read permission” entails, and provide explicit options for users to control what data Cody accesses and processes.
- Term Definitions: Clearly define the term “individual user” in the Sourcegraph Terms to ensure legal clarity and user understanding.
8. Performance Issues with VSCode Extension
Users have reported that the VSCode extension for Cody uses large amounts of RAM and system resources. Additionally, longer conversations with Cody within VSCode can cause the window to go grey, assumed to be an “out of memory” issue. This suggests that the editor tab being used by Cody attempts to exceed the memory usage allotment for tabs allowed by the VSCode API.
Recommendations:
- Optimization: Optimize the VSCode extension to reduce RAM and system resource usage. This can involve optimizing code, improving memory management, and ensuring efficient resource allocation.
- Root Cause Analysis: Investigate and address the root causes of the memory issues, ensuring that the extension operates within the memory limits imposed by the VSCode API. Thorough root cause analysis can help develop targeted solutions.
- Performance Monitoring: Implement performance monitoring and optimization tools to detect and mitigate high resource usage scenarios. These tools can provide real-time insights into the extension’s performance and help identify bottlenecks.
- User Guidelines: Provide clear guidelines and troubleshooting steps for users experiencing performance issues with the extension. This can include recommendations for optimizing their development environment and configuring the extension for better performance.
9. Contributor Concerns
Several users have expressed concerns about contributing to the source code for Cody. They feel that their pull requests (PRs) are often overlooked, and the majority do not get acknowledged. This has led to frustration and a reluctance to contribute further.
Recommendations:
- PR Review Process: Improve the process for reviewing and acknowledging PRs to ensure that all contributions are considered and responded to in a timely manner. This can involve setting up a dedicated review team and implementing SLAs (Service Level Agreements) for PR reviews.
- Clear Guidelines: Provide clear guidelines and expectations for contributors, outlining the review process, criteria for acceptance, and best practices for submitting PRs. Transparent guidelines can help contributors understand the expectations and requirements.
- Community Recognition: Foster a more inclusive and supportive contributor community by recognizing and celebrating contributions, regardless of size. Public recognition and rewards, such as highlighting top contributors in newsletters or awarding badges, can motivate and retain contributors.
- Mentorship Program: Implement a mentorship or buddy system to help new contributors navigate the contribution process and feel more supported. Mentorship programs can facilitate knowledge transfer, provide guidance, and encourage continued involvement in the project.
Conclusion
Addressing these critical issues is paramount to improving the stability, reliability, and overall user experience of Cody. Enhancing community communication by adding more staff and implementing regular updates will reduce the burden on individual staff members, preventing burnout and ensuring timely and effective communication. Clarifying the project direction through a well-defined roadmap and regular updates will align stakeholder expectations and guide development efforts.
Strategic planning of product changes and rigorous testing before feature releases will maintain software stability and user satisfaction. Resolving rate limit issues and updating documentation will ensure users have a smooth and informed experience. Providing clear and detailed privacy policies will build user trust and engagement.
Optimizing the VSCode extension for better performance will address technical issues that hamper productivity, and improving the contribution process will foster a more inclusive and supportive community, encouraging more contributions and enhancing the project’s development.
By implementing these recommendations, the development team can create a more engaged, satisfied, and collaborative community, ensuring the long-term success and growth of Cody. This comprehensive approach will not only improve the immediate user experience but also build a solid foundation for future development, innovation, and community involvement, as well as increasing the overall retention of paid users (users who are upgraded onto a Pro plan).
Best Regards,
Kynlo Akari
Independent Developer
Sourcegraph Community Expert
Drop me a message on here or on Discord and we can arrange a time!