Run a pilot
Documentation Index
Fetch the complete documentation index at: https://docs.sourcery.ai/llms.txt
Use this file to discover all available pages before exploring further.
A pilot runs Sourcery on your real pull requests for the length of the trial. At the end you’ll know whether it caught bugs the team would have missed, and whether review got faster.
Decide what the pilot has to show you
Section titled “Decide what the pilot has to show you”Before you connect anything, agree on what a good result looks like. The dashboards answer these questions, so pick your criteria from them:
- Do engineers act on Sourcery’s comments? Sourcery Acceptance Rate measures it.
- Does review get faster? Median Cycle Time and Time to First Review measure it.
- Are the security findings real, and does the team fix them? Security Reports count them, and you judge a sample yourself.
Write down the result you’d need to see against each one. You’ll come back to it at the end.
Ask for help if you want it
Section titled “Ask for help if you want it”In-app chat is on every dashboard page, and during a pilot it’s the fastest way to reach us. If a finding looks wrong or you can’t find a setting, ask there. You can also email support@sourcery.ai.
Set up
Section titled “Set up”The pilot uses Analytics and on-demand security scans, both Team plan features. Signing in with GitHub or GitLab starts a Team trial. If your account is on a Pro trial, switch it to Team on the Billing page.
Turn on reviews and scans
Section titled “Turn on reviews and scans”Connect the repositories you want to evaluate with the quickstart for your platform. Sourcery reviews every new pull request from then on.
Start a security scan too. Open Repositories and click Start Scan.
Bring in the team
Section titled “Bring in the team”Everyone in your organization has access during the trial. Invite the engineers who open and review pull requests in the pilot repositories, and whoever will act on the security findings.
Configure reviews to fit your team
Section titled “Configure reviews to fit your team”Sourcery reviews sensibly out of the box, and you can configure it further. Open Review Settings to choose what it posts and which pull requests it reviews. Write your team’s recurring conventions as review rules. Set these up at the start. The results then reflect how your team runs Sourcery.
Read your stats at the end
Section titled “Read your stats at the end”Go back to the criteria you wrote down and read each one off the dashboards.
Code reviews
Section titled “Code reviews”The Analytics page shows the code-review numbers. It’s available for GitHub; if you’re piloting on GitLab, read the code-review results from the reviews on the merge requests themselves.
Sourcery Acceptance Rate, on the Overview tab, is the share of Sourcery’s comments that led to a code change. It shows whether the comments are worth acting on.
Median Cycle Time and Time to First Review, both on the Overview tab, show whether review got faster. Each shows the change against the previous period.
Security findings
Section titled “Security findings”The Security Reports page covers the scan, on GitHub and GitLab alike. The Risk Impact tab shows Open Issues and the breakdown by severity, from Critical down to Low. The Trends tab shows New against Resolved, the findings that came in during the pilot and the ones that got fixed.
The report counts findings and can’t tell you which ones are false positives. Judge that yourself on a sample.
Check in with your teammates
Section titled “Check in with your teammates”Near the end of the pilot, ask the engineers whether the comments were worth reading. If the team wants to keep Sourcery, that tells you as much as any number.
Optional extras
Section titled “Optional extras”If Jira is your tracker, add Jira autocreate to the pilot. If your engineers fix issues with coding agents, try agent fix prompts.
What it costs
Section titled “What it costs”Costs shouldn’t be a surprise at the end of a pilot. Pricing is public on the plans page.