Sourcery Custom Rule Fields¶
Rule schema¶
The following fields are supported in a rule:
Field | Type | Required | Description |
---|---|---|---|
id | string | Required | Unique, descriptive identifier, e.g. raise-not-implemented |
description | string | Required | A description of why this rule triggered and how to resolve it |
pattern | string | Required | Search for code matching this expression |
condition | string | Optional | Filter pattern matches to those that pass this condition |
replacement | string | Optional | Replace the matched code with this |
paths | object | Optional | Paths to include or exclude when running this rule |
tests | list | Optional | Tests to run and check if the rule is correct |
ID¶
Required string.
The ID has to be unique. It can't collide with:
- other custom rules' IDs
- Sourcery core rules' IDs
Description¶
Required string.
The description
is displayed in the plugins and in the command line.
Try to keep it only 1 line long.
Descriptions with a captured name¶
Descriptions may contain captured names from the pattern
field.
When the rule description is displayed, the placeholders will be replaced by the
contents of their captures.
For instance, the following rule
rules:
- id: do-not-assign-to-self
pattern: ${var} = ${var}
description: Variable "${var}" should not be assigned to itself
when applied to the snippet
x = x
will show a comment with the description
Variable "x" should not be assigned to itself
Descriptions with multiple captures¶
You can use multiple captures, and each capture may appear more than once in the description:
rules:
- id: do-not-assign-to-self
pattern: '${var}: ${ann?} = ${var}'
description: |
Variable "${var}" of type "${ann}" should not be assigned to itself.
Assign "${var}" to something else.
This rule, when applied to
x: int = x
will show the description
Variable "x" of type "int" should not be assigned to itself.
Assign "x" to something else.
In addition, if applied to the snippet
x = x
the same rule would show as description
Variable "x" of type "<no-match>" should not be assigned to itself.
Assign "x" to something else.
because there is no match for the ann
capture.
Errors in descriptions with captures¶
Descriptions can only use names that are present in the rule pattern, so the following rule:
rules:
- id: do-not-assign-to-self
pattern: '${var}: ${ann?} = ${var}'
description: Variable "${another}" should not be assigned to itself
would raise an error:
Name not in pattern: "another". Available names are: "ann", "var"
Pattern¶
Required string.
The pattern
defines the code Sourcery is looking for when it searches for
applications of your rule.
It needs to be valid Python code.
It can be :
- a simple string with valid Python code, e.g.
raise NotImplemented
- or it can contain captures, e.g.
print(${name})
For more about captures, see our guide about the pattern syntax
Condition¶
Optional string.
The condition
filters pattern matches. It applies conditions to captures like
so:
rule:
- id: no-global-variables
pattern: ${var} = ${value}
condition: |
var.in_module_scope()
and not var.is_upper_case()
and not var.starts_with("_")
description: Don't declare `${var}` as a global variable
See the full list of condition functions.
Replacement¶
Optional string.
The replacement
guides Sourcery's behaviour when it finds some code that
matches the pattern
:
- If your rule contains a replacement: Sourcery will fix the issue.
- If your rule doesn't contain a replacement: Sourcery will only flag the issue.
The replacement
needs to be valid Python code. It can refer to the captures
declared by the pattern
, but can't declare new captures.
Empty Replacement¶
The replacement
can also be an empty string. In that case, Sourcery deletes
any code that matches
the pattern.
For example:
rules:
- id: remove_breakpoint
description: Remove breakpoints from production code
pattern: breakpoint()
replacement: ''
If a code block contained only the code matching the pattern
, applying such a
rule would remove the whole content of that block. In such cases, Sourcery won't
apply the rule. For example, the remove_breakpoint
rule won't trigger for this
code snippet:
if improbable_condition():
breakpoint()
continue_logic()
Explanation¶
Optional string.
The explanation
provides background information to this rule.
The explanation
is displayed in both the VSCode and
PyCharm plugins. They both support the markdown format:
Feel free to add code snippets, links, listings to your explanations.
Frequent usages:
- The motivation behind this rule.
- Links to docs, articles explaining why this change is a good idea.
- For rules without a replacement: some guideline how to solve the issue.
Paths¶
Optional object.
The paths
field defines the scope of the rule. It provides two options:
include
: Run this rule only on the specified paths.exclude
: Run this rule everywhere except of the specified paths.
Exclude paths a rule runs on¶
To ignore specific files for a rule, set the paths
key:
rules:
- id: no-asserts
pattern: assert ${condition}, ${message?}
description: Don't use asserts in app code
paths:
exclude:
- setup.py
- test/
- benchmark/*/*.py
The above rule will run on all files except:
- The
setup.py
file - Any files within the
test
directory - Any file matching the
benchmark/*/*.py
glob syntax
Note
Python's pathlib glob syntax is used to match against the given file.
Include paths a rule runs on¶
If you only want to run a rule on specific paths, use paths.include
:
rules:
- id: no-asserts
pattern: import app
description: Don't import app into lib directory
paths:
include:
- lib
Note
If both include and exclude are used, exclude takes precedence over include.
Example:
rules:
- id: hello-world
pattern: print("Hello world")
description: Include an exclamation in "Hello world!"
paths:
include:
- test_*.py
- test/*
exclude:
- test_abc.py
- test/app
These files will be excluded by exclude
even though they match include
:
test/app/test_def.py
test_abc.py
These files will be included by include
:
test_def.py
test/abc.py
These files are not included because they don't match include
test/ternary/test_abc.py
abc.py
Tests¶
Optional list.
Custom rule testing allows you to check that your rules are working as expected by providing example snippets that you would wish Sourcery to refactor or not in your codebase.
We strongly recommend that you write tests for your rules to ensure that they work exactly how you expect them to. You may even use tests to write your rule in a TDD fashion by first writing tests, and then implementing a pattern that matches them.
Always try to provide realistic examples from your codebase to make sure that Sourcery will help you in the most effective manner possible.
How to write tests¶
You can add tests to your rule by providing a tests
field, which must be a
list of either match
or no-match
examples. Match examples may be optionally
accompanied by a expect
key if the rule contains a replacement
.
Match examples¶
Match examples are marked by the key match
. Sourcery will test that your rule
is correctly applied to the example code you provide, and will issue an error if
it does not. Use match
examples to check that your rule appears correctly
where you expect it to.
For example:
rules:
- id: do-not-call-print
pattern: print(${arg})
description: Do not call `print` - prefer using logging functions instead
tests:
- match: print(5)
- match: print("Hey, you should use `logger.info`!")
Match-expect examples¶
If your rule contains a replacement
field, you can also optionally provide a
expect
field to the match examples. Sourcery will then check that not only
your rule is applied to the match
field, but that it also outputs the code in
expect
:
rules:
- id: do-not-call-print
pattern: print(${arg})
replacement: log(${arg})
description: Do not call `print` - prefer using logging functions instead
tests:
- match: print("Match this AND check for the replacement")
expect: log("Match this AND check for the replacement")
- match: print("Match this, but DO NOT check for the replacement")
Note
There is no dash (-
) before the expect
key.
No-match examples¶
Use no-match
examples to check that your rule is not too general, appearing in
cases where it should not. This is the opposite of match
examples: Sourcery
will test that your rule does not get applied to the example you provide, and
will issue an error if it does. Use them to eliminate false-positives by
including snippets that you know that are correct and should not be refactored
by your rule.
In the following example, we include a match
example where we do want
do-not-call-print
to be applied. However, we also include a no-match
test
because we know that calls to Logger.print
are fine in our codebase, and
should not be flagged as issues by Sourcery:
rules:
- id: do-not-call-print
pattern: print(${arg})
description: Do not call `print` - prefer using logging functions instead
tests:
- match: print("This is bad, we don't want calls to `print`")
- no-match: custom_logger.print("But it is OK if `print` is just a method")
More examples¶
You can have multiple (or even zero) match
and no-match
tests, and they can
appear in any order. In addition, test examples can be multiline strings. This
way, all the following examples are valid:
rules:
- id: do-not-call-print
pattern: print(${arg})
description: Do not call `print` - prefer using logging functions instead
tests:
- match: print(3)
- match: print(10)
- match: |
def validate_processing_mode(mode):
if mode == "legacy-mode":
print("Do not use 'legacy-mode' anymore")
else:
print(f"Mode '{mode}' successfully validated")
- id: replace-deprecated-function-with-new-one
pattern: my_deprecated_function(${arg})
replacement: new_function(${arg}, mode='legacy')
description: |
Function `my_deprecated_function` is deprecated.
Use `new_function` with `mode='legacy'` instead.
tests:
- no-match: this_function_is_ok(3)
- match: my_deprecated_function(3)
- match: |
if processing_mode == LEGACY_MODE:
my_deprecated_function(data)
expect: |
if processing_mode == LEGACY_MODE:
new_function(data, mode='legacy')
- no-match: print("Note that `do-not-call-print` is not triggered here")
Note that rules are never applied to other rules' tests. This helps you to ensure that the behavior you are testing derives solely from the rule you are writing.
Note
The values for match
, expect
and no-match
must always be valid Python code. Sourcery will issue parsing errors if they are
not.
Failing tests¶
If any of your tests fails, this indicates that your rule is not behaving as expected. Sourcery will issue configuration errors for each failing test in your configuration file. Those errors appear in your IDE's problems panel, or inline with your YAML file if you use extensions like Error Lens.
For example, the following rule has a failing test:
rules:
- id: replace-map-with-generator
description: |
Calling map with a lambda is confusing, prefer writing a generator instead
pattern: 'map(lambda ${arg}: ${expr}, items)'
tests:
- match: 'map(lambda x: x**2, items)' # SUCCESS
- match: 'map(lambda x: x**2, other_items)' # ERROR: (rules -> 0 -> tests -> 1 -> match) Match example did not trigger
For this file, Sourcery will issue
(rules -> 0 -> tests -> 1 -> match) Match example did not trigger
. This error
can be navigated as:
- in the
rules
field - the first rule (corresponding to index
0
since we use 0-based indexing) - in the
tests
field - the second test (corresponding to index
1
) - in the
match
key
Indeed, in the example above, the problem is that our pattern is not general
enough: it accept any one-element lambdas, but it requires the iterable to be a
variable literally named items
. To solve this, you can replace the literal
items
in the capture with a general capture ${items}
:
rules:
- id: replace-map-with-generator
description: Calling map with a lambda is confusing, prefer writing a generator
instead.
pattern: 'map(lambda ${arg}: ${expr}, ${items})'
tests:
- match: 'map(lambda x: x**2, items)' # SUCCESS
- match: 'map(lambda x: x**2, other_items)' # SUCCESS
The following list contains the most common errors you may get while writing your rules:
- Invalid syntax: this is normally caused by syntax errors in the Python code you provided as example
- Match example did not trigger: Sourcery was unable to find a match for
your rule in the
match
example you provided - Match example output did not match expected result. Got
code
: this happens in rules with replacements where Sourcery was able to find a match in your example, but the replaced code is not equal to what you expected in theexpect
field - No-match example triggered: Sourcery found an unexpected match in the
no-match
example you provided