Sourcery Custom Rule Fields¶

Rule schema¶

The following fields are supported in a rule:

Field	Type	Required	Description
id	string	Required	Unique, descriptive identifier, e.g. `raise-not-implemented`
description	string	Required	A description of why this rule triggered and how to resolve it
pattern	string	Required	Search for code matching this expression
condition	string	Optional	Filter pattern matches to those that pass this condition
replacement	string	Optional	Replace the matched code with this
paths	object	Optional	Paths to include or exclude when running this rule
tests	list	Optional	Tests to run and check if the rule is correct

ID¶

Required string.

The ID has to be unique. It can't collide with:

other custom rules' IDs
Sourcery core rules' IDs

Description¶

Required string.

The description is displayed in the plugins and in the command line.

Try to keep it only 1 line long.

Descriptions with a captured name¶

Descriptions may contain captured names from the pattern field. When the rule description is displayed, the placeholders will be replaced by the contents of their captures.

For instance, the following rule

rules:
  - id: do-not-assign-to-self
    pattern: ${var} = ${var}
    description: Variable "${var}" should not be assigned to itself

when applied to the snippet

x = x

will show a comment with the description Variable "x" should not be assigned to itself

Descriptions with multiple captures¶

You can use multiple captures, and each capture may appear more than once in the description:

rules:
  - id: do-not-assign-to-self
    pattern: '${var}: ${ann?} = ${var}'
    description: |
      Variable "${var}" of type "${ann}" should not be assigned to itself.
      Assign "${var}" to something else.

This rule, when applied to

x: int = x

will show the description

Variable "x" of type "int" should not be assigned to itself.
Assign "x" to something else.

In addition, if applied to the snippet

x = x

the same rule would show as description

Variable "x" of type "<no-match>" should not be assigned to itself.
Assign "x" to something else.

because there is no match for the ann capture.

Errors in descriptions with captures¶

Descriptions can only use names that are present in the rule pattern, so the following rule:

rules:
  - id: do-not-assign-to-self
    pattern: '${var}: ${ann?} = ${var}'
    description: Variable "${another}" should not be assigned to itself

would raise an error:

Name not in pattern: "another". Available names are: "ann", "var"

Pattern¶

Required string.

The pattern defines the code Sourcery is looking for when it searches for applications of your rule.

It needs to be valid Python code.

It can be :

a simple string with valid Python code, e.g. raise NotImplemented
or it can contain captures, e.g. print(${name})

For more about captures, see our guide about the pattern syntax

Condition¶

Optional string.

The condition filters pattern matches. It applies conditions to captures like so:

rule:
  - id: no-global-variables
    pattern: ${var} = ${value}
    condition: |
      var.in_module_scope()
        and not var.is_upper_case()
        and not var.starts_with("_")
    description: Don't declare `${var}` as a global variable

See the full list of condition functions.

Replacement¶

Optional string.

The replacement guides Sourcery's behaviour when it finds some code that matches the pattern:

If your rule contains a replacement: Sourcery will fix the issue.
If your rule doesn't contain a replacement: Sourcery will only flag the issue.

The replacement needs to be valid Python code. It can refer to the captures declared by the pattern, but can't declare new captures.

Empty Replacement¶

The replacement can also be an empty string. In that case, Sourcery deletes any code that matches the pattern.

For example:

rules:
  - id: remove_breakpoint
    description: Remove breakpoints from production code
    pattern: breakpoint()
    replacement: ''

If a code block contained only the code matching the pattern, applying such a rule would remove the whole content of that block. In such cases, Sourcery won't apply the rule. For example, the remove_breakpoint rule won't trigger for this code snippet:

if improbable_condition():
    breakpoint()
continue_logic()

Explanation¶

Optional string.

The explanation provides background information to this rule.

The explanation is displayed in both the VSCode and PyCharm plugins. They both support the markdown format: Feel free to add code snippets, links, listings to your explanations.

custom rule with explanation in VSCode

Frequent usages:

The motivation behind this rule.
Links to docs, articles explaining why this change is a good idea.
For rules without a replacement: some guideline how to solve the issue.

Paths¶

Optional object.

The paths field defines the scope of the rule. It provides two options:

include: Run this rule only on the specified paths.
exclude: Run this rule everywhere except of the specified paths.

Exclude paths a rule runs on¶

To ignore specific files for a rule, set the paths key:

rules:
  - id: no-asserts
    pattern: assert ${condition}, ${message?}
    description: Don't use asserts in app code
    paths:
      exclude:
        - setup.py
        - test/
        - benchmark/*/*.py

The above rule will run on all files except:

The setup.py file
Any files within the test directory
Any file matching the benchmark/*/*.py glob syntax

Note

Python's pathlib glob syntax is used to match against the given file.

Include paths a rule runs on¶

If you only want to run a rule on specific paths, use paths.include:

rules:
  - id: no-asserts
    pattern: import app
    description: Don't import app into lib directory
    paths:
      include:
        - lib

Note

If both include and exclude are used, exclude takes precedence over include.

Example:

rules:
  - id: hello-world
    pattern: print("Hello world")
    description: Include an exclamation in "Hello world!"
    paths:
      include:
        - test_*.py
        - test/*
      exclude:
        - test_abc.py
        - test/app

These files will be excluded by exclude even though they match include:

test/app/test_def.py
test_abc.py

These files will be included by include:

test_def.py
test/abc.py

These files are not included because they don't match include

test/ternary/test_abc.py
abc.py

Tests¶

Optional list.

Custom rule testing allows you to check that your rules are working as expected by providing example snippets that you would wish Sourcery to refactor or not in your codebase.

We strongly recommend that you write tests for your rules to ensure that they work exactly how you expect them to. You may even use tests to write your rule in a TDD fashion by first writing tests, and then implementing a pattern that matches them.

Always try to provide realistic examples from your codebase to make sure that Sourcery will help you in the most effective manner possible.

How to write tests¶

You can add tests to your rule by providing a tests field, which must be a list of either match or no-match examples. Match examples may be optionally accompanied by a expect key if the rule contains a replacement.

Match examples¶

Match examples are marked by the key match. Sourcery will test that your rule is correctly applied to the example code you provide, and will issue an error if it does not. Use match examples to check that your rule appears correctly where you expect it to.

For example:

rules:
  - id: do-not-call-print
    pattern: print(${arg})
    description: Do not call `print` - prefer using logging functions instead
    tests:
      - match: print(5)
      - match: print("Hey, you should use `logger.info`!")

Match-expect examples¶

If your rule contains a replacement field, you can also optionally provide a expect field to the match examples. Sourcery will then check that not only your rule is applied to the match field, but that it also outputs the code in expect:

rules:
  - id: do-not-call-print
    pattern: print(${arg})
    replacement: log(${arg})
    description: Do not call `print` - prefer using logging functions instead
    tests:
      - match: print("Match this AND check for the replacement")
        expect: log("Match this AND check for the replacement")
      - match: print("Match this, but DO NOT check for the replacement")

Note

There is no dash (-) before the expect key.

No-match examples¶

Use no-match examples to check that your rule is not too general, appearing in cases where it should not. This is the opposite of match examples: Sourcery will test that your rule does not get applied to the example you provide, and will issue an error if it does. Use them to eliminate false-positives by including snippets that you know that are correct and should not be refactored by your rule.

In the following example, we include a match example where we do want do-not-call-print to be applied. However, we also include a no-match test because we know that calls to Logger.print are fine in our codebase, and should not be flagged as issues by Sourcery:

rules:
  - id: do-not-call-print
    pattern: print(${arg})
    description: Do not call `print` - prefer using logging functions instead
    tests:
      - match: print("This is bad, we don't want calls to `print`")
      - no-match: custom_logger.print("But it is OK if `print` is just a method")

More examples¶

You can have multiple (or even zero) match and no-match tests, and they can appear in any order. In addition, test examples can be multiline strings. This way, all the following examples are valid:

rules:
  - id: do-not-call-print
    pattern: print(${arg})
    description: Do not call `print` - prefer using logging functions instead
    tests:
      - match: print(3)
      - match: print(10)
      - match: |
          def validate_processing_mode(mode):
              if mode == "legacy-mode":
                  print("Do not use 'legacy-mode' anymore")
              else:
                  print(f"Mode '{mode}' successfully validated")
  - id: replace-deprecated-function-with-new-one
    pattern: my_deprecated_function(${arg})
    replacement: new_function(${arg}, mode='legacy')
    description: |
      Function `my_deprecated_function` is deprecated.
      Use `new_function` with `mode='legacy'` instead.
    tests:
      - no-match: this_function_is_ok(3)
      - match: my_deprecated_function(3)
      - match: |
          if processing_mode == LEGACY_MODE:
              my_deprecated_function(data)
        expect: |
          if processing_mode == LEGACY_MODE:
              new_function(data, mode='legacy')
      - no-match: print("Note that `do-not-call-print` is not triggered here")

Note that rules are never applied to other rules' tests. This helps you to ensure that the behavior you are testing derives solely from the rule you are writing.

Note

The values for match, expect and no-match must always be valid Python code. Sourcery will issue parsing errors if they are not.

Failing tests¶

If any of your tests fails, this indicates that your rule is not behaving as expected. Sourcery will issue configuration errors for each failing test in your configuration file. Those errors appear in your IDE's problems panel, or inline with your YAML file if you use extensions like Error Lens.

For example, the following rule has a failing test:

rules:
  - id: replace-map-with-generator
    description: |
      Calling map with a lambda is confusing, prefer writing a generator instead
    pattern: 'map(lambda ${arg}: ${expr}, items)'
    tests:
      - match: 'map(lambda x: x**2, items)' # SUCCESS
      - match: 'map(lambda x: x**2, other_items)' # ERROR: (rules -> 0 -> tests -> 1 -> match) Match example did not trigger

For this file, Sourcery will issue (rules -> 0 -> tests -> 1 -> match) Match example did not trigger. This error can be navigated as:

in the rules field
the first rule (corresponding to index 0 since we use 0-based indexing)
in the tests field
the second test (corresponding to index 1)
in the match key

Indeed, in the example above, the problem is that our pattern is not general enough: it accept any one-element lambdas, but it requires the iterable to be a variable literally named items. To solve this, you can replace the literal items in the capture with a general capture ${items}:

rules:
  - id: replace-map-with-generator
    description: Calling map with a lambda is confusing, prefer writing a generator
      instead.
    pattern: 'map(lambda ${arg}: ${expr}, ${items})'
    tests:
      - match: 'map(lambda x: x**2, items)' # SUCCESS
      - match: 'map(lambda x: x**2, other_items)' # SUCCESS

The following list contains the most common errors you may get while writing your rules:

Invalid syntax: this is normally caused by syntax errors in the Python code you provided as example
Match example did not trigger: Sourcery was unable to find a match for your rule in the match example you provided
Match example output did not match expected result. Got code: this happens in rules with replacements where Sourcery was able to find a match in your example, but the replaced code is not equal to what you expected in the expect field
No-match example triggered: Sourcery found an unexpected match in the no-match example you provided