Sorry, something went wrong.
|
@zooba here's the promised walk-and-match implementation! |
Sorry, something went wrong.
There was a problem hiding this comment.
This seems fine, there's potentially a few simplifications, but it looks like a great improvement over the existing code.
Do we need any new tests to specifically trigger anything that behaves differently?
Sorry, something went wrong.
|
Thanks for the review! I've added a few more tests exercising .. and ** segments. |
Sorry, something went wrong.
This PR introduces a 'walk-and-match' strategy for handling glob patterns that include a non-terminal ** wildcard, such as **/*.py. For this example, the previous implementation recursively walked directories using os.scandir() when it expanded the ** component, and then scanned those same directories again when expanded the *.py component. This is wasteful.
In the new implementation, any components following a ** wildcard are used to build a re.Pattern object, which is used to filter the results of the recursive walk. A pattern like **/*.py uses half the number of os.scandir() calls; a pattern like **/*/*.py a third, etc.
This new algorithm does not apply if either:
In these cases we fall back to the old implementation.
This PR also replaces selector classes with selector functions. These generators directly yield results rather calling through to their successors. A new internal Path._glob() method takes care to chain these generators together, which simplifies the lazy algorithm and slightly improves performance. It should also be easier to understand and maintain.
Performance for the original #102613 repro case, with 400 nested a/ directories, and matching treatment of symlinks and hidden files:
These results were from an SSD. The improvement will be greater for slow storage (e.g. network-mounted volumes).