Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/daily.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
strategy:
matrix:
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
fail-fast: false

steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/meta_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
with:
version: PATH
python-platform: ${{ matrix.python-platform }}
python-version: "3.9" # Oldest version supported for running scripts and tests
python-version: "3.10" # Oldest version supported for running scripts and tests
project: ./pyrightconfig.scripts_and_tests.json
stubsabot-dry-run:
name: Stubsabot dry run
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/stubtest_stdlib.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
strategy:
matrix:
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
fail-fast: false

steps:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
strategy:
matrix:
platform: ["linux", "win32", "darwin"]
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
fail-fast: false
steps:
- uses: actions/checkout@v6
Expand Down Expand Up @@ -84,7 +84,7 @@ jobs:
strategy:
matrix:
python-platform: ["Linux", "Windows", "Darwin"]
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
fail-fast: false
steps:
- uses: actions/checkout@v6
Expand Down
44 changes: 27 additions & 17 deletions tests/README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,28 @@
This directory contains several tests:

- `tests/mypy_test.py`
tests the stubs with [mypy](https://github.com/python/mypy/)
tests the stubs with [mypy](https://github.com/python/mypy/)
- `tests/pyright_test.py` tests the stubs with
[pyright](https://github.com/microsoft/pyright).
[pyright](https://github.com/microsoft/pyright).
- `tests/regr_test.py` runs mypy against the test cases for typeshed's
stubs, guarding against accidental regressions.
stubs, guarding against accidental regressions.
- `tests/check_typeshed_structure.py` checks that typeshed's directory
structure and metadata files are correct.
structure and metadata files are correct.
- `tests/stubtest_stdlib.py` checks standard library stubs against the
objects at runtime.
objects at runtime.
- `tests/stubtest_third_party.py` checks third-party stubs against the
objects at runtime.
objects at runtime.
- `tests/typecheck_typeshed.py` runs mypy against typeshed's own code
in the `tests` and `scripts` directories.
in the `tests` and `scripts` directories.

To run the tests, follow the [setup instructions](../CONTRIBUTING.md#preparing-the-environment)
in the `CONTRIBUTING.md` document. In particular, you have to run with Python 3.9+.
in the `CONTRIBUTING.md` document.

In order for `pyright_test` to work correctly, some third-party stubs
may require extra dependencies external to typeshed to be installed in your virtual environment
prior to running the test.
You can list or install all of a stubs package's external dependencies using the following script:

```bash
(.venv)$ python tests/get_external_stub_requirements.py <third_party_stub> # List external dependencies for <third_party_stub>
(.venv)$ python tests/get_external_stub_requirements.py <third_party_stub1> <third_party_stub2> # List external dependencies for <third_party_stub1> and <third_party_stub2>
Expand All @@ -31,6 +33,7 @@ You can list or install all of a stubs package's external dependencies using the
## Run all tests for a specific stub

Run using:

```bash
(.venv)$ python3 tests/runtests.py <stdlib-or-stubs>/<stub-to-test>
```
Expand All @@ -48,9 +51,10 @@ whether or not the test passes will depend on the exact version of Python
you're using, as well as various other details regarding your local environment.
For more information, see the docs on [`stubtest_stdlib.py`](#stubtest_stdlibpy) below.

## mypy\_test.py
## mypy_test.py

Run using:

```bash
(.venv)$ python3 tests/mypy_test.py
```
Expand All @@ -65,11 +69,12 @@ imported but doesn't check whether stubs match their implementation
Run `python tests/mypy_test.py --help` for information on the various configuration options
for this script.

## pyright\_test.py
## pyright_test.py

This test requires [Node.js](https://nodejs.org) to be installed. Although
typeshed runs pyright in CI, it does not currently use this script. However,
this script uses the same pyright version and configuration as the CI.

```bash
(.venv)$ python3 tests/pyright_test.py # Check all files
(.venv)$ python3 tests/pyright_test.py stdlib/sys.pyi # Check one file
Expand All @@ -81,7 +86,7 @@ checks that would typically fail on incomplete stubs (such as `Unknown` checks).
In typeshed's CI, pyright is run with these configuration settings on a subset of
the stubs in typeshed (including the standard library).

## regr\_test.py
## regr_test.py

This test runs mypy against the test cases for typeshed's stdlib and third-party
stubs. See [the REGRESSION.md document](./REGRESSION.md)
Expand All @@ -90,18 +95,20 @@ for more information about what
these test cases are for and how they work. Run `python tests/regr_test.py --help`
for information on the various configuration options.

## check\_typeshed\_structure.py
## check_typeshed_structure.py

This checks that typeshed's directory structure and metadata files are correct.

Run using:

```bash
$ python3 tests/check_typeshed_structure.py
```

## stubtest\_stdlib.py
## stubtest_stdlib.py

Run using

```bash
(.venv)$ python3 tests/stubtest_stdlib.py
```
Expand All @@ -116,7 +123,7 @@ test it automatically (or
[running the test via Github Actions](https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow#running-a-workflow)
on your typeshed fork).

As a convenience, stubtest\_stdlib.py will look for local-only allowlist files
As a convenience, stubtest_stdlib.py will look for local-only allowlist files
and use those if they are present. Only version-specific local allowlists are supported.
An example local allowlist file is
`stdlib/@tests/stubtest_allowlists/py312.txt.local`. Use caution when taking advantage of this feature;
Expand All @@ -130,13 +137,14 @@ can add to the allowlists for each affected Python version in
`stdlib/@tests/stubtest_allowlists`. Please file issues for stubtest false positives
at [mypy](https://github.com/python/mypy/issues).

## stubtest\_third\_party.py
## stubtest_third_party.py

:warning: This script downloads and executes arbitrary code from PyPI. Only run
this script locally if you know you can trust the packages you're running
stubtest on.

Run using

```bash
(.venv)$ python3 tests/stubtest_third_party.py
```
Expand All @@ -151,6 +159,7 @@ check on the command line:

If you have the runtime package installed in your local virtual environment, you can also run stubtest
directly, with

```bash
(.venv)$ MYPYPATH=<path-to-module-stubs> python3 -m mypy.stubtest \
--custom-typeshed-dir <path-to-typeshed> \
Expand All @@ -177,7 +186,7 @@ considered "incomplete".
You can help make typeshed's stubs more complete by removing
`ignore_missing_stub = true` from the `tests/METADATA.toml` file for a
third-party stubs distribution, running stubtest, and then adding things that
stubtest reports to be missing to the stub. However, note that not *everything*
stubtest reports to be missing to the stub. However, note that not _everything_
that stubtest reports to be missing should necessarily be added to the stub.
For some implementation details, it is often better to add allowlist entries
for missing objects rather than trying to match the runtime in every detail.
Expand All @@ -199,9 +208,10 @@ For Django stubs specifically, you'll need to create a `django_settings.py` file
that contains the Django settings required by the plugin. This file will be referenced by the plugin
configuration to properly validate Django-specific types during stubtest execution.

## typecheck\_typeshed.py
## typecheck_typeshed.py

Run using

```bash
(.venv)$ python3 tests/typecheck_typeshed.py
```
Expand Down
12 changes: 6 additions & 6 deletions tests/REGRESSION.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ the annotations correctly. Examples of tests like these are
Other test cases, such as the samples for `ExitStack` in
`stdlib/@tests/test_cases/check_contextlib.py` and the samples for `LogRecord`
in `stdlib/@tests/test_cases/check_logging.py`, do not relate to
stubs where the annotations are particularly complex, but they *do* relate to
stubs where the annotations are particularly complex, but they _do_ relate to
stubs where decisions have been taken that might be slightly unusual. These
test cases serve a different purpose: to check that type checkers do not emit
false-positive errors for idiomatic usage of these classes.
Expand Down Expand Up @@ -65,7 +65,7 @@ mypy's
[`--warn-unused-ignores`](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-warn-unused-ignores)
setting and pyright's
[`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings)
setting) to test instances where a type checker *should* emit some kind of
setting) to test instances where a type checker _should_ emit some kind of
error, if the stubs are correct. Both settings are enabled by default for
all `@tests/test_cases/` subdirectories in typeshed.

Expand Down Expand Up @@ -121,13 +121,13 @@ Some tests will only pass on mypy
with a specific Python version passed on the command line to the `tests/regr_test.py` script.
To mark a test-case file as being skippable on lower versions of Python,
append `-py3*` to the filename.
For example, if `foo` is a stdlib feature that's new in Python 3.11,
For example, if `foo` is a stdlib feature that's new in Python 3.14,
test cases for `foo` should be put in a file named
`stdlib/@tests/test_cases/check_foo-py311.py`.
`stdlib/@tests/test_cases/check_foo-py314.py`.
This means that mypy will only run the test case
if `--python-version 3.11`, `--python-version 3.12`, etc.
if `--python-version 3.14`, `--python-version 3.15`, etc.
is passed on the command line to `tests/regr_test.py`,
but it _won't_ run the test case if e.g. `--python-version 3.9`
but it _won't_ run the test case if e.g. `--python-version 3.13`
is passed on the command line.

However, `if sys.version_info >= (3, target):` is still required for `pyright`
Expand Down
2 changes: 1 addition & 1 deletion tests/mypy_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
print_error("Cannot import mypy. Did you install it?")
sys.exit(1)

SUPPORTED_VERSIONS = ["3.14", "3.13", "3.12", "3.11", "3.10", "3.9"]
SUPPORTED_VERSIONS = ["3.14", "3.13", "3.12", "3.11", "3.10"]
SUPPORTED_PLATFORMS = ("linux", "win32", "darwin")
DIRECTORIES_TO_TEST = [STDLIB_PATH, STUBS_PATH]

Expand Down
6 changes: 3 additions & 3 deletions tests/regr_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
TYPESHED = "typeshed"

SUPPORTED_PLATFORMS = ["linux", "darwin", "win32"]
SUPPORTED_VERSIONS = ["3.14", "3.13", "3.12", "3.11", "3.10", "3.9"]
SUPPORTED_VERSIONS = ["3.14", "3.13", "3.12", "3.11", "3.10"]


def distribution_with_test_cases(distribution_name: str) -> DistributionTests:
Expand Down Expand Up @@ -213,8 +213,8 @@ def run_testcases(

flags.extend(["--custom-typeshed-dir", str(custom_typeshed)])

# If the test-case filename ends with -py39,
# only run the test if --python-version was set to 3.9 or higher (for example)
# If the test-case filename ends with -py314,
# only run the test if --python-version was set to 3.14 or higher (for example)
for path in new_test_case_dir.rglob("*.py"):
if match := re.fullmatch(r".*-py3(\d{1,2})", path.stem):
minor_version_required = int(match[1])
Expand Down
2 changes: 1 addition & 1 deletion tests/runtests.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ def main() -> None:
parser.add_argument(
"--python-version",
default=None,
choices=("3.9", "3.10", "3.11", "3.12", "3.13", "3.14"),
choices=("3.10", "3.11", "3.12", "3.13", "3.14"),
# We're using the oldest fully supported version because it's the most likely to produce errors
# due to unsupported syntax, feature, or bug in a tool.
help="Target Python version for the test (defaults to oldest supported Python version).",
Expand Down
2 changes: 1 addition & 1 deletion tests/typecheck_typeshed.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
ReturnCode: TypeAlias = int

SUPPORTED_PLATFORMS = ("linux", "darwin", "win32")
SUPPORTED_VERSIONS = ("3.14", "3.13", "3.12", "3.11", "3.10", "3.9")
SUPPORTED_VERSIONS = ("3.14", "3.13", "3.12", "3.11", "3.10")
LOWEST_SUPPORTED_VERSION = min(SUPPORTED_VERSIONS, key=lambda x: int(x.split(".")[1]))
DIRECTORIES_TO_TEST = ("scripts", "tests")
EMPTY: list[str] = []
Expand Down