Skip to content

test discovery 2.0 #25760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eleanorjboyd merged 29 commits into from
Feb 6, 2026
Merged

test discovery 2.0 #25760

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
ad3ae47
Checkpoint from VS Code for cloud agent session
eleanorjboyd Jan 7, 2026
7af8ea3
Phase 1: Add core infrastructure for project-based testing
Copilot Jan 7, 2026
460faa8
Phase 2: Add project discovery integration
Copilot Jan 7, 2026
b670e8e
Update activate() to support project-based testing
Copilot Jan 7, 2026
8e13f1d
updates
eleanorjboyd Jan 8, 2026
133d97e
formatting
eleanorjboyd Jan 8, 2026
aba0183
remove doc on design
eleanorjboyd Jan 8, 2026
61745de
remove unneeded
eleanorjboyd Jan 8, 2026
3b7cbf9
adding tests for helpers
eleanorjboyd Jan 8, 2026
cf2e75c
testing and refinement
eleanorjboyd Jan 8, 2026
28b34dc
tests for controller
eleanorjboyd Jan 8, 2026
7b81f07
separators and update api calls
eleanorjboyd Jan 8, 2026
b2a3a8e
checkpoint- project test nodes
eleanorjboyd Jan 9, 2026
45675a4
second checkpoint- ignore implemented
eleanorjboyd Jan 9, 2026
267007b
cleanup cleanup everybody everywhere
eleanorjboyd Jan 9, 2026
2abfbbe
remove comments
eleanorjboyd Feb 3, 2026
4e7a325
refinement
eleanorjboyd Feb 3, 2026
225ff12
remove unittest refs
eleanorjboyd Feb 3, 2026
29533cf
cleanup
eleanorjboyd Feb 3, 2026
ca14068
pytest tests
eleanorjboyd Feb 3, 2026
34965e3
test fixes
eleanorjboyd Feb 3, 2026
7c3c879
fix
eleanorjboyd Feb 3, 2026
42cd011
address comments
eleanorjboyd Feb 3, 2026
145ccc8
fixes
eleanorjboyd Feb 4, 2026
ef32ac2
testing logging
eleanorjboyd Feb 4, 2026
b4563cd
test fix
eleanorjboyd Feb 5, 2026
d472e4b
remove unneeded edits
eleanorjboyd Feb 6, 2026
0718994
lots of fun fixes
eleanorjboyd Feb 6, 2026
9050b23
fix
eleanorjboyd Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/instructions/testing-workflow.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -578,3 +578,4 @@ envConfig.inspect
- When mocking `testController.createTestItem()` in unit tests, use `typemoq.It.isAny()` for parameters when testing handler behavior (not ID/label generation logic), but consider using specific matchers (e.g., `It.is((id: string) => id.startsWith('_error_'))`) when the actual values being passed are important for correctness - this balances test precision with maintainability (2)
- Remove unused variables from test code immediately - leftover tracking variables like `validationCallCount` that aren't referenced indicate dead code that should be simplified (1)
- Use `Uri.file(path).fsPath` for both sides of path comparisons in tests to ensure cross-platform compatibility - Windows converts forward slashes to backslashes automatically (1)
- When tests fail with "Cannot stub non-existent property", the method likely moved to a different class during refactoring - find the class that owns the method and test that class directly instead of stubbing on the original class (1)
49 changes: 49 additions & 0 deletions .github/instructions/testing_feature_area.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ This document maps the testing support in the extension: discovery, execution (r
- `src/client/testing/serviceRegistry.ts` — DI/wiring for testing services.
- Workspace orchestration
- `src/client/testing/testController/workspaceTestAdapter.ts` — `WorkspaceTestAdapter` (provider-agnostic entry used by controller).
- **Project-based testing (multi-project workspaces)**
- `src/client/testing/testController/common/testProjectRegistry.ts` — `TestProjectRegistry` (manages project lifecycle, discovery, and nested project handling).
- `src/client/testing/testController/common/projectAdapter.ts` — `ProjectAdapter` interface (represents a single Python project with its own test infrastructure).
- `src/client/testing/testController/common/projectUtils.ts` — utilities for project ID generation, display names, and shared adapter creation.
- Provider adapters
- Unittest
- `src/client/testing/testController/unittest/testDiscoveryAdapter.ts`
Expand Down Expand Up @@ -151,6 +155,51 @@ The adapters in the extension don't implement test discovery/run logic themselve
- Settings are consumed by `src/client/testing/common/testConfigurationManager.ts`, `src/client/testing/configurationFactory.ts`, and adapters under `src/client/testing/testController/*` which read settings to build CLI args and env for subprocesses.
- The setting definitions and descriptions are in `package.json` and localized strings in `package.nls.json`.

## Project-based testing (multi-project workspaces)

Project-based testing enables multi-project workspace support where each Python project gets its own test tree root with its own Python environment.

> **⚠️ Note: unittest support for project-based testing is NOT yet implemented.** Project-based testing currently only works with pytest. unittest support will be added in a future PR.

### Architecture

- **TestProjectRegistry** (`testProjectRegistry.ts`): Central registry that:

- Discovers Python projects via the Python Environments API
- Creates and manages `ProjectAdapter` instances per workspace
- Computes nested project relationships and configures ignore lists
- Falls back to "legacy" single-adapter mode when API unavailable

- **ProjectAdapter** (`projectAdapter.ts`): Interface representing a single project with:
- Project identity (ID, name, URI from Python Environments API)
- Python environment with execution details
- Test framework adapters (discovery/execution)
- Nested project ignore paths (for parent projects)

### How it works

1. **Activation**: When the extension activates, `PythonTestController` checks if the Python Environments API is available.
2. **Project discovery**: `TestProjectRegistry.discoverAndRegisterProjects()` queries the API for all Python projects in each workspace.
3. **Nested handling**: `configureNestedProjectIgnores()` identifies child projects and adds their paths to parent projects' ignore lists.
4. **Test discovery**: For each project, the controller calls `project.discoveryAdapter.discoverTests()` with the project's URI. The adapter sets `PROJECT_ROOT_PATH` environment variable for the Python runner.
5. **Python side**: `get_test_root_path()` in `vscode_pytest/__init__.py` returns `PROJECT_ROOT_PATH` (if set) or falls back to `cwd`.
6. **Test tree**: Each project gets its own root node in the Test Explorer, with test IDs scoped by project ID using the `||` separator.

### Logging prefix

All project-based testing logs use the `[test-by-project]` prefix for easy filtering in the output channel.

### Key files

- Python side: `python_files/vscode_pytest/__init__.py` — `get_test_root_path()` function and `PROJECT_ROOT_PATH` environment variable.
- TypeScript: `testProjectRegistry.ts`, `projectAdapter.ts`, `projectUtils.ts`, and the discovery adapters.

### Tests

- `src/test/testing/testController/common/testProjectRegistry.unit.test.ts` — TestProjectRegistry tests
- `src/test/testing/testController/common/projectUtils.unit.test.ts` — Project utility function tests
- `python_files/tests/pytestadapter/test_get_test_root_path.py` — Python-side get_test_root_path() tests

## Coverage support (how it works)

- Coverage is supported by running the Python helper scripts with coverage enabled and then collecting a coverage payload from the runner.
Expand Down
211 changes: 211 additions & 0 deletions python_files/tests/pytestadapter/expected_discovery_test_output.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1870,3 +1870,214 @@
],
"id_": TEST_DATA_PATH_STR,
}

# =====================================================================================
# PROJECT_ROOT_PATH environment variable tests
# These test the project-based testing feature where PROJECT_ROOT_PATH changes
# the test tree root from cwd to the specified project path.
Copy link
Member Author

@eleanorjboyd eleanorjboyd Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the diagram- outline where the projects are configured in this setup

# =====================================================================================

# This is the expected output for unittest_folder when PROJECT_ROOT_PATH is set to unittest_folder.
# The root of the tree is unittest_folder (not .data), simulating project-based testing.
#
# **Project Configuration:**
# In the VS Code Python extension, projects are defined by the Python Environments extension.
# Each project has a root directory (identified by pyproject.toml, setup.py, etc.).
# When PROJECT_ROOT_PATH is set, pytest uses that path as the test tree root instead of cwd.
#
# **Test Tree Structure:**
# Without PROJECT_ROOT_PATH (legacy mode):
# └── .data (cwd = workspace root)
# └── unittest_folder
# └── test_add.py, test_subtract.py...
#
# With PROJECT_ROOT_PATH set to unittest_folder (project-based mode):
# └── unittest_folder (ROOT - set via PROJECT_ROOT_PATH env var)
# ├── test_add.py
# │ └── TestAddFunction
# │ ├── test_add_negative_numbers
# │ └── test_add_positive_numbers
# │ └── TestDuplicateFunction
# │ └── test_dup_a
# └── test_subtract.py
# └── TestSubtractFunction
# ├── test_subtract_negative_numbers
# └── test_subtract_positive_numbers
# └── TestDuplicateFunction
# └── test_dup_s
#
# Note: This reuses the unittest_folder paths defined earlier in this file.
project_root_unittest_folder_expected_output = {
"name": "unittest_folder",
"path": os.fspath(unittest_folder_path),
"type_": "folder",
"children": [
{
"name": "test_add.py",
"path": os.fspath(test_add_path),
"type_": "file",
"id_": os.fspath(test_add_path),
"children": [
{
"name": "TestAddFunction",
"path": os.fspath(test_add_path),
"type_": "class",
"children": [
{
"name": "test_add_negative_numbers",
"path": os.fspath(test_add_path),
"lineno": find_test_line_number(
"test_add_negative_numbers",
os.fspath(test_add_path),
),
"type_": "test",
"id_": get_absolute_test_id(
"test_add.py::TestAddFunction::test_add_negative_numbers",
test_add_path,
),
"runID": get_absolute_test_id(
"test_add.py::TestAddFunction::test_add_negative_numbers",
test_add_path,
),
},
{
"name": "test_add_positive_numbers",
"path": os.fspath(test_add_path),
"lineno": find_test_line_number(
"test_add_positive_numbers",
os.fspath(test_add_path),
),
"type_": "test",
"id_": get_absolute_test_id(
"test_add.py::TestAddFunction::test_add_positive_numbers",
test_add_path,
),
"runID": get_absolute_test_id(
"test_add.py::TestAddFunction::test_add_positive_numbers",
test_add_path,
),
},
],
"id_": get_absolute_test_id(
"test_add.py::TestAddFunction",
test_add_path,
),
"lineno": find_class_line_number("TestAddFunction", test_add_path),
},
{
"name": "TestDuplicateFunction",
"path": os.fspath(test_add_path),
"type_": "class",
"children": [
{
"name": "test_dup_a",
"path": os.fspath(test_add_path),
"lineno": find_test_line_number(
"test_dup_a",
os.fspath(test_add_path),
),
"type_": "test",
"id_": get_absolute_test_id(
"test_add.py::TestDuplicateFunction::test_dup_a",
test_add_path,
),
"runID": get_absolute_test_id(
"test_add.py::TestDuplicateFunction::test_dup_a",
test_add_path,
),
},
],
"id_": get_absolute_test_id(
"test_add.py::TestDuplicateFunction",
test_add_path,
),
"lineno": find_class_line_number("TestDuplicateFunction", test_add_path),
},
],
},
{
"name": "test_subtract.py",
"path": os.fspath(test_subtract_path),
"type_": "file",
"id_": os.fspath(test_subtract_path),
"children": [
{
"name": "TestSubtractFunction",
"path": os.fspath(test_subtract_path),
"type_": "class",
"children": [
{
"name": "test_subtract_negative_numbers",
"path": os.fspath(test_subtract_path),
"lineno": find_test_line_number(
"test_subtract_negative_numbers",
os.fspath(test_subtract_path),
),
"type_": "test",
"id_": get_absolute_test_id(
"test_subtract.py::TestSubtractFunction::test_subtract_negative_numbers",
test_subtract_path,
),
"runID": get_absolute_test_id(
"test_subtract.py::TestSubtractFunction::test_subtract_negative_numbers",
test_subtract_path,
),
},
{
"name": "test_subtract_positive_numbers",
"path": os.fspath(test_subtract_path),
"lineno": find_test_line_number(
"test_subtract_positive_numbers",
os.fspath(test_subtract_path),
),
"type_": "test",
"id_": get_absolute_test_id(
"test_subtract.py::TestSubtractFunction::test_subtract_positive_numbers",
test_subtract_path,
),
"runID": get_absolute_test_id(
"test_subtract.py::TestSubtractFunction::test_subtract_positive_numbers",
test_subtract_path,
),
},
],
"id_": get_absolute_test_id(
"test_subtract.py::TestSubtractFunction",
test_subtract_path,
),
"lineno": find_class_line_number("TestSubtractFunction", test_subtract_path),
},
{
"name": "TestDuplicateFunction",
"path": os.fspath(test_subtract_path),
"type_": "class",
"children": [
{
"name": "test_dup_s",
"path": os.fspath(test_subtract_path),
"lineno": find_test_line_number(
"test_dup_s",
os.fspath(test_subtract_path),
),
"type_": "test",
"id_": get_absolute_test_id(
"test_subtract.py::TestDuplicateFunction::test_dup_s",
test_subtract_path,
),
"runID": get_absolute_test_id(
"test_subtract.py::TestDuplicateFunction::test_dup_s",
test_subtract_path,
),
},
],
"id_": get_absolute_test_id(
"test_subtract.py::TestDuplicateFunction",
test_subtract_path,
),
"lineno": find_class_line_number("TestDuplicateFunction", test_subtract_path),
},
],
},
],
"id_": os.fspath(unittest_folder_path),
}
94 changes: 94 additions & 0 deletions python_files/tests/pytestadapter/test_discovery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,3 +386,97 @@ def test_plugin_collect(file, expected_const, extra_arg):
), (
f"Tests tree does not match expected value. \n Expected: {json.dumps(expected_const, indent=4)}. \n Actual: {json.dumps(actual_item.get('tests'), indent=4)}"
)


def test_project_root_path_env_var():
"""Test pytest discovery with PROJECT_ROOT_PATH environment variable set.

This simulates project-based testing where the test tree root should be
the project root (PROJECT_ROOT_PATH) rather than the workspace cwd.

When PROJECT_ROOT_PATH is set:
- The test tree root (name, path, id_) should match PROJECT_ROOT_PATH
- The cwd in the response should match PROJECT_ROOT_PATH
- Test files should be direct children of the root (not nested under a subfolder)
"""
# Use unittest_folder as our "project" subdirectory
project_path = helpers.TEST_DATA_PATH / "unittest_folder"

actual = helpers.runner_with_cwd_env(
[os.fspath(project_path), "--collect-only"],
helpers.TEST_DATA_PATH, # cwd is parent of project
{"PROJECT_ROOT_PATH": os.fspath(project_path)}, # Set project root
)

assert actual
actual_list: List[Dict[str, Any]] = actual
if actual_list is not None:
actual_item = actual_list.pop(0)

assert all(item in actual_item for item in ("status", "cwd", "error"))
assert actual_item.get("status") == "success", (
f"Status is not 'success', error is: {actual_item.get('error')}"
)
# cwd in response should be PROJECT_ROOT_PATH
assert actual_item.get("cwd") == os.fspath(project_path), (
f"Expected cwd '{os.fspath(project_path)}', got '{actual_item.get('cwd')}'"
)
assert is_same_tree(
actual_item.get("tests"),
expected_discovery_test_output.project_root_unittest_folder_expected_output,
["id_", "lineno", "name", "runID"],
), (
f"Tests tree does not match expected value. \n Expected: {json.dumps(expected_discovery_test_output.project_root_unittest_folder_expected_output, indent=4)}. \n Actual: {json.dumps(actual_item.get('tests'), indent=4)}"
)


@pytest.mark.skipif(
sys.platform == "win32",
reason="Symlinks require elevated privileges on Windows",
)
def test_symlink_with_project_root_path():
"""Test pytest discovery with both symlink and PROJECT_ROOT_PATH set.

This tests the combination of:
1. A symlinked test directory (--rootdir points to symlink)
2. PROJECT_ROOT_PATH set to the symlink path

This simulates project-based testing where the project root is a symlink,
ensuring test IDs and paths are correctly resolved through the symlink.
"""
with helpers.create_symlink(helpers.TEST_DATA_PATH, "root", "symlink_folder") as (
source,
destination,
):
assert destination.is_symlink()

# Run pytest with:
# - cwd being the resolved symlink path (simulating subprocess from node)
# - PROJECT_ROOT_PATH set to the symlink destination
actual = helpers.runner_with_cwd_env(
["--collect-only", f"--rootdir={os.fspath(destination)}"],
source, # cwd is the resolved (non-symlink) path
{"PROJECT_ROOT_PATH": os.fspath(destination)}, # Project root is the symlink
)

expected = expected_discovery_test_output.symlink_expected_discovery_output
assert actual
actual_list: List[Dict[str, Any]] = actual
if actual_list is not None:
actual_item = actual_list.pop(0)
try:
assert all(item in actual_item for item in ("status", "cwd", "error")), (
"Required keys are missing"
)
assert actual_item.get("status") == "success", (
f"Status is not 'success', error is: {actual_item.get('error')}"
)
# cwd should be the PROJECT_ROOT_PATH (the symlink destination)
assert actual_item.get("cwd") == os.fspath(destination), (
f"CWD does not match symlink path: expected {os.fspath(destination)}, got {actual_item.get('cwd')}"
)
assert actual_item.get("tests") == expected, "Tests do not match expected value"
except AssertionError as e:
# Print the actual_item in JSON format if an assertion fails
print(json.dumps(actual_item, indent=4))
pytest.fail(str(e))
Loading