chore/update hatch + use dependency groups

[d-v-b]

Some updates to our development environment declaration.

• pin dependencies in CI to unblock broken tests
• use the official install-hatch action on gha to install hatch instead of using pip. this should speed up our tests.

An additional off-target change that I'm adding here:

• updates our dependency declarations to declare development dependencies as dependency groups (clearly used for local development, no impact on packaging) instead of optional dependencies (optional deps installable via pip install zarr[option] syntax). Anyone who relied on pip install zarr[tests] to get pytest will need to adjust their imports, but the remediation is simple. The benefit of this change is that we can use dependency-group-aware tools like uv with this project, e.g., uv run --group tests pytest tests.

Very happy to move this into a separate PR if people don't want to mix these concerns. These latter changes require a bit more scrutiny before we should feel comfortable merging.

closes #3726

[d-v-b]

fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

[maxrjones]

fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

this is one of several times we've encountered something similar with the hatch env setup. I think we should consider switching from hatch for environment management. pixi is more explicit in its environment setup and is catching on with friends of zarr (e.g., pydata/xarray#10888).

[d-v-b]

fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

this is one of several times we've encountered something similar with the hatch env setup. I think we should consider switching from hatch for environment management. pixi is more explicit in its environment setup and is catching on with friends of zarr (e.g., pydata/xarray#10888).

Until we have non-pypi dependencies I'm not sure pixi adds much here -- i'm not sure it would prevent the issues this PR fixes

[maxrjones]

fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

this is one of several times we've encountered something similar with the hatch env setup. I think we should consider switching from hatch for environment management. pixi is more explicit in its environment setup and is catching on with friends of zarr (e.g., pydata/xarray#10888).

Until we have non-pypi dependencies I'm not sure pixi adds much here -- i'm not sure it would prevent the issues this PR fixes

You're right that the flagship feature of non-pypi deps isn't necessary for zarr-python. At the same time, the circular zarr[test] self-reference pattern is a hatch-specific footgun — pixi doesn't support that pattern at all. Dependencies are declared explicitly per feature/environment, so there's no mechanism to accidentally create a broken self-reference in the first place. This makes environment misconfiguration like the one this PR fixes much harder to introduce silently. #2872 was the other time I discovered various issues with our hatch setup.

It's still mostly a personal preference and while I'd prefer to use pixi across projects rather than hatch for just zarr-python my opinion on this is not very strong so that's fine.

[d-v-b]

That's a good point about how pixi would prevent dep declaration cycles by design. I wonder if plain uv would offer the same protection. Either way, I do think we are in a good place in this pr thanks to the new dependency group logic. If we have more trouble with hatch in the future we should definitely revisit this.