Deploy: fixes to support version history properly

[josephjclark]

Several fixes to deploy to take version history into account

• Ensure the version hash algorithm is the same in CLI and Lightning
• Ensure the new workflow_history from app state is tracked
• When checking out a project, write a forked_from map from each workflow into openfn.yaml
• When checking for divergence, ensure forked_from is the head of the workflow in the app (but only for workflows we've changed locally...
• ... and when merging, we only merge the workflows that have changed (just to be sure we don't revert anything)
• Fetch is now destructive and will override your local project file without warning (this is correct and fine!)

Note that to work properly, this needs an update on the lighting side. I think I'm gonna sync releases.

Fixes #1231

So in short:

• Deploy, fetch and checkout (? check this!) will all warn you if you're in danger of losing work

TODO:
Need to run a full checkout after the deploy step (so that new workflows and stuff are properly represented)

Testing notes:

• If there is no local change, nothing will be deployed
• Can push a local change to the app
• Can pull a remote change from the app
• If changes are made to the same workflow in the app and CLI, a divergence warning is raised
• A local change on Workflow A is compatible with a remote change on Workflow B

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[josephjclark]

So the deploy stuff seems to be working. Got a problem now where I'm getting conflicts every time I pull, even when tehre aren't conflicts (remote is ahead of local so we can just overwrite). This might be because I'm doing a generic merge - but like on deploy, the test is a little different to general merge test

Also I've messed the tests up but that's kind of anticipated - will fix soon

[josephjclark]

Bit of a revelation here after lunch.

if the provisioner is sending correct version histories, and if the version algorithm is the same in CLI and app - then we don't need forked from. The forked_from version should be at the head of each workflow history.

So if you check out a workflow, and generate the version hash for it, you know if the workflow has changed since the last fetch because the local version is not in the workflow history. You also know if the remote version is ahead of unchanged local because the local version is in the remote history, just not at the head.

If you check out locally and made an edit, the local version hash should be different to the head of history.

So I think we have all the information we need to make good decisions here, and shouldn't need to use the fork_version metadata.

Ah no that's not quite right:

• I checkout a branch
• I make a new change
• I now have a divergent local head and a history
• I fetch from the app
• How do I know if the app's history has diverged from my local history? Forked form gives me an anchor - I can see in the remote history where we were when we edited. If we don't have forked from, we know that local head and remote head are different, but we don't know the remote head changed since we checked out.

Phew. Complex. We do still need forked_version.

[josephjclark]

I've been thinking a lot about merge logic over the last couple of days and I'm worried I'm about to start building a lot into the CLI

I think really the Project package needs to support the different kinds of merge: like syncing a local to a remote is a pariticular strategy (we detect locally changed ones, merge those into the remote, and throw if there's any remote divergence in fetched from). That's different a general "merge all these workflows into that", which I don't think does any history checking.

I need to think about this more, but it's an extension of the merge mode thing I added for sandboxes/replace

[josephjclark]

Struggled a bit with version history. Aside from a bug in lightning, I've discovered that checking for divergence on fetch is impossible in the CLI unless we always include the whole version history without squashing. Because if a version is squashed, the CLI can't compare it's local project with the remote one to decide if there's been a divergence.

I panicked for minute but the answer here is: why is fetch checking for divergence? Fetch should just download the latest version and save it to the local project.yaml file. This works on the assumption that project.yaml is never edited by a human. Because that's what the project file is - just like the old state file, it's just as snapshot of the server we pulled from. It should never be modified by the user.

Where we do need a divergence test is checkout. But here, local history is sufficient. On check out (whatever we're checking out from), if our active version has different to the forked_from version, it means we've made local changes that could be lose by checkout. So checkout must fail (unless forced)

I need to ensure there's a really good doc resource that details all this version and state theory. Because it's complex and subtle. It works great when you don't think about it - but any new user coming in to dev this stuff needs to really understand it. A good usecase for AI docs.

[josephjclark]

So I think all that's left here is to update deploy to use the only-merge-new flag, plus whatever tests are needed.

[josephjclark]

I'm expecting tests to fail here - but the local testing is looking good.

The thing I think I'm missing is: on checkout, (whether direct or through pull), warn the user if local changes are about to be lost. So, local divergence.

✅ Can pull a fresh project
✅ can pull a new remote update
✅ can deploy a change
✅ warn divergence when deploying and both have changed
✅ do not warn divergence when deploying and both have changed ON DIFFERENT WORKFLOWS
❌ warn divergence when pulling and both have changed(checkout)

Need to fix that last and then re-test all these cases

[josephjclark]

✅ Can pull a fresh project
✅ can pull a new remote update
✅ can deploy a change
✅ warn divergence when deploying and both have changed
✅ do not warn divergence when deploying and both have changed ON DIFFERENT WORKFLOWS
✅ warn divergence when pulling and both have changed(checkout)

Well, this seemed to pass my manual testing... but when doing repeated edits I did see some conflicts where I didn't expect them, and on one push it seemed to push an out of date local version up to the server

Bit confused about that.

I think where we are is that this now works. I need to get the lightning changes merged next, can't release until then. So I'm gonna get the green tick, get the lightning fixes merged, then come back and do another round of QA against openfnx next week. Maybe I can solicit a little help for that.

EDIT: Oh, Lightning already has the new hashing on main! I got my PRs mixed up. So this should be stable and ready to go... I'll push through more QA