Initial version of v2 pull integration + project info v2 #282

MarcelGeo · 2025-12-08T12:06:54Z

Resolved #280

new pull mechanism using /delta endpoint, where pull changes between current version and server are calculated from delta items

tests for both v2 and v1 features
projectinfo v2 compatibility test

Fixes

handle conflicting copy of versioned files properly without rebase logic

Pull logic updates

in pull_project_async is check if server supports v2_pull
If supports: get pull delta from server
If not: calculate pull delta from files and projects config.json
Then calculate delta items between local files and server (local delta) - delta needs to be applied to server
Compare pull_delta and local_delta -> PullAction with actions needs to be done after files are downloaded from a server defined in PullActionType enum
Starting download
After download is finished -> call pull_project_finalize
Method apply_pull_actions is then responsible to finalize PullAction based on type

+ tests for both v2 and v1 features + projectinfo v2 compatibility test

wonder-sk

let's first do a bit of refactoring... the idea is that:

we introduce a new class that captures project delta (added/updated/updated_diff/deleted) files
the project delta class would be returned by get_pull_changes and get_push_changes
there would a function that can combine server changes (from get_pull_changes) and local changes (from get_push_changes) and return a list of tasks to do at the end of the pull (e.g. copy, copy conflict, apply diff, delete - like PullTask in mobile app's merginapi)
apply_pull_changes() would just go through the list of "pull tasks" and apply them, rather than again getting local changes and doing logic based on that

mergin/client.py

mergin/client_pull.py

- models.py for some classes without specific location - clenup of non-necssary functions from previous version - refactor of pull handling using PullActions

+ descriptions + tests for appl_pull_actions + restore get_pull_changes for backward compatibility with project status

mergin/client.py

mergin/client_pull.py

varmar05 · 2026-01-08T07:07:35Z

mergin/client_pull.py

+        self.size_check = size_check  # whether we want to do merged file size check
+
+    def merge(self):
+        with open(self.dest_file, "wb") as final:


don't we need to create intermediate dirs?

it was done before. I updated this to create also directory.

varmar05 · 2026-01-08T07:09:48Z

mergin/client_pull.py

+        resp = mc.get(
+            f"/v2/projects/{mp.project_id()}/raw/diff/{self.file_path}",
+        )
+        if resp.status in [200, 206]:


Do we always download diff in single request? If so, let's state it explicitly somewhere.

added comments ⬇️

mergin/client_pull.py

varmar05 · 2026-01-08T07:35:53Z

mergin/merginproject.py

+            (DeltaChangeType.UPDATE_DIFF, DeltaChangeType.UPDATE): PullActionType.COPY_CONFLICT,
+            (DeltaChangeType.UPDATE_DIFF, DeltaChangeType.DELETE): PullActionType.COPY,
+            (DeltaChangeType.UPDATE_DIFF, DeltaChangeType.UPDATE_DIFF): PullActionType.APPLY_DIFF,  # rebase
+            (DeltaChangeType.DELETE, None): PullActionType.DELETE,


why pull action type delete here and not also for other use cases?

not sure what you mean here. better to discuss @varmar05

mergin/merginproject.py

…nitial

coveralls · 2026-01-19T16:15:55Z

Pull Request Test Coverage Report for Build 21143761159

Details

605 of 704 (85.94%) changed or added relevant lines in 8 files are covered.
18 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.3%) to 82.059%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
mergin/test/test_mergin_project.py	181	182	99.45%
mergin/merginproject.py	121	123	98.37%
mergin/client.py	14	23	60.87%
mergin/test/test_client.py	60	78	76.92%
mergin/test/test_client_pull.py	19	46	41.3%
mergin/client_pull.py	140	182	76.92%

Files with Coverage Reduction	New Missed Lines	%
mergin/client_pull.py	2	79.39%
mergin/merginproject.py	16	89.21%

Totals
Change from base Build 21135933534:	-0.3%
Covered Lines:	4638
Relevant Lines:	5652

💛 - Coveralls

MarcelGeo · 2026-02-11T09:20:10Z

mergin/client.py

+        resp = self.get(f"/v2/projects/{project_id}", params)
+        resp_json = json.load(resp)
+        project_workspace = resp_json.get("workspace")
+        return ProjectResponse(


rename to ProjectInfo

MarcelGeo · 2026-02-11T09:22:21Z

mergin/client.py

+        :type since: String
+        :param to: Optional version to track history of files to, if not given latest version is used
+        :type since: String
+        :rtype: Dict


remove noise

Copilot

Pull request overview

Implements the initial v2 pull integration using the /delta endpoint (with a v1 fallback), and adds v2 project info support plus related integration/unit tests.

Changes:

Add v2 pull flow in pull_project_async based on server feature flag (v2_pull_enabled) and project delta items.
Introduce typed models for delta items, pull actions, and v2 project info responses.
Expand test coverage for pull actions/delta calculation and project info v2 compatibility.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
`mergin/client_pull.py`	Refactors pull pipeline around `ProjectDelta` + `PullAction`, adds v2 diff download path.
`mergin/client.py`	Adds `project_info_v2()` and `get_project_delta()` API wrappers and a server-version capability check.
`mergin/merginproject.py`	Adds delta/local-delta computation and applies pull via action-based execution.
`mergin/models.py`	Adds dataclasses for delta items, pull actions, and v2 project info response typing.
`mergin/common.py`	Adds enums for delta change and pull action types.
`mergin/test/test_client.py`	Updates/extends integration tests for v2 pull and v2 project info.
`mergin/test/test_mergin_project.py`	Adds unit tests for pull-action selection, delta calculation, local delta, and applying actions.
`mergin/test/test_client_pull.py`	Adds unit tests for download item creation helpers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-11T09:24:15Z

mergin/merginproject.py

+                    if self.is_versioned_file(path):
+                        os.remove(basefile)


In the DELETE action, the basefile is only removed when the live file exists. If the live file was already removed locally but the .mergin basefile still exists, this leaves stale metadata behind. Match the previous behavior by removing the basefile whenever it exists (and optionally log when the live file is already missing).

Suggested change

if self.is_versioned_file(path):

os.remove(basefile)

# always try to remove basefile for versioned files to avoid stale metadata

if self.is_versioned_file(path) and os.path.exists(basefile):

os.remove(basefile)

Copilot · 2026-02-11T09:24:15Z

mergin/common.py

+from typing import List, Dict, Any
+from dataclasses import dataclass, field


Unused imports were added (List, Dict, Any, dataclass, field) but aren’t used in this module. Consider removing them to avoid linting issues.

Suggested change

from typing import List, Dict, Any

from dataclasses import dataclass, field

Copilot · 2026-02-11T09:24:15Z

mergin/client_pull.py

+from .utils import cleanup_tmp_dir, int_version, save_to_file
+from typing import List, Optional, Tuple


Unused imports: int_version and Tuple are imported but not used in this module. Remove them to keep imports minimal and avoid lint failures.

Suggested change

from .utils import cleanup_tmp_dir, int_version, save_to_file

from typing import List, Optional, Tuple

from .utils import cleanup_tmp_dir, save_to_file

from typing import List, Optional

Copilot · 2026-02-11T09:24:16Z

mergin/client_pull.py

    total_size = 0
-    download_list = []
-    for file_to_merge in files_to_merge:
-        download_list.extend(file_to_merge.downloaded_items)
+    download_queue_items = []
+    # Diff files downloaded without chunks (downloaded as a whole, do need to be merged)
+    for diff_file in diff_files:
+        download_queue_items.append(diff_file)
+        total_size += diff_file.size
+    for file_to_merge in download_files:


total_size is computed using DownloadDiffQueueItem.size, but these items start with size = 0 and only learn their real size after download. This makes job.total_size under-report (progress can exceed 100%, and total_size == transferred_size can be wrong). Populate diff sizes up-front (e.g., parse size from the delta response into ProjectDeltaItemDiff and set it on DownloadDiffQueueItem) or adjust progress reporting to account for unknown sizes.

Copilot · 2026-02-11T09:24:16Z

mergin/client.py

+        resp_parsed = json.load(resp)
+        return ProjectDelta(
+            to_version=resp_parsed.get("to_version"),
+            items=[
+                ProjectDeltaItem(
+                    path=item["path"],
+                    size=item.get("size"),
+                    checksum=item.get("checksum"),
+                    version=item.get("version"),
+                    change=item.get("change"),
+                    diffs=(
+                        [
+                            ProjectDeltaItemDiff(
+                                id=diff.get("id"),
+                            )
+                            for diff in item.get("diffs", [])
+                        ]
+                    ),
+                )


get_project_delta() drops diff metadata (size, version) and may set size/checksum/version to None if the server omits fields. This makes it impossible to precompute download sizes for diff items and risks runtime errors where numeric sizes are expected. Parse size/version from each diff entry (and provide safe defaults like size=0, checksum="") when constructing ProjectDeltaItem/ProjectDeltaItemDiff.

Copilot · 2026-02-11T09:24:18Z

mergin/test/test_client_pull.py

+import pytest
+from mergin.common import DeltaChangeType, CHUNK_SIZE
+from mergin.models import ProjectDeltaItem, ProjectDeltaItemDiff
+from mergin.client_pull import get_download_diff_files, get_download_diff_files, get_download_items


Duplicate import of get_download_diff_files in the same import statement. Remove the duplicate to avoid lint failures and keep imports clean.

Suggested change

from mergin.client_pull import get_download_diff_files, get_download_diff_files, get_download_items

from mergin.client_pull import get_download_diff_files, get_download_items

Copilot · 2026-02-11T09:24:18Z

mergin/test/test_mergin_project.py

+    with tempfile.TemporaryDirectory() as tmp_dir:
+        mp = MerginProject(tmp_dir)
+


MerginProject is created inside the TemporaryDirectory() context but used after the context exits. Because MerginProject sets up a file logger in that directory, this can break cleanup on Windows (open file handles prevent temp dir removal) and makes the test fragile. Keep the assertions inside the with block (or explicitly close/remove the log handler).

Copilot · 2026-02-11T09:24:18Z

mergin/test/test_client.py

+    project_info = mc.project_info_v2(mp.project_id(), mp.version())
+    assert project_info.version == "v3"
+    assert project_info.id == mp.project_id()
+    f_remote = next((f for f in project_info.files if f.path == f_updated), None)


Variable f_remote is not used.

Suggested change

f_remote = next((f for f in project_info.files if f.path == f_updated), None)

Copilot · 2026-02-11T09:24:18Z

mergin/merginproject.py

 import os
 import re
 import shutil
+from typing import List, Dict, Any, Optional


Import of 'Dict' is not used.
Import of 'Any' is not used.

Suggested change

from typing import List, Dict, Any, Optional

from typing import List, Optional

Copilot · 2026-02-11T09:24:19Z

mergin/merginproject.py

+                self.log.warning("failed to create changeset for " + path)
+                # probably the database schema has been modified if geodiff cannot create changeset.
+                # we will need to do full upload of the file
+                pass


Unnecessary 'pass' statement.

Suggested change

pass

MarcelGeo · 2026-02-11T10:26:51Z

mergin/merginproject.py

+
+        for change in added:
+            result.append(
+                ProjectDeltaItem(


Rename to ProjectDeltaChange with type arg

MarcelGeo · 2026-02-11T10:49:41Z

mergin/models.py

+    """
+
+    type: PullActionType
+    pull_delta_item: ProjectDeltaItem


Add just necessary values.

Initial dirty version of v2 pull integration + porject info v2

2220abc

+ tests for both v2 and v1 features + projectinfo v2 compatibility test

MarcelGeo changed the base branch from master to v2-pull-integration December 8, 2025 12:07

MarcelGeo changed the title ~~Initial dirty version of v2 pull integration + porject info v2~~ Initial version of v2 pull integration + project info v2 Dec 8, 2025

wonder-sk reviewed Dec 8, 2025

View reviewed changes

MarcelGeo added 7 commits December 9, 2025 08:58

Separate diff files from merge files + comments @wonder-sk

001c83d

Introduce:

68b84c1

- models.py for some classes without specific location - clenup of non-necssary functions from previous version - refactor of pull handling using PullActions

Added return types fo new functions

ae3e55c

+ descriptions + tests for appl_pull_actions + restore get_pull_changes for backward compatibility with project status

Fix test test_mergin_project

55d9e5a

add optional download_path for pattern with diff files

db7be6c

solve base file missing

8e3f2ec

Handle unfinished pull

9552372

MarcelGeo requested review from varmar05 and wonder-sk December 18, 2025 17:11

varmar05 reviewed Jan 8, 2026

View reviewed changes

MarcelGeo added 9 commits January 16, 2026 16:25

comments @varmar05

fc90173

fix files to merge renamig

411d3e1

fix download base files + import

5e2eccc

fix for is_gpgkg_open results

5891088

update tests for cases where geopackage is modified

9a18374

Black 26.1.0 upgrades

9c82771

Merge remote-tracking branch 'origin/v2-pull-integration' into pull-i…

7a144b1

…nitial

fixed imports after merge

ec500a5

pull actions tests

7fb335f

MarcelGeo requested a review from Copilot February 11, 2026 09:18

Copilot started reviewing on behalf of MarcelGeo February 11, 2026 09:18 View session

MarcelGeo commented Feb 11, 2026

View reviewed changes

Copilot AI reviewed Feb 11, 2026

View reviewed changes

MarcelGeo commented Feb 11, 2026

View reviewed changes

mergin/models.py

"""

type: PullActionType

pull_delta_item: ProjectDeltaItem

Copy link

Contributor Author

MarcelGeo Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add just necessary values.

		from typing import List, Dict, Any
		from dataclasses import dataclass, field

		from .utils import cleanup_tmp_dir, int_version, save_to_file
		from typing import List, Optional, Tuple

	from mergin.client_pull import get_download_diff_files, get_download_diff_files, get_download_items
	from mergin.client_pull import get_download_diff_files, get_download_items

		with tempfile.TemporaryDirectory() as tmp_dir:
		mp = MerginProject(tmp_dir)

	from typing import List, Dict, Any, Optional
	from typing import List, Optional

Initial version of v2 pull integration + project info v2 #282

Are you sure you want to change the base?

Initial version of v2 pull integration + project info v2 #282

Uh oh!

Conversation

MarcelGeo commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wonder-sk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coveralls commented Jan 19, 2026

Pull Request Test Coverage Report for Build 21143761159

Details

💛 - Coveralls

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

MarcelGeo commented Dec 8, 2025 •

edited

Loading