-
Notifications
You must be signed in to change notification settings - Fork 154
docs: architecture guide #1391
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
docs: architecture guide #1391
Conversation
…nos/git-proxy into 1325-architecture-guide-draft
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
Co-authored-by: Kris West <kristopher.west@natwest.com> Signed-off-by: Juan Escalada <97265671+jescalada@users.noreply.github.com>
…nos/git-proxy into 1325-architecture-guide-draft
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1391 +/- ##
=======================================
Coverage 81.34% 81.34%
=======================================
Files 65 65
Lines 4648 4648
Branches 792 792
=======================================
Hits 3781 3781
Misses 852 852
Partials 15 15 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
kriswest
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some corrections, questions and updates for open PRs expected to be merged shortly for the v2 release.
| - [`preReceive`](#prereceive) | ||
| - [`getDiff`](#getdiff) | ||
| - [`gitleaks`](#gitleaks) | ||
| - [`clearBareClone`](#clearbareclone) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - [`clearBareClone`](#clearbareclone) |
Removing this from the chain in #1393 and conditionally triggering it based on whether pullRemote succeeded
| After processors in the chain are done executing, [`audit`](#audit) is called to store the action along with all of its execution steps in the database for auditing purposes. | ||
|
|
||
| Then, if the action was auto-approved or auto-rejected as a result of running [`preReceive`](#prereceive), it will attempt to auto-approve or auto-reject it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| After processors in the chain are done executing, [`audit`](#audit) is called to store the action along with all of its execution steps in the database for auditing purposes. | |
| Then, if the action was auto-approved or auto-rejected as a result of running [`preReceive`](#prereceive), it will attempt to auto-approve or auto-reject it. | |
| After processors in the chain are done executing, [`audit`](#audit) is called to store the action along with all of its execution steps in the database for auditing purposes. | |
| If [`pullRemote`](#pullremote) ran successfully and cloned the repository, then [`clearBareClone`](#clearbareclone) is run to clear up that clone, freeing disk space and ensuring that the _.remote/*_ folder created does not conflict with any future pushes involving the same SHA. | |
| Finally, if the action was auto-approved or auto-rejected as a result of running [`preReceive`](#prereceive), it will attempt to auto-approve or auto-reject it. |
|
|
||
| Action chains are a list of processors that a Git operation goes through before awaiting approval. Three action chains are currently available: | ||
|
|
||
| #### Push action chain |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the pre-processer parseAction doesn't get a mention. I think some content on it is necessary, particularly as it classifies requests (push/pull/default) and hence control what subsequent steps run.
#1393 also pulls out audit and clearBareClone into a post-processors folder as they don't quite fit as pure push processors (audit is now applied to everything, clearBareClone isn't part of the chain now but conditionally invoked by it if pullRemote was)
|
|
||
| Executed when a user makes a `git clone` or `git pull` to GitProxy: | ||
|
|
||
| - [`checkRepoInAuthorisedList`](#checkrepoinauthorisedlist) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - [`checkRepoInAuthorisedList`](#checkrepoinauthorisedlist) | |
| - [`checkRepoInAuthorisedList`](#checkrepoinauthorisedlist) | |
| - | |
| At present the pull action chain is only checking that the repository is configured in Git Proxy, ensuring that it will block pull requests for unknown repositories. |
|
|
||
| This chain is executed when making any operation other than a `git push` or `git pull`. | ||
|
|
||
| - [`checkRepoInAuthorisedList`](#checkrepoinauthorisedlist) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - [`checkRepoInAuthorisedList`](#checkrepoinauthorisedlist) | |
| - [`checkRepoInAuthorisedList`](#checkrepoinauthorisedlist) | |
| - | |
| At present the default action chain is only checking that the repository is configured in Git Proxy, ensuring that it will block all git client requests for unknown repositories. |
|
|
||
| #### `contactEmail` | ||
|
|
||
| Sets the contact email for the Open Source Program Office in the attestation form: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Sets the contact email for the Open Source Program Office in the attestation form: | |
| Sets the contact email for the Open Source Program Office (or equivalent organisational contact) in the attestation form: |
|
|
||
| #### `authorisedList` | ||
|
|
||
| Defines a list of repositories that are allowed to be pushed to through the proxy. Note that **repositories can also be added through the UI or manually editing the database**. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Defines a list of repositories that are allowed to be pushed to through the proxy. Note that **repositories can also be added through the UI or manually editing the database**. | |
| Defines a initial list of repositories that are allowed to be pushed to through the proxy. Note that **repositories can also be added through the UI, API or by manually editing the database**. |
|
|
||
| ##### Setting up JWT Authentication | ||
|
|
||
| When JWT authentication is enabled, all incoming requests must provide a valid JWT access token in the UI. This can be set in the settings tab. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clarify what type of request - presumably the API (rather than the git proxy interface)
|
|
||
| When JWT authentication is enabled, all incoming requests must provide a valid JWT access token in the UI. This can be set in the settings tab. | ||
|
|
||
| If no token, or an invalid/expired token is set, requests will fail with a `401` Unauthorized response. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| If no token, or an invalid/expired token is set, requests will fail with a `401` Unauthorized response. | |
| If no token, or an invalid/expired token is sent, requests will fail with a `401` Unauthorized response. |
|
|
||
| #### `tls` | ||
|
|
||
| Allows configuring TLS (Transport Layer Security) **for the proxy** (not for the API): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Allows configuring TLS (Transport Layer Security) **for the proxy** (not for the API): | |
| Allows configuring TLS (Transport Layer Security) **for the proxy** (rather than for the API or UI): |
A pointer on how to configure for the API & UI would be useful
| - Plugins: Push/pull plugins provide more flexibility for implementing an organization's rules. For more information, see the [guide on writing your own plugins](https://git-proxy.finos.org/docs/development/plugins). | ||
| - Processors: Custom logic may require specific data within a push that isn't available at the end of the chain (where plugins are executed). In this case, the appropriate solution is to write a processor and add it to the correct place in the chain. | ||
|
|
||
| ## The nitty gritty |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just a nit but are each individual proxying/filtering logical steps worth documenting as part of the app's architecture? This seems most appropriate for some sort of reference doc (configuration, app, proxying steps, etc).
It distracts from the actual app behaviour which the rest of the doc explains pretty well.
Maybe just a quick reference diagram of the various steps showing the action pipeline?
|
|
||
| GitProxy has several main components: | ||
|
|
||
| - Proxy (`/src/proxy`): The actual proxy for Git. Git operations performed by users are intercepted here to apply the relevant **chain**. Also loads **plugins** and adds them to the chain. Runs by default on port `8000` or `8443` if TLS is enabled. These can be changed through the `GIT_PROXY_SERVER_PORT` and `GIT_PROXY_HTTPS_SERVER_PORT` environment variables. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - Proxy (`/src/proxy`): The actual proxy for Git. Git operations performed by users are intercepted here to apply the relevant **chain**. Also loads **plugins** and adds them to the chain. Runs by default on port `8000` or `8443` if TLS is enabled. These can be changed through the `GIT_PROXY_SERVER_PORT` and `GIT_PROXY_HTTPS_SERVER_PORT` environment variables. | |
| - HTTP Proxy Express app (`/src/proxy`): The actual proxy server for Git. Git operations performed by users are intercepted here, processed by various Express middleware (such as URL rewriting) and applies the relevant **chain** of actions to the payload. Customized functionality in the form of **plugins** are inserted and added to this chain as well. |
Runs by default on port
8000or8443if TLS is enabled. These can be changed through theGIT_PROXY_SERVER_PORTandGIT_PROXY_HTTPS_SERVER_PORTenvironment variables.
Probably better documented in a configuration reference + deployment guide.
| - Chain: A set of **processors** that are applied to an action (i.e. a `git push` operation) before requesting review from a user with permission to approve pushes | ||
| - Processor: AKA `Step`. A specific step in the chain where certain rules are applied. See the [list of default processors](#processors) below for more details.` | ||
| - Plugin: A custom processor that can be added externally to extend GitProxy's default policies. See the [plugin guide](https://git-proxy.finos.org/docs/development/plugins) for more details. | ||
| - Service/API (`/src/service`): Handles UI requests, user authentication to GitProxy (not to Git), database operations and some of the logic for rejection/approval. Runs by default on port `8080`, and can be configured with the `GIT_PROXY_UI_HOST` and `GIT_PROXY_UI_PORT` environment variables. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - Service/API (`/src/service`): Handles UI requests, user authentication to GitProxy (not to Git), database operations and some of the logic for rejection/approval. Runs by default on port `8080`, and can be configured with the `GIT_PROXY_UI_HOST` and `GIT_PROXY_UI_PORT` environment variables. | |
| - Backend-for-frontend (BFF) Service API, Express app (`/src/service`): Handles UI requests, user authentication to GitProxy (not to Git), database operations and some of the logic for rejection/approval. Runs by default on port `8080`, and can be configured with the `GIT_PROXY_UI_HOST` and `GIT_PROXY_UI_PORT` environment variables. |
| - Passport: The [library](https://www.passportjs.org/) used to authenticate to the GitProxy API (not the proxy itself - this depends on the Git `user.email`). Supports multiple authentication methods by default ([Local](#local), [AD](#activedirectory), [OIDC](#openid-connect)). | ||
| - Routes: All the API endpoints used by the UI and proxy to perform operations and fetch or modify GitProxy's state. Except for custom plugin and processor development, there is no need for users or GitProxy administrators to interact with the API directly. | ||
| - Configuration (`/src/config`): Loads and validates the configuration from `proxy.config.json`, or any provided config file. Allows customising several aspects of GitProxy, including databases, authentication methods, predefined allowed repositories, commit blocking rules and more. For a full list of configurable parameters, check the [config file schema reference](https://git-proxy.finos.org/docs/configuration/reference/). | ||
| - UI (`/src/ui`): Allows user-friendly interactions with the application. Shows the list of pushes requiring approval, the list of repositories that users can contribute to, and more. Also allows users to easily review the changes in a push, and approve or reject it manually according to company policy. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - UI (`/src/ui`): Allows user-friendly interactions with the application. Shows the list of pushes requiring approval, the list of repositories that users can contribute to, and more. Also allows users to easily review the changes in a push, and approve or reject it manually according to company policy. | |
| - Web UI, React (`/src/ui`): Allows user-friendly interactions with the application. Shows the list of pushes requiring approval, the list of repositories that users can contribute to, and more. Also allows users to easily review the changes in a push, and approve or reject it manually according to company policy. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Diagram looks great, the component view and interactions are very useful!
Closes #1325. Previous PR (#1335) got stuck because of a GitHub bug where comments on orphaned commits cannot be closed.
Previous PR description: