-
Notifications
You must be signed in to change notification settings - Fork 3.2k
feat: add $push and $remove atomic operations to localAPI update for hasMany relationship fields #13997
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat: add $push and $remove atomic operations to localAPI update for hasMany relationship fields #13997
Conversation
4b53f94 to
77d79c4
Compare
d2c960c to
920fd79
Compare
…tionships Introduces atomic operations for relationship fields with hasMany: true, enabling efficient array modifications without replacing entire arrays. Key Features: - $push: Add items to relationship arrays (prevents duplicates) - $remove: Remove specific items from relationship arrays - Works with updateByID and bulk update operations - Supports polymorphic relationships, localized fields, and nested fields - Validates conflicts between data and operations parameters Implementation: - New atomic utilities (apply.ts, validate.ts) for operation processing - Updated updateByID to support operations parameter - Extended bulk update to apply operations per-document - Each document fetched fully for accurate operations - Only selected fields returned in response Testing: - Comprehensive test suite covering all operation types - Tests for select interaction and bulk updates - Error handling and validation tests Documentation: - Complete guide for high-level and low-level API usage - Examples for simple, polymorphic, and nested relationships - Bulk update documentation with query examples Related to payloadcms#13891 (database adapter-level atomic operations) Local API only. REST and GraphQL support will be added separately.
920fd79 to
031ead2
Compare
…tomic-operations-clean
| ) | ||
| } | ||
|
|
||
| const resolvedData = deepCopyObjectSimple(data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we avoid deep copying when it's not needed? Same in other files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's actually not new but just changed the position, see https://github.com/payloadcms/payload/pull/13997/files/09ba82957240505900d3ffbe2c7f62a958d6e5d4#diff-3ccb2a85d05878478da5e00ca525dbb2113206db098d4ec5a1f5618b82eec9f0L249
I assumed it is there for a reason, so I kept it.
| const applyOperationRecursively = ( | ||
| currentDoc: Record<string, unknown>, | ||
| data: Record<string, unknown>, | ||
| operationData: Record<string, unknown>, | ||
| operationFn: (currentValue: unknown[], items: unknown[]) => unknown[], | ||
| path = '', | ||
| ): void => { | ||
| for (const [fieldName, value] of Object.entries(operationData)) { | ||
| const currentPath = path ? `${path}.${fieldName}` : fieldName | ||
|
|
||
| if (Array.isArray(value)) { | ||
| // This is a leaf field - apply the operation | ||
| const currentValue = getObjectDotNotation(currentDoc, currentPath, []) | ||
|
|
||
| if (!Array.isArray(currentValue)) { | ||
| throw new Error(`Cannot execute atomic operation on non-array field "${currentPath}"`) | ||
| } | ||
|
|
||
| // Apply the operation using the callback function | ||
| const newValue = operationFn(currentValue, value) | ||
|
|
||
| // Apply the change directly to the data object | ||
| ensureNestedPath(data, fieldName) | ||
| data[fieldName] = newValue | ||
| } else if (value && typeof value === 'object') { | ||
| // This is a container field - recurse deeper | ||
| ensureNestedPath(data, fieldName) | ||
| applyOperationRecursively( | ||
| currentDoc, | ||
| data[fieldName] as Record<string, unknown>, | ||
| value as Record<string, unknown>, | ||
| operationFn, | ||
| currentPath, | ||
| ) | ||
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might be wrong, but it doesn't seem like this function actually uses DB atomic operations which are processed here
payload/packages/db-mongodb/src/updateOne.ts
Lines 57 to 71 in 89ab526
| const $inc: Record<string, number> = {} | |
| const $push: Record<string, { $each: any[] } | any> = {} | |
| const $addToSet: Record<string, { $each: any[] } | any> = {} | |
| const $pull: Record<string, { $in: any[] } | any> = {} | |
| transform({ | |
| $addToSet, | |
| $inc, | |
| $pull, | |
| $push, | |
| adapter: this, | |
| data, | |
| fields, | |
| operation: 'write', | |
| }) |
payload.update({ data: { hasManyRelationship: [...prev, new] }}) and payload.update({ data: {}, $push: { hasManyRelationship: new }) should be the same here. To actually benefit from this functionallity I think we need to rely on DB atomic operations
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right that it doesn't use the atomic operations on a database level. I don't think it is feasible to use the database operations here directly because this way we would skip all hooks and other field-level operations and introduce breaking changes once someone wants to start using atomic operations. That's why I avoided that.
Nevertheless it has significant benefits:
- On LocalAPI level it allows you to use atomic operations without considering any side effects (such as skipped hooks, dependencies to other fields, etc.).
- To achieve the same via the local API in the current setup, you would need to fetch the data, change the data, and save it. With the new atomic operations we can shortcut this.
- As a next step, we can bring this to REST and GraphQL APIs. This will eventually allow us to utilize them in the admin UI, e.g. on the bulk edit UI to allow adding / removing items from hasMany relationship fields which is currently impossible to achieve.
I agree on the performance aspect, but that's not the primary motivation here. If there is later on a way to also improve the performance, that'd certainly great, but the current way enables already a whole bunch of things which are IMO totally worth it.
Does that make sense to you?
What?
This PR adds atomic update operations (
$pushand$remove) to Local API update functions for relationship fields configured withhasMany: true. The operations work for both single-document updates (updateById) and multi-document updates. It is related to #13891, which introduced atomic operations at the database adapter layer.New API:
Why?
Right now, updating relationship arrays generally means replacing the whole array. That forces clients to first fetch the existing document, merge changes locally, and then send everything back. Atomic operations let clients express intent directly (e.g., “add this category” or “remove that relation”) without shuttling full arrays back and forth.
How?
LocalAPI
updategot a newoperationsproperty where atomic operations forhasMany: truerelationship fields can be submitted. Atomic operations are resolved inupdateById/updateinternally before collection- and field-level hooks are executed so existing hooks and validations are not affected by this change.Implementation Notes:
$pushand$removefor relationship fields only. Similar operations such as$incmight be added later.dataandoperationsConsiderations
This change proposes introducing a dedicated
operationsproperty to carry atomic updates such as$remove,$push, and$inc. Keeping these mutation operators separate fromdatapreserves the existing semantics ofdataas pure payload and avoids breaking changes - especially hook implementations - that currently assumedatacontains only field data. From a typing perspective, operations can be described with a focused, well-scoped type that models update operators without altering the shape ofdata.An obvious alternative would be to allow these operators directly within
data. While superficially simpler, that approach risks breaking existing hooks that are not prepared to receive operator syntax mixed with collection data. It would also force a significant expansion in type complexity: we’d need recursive types that permit operators at every nesting level, which would likely cascade into type errors for downstream implementations and reduce overall readability. For these reasons, the proposal favors a new operations field, keeping backward compatibility intact and making both validation and type definitions more tractable.