Add MongoDB persistence plugin #8908
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implement MongoDB as a persistence backend for Temporal Server. This plugin follows the Direct Factory pattern (like Cassandra). Features: - Full DataStoreFactory implementation with all stores - Multi-document transaction support for replica sets - Standalone mode fallback for development - Native MongoDB visibility store with query conversion - Docker Compose integration for local development - Comprehensive unit and integration tests New configuration: development-mongodb.yaml Documentation: docs/development/persistence/
|
|
sagdelen
force-pushed
the
feature/mongodb-plugin
branch
from
7705f5e to
df7822c
Compare
The historyNodeFilter function had a bug where the MaxNodeID constraint was not properly combined with the pagination $or condition. When using token-based pagination, the $or filter for getting the next page of results was not being AND-ed with the MaxNodeID upper bound. Fixed by using $and to properly combine the pagination condition with the MaxNodeID constraint when both are present. Also removed MongoDB-specific workarounds from ndc_test.go since the pagination bug is now fixed and MongoDB should return the same batch counts as other persistence backends.
- Remove visibility store caching in MongoDB factory to ensure each service gets a store with its own chasm.Registry - Convert ConverterError to InvalidArgument in buildQueryFilter for proper error handling - Add ActivityId → WorkflowID transformation in query converter (upstream bug fix matching resolve.go behavior)
- Add MongoDB replica set service to docker-compose - Add MongoDB to functional-test, functional-test-xdc, and functional-test-ndc CI matrices
QEMU is required for cross-platform Docker builds (arm64 on amd64 runner)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
This PR adds MongoDB as a new persistence backend for Temporal Server, implementing the Direct Factory pattern (similar to Cassandra).
New Components:
common/persistence/mongodb/- Complete MongoDB persistence plugin implementationclient/- MongoDB client abstraction layer with topology detectionconfig/development-mongodb.yaml- Development configuration for MongoDBschema/mongodb/temporal/- MongoDB schema initialization scriptsdocs/development/persistence/- Comprehensive documentationpersistence-plugin-development-guide.md- General guide for new persistence pluginsmongodb-persistence-plugin.md- MongoDB-specific implementation detailssql-vs-mongodb-persistence-comparison.md- Detailed comparison of SQL vs MongoDB schema design, denormalization decisions, and implementation challengesDocker Compose integration with MongoDB replica set setup
Prometheus alerts for MongoDB monitoring
Why?
MongoDB is a widely adopted document database that offers:
This expands Temporal's persistence options beyond SQL databases and Cassandra, giving users more flexibility in choosing their preferred database technology.
How did you test it?
Testing Details:
*_test.gofiles)common/persistence/tests/mongodb_test.go)make unit-test,make integration-test,make functional-testmake lint-code- 0 issuesPotential risks
Transaction Support: MongoDB transactions require replica set deployment. Standalone mode works but without transactional guarantees - documented as development-only configuration.
Query Conversion: The visibility query converter translates SQL-like queries to MongoDB aggregation pipelines. Complex queries may have edge cases not yet covered.
Performance: While MongoDB is performant, this is a new implementation without production-scale benchmarking. Performance characteristics may differ from SQL/Cassandra implementations.
New Dependency: Adds
go.mongodb.org/mongo-driverto go.mod. This is the official MongoDB Go driver with active maintenance.