Easy to use Cortex: Single binary, single process #1262

tomwilkie · 2019-03-06T17:33:56Z

Build a single binary Cortex which can be told to start as an ingester, distributor, querier etc. Declare the various dependancies between modules and use code to start and stop them in the right order. Support the loading of config from a YAML file, and ship a config that allows you to start as a single process.

This replaces all the previous binaries and lite, and avoids the problem of having two (or more) copies of the init code that can get out of sync.

The follow flags have been removed:

-ingester.max-concurrent-streams: use -server.grpc-max-concurrent-streams
-query-frontend.max-recv-message-size-bytes: use -server.grpc-max-recv-msg-size-bytes
-memcached.* on the the frontend: use -frontend.memcached.*

And you need to add the following flag when running as a set of microservices:

-target={distributor,ingester,querier,query-frontend,table-manager}

TODO:

Single binary
Figure out what to do about ruler and alertmanager.
Test.
Config for single process mode.
Add getting started guide.
Test it all still works as a bunch of microservices.

docs/getting_started.md

khaines · 2019-04-07T21:34:37Z

pkg/configs/client/client.go

+)
+
+// Client is what the ruler and altermanger needs from a config store to process rules.
+type Client interface {


This interface looks to be missing a GetTemplates method for alert template files, which were added in PR #1237.

Hi @khaines - I don't understand - I don't see a GetTemplates method in that PR (or anywhere in the codebase). What am I missing?

Yeah, I think we're good - GetAlerts returns a ConfigsResponse which contains the templates, like before.

docs/getting_started.md

pkg/cortex/cortex.go

gouthamve

LGTM, but tbh, this is a biiig refactoring and it's easy to miss things.

It should only be accepted after thorough testing.

pkg/configs/db/db.go

pkg/cortex/cortex.go

cmd/cortex/main.go

Add a new module-loading system that declaritvely states which module depends on which, and then initialises them in the correct order. This also allows for a single-process mode where all the different modules run in the same process. Also: - Remove the migrations copying, and we just have a single docker image now. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

… type. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

…ddes consistently. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

…process. This is built into the docer image to make the getting started guide a bit simplier. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

- Don't forget to register chunk encoding flags. - Table manager still needs server. - Don't require auth on gRPC query frontend Process method. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie · 2019-05-13T11:19:42Z

Questions form the slack channel:

Is the yaml config optional ?

Yes, command line flags will continue to work and take precedence.

-target doesn't seem a good way to express "take on this function"

Target was chosen for its similarity to make - is run this module and all its dependancies. I'm not particularly wedded to it though

can it have more than one target ?

As the code stands "no", although there is nothing to stop us doing that in the future.

should I be able to switch over service by service from my current running system ? I.e. are all inter-process APIs the same?

Yes & yes

bboreham

Some initial points

cmd/cortex/migrations/002_immutable_configs.up.sql

pkg/cortex/modules.go

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

bboreham · 2019-05-13T16:04:12Z

This flag has also disappeared: -alertmanager.configs.url
and -alertmanager.configs.fallback

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Also, alertmanager depends on server. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie · 2019-05-14T14:47:42Z

grafana/cortex:single-binary-4ac3bf5bb is upload for testing, should fix issues with the alertmanager flags.

bboreham · 2019-05-15T13:29:52Z

cmd/querier/main.go

-	subrouter.PathPrefix("/api/v1").Handler(middleware.AuthenticateUser.Wrap(promRouter))
-	subrouter.Path("/read").Handler(middleware.AuthenticateUser.Wrap(querier.RemoteReadHandler(queryable)))
-	subrouter.Path("/validate_expr").Handler(middleware.AuthenticateUser.Wrap(http.HandlerFunc(dist.ValidateExprHandler)))
-	subrouter.Path("/user_stats").Handler(middleware.AuthenticateUser.Wrap(http.HandlerFunc(dist.UserStatsHandler)))


This path seems to have been dropped

Good catch!

Unfortunately you added it back on distributor not querier, and on a different path.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

It moved accidentally to distributor in the single-binary change #1262 Signed-off-by: Bryan Boreham <bryan@weave.works>

tomwilkie force-pushed the single-binary branch from 7225c6a to 4fd62d2 Compare March 14, 2019 17:45

tomwilkie force-pushed the single-binary branch 4 times, most recently from 3d1daa6 to fd10a6b Compare March 25, 2019 14:02

tomwilkie force-pushed the single-binary branch from fd10a6b to fe2f248 Compare April 7, 2019 08:57

weeco reviewed Apr 7, 2019

View reviewed changes

docs/getting_started.md Outdated Show resolved Hide resolved

docs/getting_started.md Show resolved Hide resolved

khaines reviewed Apr 7, 2019

View reviewed changes

weeco reviewed Apr 7, 2019

View reviewed changes

docs/getting_started.md Show resolved Hide resolved

khaines reviewed Apr 7, 2019

View reviewed changes

pkg/cortex/cortex.go Outdated Show resolved Hide resolved

khaines reviewed Apr 7, 2019

View reviewed changes

pkg/cortex/cortex.go Outdated Show resolved Hide resolved

khaines reviewed Apr 7, 2019

View reviewed changes

pkg/cortex/cortex.go Outdated Show resolved Hide resolved

tomwilkie force-pushed the single-binary branch 5 times, most recently from b1c43ae to b5bb25a Compare April 10, 2019 19:40

tomwilkie marked this pull request as ready for review May 9, 2019 11:02

tomwilkie force-pushed the single-binary branch 5 times, most recently from 52be491 to 3c0c054 Compare May 9, 2019 14:25

gouthamve approved these changes May 13, 2019

View reviewed changes

pkg/configs/db/db.go Show resolved Hide resolved

pkg/cortex/cortex.go Outdated Show resolved Hide resolved

gouthamve reviewed May 13, 2019

View reviewed changes

cmd/cortex/main.go Outdated Show resolved Hide resolved

cmd/cortex/main.go Outdated Show resolved Hide resolved

tomwilkie added 5 commits May 13, 2019 11:02

Unify the config client between the alertmanager and the ruler.

8a019d3

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Prefix frontend memcache flags with 'frontend.'

5238d2c

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Don't register the config DB flags twice.

4fa102b

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Remove the FromStr parsing hack in schema config, do it with a custom…

ea98df4

… type. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie added 7 commits May 13, 2019 11:02

Add yaml struct tags to more config fields; Manage lifecycle of overi…

a3a7538

…ddes consistently. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Add -print.config flag, which causes Cortex to print config and exit.

067009e

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Add getting started guide and example config for running as a single …

a5dee71

…process. This is built into the docer image to make the getting started guide a bit simplier. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback & minor fixups.

f8c9ac1

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Update modules.txt

453443e

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Fixes from testing as microservices.

f43d891

- Don't forget to register chunk encoding flags. - Table manager still needs server. - Don't require auth on gRPC query frontend Process method. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback.

310897b

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie force-pushed the single-binary branch 2 times, most recently from 85b0ce8 to 415e456 Compare May 13, 2019 10:43

Update CI for moved migrations.

7cb7de9

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie force-pushed the single-binary branch from 415e456 to 7cb7de9 Compare May 13, 2019 11:16

bboreham reviewed May 13, 2019

View reviewed changes

cmd/cortex/migrations/002_immutable_configs.up.sql Outdated Show resolved Hide resolved

pkg/cortex/modules.go Show resolved Hide resolved

pkg/cortex/modules.go Outdated Show resolved Hide resolved

Review feedback

cee654f

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie added 2 commits May 14, 2019 11:49

Register alertmanager config.

491fb2a

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Allow alertmanager to start speaking directly to the DB.

4ac3bf5

Also, alertmanager depends on server. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie force-pushed the single-binary branch from 7b7fd34 to 4ac3bf5 Compare May 14, 2019 14:32

bboreham reviewed May 15, 2019

View reviewed changes

Add /user_stats handler

ccd9362

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie merged commit 4bf1dcf into cortexproject:master May 16, 2019

tomwilkie deleted the single-binary branch May 16, 2019 10:47

weeco mentioned this pull request May 18, 2019

Update k8s manifests with target flags #1398

Merged

tomwilkie mentioned this pull request Jun 2, 2019

Have a way to specify userID in other way than just from X-Scope-OrgID header #323

Closed

bboreham added a commit that referenced this pull request Jun 7, 2019

Add user_stats handler back on querier

d4867e2

It moved accidentally to distributor in the single-binary change #1262 Signed-off-by: Bryan Boreham <bryan@weave.works>

bboreham mentioned this pull request Jun 7, 2019

Add user_stats handler back on querier #1451

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Easy to use Cortex: Single binary, single process #1262

Easy to use Cortex: Single binary, single process #1262

Uh oh!

tomwilkie commented Mar 6, 2019 •

edited

Loading

Uh oh!

Uh oh!

khaines Apr 7, 2019

tomwilkie Apr 10, 2019

tomwilkie Apr 10, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gouthamve left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomwilkie commented May 13, 2019

bboreham left a comment

Uh oh!

Uh oh!

Uh oh!

bboreham commented May 13, 2019 •

edited

Loading

tomwilkie commented May 14, 2019

bboreham May 15, 2019

tomwilkie May 15, 2019

bboreham Jun 7, 2019 •

edited

Loading

Easy to use Cortex: Single binary, single process #1262

Easy to use Cortex: Single binary, single process #1262

Uh oh!

Conversation

tomwilkie commented Mar 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

khaines Apr 7, 2019

Choose a reason for hiding this comment

tomwilkie Apr 10, 2019

Choose a reason for hiding this comment

tomwilkie Apr 10, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gouthamve left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomwilkie commented May 13, 2019

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bboreham commented May 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tomwilkie commented May 14, 2019

bboreham May 15, 2019

Choose a reason for hiding this comment

tomwilkie May 15, 2019

Choose a reason for hiding this comment

bboreham Jun 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

tomwilkie commented Mar 6, 2019 •

edited

Loading

bboreham commented May 13, 2019 •

edited

Loading

bboreham Jun 7, 2019 •

edited

Loading