Open
Description
Describe the bug
query-frontend split query and send to queriers, then collect the response data and merge, then return the first limit lines sort by timestamp.
However, In some situations, if the total number of log lines returned by queriers are bigger than the limit, the last returned result may not be the first limit lines. I think problem comes from the heap sort (not sure if it's api usage problem).
I found this in our production environment, after research I create a simple test case below which can reproduce it easier.
To Reproduce
- Below is a test case, put it in codec_test.go#Test_codec_MergeResponse and run it. It would fails.
{
"loki backward special",
[]queryrangebase.Response{
&LokiResponse{
Status: loghttp.QueryStatusSuccess,
Direction: logproto.BACKWARD,
Limit: 10,
Version: 1,
Data: LokiData{
ResultType: loghttp.ResultTypeStream,
Result: []logproto.Stream{
{
Labels: `{foo="bar", level="debug"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 11), Line: "11"},
},
},
{
Labels: `{foo="bar", level="error"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 12), Line: "12"},
{Timestamp: time.Unix(0, 10), Line: "10"},
{Timestamp: time.Unix(0, 9), Line: "9"},
{Timestamp: time.Unix(0, 8), Line: "8"},
},
},
},
},
},
&LokiResponse{
Status: loghttp.QueryStatusSuccess,
Direction: logproto.BACKWARD,
Limit: 10,
Version: 1,
Data: LokiData{
ResultType: loghttp.ResultTypeStream,
Result: []logproto.Stream{
{
Labels: `{foo="bar", level="debug"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 6), Line: "6"},
},
},
{
Labels: `{foo="bar", level="error"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 7), Line: "7"},
{Timestamp: time.Unix(0, 5), Line: "5"},
{Timestamp: time.Unix(0, 4), Line: "4"},
},
},
},
},
},
&LokiResponse{
Status: loghttp.QueryStatusSuccess,
Direction: logproto.BACKWARD,
Limit: 10,
Version: 1,
Data: LokiData{
ResultType: loghttp.ResultTypeStream,
Result: []logproto.Stream{
{
Labels: `{foo="bar", level="debug"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 2), Line: "2"},
{Timestamp: time.Unix(0, 1), Line: "1"},
},
},
{
Labels: `{foo="bar", level="error"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 3), Line: "3"},
},
},
},
},
},
},
&LokiResponse{
Status: loghttp.QueryStatusSuccess,
Direction: logproto.BACKWARD,
Limit: 10,
Version: 1,
Statistics: stats.Result{Summary: stats.Summary{Splits: 3}},
Data: LokiData{
ResultType: loghttp.ResultTypeStream,
Result: []logproto.Stream{
{
Labels: `{foo="bar", level="error"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 12), Line: "12"},
{Timestamp: time.Unix(0, 10), Line: "10"},
{Timestamp: time.Unix(0, 9), Line: "9"},
{Timestamp: time.Unix(0, 8), Line: "8"},
{Timestamp: time.Unix(0, 7), Line: "7"},
{Timestamp: time.Unix(0, 5), Line: "5"},
{Timestamp: time.Unix(0, 4), Line: "4"},
{Timestamp: time.Unix(0, 3), Line: "3"},
},
},
{
Labels: `{foo="bar", level="debug"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 11), Line: "11"},
{Timestamp: time.Unix(0, 6), Line: "6"},
},
},
},
},
},
false,
},Expected behavior
There are 12 lines in total (nanoseconds 1 ~ 12), and limit is 10, expect return 12 ~ 3.