feat: export NextToken to check token info

[xujianhai666]

• easy for custom iter.ReadMapCB, we need to use NextToken to check is there more token, for example, we check ',' token to know the end.

Change-Id: Ia335036825df098503f7008bf013b9442fe14298

[codecov]

Codecov Report

Merging #442 into master will increase coverage by 0.04%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #442      +/-   ##
==========================================
+ Coverage   86.46%   86.51%   +0.04%     
==========================================
  Files          41       41              
  Lines        5106     5108       +2     
==========================================
+ Hits         4415     4419       +4     
+ Misses        555      553       -2     
  Partials      136      136

Impacted Files	Coverage Δ
iter.go	89.25% <0%> (-0.85%)	⬇️
reflect_struct_decoder.go	81.89% <0%> (+0.53%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49c900e...ae60458. Read the comment docs.

[xujianhai666]

@AllenX2018

[AllenX2018]

1. I'm sorry but I don't exactly understand why you need this function? Could you show some code about the use case for detailed description?
2. Please add test for your commit.

[xujianhai666]

1. I'm sorry but I don't exactly understand why you need this function? Could you show some code about the use case for detailed description?
2. Please add test for your commit.
   @AllenX2018

1. fastjson of alibaba is special format, I should compatible with that map format, which there is no quota between "{" or "}", more information refer by "https://github.com/apache/rocketmq-client-go/pull/481/files#diff-01a0b164a18584e31bcfdbeaba02929cR262"
2. func NextToken has been for long time, so ut could be ommited

[xujianhai666]

ping @AllenX2018

[ShannonDing]

hello, @AllenX2018 ,
we strongly hope to have this interface to parse the map from fastjson, Could you plz help to review it?

[AllenX2018]

@ShannonDing @xujianhai666 I'm sorry for the late reply and I've been busy these days. I will check it this weekend.

[AllenX2018]

I've checkout your reference information here. I'm sorry that I'm not familiar with alibaba fastjson. But I guess what you want to decode is a json like:

{
    "xxx":{
        {"yyy":"xxx"}:{"zzz","xxx"},
        {"yyy":"xxx"}:{"zzz","xxx"},
    }
}

If so, I suggest that you register an extension to decode such a "struct" type map key. You can check the example below:

func main() {
	data := []byte(`
{
	"mqTable":{
		{"whatEverKey":"key1"}:{"whatEverValue":"value1"},
		{"whatEverKey":"key2"}:{"whatEverValue":"value2"}
	}
}`)
	table := make(map[keyStruct]valueStruct)
	jsoniter.ConfigCompatibleWithStandardLibrary.RegisterExtension(&structKeyMapExt{})

	iter := jsoniter.ParseBytes(jsoniter.ConfigCompatibleWithStandardLibrary, data)
	_ = iter.ReadMapCB(func(iterator *jsoniter.Iterator, key string) bool {
		if key == "mqTable" {
			iter.ReadVal(&table)
		}
		return true
	})

	if iter.Error != nil && iter.Error != io.EOF {
		fmt.Println(iter.Error)
	} else {
		fmt.Printf("%+v\n", table)
	}
}

type keyStruct struct {
	WhatEverKey string `json:"whatEverKey"`
}

type valueStruct struct {
	WhatEverValue string `json:"whatEverValue"`
}

type structKeyMapExt struct {
	jsoniter.DummyExtension
}

func (e *structKeyMapExt) CreateMapKeyDecoder(typ reflect2.Type) jsoniter.ValDecoder {
	if typ == reflect2.TypeOf(keyStruct{}) {
		return &structKeyDecoder{}
	}

	return nil
}

type structKeyDecoder struct {
}

func (dec *structKeyDecoder) Decode(ptr unsafe.Pointer, iter *jsoniter.Iterator) {
	iter.ReadVal((*keyStruct)(ptr))
}

You can try it to decode your json. But please use the json-iterator/go with the latest commit on master branch(don't use v1.1.9 because it has a bug). Please let me know if you have any question.

[alvarolm]

@AllenX2018

I support this commit in the case of custom Unmarshal implementation, since its required to check for bytes left after unmarshaling
https://github.com/json-iterator/go/blob/master/config.go#L349

why a custom Unmarshal implementation could be needed ?
since codecs only provide unsafe.Pointer and *Iterator but not the direct value of v that may be needed to avoid reflection cost or just direct handling

thanks !

[alvarolm]

just realized that this commit will not compile, since only exports the definition without updating references, just created #481