Protobuf-net deserializes empty collection to null when the collection is a property of another type

Question

I've run in to a problem with protobuf-net and I hope it is user error and not a bug with protobuf-net.

I can serialize an empty collection (for example, IDictionary<string, string>()) and then deserialize the collection. The deserialization results in a non-null object (exactly as I serialized).

However, things get wonky if the collection belongs to another type. Serializing the custom type with a non-null empty collection results in a deserialization where the collection is null.

Below is an example of what I am working on and unit tests that illustrate the problem.

The test Test_Protobuf_EmptyDictionary_SerializeDeserialize passes. The test Test_Protobuf_EmptyCollectionAndPopulatedCollection_SerializeDeserialize fails.

[ProtoContract]
[ProtoInclude(100, typeof(ConcreteA))]
public abstract class AbstractClass
{
    [ProtoMember(1)]
    public string Name { get; set; }

    [ProtoMember(2)]
    public IDictionary<string, string> FieldsA { get; set; }

    [ProtoMember(3)]
    public IDictionary<string, string> FieldsB { get; set; }

    [ProtoMember(4)]
    public ICollection<string> FieldsC { get; set; }

    [ProtoMember(5)]
    public ICollection<string> FieldsD { get; set; } 
}

[ProtoContract]
[ProtoInclude(110, typeof(ConcreteB))]
public class ConcreteA : AbstractClass
{
    public ConcreteA() {}
}

[ProtoContract]
[ProtoInclude(120, typeof(ConcreteC))]
public class ConcreteB : ConcreteA
{
    [ProtoMember(1)]
    public int Age { get; set; }

    public ConcreteB() {}
}

[ProtoContract]
public class ConcreteC : ConcreteB
{
    [ProtoMember(1)]
    public string HairColor { get; set; }
}

[TestFixture]
public class ProtobufTests
{
    [Test]
    public void Test_Protobuf_EmptyDictionary_SerializeDeserialize()
    {
        IDictionary<string,string> dictionary = new Dictionary<string, string>();
        ICollection<string> collection = new List<string>();

        Assert.IsNotNull(dictionary);
        Assert.IsNotNull(collection);
        Assert.AreEqual(0, dictionary.Keys.Count);
        Assert.AreEqual(0, collection.Count);

        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, dictionary);
            ms.Position = 0;
            var deserialized = Serializer.Deserialize<IDictionary<string, string>>(ms);

            Assert.IsNotNull(deserialized);
            Assert.AreEqual(0, deserialized.Keys.Count);
        }

        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, collection);
            ms.Position = 0;
            var deserialized = Serializer.Deserialize<ICollection<string>>(ms);

            Assert.IsNotNull(deserialized);
            Assert.AreEqual(0, deserialized.Count);
        }
    }

    [Test]
    public void Test_Protobuf_EmptyCollectionAndPopulatedCollection_SerializeDeserialize()
    {
        ConcreteC c = new ConcreteC {
                                        FieldsA = new Dictionary<string, string>(),
                                        FieldsB = new Dictionary<string, string> {{"john", "elway"}},
                                        FieldsC = new List<string>(),
                                        FieldsD = new List<string>{"james", "jones"}
                                    };
        Assert.IsNotNull(c);
        Assert.IsNotNull(c.FieldsA);
        Assert.IsNotNull(c.FieldsB);
        Assert.IsNotNull(c.FieldsC);
        Assert.IsNotNull(c.FieldsD);
        Assert.AreEqual(0, c.FieldsA.Keys.Count);
        Assert.AreEqual(1, c.FieldsB.Keys.Count);
        Assert.AreEqual(0, c.FieldsC.Count);
        Assert.AreEqual(2, c.FieldsD.Count);

        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, c);
            ms.Position = 0;
            var deserialized = Serializer.Deserialize<ConcreteC>(ms);

            Assert.IsNotNull(deserialized);
            Assert.IsNotNull(deserialized.FieldsA); // first failing test
            Assert.IsNotNull(deserialized.FieldsB);
            Assert.IsNotNull(deserialized.FieldsC);
            Assert.IsNotNull(deserialized.FieldsD);
            Assert.AreEqual(0, deserialized.FieldsA.Keys.Count);
            Assert.AreEqual(1, deserialized.FieldsB.Keys.Count);
            Assert.AreEqual(0, deserialized.FieldsC.Count);
            Assert.AreEqual(1, deserialized.FieldsD.Count);
        }
    }
}

Marc Gravell · Accepted Answer · 2014-02-07 15:52:46Z

Ultimately, this comes down to the fact that the protobuf specification has no concept of null, and no way of expressing null. A collection is really just a repeated block (from the .proto specification); and a repeated block of 0 items is nothing whatsoever; no bytes; nothing. Since the collection member itself does not appear in protobuf, there is nowhere to say whether that thing is null vs not-null.

In xml terms, if is like using [XmlElement] to embed child items inside a parent (rather than [XmlArray] / [XmlArrayItem] - i.e.

<Foo>
    <Name>abc</Name>
    <Item>x</Item>
    <Item>y</Item>
    <Item>z</Item>
</Foo>

Here, a Foo with 0 Items would be:

<Foo>
    <Name>abc</Name>
</Foo>

from which, it is impossible to say whether the collection itself was null vs empty. Obviously protobuf isn't actually xml, but the above is meant purely as an illustrative example.

So: protobuf has no way to express this scenario, and therefore neither does protobuf-net.

In the other scenario you represent: the serialized representation of an empty list (as the root element) is: zero bytes. When you deserialize, if it finds the stream is empty, it always returns a non-null value as the root object, that being the most likely version of what you wanted.

Is there a way to change this behavior? Types that go over-the-wire which is a big use case of Protobuf tend to be immutable and usually refrain fron using Null. I would argue null should never be used and having an empty list instead should be the default value; at the very least for collection IEnumerable based fields on a type an empty list for collections would be the more common case for users. Is there an attribute or something that we could apply to the field to distinguish this case on a field-by-field basis?
@akara not currently; however, you can just initialize the property in the constructor - or in C# "current": List<Foo> Foos {get;} = new List<Foo>();
@MarcGravell Nice explanation with XmlArrayAttribute. Only difference I can think of in terms of behavior; XML Deserialize will initialize all the collection to empty collection instead of null.

felix-b · Accepted Answer · 2024-05-15 06:00:24Z

I'm sure late to the party, but just in case anyone is still looking for the answer -- this is what worked for me with protobuf-net 3.2.30 in .NET 7:

See UPDATE below

[ProtoContract] // (1) explicit contract spec
public record MyRecord
(
    [property: ProtoMember(1)] 
    int MyNum,
    
    [property: ProtoMember(2)] 
    string MyStr,
    
    [property: ProtoMember(3, IsRequired = true)]  // (2) IsRequired = true for the collection property
    Dictionary<int, MySubRecord> SubRecords
)
{
    MyRecord() : this(0, string.Empty, new())  // (3) parameterless constructor that initializes the collection
    {
    }
}

All 3 are required to get an empty collection on deserialization.

The related GitHub issue was opened and closed back in 2017. It mentions probable enhancements in protobuf-net v3, which apparently weren't implemented. I guess, there aren't any plans to make deserializing empty collections the default behavior.

UPDATE

After experimenting more with the issue, here is what I came up with:

[ProtoContract(SkipConstructor = true)]
public record MyRecord(
    [field: ProtoMember(1)] 
    int MyNum,
    [field: ProtoMember(2)] 
    string MyStr,
    [field: ProtoMember(3), NullWrappedCollection]
    Dictionary<int, MySubRecord> SubRecords
);

[ProtoContract(SkipConstructor = true)]
public record MySubRecord(
    [field: ProtoMember(1)] int SubNum,
    [field: ProtoMember(2)] string SubStr
);

Here's is what's changed:

Apply NullWrappedCollection attribute to collection-type fields where you want to deserialize a non-null empty collection
Tested with .NET 8 and not 7, although I'm not sure it matters

Other stuff related to records so not necessarily relevant:

No need for the empty constructor if you put SkipConstructor = true in the ProtoContract attribute
For some reason ProtoMember has to be applied to fields and not properties: [field: ProtoMember(...)]

increddibelly · Accepted Answer · 2020-03-20 05:41:46Z

1

make sure you have a public default constructor so protobuf-net can give you an empty object.

answered Mar 20, 2020 at 5:41

increddibelly

1,2691 gold badge15 silver badges25 bronze badges

Comments

usernotdefined · Accepted Answer · 2024-07-31 12:56:08Z

0

[NullWrappedCollection] Attribute should solve this issue. The github issue is closed

answered Jul 31, 2024 at 12:56

usernotdefined

32 bronze badges

1 Comment

Community Over a year ago

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

Protobuf-net deserializes empty collection to null when the collection is a property of another type

4 Answers 4

3 Comments

Comments

Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

1 Comment

Related