11

I need to work with YAML generated by Kubernetes and I'd like to be able to read specific properties with an XPath-like or jq-like DSL notation in C#.

The structure and nature of the YAML that Kubernetes generates is well-defined in most places, but in some cases is arbitrary and comes from user input, so it's not possible to define static types up front that can capture the entire structure of the YAML.

The most popular solution for deserializing and reading YAML in C# seems to be YamlDotNet, but it's mostly geared towards deserializing into fully-typed objects.

I'd rather not have to define a bunch of static types or do a lot of cumbersome casting just to get one or two fields or aggregate across them. My ideal approach would instead be something like:

var reader = new FileReader("my-file.yaml");
List<string> listOfPodNames = Yaml.Deserialize(reader)
                                  .Query(".pods[*].name")
                                  .AsList;
// expected result: list of all pod names as strings

Is this possible with YamlDotNet or another similar and well-supported tool in C#?

Update: I tried a number of approaches, but in the end, the one that worked best was reserializing to JSON and then querying with Json.NET, which has better support.

John Feminella
  • 281,997
  • 42
  • 326
  • 347
  • 1
    [This rudimentary documentation page](http://aaubry.net/pages/yamldotnet.html) refers to [this example](https://dotnetfiddle.net/rrR2Bb) of loading a Yaml stream in the "representation model". That sounds like what you're after, but I don't know offhand how clunky it is to work with. – Jon Skeet Jun 28 '18 at 19:34
  • 3
    Or as a horrible hack but one that'll get you going quickly: use YamlDotNet to convert it to JSON, then use Json.NET with LINQ to JSON :) – Jon Skeet Jun 28 '18 at 19:38
  • 1
    @DaisyShipton the converting to JSON is a nice idea. They also have an example in the the link you provided. – Nkosi Jun 28 '18 at 19:47
  • Could easily put the two libraries together and take advantage of the LINQ to JSON. – Nkosi Jun 28 '18 at 19:51
  • For example converting the yamlObject directly to a JObject https://dotnetfiddle.net/l1JWVQ – Nkosi Jun 28 '18 at 20:02
  • @Nkosi, hmm, I'm not sure if that example really accomplishes anything. What is needed is the ability to make a specific query without supplying a bunch of static types (as in the `.Query(".pods[*].name")` example). – John Feminella Jun 28 '18 at 23:42

4 Answers4

6

When using YamlDotNet Deserializing mechanism without specifying a target type, we always get a either a Dictionary (mapping),a List of KeyValuePairs (list) or a single KeyValuePair/string (scalar). The KeyValuePairs will either contain another Dictionary, another List or the actual value.

We now can implement a query functionality:

var data = new YamlQuery(yamlObject)
                        .On("pods")  // parent
                      // this functionality could be implemented as well wihtout much effort
                      //.Where("ignore").Equals(true)
                        .Get("name") // propery
                        .ToList<string>();

Edit: Multiple nested values

var data = new YamlQuery(yamlObject)
                .On("ressources")
                .On("pods")
                .Get("name")
                .ToList<string>();

Working example: https://dotnetfiddle.net/uNQPyl

using System.IO;
using System;
using System.Linq;
using YamlDotNet.Serialization;
using System.Collections.Generic;
using YamlDotNet.RepresentationModel;

namespace ConsoleApp1
{
    public class Program
    {
        public static void Main()
        {
            object yamlObject;
            using (var r = new StringReader(Program.Document))
                yamlObject = new Deserializer().Deserialize(r);

            var data = new YamlQuery(yamlObject)
                                .On("pods")
                                .Get("name")
                                .ToList<string>();
            Console.WriteLine("all names of pods");
            Console.WriteLine(string.Join(",", data));


            data = new YamlQuery(yamlObject)
                    .On("ressources")
                    .On("pods")
                    .Get("name")
                    .ToList<string>();
            Console.WriteLine("all names of pods in ressources");
            Console.WriteLine(string.Join(",", data));

        }

        public class YamlQuery
        {
            private object yamlDic;
            private string key;
            private object current;

            public YamlQuery(object yamlDic)
            {
                this.yamlDic = yamlDic;
            }

            public YamlQuery On(string key)
            {
                this.key = key;
                this.current = query<object>(this.current ?? this.yamlDic, this.key, null);
                return this;
            }
            public YamlQuery Get(string prop)
            {
                if (this.current == null)
                    throw new InvalidOperationException();

                this.current = query<object>(this.current, null, prop, this.key);
                return this;
            }

            public List<T> ToList<T>()
            {
                if (this.current == null)
                    throw new InvalidOperationException();

                return (this.current as List<object>).Cast<T>().ToList();
            }

            private IEnumerable<T> query<T>(object _dic, string key, string prop, string fromKey = null)
            {
                var result = new List<T>();
                if (_dic == null)
                    return result;
                if (typeof(IDictionary<object, object>).IsAssignableFrom(_dic.GetType()))
                {
                    var dic = (IDictionary<object, object>)_dic;
                    var d = dic.Cast<KeyValuePair<object, object>>();

                    foreach (var dd in d)
                    {
                        if (dd.Key as string == key)
                        {
                            if (prop == null)
                            { 
                                result.Add((T)dd.Value);
                            } else
                            { 
                                result.AddRange(query<T>(dd.Value, key, prop, dd.Key as string));
                            }
                        }
                        else if (fromKey == key && dd.Key as string == prop)
                        { 
                            result.Add((T)dd.Value);
                        }
                        else
                        { 
                            result.AddRange(query<T>(dd.Value, key, prop, dd.Key as string));
                        }
                    }
                }
                else if (typeof(IEnumerable<object>).IsAssignableFrom(_dic.GetType()))
                {
                    var t = (IEnumerable<object>)_dic;
                    foreach (var tt in t)
                    {
                        result.AddRange(query<T>(tt, key, prop, key));
                    }

                }
                return result;
            }
        }




        private const string Document = @"---
            receipt:    Oz-Ware Purchase Invoice
            date:        2007-08-06
            customer:
                given:   Dorothy
                family:  Gale

            pods:
                - name:   pod1
                  descrip:   Water Bucket (Filled)
                  price:     1.47
                  quantity:  4


                - name:   pod2
                  descrip:   High Heeled ""Ruby"" Slippers
                  price:     100.27
                  quantity:  1
                - name:   pod3
                  descrip:   High Heeled ""Ruby"" Slippers
                  ignore:    true
                  quantity:  1

            bill-to:  &id001
                street: |-
                        123 Tornado Alley
                        Suite 16
                city:   East Westville
                state:  KS
                pods:
                    - name: pod4
                      descrip:   High Heeled ""Ruby"" Slippers
                      price:     100.27
                      quantity:  
            ressources:
                      - pids:
                            - id: 1
                            - name: pid
                      - pods: 
                            - name: pod5
                              descrip:   High Heeled ""Ruby"" Slippers
                              price:     100.27
                              quantity:  
                            - name: pod6
                              descrip:   High Heeled ""Ruby"" Slippers
                              price:     100.27
                              quantity:  
            specialDelivery: >
                Follow the Yellow Brick
                Road to the Emerald City.
                Pay no attention to the
                man behind the curtain.

...";
    }

}
codeteq
  • 1,337
  • 6
  • 12
  • This doesn't seem quite right, since it's not correct that you only get a Dictionary or a List when deserializing. You can also have a scalar object, which isn't a List or a Dictionary. Furthermore it doesn't seem to accommodate more than one level of nesting, e.g. a record like `.resources.pods[*].names` doesn't seem to work with this approach. – John Feminella Jun 29 '18 at 14:18
  • no, scalar values are deserialized into key,value. The example yaml has scalar values btw. Please see my edit above for example of nested records. – codeteq Jun 29 '18 at 16:06
  • deserializing yaml files containing only a scalar object with the actual yaml data i.e. result in just a string. you will need to extract the actual yaml from the scalar value before using the YamlQuery on it. – codeteq Jun 29 '18 at 19:02
  • Try deserializing a YAML file containing only: `foo: bar`. The value of `foo` is the object `"bar"` — which is an `Object`, not a `Dictionary`. – John Feminella Jul 01 '18 at 02:42
  • 1
    @JohnFeminella there's no need to discuss about the fact that you always get Dictionaries or a List of KeyValues or in case of scalar a string. Even foo: "bar" results in a list of KeyValue Pairs. it's the nature of YamlDotNet. My code does exactly what your looking for. – codeteq Jul 01 '18 at 13:29
  • 1
    You either have a bug or have misunderstood my reference example in the earlier comment. `foo: bar` should be a scalar. `foo: - one - two ...` should be a sequence. `foo: bar: ..., baz: ...` should be a mapping. The creator of YamlDotNet agrees with this: https://github.com/aaubry/YamlDotNet/issues/332#issuecomment-399883609 – John Feminella Jul 02 '18 at 19:38
  • 1
    This is totally correct. But, as i pointed out: when deserializing without specifing a type with YamlDotNet the types of the return values will be always either a keyValuePair/string (scalar), a list (sequenz) or a dictionary(mapping) and there is no doubt about it, you can try this yourself to be sure. – codeteq Jul 02 '18 at 19:58
2

Another approach you can use is converting YAML to JSON and then query it. though it would be a more time-consuming approach you can surely query a JSON easily then YAML.

Here is how you can do it

Convert YAML To JSON

    public class ConvertYamlToJson
    {
        private readonly ITestOutputHelper output;

        public ConvertYamlToJson(ITestOutputHelper output)
        {
            this.output = output;
        }

        [Sample(
            DisplayName = "Convert YAML to JSON",
            Description = "Shows how to convert a YAML document to JSON."
        )]
        public void Main()
        {
            // convert string/file to YAML object
            var r = new StringReader(@"
scalar: a scalar
sequence:
  - one
  - two
");
            var deserializer = new DeserializerBuilder().Build();
            var yamlObject = deserializer.Deserialize(r);

            var serializer = new SerializerBuilder()
                .JsonCompatible()
                .Build();

            var json = serializer.Serialize(yamlObject);

            output.WriteLine(json);
        }
    }

Ref:- Convert YAML to JSON

Query JSON

 string json = @"
            {
              ""client_id"": ""26075235"",
              ""client_version"": ""1.0.0"",
              ""event"": ""app.uninstall"",
              ""timestamp"": 1478741247,
              ""data"": {
                ""user_id"": ""62581379"",
                ""site_id"": ""837771289247593785"",
                ""platform_app_id"": ""26075235""
              }
            }";

        JObject jo = JObject.Parse(json);

        Console.WriteLine("User ID: " + (string)jo.SelectToken("data.user_id"));

Ref:- JSON.NET JObject - how do I get value from this nested JSON structure

Mihir Dave
  • 3,869
  • 1
  • 10
  • 26
1

There is the following GitHub project: YamlDotNet.Dynamic

It leverages the dynamic type in C# and therefore you don't need to define static types.


A different approach would be to convert into Json and use Newtonsoft.Json, which supports dynamic types, too.

johannespartin
  • 374
  • 2
  • 13
-1

What do you think about that? you can cast only the keys you need :)

List<Dictionary<string, object>> mydic = new Deserializer().Deserialize(r);
                foreach(Dictionary<string, object> wiii in mydic)
                {
                    bool value = false;
                    if (wiii.ContainsKey("yourKeyname"))
                        value = (bool)wiii["yourKeyname"]; //<-- Here you can cast it in the type you wish
                }

EDIT

Add the using at the start of your code:

using YamlDotNet.Serialization;
using YamlDotNet.RepresentationModel;

with that library added to your project and edit the code with the correct YAML deserialization, should be like the initial source code.

Legion
  • 744
  • 6
  • 22
  • There is no query method in the deserialized yamlObject. please provide the more code – codeteq Jul 05 '18 at 09:13
  • Sry my bad, i haven't YamlDotNet installed, so i was thinking "teorically". Just deserialize it, and search your keys with the forech, i edit the code over. – Legion Jul 05 '18 at 10:28
  • What is Yaml.Deserialize? Which library do you use? – codeteq Jul 05 '18 at 10:35
  • I edit again my answer..... not downvorte if you not understand somethnig.... just ask..... – Legion Jul 05 '18 at 11:01