Tutorial #3: Defining The Structs and Unions

Defining Structs and Unions

Struct and Union shares a lot of the same characteristics, only that struct may be either a Union and packed data structure or both. You can define the FieldOffset in C# that allows you to overlap the fields when defining a Union. Also when attempting to marshal structs, struct should have StructLayout attribute set to LayoutKind.Sequential or LayoutKind.Explicit. The difference between the three layouts are, one is C# managed struct, a simplified layout for marshaling based on datatype and structs in ordered fashion, and explicit requires each field to have FieldOffset attribute to explicitly define where field should be located.

Now let's get started:

Copy and paste this code for Tutorial.c

#include <stdint.h>
#include <stdlib.h>

struct TutorialStruct {
    int32_t FirstField;
    int16_t SecondField;
    int16_t ThirdField;
    int64_t FourthField;
};

union TutorialUnion {
    int16_t FirstField;
    int32_t SecondField;
    int64_t ThirdField;
};

struct TutorialStruct* CreateTutorialStruct()
{
    struct TutorialStruct* tutorialStructPtr = (struct TutorialStruct*)malloc(sizeof(struct TutorialStruct));
    tutorialStructPtr->FirstField = 11;
    tutorialStructPtr->SecondField = 22;
    tutorialStructPtr->ThirdField = 33;
    tutorialStructPtr->FourthField = 44;
    return tutorialStructPtr;
}

union TutorialUnion* CreateTutorialUnion()
{
    union TutorialUnion* tutorialUnionPtr = (union TutorialUnion*)malloc(sizeof(union TutorialUnion));
    tutorialUnionPtr->FirstField = 12443;
    return tutorialUnionPtr;
}


And then let's clear Program.cs to keep it tidy:

using System;
using System.IO;
using System.Runtime.InteropServices;

namespace Tutorial1
{
    public class LibTutorial : DLSupport
    {
        public LibTutorial() : base(Path.GetDirectoryName(typeof(LibTutorial).Assembly.Location),
                                    "libtutorial.so", LoadFlag.RTLD_LAZY)
        {
        }

        static void Main()
        {
            LibTutorial lib = new LibTutorial();

        }
    }
}


First thing first, let's define our struct.

We have a clearly defined struct in C that first field is an int, second and third fields are shorts, and then the last field is a long. Let's define struct as such:


    public struct TutorialStruct
    {
        public int FirstField;
        public short SecondField;
        public short ThirdField;
        public long FourthField;
    }

Now we have to remember to add an attribute to our struct, so to keep it relatively simple, let's set our LayoutKind to sequential like so:

    [StructLayout(LayoutKind.Sequential)]


And we have:

    [StructLayout(LayoutKind.Sequential)]
    public struct TutorialStruct
    {
        public int FirstField;
        public short SecondField;
        public short ThirdField;
        public long FourthField;
    }

Next, we need to define a union which means fields are going to be overlapping over one another, so to define that, we need to set the offset to 0 and set the LayoutKind to Explicit like so:

     [StructLayout(LayoutKind.Explicit)]
    public struct TutorialUnioni
    {
        [FieldOffset(0)]
        public short FirstField;
        [FieldOffset(0)]
        public int SecondField;
        [FieldOffset(0)]
        public long ThirdField;
    }

Now we have to define our functions that would recieve the pointer of those structs from C. Since it's a pointer, we will need to define those pointer as IntPtr and marshal a struct from those.

So we have:

        public CreateTutorialStruct_dt CreateTutorialStruct;
        public CreateTutorialUnion_dt CreateTutorialUnion;
        public delegate IntPtr CreateTutorialStruct_dt();
        public delegate IntPtr CreateTutorialUnion_dt();

and

        public LibTutorial() : base(Path.GetDirectoryName(typeof(LibTutorial).Assembly.Location),
                                    "libtutorial.so", LoadFlag.RTLD_LAZY)
        {
            CreateTutorialStruct = GetFunctionDelegate<CreateTutorialStruct_dt>("CreateTutorialStruct");
            CreateTutorialUnion = GetFunctionDelegate<CreateTutorialUnion_dt>("CreateTutorialUnion");
        }


Now we have our structs and functions defined, now let's try calling them and marshaling them.

So first, let's call the functions we've defined and assigned in Main method:

            var lib = new LibTutorial();
            var tutorialStructPtr = lib.CreateTutorialStruct();
            var tutorialUnionPtr = lib.CreateTutorialUnion();

then we'll use one of Marshal's function, PtrToStructure which accepts a generic argument of struct type we're going to use and a parameter to let Marshal what to marshal from:

            var tutorialStruct = Marshal.PtrToStructure<TutorialStruct>(tutorialStructPtr);
            var tutorialUnion = Marshal.PtrToStructure<TutorialUnion>(tutorialUnionPtr);


Now we have our structs, we need to display the fields of those structs! I've written a little reflection magic to quickly iterate the fields in a struct and output it's value:

            Console.WriteLine("tutorialStruct");
            foreach (var field in tutorialStruct.GetType().GetFields())
                Console.WriteLine("\t{0}", field.GetValue(tutorialStruct));
            Console.WriteLine("tutorialUnion");
            foreach (var field in tutorialUnion.GetType().GetFields())
                Console.WriteLine("\t{0}", field.GetValue(tutorialUnion));

Your code should looks like this:

using System;
using System.IO;
using System.Runtime.InteropServices;

namespace Tutorial3
{
    public class LibTutorial : DLSupport
    {
        public LibTutorial() : base(Path.GetDirectoryName(typeof(LibTutorial).Assembly.Location),
                                    "libtutorial.so", LoadFlag.RTLD_LAZY)
        {
            CreateTutorialStruct = GetFunctionDelegate<CreateTutorialStruct_dt>("CreateTutorialStruct");
            CreateTutorialUnion = GetFunctionDelegate<CreateTutorialUnion_dt>("CreateTutorialUnion");
        }

        public CreateTutorialStruct_dt CreateTutorialStruct;
        public CreateTutorialUnion_dt CreateTutorialUnion;
        public delegate IntPtr CreateTutorialStruct_dt();
        public delegate IntPtr CreateTutorialUnion_dt();

        static void Main()
        {
            var lib = new LibTutorial();
            var tutorialStructPtr = lib.CreateTutorialStruct();
            var tutorialUnionPtr = lib.CreateTutorialUnion();

            var tutorialStruct = Marshal.PtrToStructure<TutorialStruct>(tutorialStructPtr);
            var tutorialUnion = Marshal.PtrToStructure<TutorialUnion>(tutorialUnionPtr);

            Console.WriteLine("tutorialStruct");
            foreach (var field in tutorialStruct.GetType().GetFields())
                Console.WriteLine("\t{0}", field.GetValue(tutorialStruct));
            Console.WriteLine("tutorialUnion");
            foreach (var field in tutorialUnion.GetType().GetFields())
                Console.WriteLine("\t{0}", field.GetValue(tutorialUnion));
        }
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct TutorialStruct
    {
        public int FirstField;
        public short SecondField;
        public short ThirdField;
        public long FourthField;
    }

    [StructLayout(LayoutKind.Explicit)]
    public struct TutorialUnion
    {
        [FieldOffset(0)]
        public short FirstField;
        [FieldOffset(0)]
        public int SecondField;
        [FieldOffset(0)]
        public long ThirdField;
    }
}

Now you try build and run and see the result for yourself which should output:

Now let's talk about some of the quirks of C# when marshaling struct. When using MarshalAs attribute for struct such as this:

        [return: MarshalAs(UnmanagedType.LPStruct)]
        public delegate TutorialStruct CreateTutorialStruct_dt();
        [return: MarshalAs(UnmanagedType.LPStruct)]
        public delegate TutorialUnion CreateTutorialUnion_dt();

It does not do it's intended purpose of deferencing struct pointer by default, you have to use Marshal.PtrToStructure to deference it.

There is another quirk that you have to keep in mind, when working with struct that maybe defined as this:

struct ComplexStruct
{
    int32_t FirstField;
    union {
        int16_t ShortField;
        int64_t LongField;
    }
};

You have to keep an account of offset in bytes for Unions that may be nested inside a struct, while you can set the same offset for Short and Long fields, you have to calculate the offset by 4 so it wouldn't overlap with FirstField which would be defined as such:

    [StructLayout(LayoutKind.Explicit)]
    public struct ComplexStruct
    {
        [FieldOffset(0)]
        public int FirstField;
        [FieldOffset(4)]
        public short ShortField;
        [FieldOffset(4)]
        public long LongField;
    }

Also keep in mind that structs can be defined inside another struct and Marshal will be able to marshal it correctly if struct is passed by value, but if by reference, it need to be stored as IntPtr and then Marshaled with Marshal.PtrToStructure.

Also feel free to access the tutorials github repository here.

Tutorial #2: The Difference between Approaches

What is DLSupport and DL Library?

Open up DLSupport.cs in MonoDevelop and you'll see three functions that are imported from libdl:

    [DllImport("libdl.so.2")]
    static extern IntPtr dlopen(string filename, int flags);

    [DllImport("libdl.so.2")]
    static extern IntPtr dlsym(IntPtr handle, string symbol);

    [DllImport("libdl.so.2")]
    static extern int dlclose(IntPtr handle);

The dlopen function opens a shared object and load it into memory and return a handle which is a pointer so it could be used in other DL functions. The dlsym function retrieve a symbol pointer directly from the loaded library handle with a symbol name which you can view what are exported symbols in the library with the following command in terminal:

using libdl.so.2 as an example

objdump -T /usr/lib/libdl.so.2

which will display a list of symbols that you can call in the library.

dlclose is a dangerous command, because when you execute dlopen and recieving a handle, other part of the code would get the same exact handle when it already been opened previously in the same process. So when you close that handle, that handle will disrupt other codes using it. This is the case with Mono, because Mono is using DL library when you use DllImport attribute.

With this in mind, this is not the case with older Dotnet Core. an older Dotnet Core does not use DL Library to dynamically load the library and retrieve the symbol for DllImport, so that means when you execute functions imported from the same library with DllImport while using delegate approach for accessing variables in the same library, you'll have two very drastically different result from one another. This again, is why it is called the Delegate Approach, it's verbose and necessary if you intend to make it work across Mono and Dotnet Core.

In the most up to date Dotnet Core, you can use DllImport mixed in with Delegate Approach which means you can avoid definding delegate types and functions and instead use DllImport with ease.

The rest of the functions in DLSupport are straightforward such as GetFunctionDelegate and GetSymbolPointer.

To learn more about DL Functions, you can read it here.

Let's Try It Out Anyway

Let's delete the following lines:

        public delegate void IncrementVariableInC_dt();
 

        IncrementVariableInC_dt IncrementVariableInC;
 

        IncrementVariableInC = GetFunctionDelegate<IncrementVariableInC_dt>("IncrementVariableInC");


And instead, let's import the same function using DllImport!

        [DllImport("libtutorial.so")]
        public static extern void IncrementVariableInC();

 And finally, let's fix the function call in Main method by simply removing "lib.":

            IncrementVariableInC();

Build and run program, the output will be 2 which means Mono is using the same library handle as you do.

Now let's try it on Dotnet Core:

There are few changes that have to be made to be compatible on Dotnet Core:

First thing first, create a new directory and run command in terminal in that directory:

dotnet new console

Then copy and paste the code below into Program.cs:

using System;
using System.IO;
using System.Reflection;
using System.Runtime.InteropServices;

public abstract class DLSupport
{
    [DllImport("libdl.so.2")]
    static extern IntPtr dlopen(string filename, int flags);

    [DllImport("libdl.so.2")]
    static extern IntPtr dlsym(IntPtr handle, string symbol);

    [DllImport("libdl.so.2")]
    static extern int dlclose(IntPtr handle);

    protected IntPtr handle;

    public DLSupport(string dir, string lib, LoadFlag flag)
    {
        var path = Path.Combine(dir, lib);
        if (!File.Exists(path))
            throw new FileNotFoundException("Library could not be found!");
        handle = dlopen(path, (int)flag);
        if (handle == IntPtr.Zero)
            throw new DllNotFoundException("Library cannot be loaded with given flag.");
    }

    public DLSupport(string lib, LoadFlag flag)
    {
        handle = dlopen(lib, (int)flag);
        if (handle == IntPtr.Zero)
            throw new DllNotFoundException("Library cannot be found or loaded with given flag.");
    }

    protected T GetFunctionDelegate<T>(string symbol, bool SuppressSymbolError = false)
    {
        var ptr = dlsym(handle, symbol);
        if (!SuppressSymbolError && ptr == IntPtr.Zero)
            throw new Exception(string.Format("Symbol cannot be found for type: {0}", typeof(T).Name));
        return Marshal.GetDelegateForFunctionPointer<T>(ptr);
    }

    protected IntPtr GetSymbolPointer(string symbol)
    {
        return dlsym(handle, symbol);
    }

    /// <summary>
    /// Should not be invoked unless none of DllImport uses the same library as this handle.
    /// The runtime will fail if you attempt to use DllImported functions that happens to be using the same
    /// handle as this.
    /// </summary>
    protected void DangerousDispose()
    {
        dlclose(handle);
    }
}

public enum LoadFlag : int
{
    RTLD_LAZY = 0x00001,
    /// <summary>
    /// Immediate function call binding.
    /// </summary>
    RTLD_NOW = 0x00002,
    /// <summary>
    /// Mask of binding time value.
    /// </summary>
    RTLD_BINDING_MASK = 0x3,
    /// <summary>
    /// Do not load the object.
    /// </summary>
    RTLD_NOLOAD = 0x00004,
    /// <summary>
    /// Use deep binding.
    /// </summary>
    RTLD_DEEPBIND = 0x00008,

    /// <summary>
    /// If the following bit is set in the MODE argument to `dlopen',
    /// the symbols of the loaded object and its dependencies are made
    /// visible as if the object were linked directly into the program.
    /// </summary>
    RTLD_GLOBAL = 0x00100,

    /// <summary>
    /// Unix98 demands the following flag which is the inverse to RTLD_GLOBAL.
    /// The implementation does this by default and so we can define the
    /// value to zero.
    /// </summary>
    RTLD_LOCAL = 0,

    /// <summary>
    /// Do not delete object when closed.
    /// </summary>
    RTLD_NODELETE = 0x01000
}

namespace Tutorial1
{
    public class LibTutorial : DLSupport
    {
        IntPtr VariableInCPtr { get; set; }

        int VariableInC
        {
            get
            {
                if (VariableInCPtr == IntPtr.Zero)
                    throw new Exception("Pointer for VariableInC is not assigned!");
                return Marshal.ReadInt32(VariableInCPtr);
            }

            set
            {
                if (VariableInCPtr == IntPtr.Zero)
                    throw new Exception("Pointer for VariableInC is not assigned!");
                Marshal.WriteInt32(VariableInCPtr, value);
            }
        }
        public LibTutorial() : base(Path.GetDirectoryName(typeof(LibTutorial).GetTypeInfo().Assembly.Location),
                                    "libtutorial.so", LoadFlag.RTLD_LAZY)
        {
            VariableInCPtr = GetSymbolPointer("VariableInC");
        }

        [DllImport("libtutorial.so")]
        public static extern void IncrementVariableInC();

        static void Main()
        {
            LibTutorial lib = new LibTutorial();

            for (int I = 0; I < 100; I++)
            IncrementVariableInC();
            
            Console.WriteLine(lib.VariableInC);
        }
    }
}



Before you can build and run it, you need to copy libtutorial.so from other Tutorial project folder under bin/Debug folder into bin/Debug/netcoreapp1.1 of Dotnet Core project folder.

Finally run the commands below:

dotnet restore

dotnet build

dotnet run

In older verion before 1.0.1-rc4-004953, Dotnet Core will not be using DL library and therefore the output would be a 1 and appearently in current up to date version of Dotnet Core, it is using DL Library which therefore the output would be a 101.

To view the codes above, check out the github repository here.

Onto Tutorial #3