Jekyll2023-08-12T02:05:45+10:00https://dasith.me/feed.xmlDasith’s Gossip Protocol - Adventures in a #distributed worldDasith Wijesiriwardena (@dasiths) - The stories of a .NET developer with a focus on distributed systems and the cloudDasith WijesiriwardenaWhat is ORAS and why should you care?2023-06-04T22:06:00+10:002023-06-04T22:06:00+10:00https://dasith.me/2023/06/04/what-is-oras<p>Most systems we build today are delivered as containers. Container registries and associated technologies are an important cog in this ecosystem. As the container ecosystem matures, there is an increased need to consume associated artefacts like Helm packages, software bill of materials, evidence of provenance, machine learning data sets etc from the same storage. There are even upcoming use cases like WebAssembly libraries that need a home. Container registries have evolved to become more than their initial need.</p>
<p>The <a href="https://github.com/opencontainers/wg-reference-types">OCI Working Group for Reference Types</a> are planning changes to the OCI spec to support these scenarios. In this post we will have a look at how we got here and how projects like ORAS are driving innovation when it comes to storing artefacts and how it’s redefining what a container registry is.</p>
<p><em>Note: There have been some recent updates to the OCI image spec and ORAS (August 2023) and they are covered <a href="#update-04-aug-2023">here</a>.</em></p>
<ul>
<li><a href="#intro-to-oci">Intro to OCI</a></li>
<li><a href="#comparing-docker-image-v2-schema-2-vs-oci-10-image-schema">Comparing Docker Image v2 schema 2 vs OCI 1.0 Image schema</a>
<ul>
<li><a href="#same-story-with-the-index-manifest">Same story with the Index Manifest</a></li>
</ul>
</li>
<li><a href="#thats-great-for-images-but-what-about-other-artefacts">That’s great for images, but what about other artefacts?</a></li>
<li><a href="#enter-oci-v11-specification">Enter OCI v1.1 Specification</a>
<ul>
<li><a href="#not-all-good-news-though">Not All Good News Though</a></li>
</ul>
</li>
<li><a href="#pushing-this-further-with-oras">Pushing This Further With ORAS</a></li>
<li><a href="#how-does-oras-extend-the-oci-11-spec">How Does ORAS Extend The OCI 1.1 Spec?</a>
<ul>
<li><a href="#oras-artefact-manifest">ORAS Artefact Manifest</a></li>
</ul>
</li>
<li><a href="#oras-artefact-spec-future">ORAS Artefact Spec Future</a>
<ul>
<li><a href="#update-04-aug-2023">Update: 04-Aug-2023</a></li>
<li><a href="#update-12-aug-2023">Update: 12-Aug-2023</a>
<ul>
<li><a href="#what-this-means-for-oras">What this means for ORAS?</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#oras-use-cases-and-adopters">ORAS Use Cases And Adopters</a>
<ul>
<li><a href="#supply-chain-artefacts">Supply Chain Artefacts</a></li>
</ul>
</li>
<li><a href="#using-oras-cli">Using ORAS CLI</a></li>
<li><a href="#closing">Closing</a></li>
</ul>
<h2 id="intro-to-oci">Intro to OCI</h2>
<p>You have no doubt heard of Docker and containers. Since <a href="https://www.informationweek.com/cloud/open-container-initiative-finds-footing-in-linux-foundation">Docker donated their technology to the open source community</a>, a large community of people including tech giants have come together to make containers the defacto unit of software delivery.</p>
<p>The <a href="https://opencontainers.org/about/overview/">Open Container Initiative (OCI) was launched in 2015 by Docker</a> and other industry leaders as an open governance structure project. Over the years Docker has <a href="https://www.docker.com/blog/donating-docker-distribution-to-the-cncf/">kept donating more stuff</a> to the open source community.</p>
<p>But <a href="https://www.docker.com/blog/demystifying-open-container-initiative-oci-specifications/">OCI is not a replacement for Docker</a>. Docker is a platform while OCI exists with the sole purpose of creating open industry standards around container formats and runtimes.</p>
<p>From the OCI website: https://opencontainers.org/about/overview/</p>
<blockquote>
<p>The OCI currently contains three specifications: the Runtime Specification (runtime-spec), the Image Specification (image-spec) and the Distribution Specification (distribution-spec).</p>
</blockquote>
<p>Over the years OCI have defined their own specification and standards to support various technical and business needs.</p>
<h2 id="comparing-docker-image-v2-schema-2-vs-oci-10-image-schema">Comparing Docker Image v2 schema 2 vs OCI 1.0 Image schema</h2>
<ul>
<li><a href="https://docs.docker.com/registry/spec/manifest-v2-2/#example-image-manifest">Docker image manifest spec</a></li>
<li><a href="https://github.com/opencontainers/image-spec/blob/v1.0/manifest.md#example-image-manifest">OCI image manifest spec</a></li>
</ul>
<p><a href="/assets/images/docker_vs_oci_image_manifest.png"><img src="/assets/images/docker_vs_oci_image_manifest.png" alt="Docker vs OCI image manifest" /></a>
<em>Click to enlarge</em>.</p>
<p>As you can observe the key differences are just in the <code class="language-plaintext highlighter-rouge">mediaType</code> fields. Instead of the <code class="language-plaintext highlighter-rouge">application/vnd.docker.*</code> the OCI spec has <code class="language-plaintext highlighter-rouge">application/vnd.oci.*</code>. The OCI spec additionally supports annotations as well.</p>
<h3 id="same-story-with-the-index-manifest">Same story with the Index Manifest</h3>
<p>The image index (fat manifest) is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. This is useful when <a href="https://learn.microsoft.com/en-us/azure/container-registry/push-multi-architecture-images#manifest-list">storing multi architecture images</a>.</p>
<ul>
<li><a href="https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list">Docker manifest list spec</a></li>
<li><a href="https://github.com/opencontainers/image-spec/blob/v1.0/image-index.md">OCI image index spec</a></li>
</ul>
<p>I won’t do a side by side comparison here but you will see the same differences in <code class="language-plaintext highlighter-rouge">mediaType</code> there as well.</p>
<h2 id="thats-great-for-images-but-what-about-other-artefacts">That’s great for images, but what about other artefacts?</h2>
<p>We live in a container world, in fact <a href="https://community.f5.com/t5/technical-articles/it-s-a-kubernetes-world-and-i-m-just-living-in-it/tac-p/313021">we live in a Kubernetes world</a>. So container registries have become paramount in this ecosystem.</p>
<p>But your software system might not be composed of just container images. What about thing like Helm Charts? You may also have files or other supply chain assets like <a href="https://en.wikipedia.org/wiki/Software_supply_chain">SBOMs</a> as well.</p>
<p>If you need those files inside your k8s cluster, you used to have 2 options.</p>
<ul>
<li>Store the file in some blob storage and allow the cluster to pull it down as required. But what about versioning, replication, edge and disconnected scenarios etc?</li>
<li>Store your file inside a container image and store it in a container registry. At least this way the dependencies are in the same place as the container image. But this feels like cheating.</li>
</ul>
<p>As the world kept moving more and more workloads to k8s, the industry realized <strong>we need a way to store more than container images in container registries and we needed to support that as a first class concept.</strong></p>
<p>Think about it, the container registry is the best place to store it. Artefacts can be versioned and the inherent nature of the registry where manifests and blob content can be stored separately made it ideal.</p>
<p><strong>Container registries needed to metamorphosize into artefact registries.</strong></p>
<p>Steve Lasker makes this argument more eloquently than I did.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/BpKF_0M37-0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<h2 id="enter-oci-v11-specification">Enter OCI v1.1 Specification</h2>
<p>With OCI v1.1 spec we finally <a href="https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidelines-for-artifact-usage">got support for artefacts</a> as a first class concept.</p>
<blockquote>
<p>Content other than OCI container images MAY be packaged using the image manifest. When this is done, the <code class="language-plaintext highlighter-rouge">config.mediaType</code> value MUST be set to a value specific to the artifact type or the empty value. If the <code class="language-plaintext highlighter-rouge">config.mediaType</code> is set to the empty value, the <code class="language-plaintext highlighter-rouge">artifactType</code> MUST be defined. If the artifact does not need layers, a single layer SHOULD be included with a non-zero size. The suggested content for an unused <code class="language-plaintext highlighter-rouge">layers</code> array is the <a href="https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidance-for-an-empty-descriptor">empty descriptor</a>.</p>
</blockquote>
<ul>
<li>an [image].<code class="language-plaintext highlighter-rouge">artifactType</code> field was also introduced.
<blockquote>
<p>This OPTIONAL property contains the type of an artifact when the manifest is used for an artifact. This MUST be set when <code class="language-plaintext highlighter-rouge">config.mediaType</code> is set to the <a href="https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidance-for-an-empty-descriptor">empty value</a>. If defined, the value MUST comply with RFC 6838, including the <a href="https://tools.ietf.org/html/rfc6838#section-4.2">naming requirements</a> in its section 4.2, and MAY be registered with <a href="https://www.iana.org/assignments/media-types/media-types.xhtml">IANA</a>. Implementations storing or copying image manifests MUST NOT error on encountering an artifactType that is unknown to the implementation.</p>
</blockquote>
</li>
<li>
<p>This meant artefact authors could now leverage the existing <code class="language-plaintext highlighter-rouge">image manifest</code> to store artefacts in a way that works with the Content Addressable Storage (CAS) capabilities of <a href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md">OCI Distribution</a>.</p>
</li>
<li>The OCI image manifest 1.1 spec also introduced the <code class="language-plaintext highlighter-rouge">subject</code> field.
<blockquote>
<p>This OPTIONAL property specifies a <a href="https://github.com/opencontainers/image-spec/blob/main/descriptor.md">descriptor</a> of another manifest. This value, used by the <a href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers"><code class="language-plaintext highlighter-rouge">referrers</code> API</a>, indicates a relationship to the specified manifest.</p>
</blockquote>
<p>This would allow artefacts/manifests to be linked. i.e. An SBOM could be linked/attached to the container image it represented.</p>
</li>
<li>The OCI distribution spec 1.1 introduced the <a href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers">Referrers API</a>. This allowed clients to query for related artefacts.</li>
</ul>
<h3 id="not-all-good-news-though">Not All Good News Though</h3>
<ul>
<li>
<p>The use of the <code class="language-plaintext highlighter-rouge">config.mediaType</code> was not ideal. the ideal field would have been [image].<code class="language-plaintext highlighter-rouge">mediaType</code> (top-level) but for backwards compatibility reasons they could not. More about that in <a href="https://dlorenc.medium.com/oci-artifacts-explained-8f4a77945c13">this post by Dan Lorenc here</a>.</p>
</li>
<li>
<p>This resulted in a lot of artefacts implementations simply leaving the <code class="language-plaintext highlighter-rouge">[image].mediaType</code> empty and relying on the config blob to be set to a custom type. Not all the registries supported this or had limits on what type of values were supported.</p>
</li>
</ul>
<h2 id="pushing-this-further-with-oras">Pushing This Further With ORAS</h2>
<p>The <a href="https://oras.land/">ORAS (OCI Registry As Storage)</a> project aims to “Distribute Artifacts Across OCI Registries With Ease”.</p>
<p>ORAS extends the OCI 1.1 specification and allows artefacts to be used in an easily discoverable way. This is done by storing independent but softly linked artefacts without making any changes to the existing image manifest. This makes it ideal for supply chain scenarios where you have many artefacts accompanying container image.</p>
<p>The below object graph shows such a scenario where a container image, SBOM and their signatures to verify provenance. They are associated with the container image using the <code class="language-plaintext highlighter-rouge">subject</code> field.</p>
<p><img src="https://github.com/oras-project/artifacts-spec/raw/v1.0.0-rc.2/media/net-monitor-graph.svg" alt="Artefact association" /></p>
<h2 id="how-does-oras-extend-the-oci-11-spec">How Does ORAS Extend The OCI 1.1 Spec?</h2>
<p>The following is from the “Comparing the ORAS Artifact Manifest and OCI Image Manifest” <a href="https://github.com/oras-project/artifacts-spec/blob/main/README.md#comparing-the-oras-artifact-manifest-and-oci-image-manifest">section</a>.</p>
<blockquote>
<p>OCI Artifacts defines how to implement stand-alone artifacts that can fit within the constraints of the image-spec. OCI Artifacts uses the <code class="language-plaintext highlighter-rouge">manifest.config.mediaType</code> to identify the artifact is something other than a container image. While this validated the ability to generalize the <strong>C</strong>ontent <strong>A</strong>ddressable <strong>S</strong>torage (CAS) capabilities of <a href="https://github.com/opencontainers/distribution-spec">OCI Distribution</a>, a new set of artifacts require additional capabilities that aren’t constrained to the image-spec. ORAS Artifacts provide a more generic means to store a wider range of artifact types, including references between artifacts.</p>
</blockquote>
<blockquote>
<p>The addition of a new manifest does not change, nor impact the <code class="language-plaintext highlighter-rouge">image.manifest</code>.
By defining the <code class="language-plaintext highlighter-rouge">artifact.manifest</code> and the <code class="language-plaintext highlighter-rouge">referrers/</code> api, registries and clients opt-into new capabilities, without breaking existing registry and client behaviour.</p>
</blockquote>
<p>The high-level differences between the <code class="language-plaintext highlighter-rouge">oci.image.manifest</code> and the <code class="language-plaintext highlighter-rouge">oras.artifact.manifest</code>:</p>
<table>
<thead>
<tr>
<th>OCI Image Manifest</th>
<th>ORAS Artifacts Manifest</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">config</code> REQUIRED</td>
<td><code class="language-plaintext highlighter-rouge">config</code> OPTIONAL as it’s just another entry in the <code class="language-plaintext highlighter-rouge">blobs</code> collection with a config <code class="language-plaintext highlighter-rouge">mediaType</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">layers</code> REQUIRED</td>
<td><code class="language-plaintext highlighter-rouge">blobs</code> are OPTIONAL, which were renamed from <code class="language-plaintext highlighter-rouge">layers</code> to reflect general usage</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">layers</code> ORDINAL</td>
<td><code class="language-plaintext highlighter-rouge">blobs</code> are defined by the specific artifact spec. For example, Helm utilizes two independent, non-ordinal blobs, while other artifact types like container images may require blobs to be ordinal</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">manifest.config.mediaType</code> used to uniquely identify artifact types.</td>
<td><code class="language-plaintext highlighter-rouge">manifest.artifactType</code> added to lift the workaround for using <code class="language-plaintext highlighter-rouge">manifest.config.mediaType</code> on a REQUIRED, but not always used <code class="language-plaintext highlighter-rouge">config</code> property. Decoupling <code class="language-plaintext highlighter-rouge">config.mediaType</code> from <code class="language-plaintext highlighter-rouge">artifactType</code> enables artifacts to OPTIONALLY share config schemas.</td>
</tr>
<tr>
<td> </td>
<td><code class="language-plaintext highlighter-rouge">subject</code> OPTIONAL, enabling an artifact to extend another artifact (SBOM, Signatures, Nydus, Scan Results)</td>
</tr>
<tr>
<td> </td>
<td><code class="language-plaintext highlighter-rouge">/referrers</code> api for discovering referenced artifacts, with the ability to filter by <code class="language-plaintext highlighter-rouge">artifactType</code></td>
</tr>
<tr>
<td> </td>
<td>Lifecycle management defined, starting to provide standard expectations for how users can manage their content</td>
</tr>
</tbody>
</table>
<p>For more info, see:</p>
<ul>
<li><a href="https://github.com/oras-project/artifacts-spec/discussions/91">Proposal: Decoupling Registries from Specific Artifact Specs #91</a></li>
<li><a href="https://github.com/opencontainers/artifacts/discussions/41">Discussion of a new manifest #41</a></li>
</ul>
<h3 id="oras-artefact-manifest">ORAS Artefact Manifest</h3>
<p>The ORAS Artifact manifest is similar to the OCI image manifest, but removes constraints defined on the image-manifest such as a required config object and required & ordinal layers</p>
<p>ORAS artefact manifest introduced their own <code class="language-plaintext highlighter-rouge">mediaType</code> field with the value <code class="language-plaintext highlighter-rouge">application/vnd.cncf.oras.artifact.manifest.v1+json</code></p>
<p>Full spec can be <a href="https://github.com/oras-project/artifacts-spec/blob/main/artifact-manifest.md">found here</a>.</p>
<h2 id="oras-artefact-spec-future">ORAS Artefact Spec Future</h2>
<p>There are no future releases or work items planned.</p>
<blockquote>
<p>The output of this project has been proposed to the <a href="https://github.com/opencontainers/wg-reference-types">OCI Reference Types Working Group</a>. Future discussions about artifacts in OCI registries should happen in the <a href="https://github.com/opencontainers/distribution-spec">OCI distribution-spec</a> & <a href="https://github.com/opencontainers/image-spec">image-spec</a> repositories.</p>
</blockquote>
<p>The idea is to get the proposed changes adopted via the OCI spec upstream and make the artefact use common across all registries and clients that way.</p>
<h3 id="update-04-aug-2023">Update: 04-Aug-2023</h3>
<p>The OCI working group have <a href="https://opencontainers.org/posts/blog/2023-07-07-summary-of-upcoming-changes-in-oci-image-and-distribution-specs-v-1-1/">made an announcement</a> on what proposals from ORAS they have incorporated.</p>
<p>These include</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">artifactType</code> as a top level field. Preferred over <code class="language-plaintext highlighter-rouge">config.mediaType</code> for new artefacts.</li>
<li><code class="language-plaintext highlighter-rouge">subject</code> field to be used establishing relationships between.</li>
<li><code class="language-plaintext highlighter-rouge">/v2/<name>/referrers/<digest></code> referrers API endpoint to query relationships based on the <code class="language-plaintext highlighter-rouge">subject</code> descriptor.</li>
</ul>
<p>I have created a <a href="https://github.com/opencontainers/image-spec/pull/1100">pull request for the OCI image spec repo</a> to update its artefact usage guidance.</p>
<h3 id="update-12-aug-2023">Update: 12-Aug-2023</h3>
<ul>
<li>My changes from the <a href="https://github.com/opencontainers/image-spec/pull/1100">above PR</a> have been incorporated into a new PR which can be <a href="https://github.com/opencontainers/image-spec/pull/1101">found here</a>.</li>
<li>The ORAS project is also updating its guidance based on that. The PR for that <a href="https://github.com/oras-project/oras-www/pull/248">is here</a>.</li>
</ul>
<p>This was my first time contributing to the OCI (opencontainers) project and ORAS and I enjoyed the conversation and process of PR review very much.</p>
<p>If you see a gap in the guidance or spec, please feel free to create an issue or a PR to fix it. The folks over there are a good bunch of people to work with.</p>
<h4 id="what-this-means-for-oras">What this means for ORAS?</h4>
<p>This means the ORAS artefact manifest spec will now considered to be deprecated. You can start using the OCI 1.1 image spec to store artefacts. The intention of the project has been satisfied in getting the OCI image spec to adopt some of its (ORAS artefact spec) recommendations.</p>
<p>You can keep using the ORAS CLI and SDK tools to interact with OCI 1.1 registries. In fact this is the preferred way rather than writing your own logic based on the runtime spec. ORAS SDK handles everything for you.</p>
<h2 id="oras-use-cases-and-adopters">ORAS Use Cases And Adopters</h2>
<ul>
<li><a href="https://v3.helm.sh/docs/topics/registries/">Helm</a>: Store packages.</li>
<li><a href="https://docs.sylabs.io/guides/3.1/user-guide/cli/singularity.html">Project Singularity</a>: Store Singularity Images.</li>
<li><a href="https://github.com/notaryproject/notation">Notation</a>: Store Signature used in secure supply chain.</li>
<li><a href="https://github.com/engineerd/wasm-to-oci">WASM to OCI</a> - Store WebAssembly modules in OCI registries.</li>
</ul>
<p>A full list can be <a href="https://oras.land/docs/category/oras-commands/">found here</a>.</p>
<h3 id="supply-chain-artefacts">Supply Chain Artefacts</h3>
<p>There are some examples below on how to use ORAS to store supply chain artefacts and sign them using Notation.</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=7RvFj_RWE7c&ab_channel=CNCF%5BCloudNativeComputingFoundation%5D">CNCF Webinar - Secure Container Supply Chain with Notation, ORAS, and Ratify</a></li>
<li><a href="https://learn.microsoft.com/en-us/azure/container-registry/container-registry-oci-artifacts">Push and pull OCI artifacts using an Azure container registry</a></li>
<li><a href="https://learn.microsoft.com/en-us/azure/container-registry/container-registry-oras-artifacts">Push and pull supply chain artifacts using Azure Registry (Preview)</a></li>
<li><a href="https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tutorial-sign-build-push">Build, sign, and verify container images using Notary and Azure Key Vault (Preview)</a></li>
</ul>
<h2 id="using-oras-cli">Using ORAS CLI</h2>
<p>To install ORAS CLI on Linux:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">VERSION</span><span class="o">=</span><span class="s2">"1.0.0"</span>
curl <span class="nt">-LO</span> <span class="s2">"https://github.com/oras-project/oras/releases/download/v</span><span class="k">${</span><span class="nv">VERSION</span><span class="k">}</span><span class="s2">/oras_</span><span class="k">${</span><span class="nv">VERSION</span><span class="k">}</span><span class="s2">_linux_amd64.tar.gz"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> oras-install/
<span class="nb">tar</span> <span class="nt">-zxf</span> oras_<span class="k">${</span><span class="nv">VERSION</span><span class="k">}</span>_<span class="k">*</span>.tar.gz <span class="nt">-C</span> oras-install/
<span class="nb">sudo mv </span>oras-install/oras /usr/local/bin/
<span class="nb">rm</span> <span class="nt">-rf</span> oras_<span class="k">${</span><span class="nv">VERSION</span><span class="k">}</span>_<span class="k">*</span>.tar.gz oras-install/
</code></pre></div></div>
<p>Other platforms are <a href="https://oras.land/docs/installation">listed here</a>.</p>
<p>You will need an compatible registry like <a href="https://zotregistry.io/">Zot</a>. A list of <a href="https://oras.land/docs/adopters">supported registries</a> are listed here.</p>
<p>To run <code class="language-plaintext highlighter-rouge">Zot</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">-d</span> <span class="nt">-p</span> 5000:5000 <span class="nt">--name</span> oras-quickstart ghcr.io/project-zot/zot-linux-amd64:latest
</code></pre></div></div>
<p>Create a sample file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"hello world"</span> <span class="o">></span> artifact.txt
</code></pre></div></div>
<p>Push the artefact:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oras push <span class="nt">--plain-http</span> localhost:5000/hello-artifact:v1 <span class="se">\</span>
<span class="nt">--artifact-type</span> application/vnd.acme.rocket.config <span class="se">\</span>
artifact.txt:text/plain
Uploading a948904f2f0f artifact.txt
Uploaded a948904f2f0f artifact.txt
Pushed <span class="o">[</span>registry] localhost:5000/hello-artifact:v1
Digest: sha256:bcdd6799fed0fca0eaedfc1c642f3d1dd7b8e78b43986a89935d6fe217a09cee
</code></pre></div></div>
<p>Attach an artefact:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"hello world"</span> <span class="o">></span> hi.txt
oras attach <span class="nt">--artifact-type</span> doc/example localhost:5000/hello-artifact:v1 hi.txt
</code></pre></div></div>
<p>Pull an artefact:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oras pull localhost:5000/hello-artifact:v1
Downloading a948904f2f0f artifact.txt
Downloaded a948904f2f0f artifact.txt
Pulled <span class="o">[</span>registry] localhost:5000/hello-artifact:v1
Digest: sha256:19e1b5170646a1500a1ac56bad28675ab72dc49038e69ba56eb7556ec478859f
</code></pre></div></div>
<p>Discover the referrers:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oras discover localhost:5000/hello-artifact:v1
Discovered 1 artifact referencing v1
Digest: sha256:327db68f73d0ed53d528d927a6703c00739d7c1076e50762c3f6641b51b76fdc
Artifact Type Digest
doc/example sha256:bcdd6799fed0fca0eaedfc1c642f3d1dd7b8e78b43986a89935d6fe217a09cee
</code></pre></div></div>
<ul>
<li>ORAS commands are <a href="https://oras.land/docs/category/oras-commands/">listed here</a>.</li>
<li>More use cases and custom manifest configs are <a href="https://oras.land/docs/category/how-to-guides">covered here</a>.</li>
</ul>
<h2 id="closing">Closing</h2>
<p>Hope this post gave you a deeper understanding of the state of artefacts in container registries and how the OCI 1.1 spec and projects like ORAS are trying to push the industry in a direction that allows for standardised registries and clients.</p>
<p>If you have any feedback or questions, please reach out to me on twitter @dasiths or post them here.</p>
<p>Happy coding.</p>Dasith WijesiriwardenaMost systems we build today are delivered as containers. Container registries and associated technologies are an important cog in this ecosystem. As the container ecosystem matures, there is an increased need to consume associated artefacts like Helm packages, software bill of materials, evidence of provenance, machine learning data sets etc from the same storage. There are even upcoming use cases like WebAssembly libraries that need a home. Container registries have evolved to become more than their initial need. The OCI Working Group for Reference Types are planning changes to the OCI spec to support these scenarios. In this post we will have a look at how we got here and how projects like ORAS are driving innovation when it comes to storing artefacts and how it’s redefining what a container registry is. Note: There have been some recent updates to the OCI image spec and ORAS (August 2023) and they are covered here. Intro to OCI Comparing Docker Image v2 schema 2 vs OCI 1.0 Image schema Same story with the Index Manifest That’s great for images, but what about other artefacts? Enter OCI v1.1 Specification Not All Good News Though Pushing This Further With ORAS How Does ORAS Extend The OCI 1.1 Spec? ORAS Artefact Manifest ORAS Artefact Spec Future Update: 04-Aug-2023 Update: 12-Aug-2023 What this means for ORAS? ORAS Use Cases And Adopters Supply Chain Artefacts Using ORAS CLI Closing Intro to OCI You have no doubt heard of Docker and containers. Since Docker donated their technology to the open source community, a large community of people including tech giants have come together to make containers the defacto unit of software delivery. The Open Container Initiative (OCI) was launched in 2015 by Docker and other industry leaders as an open governance structure project. Over the years Docker has kept donating more stuff to the open source community. But OCI is not a replacement for Docker. Docker is a platform while OCI exists with the sole purpose of creating open industry standards around container formats and runtimes. From the OCI website: https://opencontainers.org/about/overview/ The OCI currently contains three specifications: the Runtime Specification (runtime-spec), the Image Specification (image-spec) and the Distribution Specification (distribution-spec). Over the years OCI have defined their own specification and standards to support various technical and business needs. Comparing Docker Image v2 schema 2 vs OCI 1.0 Image schema Docker image manifest spec OCI image manifest spec Click to enlarge. As you can observe the key differences are just in the mediaType fields. Instead of the application/vnd.docker.* the OCI spec has application/vnd.oci.*. The OCI spec additionally supports annotations as well. Same story with the Index Manifest The image index (fat manifest) is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. This is useful when storing multi architecture images. Docker manifest list spec OCI image index spec I won’t do a side by side comparison here but you will see the same differences in mediaType there as well. That’s great for images, but what about other artefacts? We live in a container world, in fact we live in a Kubernetes world. So container registries have become paramount in this ecosystem. But your software system might not be composed of just container images. What about thing like Helm Charts? You may also have files or other supply chain assets like SBOMs as well. If you need those files inside your k8s cluster, you used to have 2 options. Store the file in some blob storage and allow the cluster to pull it down as required. But what about versioning, replication, edge and disconnected scenarios etc? Store your file inside a container image and store it in a container registry. At least this way the dependencies are in the same place as the container image. But this feels like cheating. As the world kept moving more and more workloads to k8s, the industry realized we need a way to store more than container images in container registries and we needed to support that as a first class concept. Think about it, the container registry is the best place to store it. Artefacts can be versioned and the inherent nature of the registry where manifests and blob content can be stored separately made it ideal. Container registries needed to metamorphosize into artefact registries. Steve Lasker makes this argument more eloquently than I did. Enter OCI v1.1 Specification With OCI v1.1 spec we finally got support for artefacts as a first class concept. Content other than OCI container images MAY be packaged using the image manifest. When this is done, the config.mediaType value MUST be set to a value specific to the artifact type or the empty value. If the config.mediaType is set to the empty value, the artifactType MUST be defined. If the artifact does not need layers, a single layer SHOULD be included with a non-zero size. The suggested content for an unused layers array is the empty descriptor. an [image].artifactType field was also introduced. This OPTIONAL property contains the type of an artifact when the manifest is used for an artifact. This MUST be set when config.mediaType is set to the empty value. If defined, the value MUST comply with RFC 6838, including the naming requirements in its section 4.2, and MAY be registered with IANA. Implementations storing or copying image manifests MUST NOT error on encountering an artifactType that is unknown to the implementation. This meant artefact authors could now leverage the existing image manifest to store artefacts in a way that works with the Content Addressable Storage (CAS) capabilities of OCI Distribution. The OCI image manifest 1.1 spec also introduced the subject field. This OPTIONAL property specifies a descriptor of another manifest. This value, used by the referrers API, indicates a relationship to the specified manifest. This would allow artefacts/manifests to be linked. i.e. An SBOM could be linked/attached to the container image it represented. The OCI distribution spec 1.1 introduced the Referrers API. This allowed clients to query for related artefacts. Not All Good News Though The use of the config.mediaType was not ideal. the ideal field would have been [image].mediaType (top-level) but for backwards compatibility reasons they could not. More about that in this post by Dan Lorenc here. This resulted in a lot of artefacts implementations simply leaving the [image].mediaType empty and relying on the config blob to be set to a custom type. Not all the registries supported this or had limits on what type of values were supported. Pushing This Further With ORAS The ORAS (OCI Registry As Storage) project aims to “Distribute Artifacts Across OCI Registries With Ease”. ORAS extends the OCI 1.1 specification and allows artefacts to be used in an easily discoverable way. This is done by storing independent but softly linked artefacts without making any changes to the existing image manifest. This makes it ideal for supply chain scenarios where you have many artefacts accompanying container image. The below object graph shows such a scenario where a container image, SBOM and their signatures to verify provenance. They are associated with the container image using the subject field. How Does ORAS Extend The OCI 1.1 Spec? The following is from the “Comparing the ORAS Artifact Manifest and OCI Image Manifest” section. OCI Artifacts defines how to implement stand-alone artifacts that can fit within the constraints of the image-spec. OCI Artifacts uses the manifest.config.mediaType to identify the artifact is something other than a container image. While this validated the ability to generalize the Content Addressable Storage (CAS) capabilities of OCI Distribution, a new set of artifacts require additional capabilities that aren’t constrained to the image-spec. ORAS Artifacts provide a more generic means to store a wider range of artifact types, including references between artifacts. The addition of a new manifest does not change, nor impact the image.manifest. By defining the artifact.manifest and the referrers/ api, registries and clients opt-into new capabilities, without breaking existing registry and client behaviour. The high-level differences between the oci.image.manifest and the oras.artifact.manifest: OCI Image Manifest ORAS Artifacts Manifest config REQUIRED config OPTIONAL as it’s just another entry in the blobs collection with a config mediaType layers REQUIRED blobs are OPTIONAL, which were renamed from layers to reflect general usage layers ORDINAL blobs are defined by the specific artifact spec. For example, Helm utilizes two independent, non-ordinal blobs, while other artifact types like container images may require blobs to be ordinal manifest.config.mediaType used to uniquely identify artifact types. manifest.artifactType added to lift the workaround for using manifest.config.mediaType on a REQUIRED, but not always used config property. Decoupling config.mediaType from artifactType enables artifacts to OPTIONALLY share config schemas. subject OPTIONAL, enabling an artifact to extend another artifact (SBOM, Signatures, Nydus, Scan Results) /referrers api for discovering referenced artifacts, with the ability to filter by artifactType Lifecycle management defined, starting to provide standard expectations for how users can manage their content For more info, see: Proposal: Decoupling Registries from Specific Artifact Specs #91 Discussion of a new manifest #41 ORAS Artefact Manifest The ORAS Artifact manifest is similar to the OCI image manifest, but removes constraints defined on the image-manifest such as a required config object and required & ordinal layers ORAS artefact manifest introduced their own mediaType field with the value application/vnd.cncf.oras.artifact.manifest.v1+json Full spec can be found here. ORAS Artefact Spec Future There are no future releases or work items planned. The output of this project has been proposed to the OCI Reference Types Working Group. Future discussions about artifacts in OCI registries should happen in the OCI distribution-spec & image-spec repositories. The idea is to get the proposed changes adopted via the OCI spec upstream and make the artefact use common across all registries and clients that way. Update: 04-Aug-2023 The OCI working group have made an announcement on what proposals from ORAS they have incorporated. These include artifactType as a top level field. Preferred over config.mediaType for new artefacts. subject field to be used establishing relationships between. /v2/<name>/referrers/<digest> referrers API endpoint to query relationships based on the subject descriptor. I have created a pull request for the OCI image spec repo to update its artefact usage guidance. Update: 12-Aug-2023 My changes from the above PR have been incorporated into a new PR which can be found here. The ORAS project is also updating its guidance based on that. The PR for that is here. This was my first time contributing to the OCI (opencontainers) project and ORAS and I enjoyed the conversation and process of PR review very much. If you see a gap in the guidance or spec, please feel free to create an issue or a PR to fix it. The folks over there are a good bunch of people to work with. What this means for ORAS? This means the ORAS artefact manifest spec will now considered to be deprecated. You can start using the OCI 1.1 image spec to store artefacts. The intention of the project has been satisfied in getting the OCI image spec to adopt some of its (ORAS artefact spec) recommendations. You can keep using the ORAS CLI and SDK tools to interact with OCI 1.1 registries. In fact this is the preferred way rather than writing your own logic based on the runtime spec. ORAS SDK handles everything for you. ORAS Use Cases And Adopters Helm: Store packages. Project Singularity: Store Singularity Images. Notation: Store Signature used in secure supply chain. WASM to OCI - Store WebAssembly modules in OCI registries. A full list can be found here. Supply Chain Artefacts There are some examples below on how to use ORAS to store supply chain artefacts and sign them using Notation. CNCF Webinar - Secure Container Supply Chain with Notation, ORAS, and Ratify Push and pull OCI artifacts using an Azure container registry Push and pull supply chain artifacts using Azure Registry (Preview) Build, sign, and verify container images using Notary and Azure Key Vault (Preview) Using ORAS CLI To install ORAS CLI on Linux: VERSION="1.0.0" curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz" mkdir -p oras-install/ tar -zxf oras_${VERSION}_*.tar.gz -C oras-install/ sudo mv oras-install/oras /usr/local/bin/ rm -rf oras_${VERSION}_*.tar.gz oras-install/ Other platforms are listed here. You will need an compatible registry like Zot. A list of supported registries are listed here. To run Zot: docker run -d -p 5000:5000 --name oras-quickstart ghcr.io/project-zot/zot-linux-amd64:latest Create a sample file: echo "hello world" > artifact.txt Push the artefact: oras push --plain-http localhost:5000/hello-artifact:v1 \ --artifact-type application/vnd.acme.rocket.config \ artifact.txt:text/plain Uploading a948904f2f0f artifact.txt Uploaded a948904f2f0f artifact.txt Pushed [registry] localhost:5000/hello-artifact:v1 Digest: sha256:bcdd6799fed0fca0eaedfc1c642f3d1dd7b8e78b43986a89935d6fe217a09cee Attach an artefact: echo "hello world" > hi.txt oras attach --artifact-type doc/example localhost:5000/hello-artifact:v1 hi.txt Pull an artefact: oras pull localhost:5000/hello-artifact:v1 Downloading a948904f2f0f artifact.txt Downloaded a948904f2f0f artifact.txt Pulled [registry] localhost:5000/hello-artifact:v1 Digest: sha256:19e1b5170646a1500a1ac56bad28675ab72dc49038e69ba56eb7556ec478859f Discover the referrers: oras discover localhost:5000/hello-artifact:v1 Discovered 1 artifact referencing v1 Digest: sha256:327db68f73d0ed53d528d927a6703c00739d7c1076e50762c3f6641b51b76fdc Artifact Type Digest doc/example sha256:bcdd6799fed0fca0eaedfc1c642f3d1dd7b8e78b43986a89935d6fe217a09cee ORAS commands are listed here. More use cases and custom manifest configs are covered here. Closing Hope this post gave you a deeper understanding of the state of artefacts in container registries and how the OCI 1.1 spec and projects like ORAS are trying to push the industry in a direction that allows for standardised registries and clients. If you have any feedback or questions, please reach out to me on twitter @dasiths or post them here. Happy coding.Lessons learned from doing EdgeDevOps (GitOps) in the bush, air and underwater - API Days Australia 20222023-01-06T22:06:00+11:002023-01-06T22:06:00+11:00https://dasith.me/2023/01/06/edge-devops-apidays-australia-2022<p>I recently spoke at <a href="https://www.apidays.global/australia/">API Days Australia</a> about my experiences building distributed systems and some challenges my team faced deploying and running them on the edge.</p>
<p>It is not an exaggeration to say that most modern systems that teams build are running on the cloud in a distributed architecture. There are some well-known successful practices around DevOps for these cloud native solutions as well. But what happens when you want to use the same workflows to deploy and run on the edge where connectivity might be intermittent or not available (air gapped systems)?</p>
<p>How do we run Kubernetes on the edge and use our favourite GitOps workflows? In this talk we spoke about some of the techniques and practices we have been using to build and run workloads on Azure Edge and other edge devices. During this talk we elaborated on the challenges faced running Kubernetes on the edge and some practical solutions, starting off from your development environment, to continuously having your code deployed and running on a fleet of devices in an automated way regardless if it’s a mobile platform, drone or a submarine.</p>
<p>My team at Microsoft CSE (Commercial Software Engineering) have been building software that run on Kubernetes on the edge. This has posed a plethora of challenges and edge cases for us to solve.</p>
<p>In this talk we dived in to the best practices and practical solutions we have discovered along the way. This will help any team building software systems to run on edge devices that have intermittent connectivity or no connectivity (air gapped).</p>
<h2 id="about-api-days">About API Days</h2>
<p>This is the fifth time I spoke at API Days and I wanted to get my dev crew from the <a href="https://microsoft.github.io/code-with-engineering-playbook/CSE/">Microsoft Commercial Software Engineering</a> involved in the talk. So I reached out to the organising committee and they gave us the green light to present this as a team. We were thrilled as this was the first in person conference for us since the Covid restrictions. We hope you liked the format we presented it in.</p>
<h2 id="recording--slide-deck">Recording & Slide deck</h2>
<iframe width="1280" height="720" src="https://www.youtube.com/embed/PYpHWBQapSs" title="Apidays Australia 2022 - Lessons from doing EdgeDevOps (GitOps) in the bush, air, and underwater." frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<p><br /></p>
<iframe class="speakerdeck-iframe" frameborder="0" src="https://speakerdeck.com/player/4d51700c463744cfa01e212c3d8c0930" title="Lessons learned from doing EdgeDevOps (GitOps) in the bush, air and underwater - API Days Australia 2022" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true" style="border: 0px; background: padding-box padding-box rgba(0, 0, 0, 0.1); margin: 0px; padding: 0px; border-radius: 6px; box-shadow: rgba(0, 0, 0, 0.2) 0px 5px 40px; width: 560px; height: 314px;" data-ratio="1.78343949044586"></iframe>
<p><br /><br /></p>
<p>If you have any thoughts or comments please leave them here. Thanks for taking the time to read this post.</p>Dasith WijesiriwardenaI recently spoke at API Days Australia about my experiences building distributed systems and some challenges my team faced deploying and running them on the edge. It is not an exaggeration to say that most modern systems that teams build are running on the cloud in a distributed architecture. There are some well-known successful practices around DevOps for these cloud native solutions as well. But what happens when you want to use the same workflows to deploy and run on the edge where connectivity might be intermittent or not available (air gapped systems)? How do we run Kubernetes on the edge and use our favourite GitOps workflows? In this talk we spoke about some of the techniques and practices we have been using to build and run workloads on Azure Edge and other edge devices. During this talk we elaborated on the challenges faced running Kubernetes on the edge and some practical solutions, starting off from your development environment, to continuously having your code deployed and running on a fleet of devices in an automated way regardless if it’s a mobile platform, drone or a submarine. My team at Microsoft CSE (Commercial Software Engineering) have been building software that run on Kubernetes on the edge. This has posed a plethora of challenges and edge cases for us to solve. In this talk we dived in to the best practices and practical solutions we have discovered along the way. This will help any team building software systems to run on edge devices that have intermittent connectivity or no connectivity (air gapped). About API Days This is the fifth time I spoke at API Days and I wanted to get my dev crew from the Microsoft Commercial Software Engineering involved in the talk. So I reached out to the organising committee and they gave us the green light to present this as a team. We were thrilled as this was the first in person conference for us since the Covid restrictions. We hope you liked the format we presented it in. Recording & Slide deck If you have any thoughts or comments please leave them here. Thanks for taking the time to read this post.Instrument MQTT based python messaging app using Open Telemetry2023-01-06T22:06:00+11:002023-01-06T22:06:00+11:00https://dasith.me/2023/01/06/instrument-mqtt-open-telemetry-python<p>Some time back I did a <a href="https://dasith.me/2022/01/23/open-telemetry-apidays-australia-2021/">bit of an intro to OpenTelemetry</a> and in there I covered some basics like what Signals and Context Propagation are. I also spoke about how concepts like Tracing, Spans and Instrumentation interrelate to one another. I even put some <a href="https://github.com/dasiths/OpenTelemetryDistributedTracingSample">code samples up at GitHub</a> to demo this.</p>
<p>Most if not all of those code samples are in .NET and they demo tracing and baggage. Since I did that talk in 2021 the OpenTelemetry community have decided to <a href="https://www.honeycomb.io/blog/opentelemetry-logs-go-etc">add logs as a signal</a>.</p>
<h2 id="logs-are-a-signal">Logs Are a Signal</h2>
<p>There are <a href="https://opentelemetry.io/docs/concepts/signals/">4 types of signals</a> as of the time of writing this.</p>
<ol>
<li>Tracing</li>
<li>Metrics</li>
<li>Baggage</li>
<li><a href="https://opentelemetry.io/docs/reference/specification/logs/">Logs</a></li>
</ol>
<p>The Logs have the <a href="https://opentelemetry.io/docs/reference/specification/logs/data-model/">same specification as a <code class="language-plaintext highlighter-rouge">span event</code> we used to know before</a>.</p>
<h2 id="instrumenting-python-and-paho-mqtt-client">Instrumenting Python (and Paho MQTT Client)</h2>
<p>I recently had to instrument an existing app written in python that uses MQTT protocol to communicate.</p>
<p>There were a few things I needed to do</p>
<ul>
<li>Instrument the python app(s) using OTEL Python SDK for Tracing, Metrics and Logs</li>
<li>Figure out how context propagation works with the MQTT protocol (if the python MQTT client I used isn’t already instrumented. Spoiler, it wasn’t)</li>
<li>Decide if
<ul>
<li>I use specific exporters directly from the python app (No OTEL Collector) or</li>
<li>Export to an OTEL Collector in OTLP format and then export it to specific tool from there. Spoiler. I chose the <a href="https://opentelemetry.io/img/otel_diagram.png">OTEL Collector approach</a>.</li>
</ul>
</li>
<li>Deploy OTEL Collector to k8s/Docker Compose and configure it to export to my tools like Jaeger and Prometheus.
<ul>
<li>Configuring OTEL Collector with exporters</li>
<li>Configuring Prometheus to scrape from my OTEL collector</li>
<li>Setting up Grafana to add Prometheus as a data source</li>
<li>Setting up Azure Monitor Exporter</li>
</ul>
</li>
</ul>
<h2 id="otel-python-sdk">OTEL Python SDK</h2>
<p>The OTEL <a href="https://opentelemetry.io/docs/instrumentation/python/getting-started/">official documentation</a> is a good place to start. There are some <a href="https://opentelemetry.io/docs/instrumentation/python/exporters/">examples of how to setup and use traces/metrics</a>. If you need something more specific, there are <a href="https://opentelemetry-python.readthedocs.io/en/stable/examples/">more examples</a> here.</p>
<p>For brevity let’s look at some simple code examples.</p>
<p>First, install these packages</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>opentelemetry-api
pip <span class="nb">install </span>opentelemetry-sdk
pip <span class="nb">install </span>opentelemetry-exporter-otlp
</code></pre></div></div>
<h3 id="traces">Traces</h3>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">opentelemetry</span> <span class="kn">import</span> <span class="n">trace</span>
<span class="kn">from</span> <span class="nn">opentelemetry.trace.propagation.tracecontext</span> <span class="kn">import</span> <span class="n">TraceContextTextMapPropagator</span>
<span class="kn">from</span> <span class="nn">opentelemetry.trace</span> <span class="kn">import</span> <span class="n">Status</span><span class="p">,</span> <span class="n">StatusCode</span><span class="p">,</span> <span class="n">SpanKind</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.resources</span> <span class="kn">import</span> <span class="n">SERVICE_NAME</span><span class="p">,</span> <span class="n">SERVICE_INSTANCE_ID</span><span class="p">,</span> <span class="n">Resource</span>
<span class="kn">from</span> <span class="nn">opentelemetry.semconv.trace</span> <span class="kn">import</span> <span class="n">SpanAttributes</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.trace</span> <span class="kn">import</span> <span class="n">TracerProvider</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.trace.export</span> <span class="kn">import</span> <span class="p">(</span>
<span class="n">BatchSpanProcessor</span><span class="p">,</span>
<span class="n">ConsoleSpanExporter</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span> <span class="nn">opentelemetry.exporter.otlp.proto.grpc.trace_exporter</span> <span class="kn">import</span> <span class="n">OTLPSpanExporter</span>
<span class="n">OTLP_endpoint</span> <span class="o">=</span> <span class="s">"http://127.0.0.1:4317"</span>
<span class="k">def</span> <span class="nf">add_console_exporter</span><span class="p">(</span><span class="n">provider</span><span class="p">:</span> <span class="n">TracerProvider</span><span class="p">):</span>
<span class="n">processor</span> <span class="o">=</span> <span class="n">BatchSpanProcessor</span><span class="p">(</span><span class="n">span_exporter</span><span class="o">=</span><span class="n">ConsoleSpanExporter</span><span class="p">(),</span> <span class="n">schedule_delay_millis</span><span class="o">=</span><span class="mi">1000</span><span class="p">)</span>
<span class="n">provider</span><span class="p">.</span><span class="n">add_span_processor</span><span class="p">(</span><span class="n">processor</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">add_otlp_exporter</span><span class="p">(</span><span class="n">provider</span><span class="p">:</span> <span class="n">TracerProvider</span><span class="p">):</span>
<span class="n">otlp_exporter</span> <span class="o">=</span> <span class="n">OTLPSpanExporter</span><span class="p">(</span><span class="n">endpoint</span><span class="o">=</span><span class="n">OTLP_endpoint</span><span class="p">,</span> <span class="n">insecure</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">otlp_span_processor</span> <span class="o">=</span> <span class="n">BatchSpanProcessor</span><span class="p">(</span><span class="n">span_exporter</span><span class="o">=</span><span class="n">otlp_exporter</span><span class="p">,</span> <span class="n">schedule_delay_millis</span><span class="o">=</span><span class="mi">1000</span><span class="p">)</span>
<span class="n">provider</span><span class="p">.</span><span class="n">add_span_processor</span><span class="p">(</span><span class="n">otlp_span_processor</span><span class="p">)</span>
<span class="n">resource</span> <span class="o">=</span> <span class="n">Resource</span><span class="p">.</span><span class="n">create</span><span class="p">({</span><span class="n">SERVICE_NAME</span><span class="p">:</span> <span class="s">"Service1"</span><span class="p">,</span> <span class="n">SERVICE_INSTANCE_ID</span><span class="p">:</span> <span class="s">"1"</span><span class="p">})</span>
<span class="n">provider</span> <span class="o">=</span> <span class="n">TracerProvider</span><span class="p">(</span>
<span class="c1"># This can also be read from envrionment variables https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/
</span> <span class="n">resource</span><span class="o">=</span><span class="n">resource</span>
<span class="p">)</span>
<span class="c1"># setup the exporters
</span><span class="n">add_console_exporter</span><span class="p">(</span><span class="n">provider</span><span class="p">)</span>
<span class="n">add_otlp_exporter</span><span class="p">(</span><span class="n">provider</span><span class="p">)</span>
<span class="c1"># Sets the global default tracer provider
</span><span class="n">trace</span><span class="p">.</span><span class="n">set_tracer_provider</span><span class="p">(</span><span class="n">provider</span><span class="p">)</span>
<span class="c1"># Creates a tracer from the global tracer provider
</span><span class="n">tracer</span> <span class="o">=</span> <span class="n">trace</span><span class="p">.</span><span class="n">get_tracer</span><span class="p">(</span><span class="s">"Service1"</span><span class="p">)</span>
<span class="c1"># Use atrribute function decorator to indicate a new span
</span><span class="o">@</span><span class="n">tracer</span><span class="p">.</span><span class="n">start_as_current_span</span><span class="p">(</span><span class="s">"Service1_Create_Message"</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="n">SpanKind</span><span class="p">.</span><span class="n">INTERNAL</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">some_function</span><span class="p">(</span><span class="n">msg</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">publish_message</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
<span class="n">current_span</span> <span class="o">=</span> <span class="n">trace</span><span class="p">.</span><span class="n">get_current_span</span><span class="p">()</span>
<span class="n">current_span</span><span class="p">.</span><span class="n">set_status</span><span class="p">(</span><span class="n">Status</span><span class="p">(</span><span class="n">StatusCode</span><span class="p">.</span><span class="n">ERROR</span><span class="p">))</span>
<span class="n">current_span</span><span class="p">.</span><span class="n">record_exception</span><span class="p">(</span><span class="n">ex</span><span class="p">)</span>
<span class="k">raise</span>
<span class="n">publish_message</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="o">@</span><span class="n">tracer</span><span class="p">.</span><span class="n">start_as_current_span</span><span class="p">(</span><span class="s">"Service1_Publish_Message"</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="n">SpanKind</span><span class="p">.</span><span class="n">CLIENT</span><span class="p">,</span> <span class="n">attributes</span><span class="o">=</span><span class="p">{</span><span class="n">SpanAttributes</span><span class="p">.</span><span class="n">MESSAGING_PROTOCOL</span><span class="p">:</span> <span class="s">"MQTT"</span><span class="p">})</span>
<span class="k">def</span> <span class="nf">publish_message</span><span class="p">(</span><span class="n">payload</span><span class="p">):</span>
<span class="c1"># Do something here
</span> <span class="c1"># Another way to start a new span is to call tracer.start_as_current_span
</span> <span class="n">tracer</span><span class="p">.</span><span class="n">start_as_current_span</span><span class="p">(</span><span class="s">"publish_message"</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="n">SpanKind</span><span class="p">.</span><span class="n">PRODUCER</span><span class="p">):</span>
<span class="c1"># do the work here
</span></code></pre></div></div>
<h3 id="metrics">Metrics</h3>
<p>It’s the same pattern for metrics</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">opentelemetry</span> <span class="kn">import</span> <span class="n">metrics</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.metrics</span> <span class="kn">import</span> <span class="n">MeterProvider</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.metrics.export</span> <span class="kn">import</span> <span class="n">PeriodicExportingMetricReader</span><span class="p">,</span> <span class="n">ConsoleMetricExporter</span>
<span class="kn">from</span> <span class="nn">opentelemetry.exporter.otlp.proto.grpc.metric_exporter</span> <span class="kn">import</span> <span class="n">OTLPMetricExporter</span>
<span class="n">OTLP_endpoint</span> <span class="o">=</span> <span class="s">"http://127.0.0.1:4317"</span>
<span class="n">console_metric_reader</span> <span class="o">=</span> <span class="n">PeriodicExportingMetricReader</span><span class="p">(</span><span class="n">exporter</span><span class="o">=</span><span class="n">ConsoleMetricExporter</span><span class="p">(),</span> <span class="n">export_interval_millis</span><span class="o">=</span><span class="mi">1000</span><span class="p">)</span>
<span class="n">otlp_metric_reader</span> <span class="o">=</span> <span class="n">PeriodicExportingMetricReader</span><span class="p">(</span><span class="n">exporter</span><span class="o">=</span><span class="n">OTLPMetricExporter</span><span class="p">(</span><span class="n">endpoint</span><span class="o">=</span><span class="n">OTLP_endpoint</span><span class="p">,</span> <span class="n">insecure</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span>
<span class="n">export_interval_millis</span><span class="o">=</span><span class="mi">1000</span><span class="p">)</span>
<span class="n">meter_provider</span> <span class="o">=</span> <span class="n">MeterProvider</span><span class="p">(</span><span class="n">resource</span><span class="o">=</span><span class="n">resource</span><span class="p">,</span>
<span class="n">metric_readers</span><span class="o">=</span><span class="p">[</span><span class="n">console_metric_reader</span><span class="p">,</span> <span class="n">otlp_metric_reader</span><span class="p">])</span>
<span class="n">metrics</span><span class="p">.</span><span class="n">set_meter_provider</span><span class="p">(</span><span class="n">meter_provider</span><span class="o">=</span><span class="n">meter_provider</span><span class="p">)</span>
<span class="c1"># Create meter from global meter provider
</span><span class="n">meter</span> <span class="o">=</span> <span class="n">metrics</span><span class="p">.</span><span class="n">get_meter</span><span class="p">(</span><span class="s">"Service1"</span><span class="p">,</span> <span class="s">"1.0"</span><span class="p">)</span>
<span class="n">counter</span> <span class="o">=</span> <span class="n">meter</span><span class="p">.</span><span class="n">create_counter</span><span class="p">(</span><span class="s">"message_count"</span><span class="p">,</span> <span class="s">"messages"</span><span class="p">,</span> <span class="s">"number of messages"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">some_function</span><span class="p">():</span>
<span class="c1"># increase the counter
</span> <span class="n">counter</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="logging">Logging</h3>
<p>Example from https://github.com/open-telemetry/opentelemetry-python/blob/main/docs/examples/logs/example.py</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">opentelemetry</span> <span class="kn">import</span> <span class="n">trace</span>
<span class="kn">from</span> <span class="nn">opentelemetry._logs</span> <span class="kn">import</span> <span class="n">set_logger_provider</span>
<span class="kn">from</span> <span class="nn">opentelemetry.exporter.otlp.proto.grpc._log_exporter</span> <span class="kn">import</span> <span class="p">(</span>
<span class="n">OTLPLogExporter</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk._logs</span> <span class="kn">import</span> <span class="n">LoggerProvider</span><span class="p">,</span> <span class="n">LoggingHandler</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk._logs.export</span> <span class="kn">import</span> <span class="n">BatchLogRecordProcessor</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.resources</span> <span class="kn">import</span> <span class="n">Resource</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.trace</span> <span class="kn">import</span> <span class="n">TracerProvider</span>
<span class="kn">from</span> <span class="nn">opentelemetry.sdk.trace.export</span> <span class="kn">import</span> <span class="p">(</span>
<span class="n">BatchSpanProcessor</span><span class="p">,</span>
<span class="n">ConsoleSpanExporter</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">trace</span><span class="p">.</span><span class="n">set_tracer_provider</span><span class="p">(</span><span class="n">TracerProvider</span><span class="p">())</span>
<span class="n">trace</span><span class="p">.</span><span class="n">get_tracer_provider</span><span class="p">().</span><span class="n">add_span_processor</span><span class="p">(</span>
<span class="n">BatchSpanProcessor</span><span class="p">(</span><span class="n">ConsoleSpanExporter</span><span class="p">())</span>
<span class="p">)</span>
<span class="n">logger_provider</span> <span class="o">=</span> <span class="n">LoggerProvider</span><span class="p">(</span>
<span class="n">resource</span><span class="o">=</span><span class="n">Resource</span><span class="p">.</span><span class="n">create</span><span class="p">(</span>
<span class="p">{</span>
<span class="s">"service.name"</span><span class="p">:</span> <span class="s">"shoppingcart"</span><span class="p">,</span>
<span class="s">"service.instance.id"</span><span class="p">:</span> <span class="s">"instance-12"</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">),</span>
<span class="p">)</span>
<span class="n">set_logger_provider</span><span class="p">(</span><span class="n">logger_provider</span><span class="p">)</span>
<span class="n">exporter</span> <span class="o">=</span> <span class="n">OTLPLogExporter</span><span class="p">(</span><span class="n">insecure</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">logger_provider</span><span class="p">.</span><span class="n">add_log_record_processor</span><span class="p">(</span><span class="n">BatchLogRecordProcessor</span><span class="p">(</span><span class="n">exporter</span><span class="p">))</span>
<span class="n">handler</span> <span class="o">=</span> <span class="n">LoggingHandler</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="p">.</span><span class="n">NOTSET</span><span class="p">,</span> <span class="n">logger_provider</span><span class="o">=</span><span class="n">logger_provider</span><span class="p">)</span>
<span class="c1"># Attach OTLP handler to root logger
</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">().</span><span class="n">addHandler</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span>
<span class="c1"># Log directly
</span> <span class="n">logging</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"Jackdaws love my big sphinx of quartz."</span><span class="p">)</span>
<span class="c1"># Create different namespaced loggers
</span> <span class="n">logger1</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s">"myapp.area1"</span><span class="p">)</span>
<span class="n">logger2</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s">"myapp.area2"</span><span class="p">)</span>
<span class="n">logger1</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Quick zephyrs blow, vexing daft Jim."</span><span class="p">)</span>
<span class="n">logger1</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"How quickly daft jumping zebras vex."</span><span class="p">)</span>
<span class="n">logger2</span><span class="p">.</span><span class="n">warning</span><span class="p">(</span><span class="s">"Jail zesty vixen who grabbed pay from quack."</span><span class="p">)</span>
<span class="n">logger2</span><span class="p">.</span><span class="n">error</span><span class="p">(</span><span class="s">"The five boxing wizards jump quickly."</span><span class="p">)</span>
<span class="c1"># Trace context correlation
</span> <span class="n">tracer</span> <span class="o">=</span> <span class="n">trace</span><span class="p">.</span><span class="n">get_tracer</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="k">with</span> <span class="n">tracer</span><span class="p">.</span><span class="n">start_as_current_span</span><span class="p">(</span><span class="s">"foo"</span><span class="p">):</span>
<span class="c1"># Do something
</span> <span class="n">logger2</span><span class="p">.</span><span class="n">error</span><span class="p">(</span><span class="s">"Hyderabad, we have a major problem."</span><span class="p">)</span>
<span class="n">logger_provider</span><span class="p">.</span><span class="n">shutdown</span><span class="p">()</span>
</code></pre></div></div>
<p>If you’re looking to easily instrument a popular python library, the <a href="https://github.com/open-telemetry/opentelemetry-python-contrib">open telemetry python contrib repo</a> is the one stop shop for most auto-instrumentation libraries.</p>
<p>For example, here is how you would instrument the <code class="language-plaintext highlighter-rouge">requests</code> package for http calls.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">opentelemetry.instrumentation.requests</span> <span class="kn">import</span> <span class="n">RequestsInstrumentor</span>
<span class="c1"># You can optionally pass a custom TracerProvider to instrument().
</span> <span class="n">RequestsInstrumentor</span><span class="p">().</span><span class="n">instrument</span><span class="p">()</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="s">"https://www.example.org/"</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="mqtt-trace-context-propagation">MQTT Trace Context Propagation</h2>
<p>I am using the <a href="https://www.eclipse.org/paho/index.php?page=clients/python/index.php">paho-mqtt</a> library as my MQTT client SDK.</p>
<p>While this is the most popular MQTT library for Python, I couldn’t find any auto-instrumentation libraries for it in the official contrib repo or anywhere else.</p>
<p>So, I decided to manually instrument it.</p>
<h3 id="propagate-context-injection-and-extraction">Propagate Context (Injection and Extraction)</h3>
<p>One of challenges when manually instrumenting a library that sends data over the wire is to figure out where to store the trace context. I initially thought I would need to define my own envelope like below.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"trace_context"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"traceparent"</span><span class="p">:</span><span class="s2">"00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"</span><span class="p">,</span><span class="w">
</span><span class="nl">"tracestate"</span><span class="p">:</span><span class="s2">"congo=BleGNlZWRzIHRohbCBwbGVhc3VyZS4"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"payload"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Then inject the trace context on publish, extract and hydrate a new span upon receival. That would technically work but I stumbled upon this <strong>draft</strong> <a href="https://w3c.github.io/trace-context-mqtt/">W3C specification for MQTT Trace Context</a>.</p>
<p>According to that I have 2 options (for JSON) depending on what MQTT protocol version I want to use.</p>
<ul>
<li>MQTT v3 (recommendation): Use the payload of the messages and embed the trace context in the <a href="https://w3c.github.io/trace-context-mqtt/#json-payload">root level along with other payload data</a>.</li>
<li>MQTT v5 (specification): Use <a href="https://w3c.github.io/trace-context-mqtt/#mqtt-v5-0-format"><code class="language-plaintext highlighter-rouge">User Properties</code> to embed the trace context</a>. User Properties is a <a href="http://www.steves-internet-guide.com/examining-mqttv5-user-properties/">new feature</a> of MQTT v5.</li>
</ul>
<p>With this information in mind, I decided to go with the latter approach of using MQTT v5 with User Properties.</p>
<h3 id="paho-mqtt-v5-example">Paho MQTT V5 Example</h3>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">paho.mqtt.client</span> <span class="k">as</span> <span class="n">mqtt</span>
<span class="kn">from</span> <span class="nn">paho.mqtt.properties</span> <span class="kn">import</span> <span class="n">Properties</span>
<span class="kn">from</span> <span class="nn">paho.mqtt.packettypes</span> <span class="kn">import</span> <span class="n">PacketTypes</span>
<span class="kn">from</span> <span class="nn">opentelemetry.trace.propagation.tracecontext</span> <span class="kn">import</span> <span class="n">TraceContextTextMapPropagator</span>
<span class="c1"># Use the trace and metrics examples above to setup trace and metric providers here.
</span>
<span class="c1"># Connect to mqtt v5 server and subscribe to messages as shown in http://www.steves-internet-guide.com/into-mqtt-python-client/
</span>
<span class="c1"># Publishing with trace context
</span><span class="o">@</span><span class="n">tracer</span><span class="p">.</span><span class="n">start_as_current_span</span><span class="p">(</span><span class="s">"Service2_Publish_Message"</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="n">SpanKind</span><span class="p">.</span><span class="n">PRODUCER</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">publish_message</span><span class="p">(</span><span class="n">payload</span><span class="p">):</span>
<span class="c1"># We are injecting the current propagation context into the mqtt message as per https://w3c.github.io/trace-context-mqtt/#mqtt-v5-0-format
</span> <span class="n">carrier</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">propagator</span> <span class="o">=</span> <span class="n">TraceContextTextMapPropagator</span><span class="p">()</span>
<span class="n">propagator</span><span class="p">.</span><span class="n">inject</span><span class="p">(</span><span class="n">carrier</span><span class="o">=</span><span class="n">carrier</span><span class="p">)</span>
<span class="n">properties</span> <span class="o">=</span> <span class="n">Properties</span><span class="p">(</span><span class="n">PacketTypes</span><span class="p">.</span><span class="n">PUBLISH</span><span class="p">)</span>
<span class="n">properties</span><span class="p">.</span><span class="n">UserProperty</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">carrier</span><span class="p">.</span><span class="n">items</span><span class="p">())</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Carrier after injecting span context"</span><span class="p">,</span> <span class="n">properties</span><span class="p">.</span><span class="n">UserProperty</span><span class="p">)</span>
<span class="c1"># publish
</span> <span class="n">client</span><span class="p">.</span><span class="n">publish</span><span class="p">(</span><span class="s">"otel-demo/output2"</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">properties</span><span class="o">=</span><span class="n">properties</span><span class="p">,</span> <span class="n">retain</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="c1"># Receiving message
</span><span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">userdata</span><span class="p">,</span> <span class="n">msg</span><span class="p">):</span>
<span class="n">payload</span> <span class="o">=</span> <span class="n">msg</span><span class="p">.</span><span class="n">payload</span><span class="p">.</span><span class="n">decode</span><span class="p">(</span><span class="s">"utf-8"</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"MQTT msg recieved: </span><span class="si">{</span><span class="n">payload</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="n">counter</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">labels</span><span class="p">)</span>
<span class="c1"># We need to extract the propagation context from user properties https://w3c.github.io/trace-context-mqtt/#trace-context-fields-placement-in-a-message
</span> <span class="n">prop</span> <span class="o">=</span> <span class="n">TraceContextTextMapPropagator</span><span class="p">()</span>
<span class="n">user_properties</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">msg</span><span class="p">.</span><span class="n">properties</span><span class="p">.</span><span class="n">UserProperty</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Carrier with span context"</span><span class="p">,</span> <span class="n">user_properties</span><span class="p">)</span>
<span class="n">ctx</span> <span class="o">=</span> <span class="n">prop</span><span class="p">.</span><span class="n">extract</span><span class="p">(</span><span class="n">carrier</span><span class="o">=</span><span class="n">user_properties</span><span class="p">)</span>
<span class="c1"># Create a new span with context extracted from message
</span> <span class="k">with</span> <span class="n">tracer</span><span class="p">.</span><span class="n">start_as_current_span</span><span class="p">(</span><span class="s">"Service2_Receive_Message"</span><span class="p">,</span> <span class="n">context</span><span class="o">=</span><span class="n">ctx</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="n">SpanKind</span><span class="p">.</span><span class="n">SERVER</span><span class="p">):</span>
<span class="n">current_span</span> <span class="o">=</span> <span class="n">trace</span><span class="p">.</span><span class="n">get_current_span</span><span class="p">()</span>
<span class="n">current_span</span><span class="p">.</span><span class="n">add_event</span><span class="p">(</span><span class="s">"Gonna try to do something!"</span><span class="p">)</span> <span class="c1"># Events are are primitive logs
</span> <span class="c1"># Do something here
</span> <span class="n">current_span</span><span class="p">.</span><span class="n">add_event</span><span class="p">(</span><span class="s">"Processed message!"</span><span class="p">)</span>
<span class="k">pass</span>
</code></pre></div></div>
<h3 id="summary">Summary</h3>
<p>The above code samples should now allow you to setup tracing, metrics and logging for a python app, instrument paho-mqtt library for trace context propagation and then export telemetry to a OTLP endpoint (OTEL Collector).</p>
<p>You can find the code samples <a href="https://github.com/dasiths/OpenTelemetryDistributedTracingSample/tree/master/python">here</a>.</p>
<h2 id="otel-architecture">OTEL Architecture</h2>
<p>There are 2 ways of exporting OTEL specific telemetry out of your application and getting them displayed in an observability tool like Zipkin, Jaeger, Prometheus, Azure Monitor etc.</p>
<ul>
<li>Export it directly to the tool of your choice using an exporter library. (See this <a href="https://opentelemetry-python.readthedocs.io/en/latest/exporter/zipkin/zipkin.html">example for ZipKin</a>).</li>
<li><a href="https://opentelemetry-python.readthedocs.io/en/latest/exporter/otlp/otlp.html">Export it using the OTLP format</a> to a OTEL Collector instance, and then <a href="https://opentelemetry.io/docs/collector/configuration/">configure the OTEL Collector</a> to export the telemetry from there to the observability frontend of your choice. <img src="/assets/images/otel_diagram.png" alt="Example from https://opentelemetry.io/docs/" /></li>
</ul>
<p>I prefer the latter option because it allows me to change my observability tools at anytime during the lifetime of the application without any code changes to the app. I only need to update the OTEL Collector configuration and redeploy the collector instance. It is much more enterprise friendly and less coupled to the app this way. Your OPS team will like this approach as it gives them control over observability without having to touch your code.</p>
<h2 id="deploying-the-otel-collector-in-k8s-or-docker-compose">Deploying The OTEL Collector in K8s or Docker Compose</h2>
<p>If you’re using the basic built in exporters like Zipkin and Prometheus you can use the <a href="https://opentelemetry.io/docs/k8s-operator/">OTEL Collector Operator for K8s</a>.</p>
<p>In my case I wanted to export to Azure Monitor so I had to use the <code class="language-plaintext highlighter-rouge">contrib</code> variant from <code class="language-plaintext highlighter-rouge">otel/opentelemetry-collector-contrib</code> docker hub image.</p>
<p>If you want to use the contrib variant, an <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/examples/kubernetes/otel-collector.yaml">example with k8s manifests can be found here</a>.</p>
<p>Here are the assets from my example which used docker compose.</p>
<p>Docker Compose File</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="na">prometheus-data</span><span class="pi">:</span> <span class="pi">{}</span>
<span class="na">grafana-data</span><span class="pi">:</span> <span class="pi">{}</span>
<span class="na">services</span><span class="pi">:</span>
<span class="c1"># Jaeger</span>
<span class="na">jaeger</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">jaegertracing/all-in-one:latest</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">16686:16686"</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">14250"</span>
<span class="c1">#Zipkin</span>
<span class="na">zipkin</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">openzipkin/zipkin</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">zipkin</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">9411:9411</span>
<span class="na">otel-collector</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">otel/opentelemetry-collector-contrib:0.50.0</span>
<span class="c1">#image: otel/opentelemetry-collector</span>
<span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">--config=/etc/otel-collector-config.yaml"</span><span class="pi">]</span>
<span class="na">volumes</span><span class="pi">:</span> <span class="c1"># mount your config here</span>
<span class="pi">-</span> <span class="s">${HOST_PROJECT_PATH}/otel-example/otel-collector-config.yaml:/etc/otel-collector-config.yaml</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="c1"># - "1888:1888" # pprof extension</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">8888:8888"</span> <span class="c1"># Prometheus metrics exposed by the collector</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">8889:8889"</span> <span class="c1"># Prometheus exporter metrics</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">13133:13133"</span> <span class="c1"># health_check extension</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">4317:4317"</span> <span class="c1"># OTLP gRPC receiver</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">4318:4318"</span> <span class="c1"># OTLP http receiver</span>
<span class="c1"># - "55679:55679" # zpages extension</span>
<span class="na">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">jaeger</span>
<span class="pi">-</span> <span class="s">zipkin</span>
<span class="na">prometheus</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">prom/prometheus:v2.30.3</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">9000:9090</span>
<span class="na">volumes</span><span class="pi">:</span> <span class="c1"># mount your config here</span>
<span class="pi">-</span> <span class="s">${HOST_PROJECT_PATH}/otel-example/prometheus:/etc/prometheus</span>
<span class="pi">-</span> <span class="s">prometheus-data:${HOST_PROJECT_PATH}/otel-example/prometheus</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">--web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml</span>
<span class="na">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">otel-collector</span>
<span class="na">grafana</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">grafana/grafana:7.5.7</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">3000:3000</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
<span class="na">volumes</span><span class="pi">:</span> <span class="c1"># mount your config here</span>
<span class="pi">-</span> <span class="s">${HOST_PROJECT_PATH}/otel-example/grafana:/etc/grafana/provisioning/datasources</span>
<span class="pi">-</span> <span class="s">grafana-data:/var/lib/grafana</span>
<span class="na">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">prometheus</span>
</code></pre></div></div>
<p>OTEL Config</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">receivers</span><span class="pi">:</span>
<span class="na">otlp</span><span class="pi">:</span>
<span class="na">protocols</span><span class="pi">:</span>
<span class="na">grpc</span><span class="pi">:</span>
<span class="na">zipkin</span><span class="pi">:</span>
<span class="na">exporters</span><span class="pi">:</span>
<span class="na">azuremonitor</span><span class="pi">:</span>
<span class="na">instrumentation_key</span><span class="pi">:</span> <span class="s">your-app-insights-key</span>
<span class="na">jaeger</span><span class="pi">:</span>
<span class="na">endpoint</span><span class="pi">:</span> <span class="s">jaeger:14250</span>
<span class="na">tls</span><span class="pi">:</span>
<span class="na">insecure</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">logging</span><span class="pi">:</span>
<span class="na">zipkin</span><span class="pi">:</span>
<span class="na">endpoint</span><span class="pi">:</span> <span class="s2">"</span><span class="s">http://zipkin:9411/api/v2/spans"</span>
<span class="na">prometheus</span><span class="pi">:</span>
<span class="na">endpoint</span><span class="pi">:</span> <span class="s">0.0.0.0:8889</span>
<span class="na">const_labels</span><span class="pi">:</span>
<span class="na">label1</span><span class="pi">:</span> <span class="s">value1</span>
<span class="na">send_timestamps</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">metric_expiration</span><span class="pi">:</span> <span class="s">180m</span>
<span class="na">resource_to_telemetry_conversion</span><span class="pi">:</span>
<span class="na">enabled</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">processors</span><span class="pi">:</span>
<span class="na">batch</span><span class="pi">:</span>
<span class="na">extensions</span><span class="pi">:</span>
<span class="na">health_check</span><span class="pi">:</span>
<span class="na">pprof</span><span class="pi">:</span>
<span class="na">zpages</span><span class="pi">:</span>
<span class="na">service</span><span class="pi">:</span>
<span class="na">extensions</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">pprof</span><span class="pi">,</span> <span class="nv">zpages</span><span class="pi">,</span> <span class="nv">health_check</span><span class="pi">]</span>
<span class="na">pipelines</span><span class="pi">:</span>
<span class="na">traces</span><span class="pi">:</span>
<span class="na">receivers</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">otlp</span><span class="pi">,</span> <span class="nv">zipkin</span><span class="pi">]</span>
<span class="na">exporters</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">zipkin</span><span class="pi">,</span> <span class="nv">jaeger</span><span class="pi">,</span> <span class="nv">logging</span><span class="pi">,</span> <span class="nv">azuremonitor</span><span class="pi">]</span>
<span class="na">processors</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">batch</span><span class="pi">]</span>
<span class="na">metrics</span><span class="pi">:</span>
<span class="na">receivers</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">otlp</span><span class="pi">]</span>
<span class="na">processors</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">batch</span><span class="pi">]</span>
<span class="na">exporters</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">logging</span><span class="pi">,</span> <span class="nv">prometheus</span><span class="pi">]</span>
</code></pre></div></div>
<p>Prometheus Config</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">global</span><span class="pi">:</span>
<span class="na">scrape_interval</span><span class="pi">:</span> <span class="s">30s</span>
<span class="na">scrape_timeout</span><span class="pi">:</span> <span class="s">10s</span>
<span class="na">scrape_configs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">job_name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">otel-prometheus"</span>
<span class="na">static_configs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">targets</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">otel-collector:8889"</span><span class="pi">]</span>
</code></pre></div></div>
<p>Grafana Config</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">datasources</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Prometheus</span>
<span class="na">access</span><span class="pi">:</span> <span class="s">proxy</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">prometheus</span>
<span class="na">url</span><span class="pi">:</span> <span class="s">http://prometheus:9090</span>
<span class="na">isDefault</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div></div>
<p>You can use the above manifests as a guide when deploying to k8s or docker compose and I recommend reading through the various options to understand how the OTEL Collector config and other push/pull exporters are composed together.</p>
<h3 id="bonus-reading">Bonus Reading</h3>
<p>Have a look at how <a href="https://docs.dapr.io/operations/monitoring/tracing/otel-collector/open-telemetry-collector/">Dapr configures the OTEL Collector</a> to capture telemetry and forwards it to a observability front end like Zipkin. Everything is setup to run in k8s.</p>
<h2 id="finishing-up">Finishing Up</h2>
<p>We looked at how to instrument a python app using MQTT and how to export telemetry via an OTEL Collector instance. Hopefully this serves as a starting point to help you orient yourself with the basic concepts of OTEL Signals and telemetry exporting. The code samples will be uploaded to https://github.com/dasiths/OpenTelemetryDistributedTracingSample/tree/master/python</p>
<p>If you have any questions please reach out to me via twitter @dasiths. Happy coding.</p>Dasith WijesiriwardenaSome time back I did a bit of an intro to OpenTelemetry and in there I covered some basics like what Signals and Context Propagation are. I also spoke about how concepts like Tracing, Spans and Instrumentation interrelate to one another. I even put some code samples up at GitHub to demo this. Most if not all of those code samples are in .NET and they demo tracing and baggage. Since I did that talk in 2021 the OpenTelemetry community have decided to add logs as a signal. Logs Are a Signal There are 4 types of signals as of the time of writing this. Tracing Metrics Baggage Logs The Logs have the same specification as a span event we used to know before. Instrumenting Python (and Paho MQTT Client) I recently had to instrument an existing app written in python that uses MQTT protocol to communicate. There were a few things I needed to do Instrument the python app(s) using OTEL Python SDK for Tracing, Metrics and Logs Figure out how context propagation works with the MQTT protocol (if the python MQTT client I used isn’t already instrumented. Spoiler, it wasn’t) Decide if I use specific exporters directly from the python app (No OTEL Collector) or Export to an OTEL Collector in OTLP format and then export it to specific tool from there. Spoiler. I chose the OTEL Collector approach. Deploy OTEL Collector to k8s/Docker Compose and configure it to export to my tools like Jaeger and Prometheus. Configuring OTEL Collector with exporters Configuring Prometheus to scrape from my OTEL collector Setting up Grafana to add Prometheus as a data source Setting up Azure Monitor Exporter OTEL Python SDK The OTEL official documentation is a good place to start. There are some examples of how to setup and use traces/metrics. If you need something more specific, there are more examples here. For brevity let’s look at some simple code examples. First, install these packages pip install opentelemetry-api pip install opentelemetry-sdk pip install opentelemetry-exporter-otlp Traces from opentelemetry import trace from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator from opentelemetry.trace import Status, StatusCode, SpanKind from opentelemetry.sdk.resources import SERVICE_NAME, SERVICE_INSTANCE_ID, Resource from opentelemetry.semconv.trace import SpanAttributes from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import ( BatchSpanProcessor, ConsoleSpanExporter, ) from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter OTLP_endpoint = "http://127.0.0.1:4317" def add_console_exporter(provider: TracerProvider): processor = BatchSpanProcessor(span_exporter=ConsoleSpanExporter(), schedule_delay_millis=1000) provider.add_span_processor(processor) def add_otlp_exporter(provider: TracerProvider): otlp_exporter = OTLPSpanExporter(endpoint=OTLP_endpoint, insecure=True) otlp_span_processor = BatchSpanProcessor(span_exporter=otlp_exporter, schedule_delay_millis=1000) provider.add_span_processor(otlp_span_processor) resource = Resource.create({SERVICE_NAME: "Service1", SERVICE_INSTANCE_ID: "1"}) provider = TracerProvider( # This can also be read from envrionment variables https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/ resource=resource ) # setup the exporters add_console_exporter(provider) add_otlp_exporter(provider) # Sets the global default tracer provider trace.set_tracer_provider(provider) # Creates a tracer from the global tracer provider tracer = trace.get_tracer("Service1") # Use atrribute function decorator to indicate a new span @tracer.start_as_current_span("Service1_Create_Message", kind=SpanKind.INTERNAL) def some_function(msg): try: publish_message(msg) except Exception as ex: current_span = trace.get_current_span() current_span.set_status(Status(StatusCode.ERROR)) current_span.record_exception(ex) raise publish_message(msg) @tracer.start_as_current_span("Service1_Publish_Message", kind=SpanKind.CLIENT, attributes={SpanAttributes.MESSAGING_PROTOCOL: "MQTT"}) def publish_message(payload): # Do something here # Another way to start a new span is to call tracer.start_as_current_span tracer.start_as_current_span("publish_message", kind=SpanKind.PRODUCER): # do the work here Metrics It’s the same pattern for metrics from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader, ConsoleMetricExporter from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter OTLP_endpoint = "http://127.0.0.1:4317" console_metric_reader = PeriodicExportingMetricReader(exporter=ConsoleMetricExporter(), export_interval_millis=1000) otlp_metric_reader = PeriodicExportingMetricReader(exporter=OTLPMetricExporter(endpoint=OTLP_endpoint, insecure=True), export_interval_millis=1000) meter_provider = MeterProvider(resource=resource, metric_readers=[console_metric_reader, otlp_metric_reader]) metrics.set_meter_provider(meter_provider=meter_provider) # Create meter from global meter provider meter = metrics.get_meter("Service1", "1.0") counter = meter.create_counter("message_count", "messages", "number of messages") def some_function(): # increase the counter counter.add(1) Logging Example from https://github.com/open-telemetry/opentelemetry-python/blob/main/docs/examples/logs/example.py import logging from opentelemetry import trace from opentelemetry._logs import set_logger_provider from opentelemetry.exporter.otlp.proto.grpc._log_exporter import ( OTLPLogExporter, ) from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler from opentelemetry.sdk._logs.export import BatchLogRecordProcessor from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import ( BatchSpanProcessor, ConsoleSpanExporter, ) trace.set_tracer_provider(TracerProvider()) trace.get_tracer_provider().add_span_processor( BatchSpanProcessor(ConsoleSpanExporter()) ) logger_provider = LoggerProvider( resource=Resource.create( { "service.name": "shoppingcart", "service.instance.id": "instance-12", } ), ) set_logger_provider(logger_provider) exporter = OTLPLogExporter(insecure=True) logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter)) handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider) # Attach OTLP handler to root logger logging.getLogger().addHandler(handler) # Log directly logging.info("Jackdaws love my big sphinx of quartz.") # Create different namespaced loggers logger1 = logging.getLogger("myapp.area1") logger2 = logging.getLogger("myapp.area2") logger1.debug("Quick zephyrs blow, vexing daft Jim.") logger1.info("How quickly daft jumping zebras vex.") logger2.warning("Jail zesty vixen who grabbed pay from quack.") logger2.error("The five boxing wizards jump quickly.") # Trace context correlation tracer = trace.get_tracer(__name__) with tracer.start_as_current_span("foo"): # Do something logger2.error("Hyderabad, we have a major problem.") logger_provider.shutdown() If you’re looking to easily instrument a popular python library, the open telemetry python contrib repo is the one stop shop for most auto-instrumentation libraries. For example, here is how you would instrument the requests package for http calls. import requests from opentelemetry.instrumentation.requests import RequestsInstrumentor # You can optionally pass a custom TracerProvider to instrument(). RequestsInstrumentor().instrument() response = requests.get(url="https://www.example.org/") MQTT Trace Context Propagation I am using the paho-mqtt library as my MQTT client SDK. While this is the most popular MQTT library for Python, I couldn’t find any auto-instrumentation libraries for it in the official contrib repo or anywhere else. So, I decided to manually instrument it. Propagate Context (Injection and Extraction) One of challenges when manually instrumenting a library that sends data over the wire is to figure out where to store the trace context. I initially thought I would need to define my own envelope like below. { "trace_context": { "traceparent":"00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", "tracestate":"congo=BleGNlZWRzIHRohbCBwbGVhc3VyZS4" }, "payload": "" } Then inject the trace context on publish, extract and hydrate a new span upon receival. That would technically work but I stumbled upon this draft W3C specification for MQTT Trace Context. According to that I have 2 options (for JSON) depending on what MQTT protocol version I want to use. MQTT v3 (recommendation): Use the payload of the messages and embed the trace context in the root level along with other payload data. MQTT v5 (specification): Use User Properties to embed the trace context. User Properties is a new feature of MQTT v5. With this information in mind, I decided to go with the latter approach of using MQTT v5 with User Properties. Paho MQTT V5 Example import paho.mqtt.client as mqtt from paho.mqtt.properties import Properties from paho.mqtt.packettypes import PacketTypes from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator # Use the trace and metrics examples above to setup trace and metric providers here. # Connect to mqtt v5 server and subscribe to messages as shown in http://www.steves-internet-guide.com/into-mqtt-python-client/ # Publishing with trace context @tracer.start_as_current_span("Service2_Publish_Message", kind=SpanKind.PRODUCER) def publish_message(payload): # We are injecting the current propagation context into the mqtt message as per https://w3c.github.io/trace-context-mqtt/#mqtt-v5-0-format carrier = {} propagator = TraceContextTextMapPropagator() propagator.inject(carrier=carrier) properties = Properties(PacketTypes.PUBLISH) properties.UserProperty = list(carrier.items()) print("Carrier after injecting span context", properties.UserProperty) # publish client.publish("otel-demo/output2", payload, properties=properties, retain=True) # Receiving message def on_message(client, userdata, msg): payload = msg.payload.decode("utf-8") print(f"MQTT msg recieved: {payload}") counter.add(1, labels) # We need to extract the propagation context from user properties https://w3c.github.io/trace-context-mqtt/#trace-context-fields-placement-in-a-message prop = TraceContextTextMapPropagator() user_properties = dict(msg.properties.UserProperty) print("Carrier with span context", user_properties) ctx = prop.extract(carrier=user_properties) # Create a new span with context extracted from message with tracer.start_as_current_span("Service2_Receive_Message", context=ctx, kind=SpanKind.SERVER): current_span = trace.get_current_span() current_span.add_event("Gonna try to do something!") # Events are are primitive logs # Do something here current_span.add_event("Processed message!") pass Summary The above code samples should now allow you to setup tracing, metrics and logging for a python app, instrument paho-mqtt library for trace context propagation and then export telemetry to a OTLP endpoint (OTEL Collector). You can find the code samples here. OTEL Architecture There are 2 ways of exporting OTEL specific telemetry out of your application and getting them displayed in an observability tool like Zipkin, Jaeger, Prometheus, Azure Monitor etc. Export it directly to the tool of your choice using an exporter library. (See this example for ZipKin). Export it using the OTLP format to a OTEL Collector instance, and then configure the OTEL Collector to export the telemetry from there to the observability frontend of your choice. I prefer the latter option because it allows me to change my observability tools at anytime during the lifetime of the application without any code changes to the app. I only need to update the OTEL Collector configuration and redeploy the collector instance. It is much more enterprise friendly and less coupled to the app this way. Your OPS team will like this approach as it gives them control over observability without having to touch your code. Deploying The OTEL Collector in K8s or Docker Compose If you’re using the basic built in exporters like Zipkin and Prometheus you can use the OTEL Collector Operator for K8s. In my case I wanted to export to Azure Monitor so I had to use the contrib variant from otel/opentelemetry-collector-contrib docker hub image. If you want to use the contrib variant, an example with k8s manifests can be found here. Here are the assets from my example which used docker compose. Docker Compose File version: "3" volumes: prometheus-data: {} grafana-data: {} services: # Jaeger jaeger: image: jaegertracing/all-in-one:latest ports: - "16686:16686" - "14250" #Zipkin zipkin: image: openzipkin/zipkin container_name: zipkin ports: - 9411:9411 otel-collector: image: otel/opentelemetry-collector-contrib:0.50.0 #image: otel/opentelemetry-collector command: ["--config=/etc/otel-collector-config.yaml"] volumes: # mount your config here - ${HOST_PROJECT_PATH}/otel-example/otel-collector-config.yaml:/etc/otel-collector-config.yaml ports: # - "1888:1888" # pprof extension - "8888:8888" # Prometheus metrics exposed by the collector - "8889:8889" # Prometheus exporter metrics - "13133:13133" # health_check extension - "4317:4317" # OTLP gRPC receiver - "4318:4318" # OTLP http receiver # - "55679:55679" # zpages extension depends_on: - jaeger - zipkin prometheus: image: prom/prometheus:v2.30.3 ports: - 9000:9090 volumes: # mount your config here - ${HOST_PROJECT_PATH}/otel-example/prometheus:/etc/prometheus - prometheus-data:${HOST_PROJECT_PATH}/otel-example/prometheus command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml depends_on: - otel-collector grafana: image: grafana/grafana:7.5.7 ports: - 3000:3000 restart: unless-stopped volumes: # mount your config here - ${HOST_PROJECT_PATH}/otel-example/grafana:/etc/grafana/provisioning/datasources - grafana-data:/var/lib/grafana depends_on: - prometheus OTEL Config receivers: otlp: protocols: grpc: zipkin: exporters: azuremonitor: instrumentation_key: your-app-insights-key jaeger: endpoint: jaeger:14250 tls: insecure: true logging: zipkin: endpoint: "http://zipkin:9411/api/v2/spans" prometheus: endpoint: 0.0.0.0:8889 const_labels: label1: value1 send_timestamps: true metric_expiration: 180m resource_to_telemetry_conversion: enabled: true processors: batch: extensions: health_check: pprof: zpages: service: extensions: [pprof, zpages, health_check] pipelines: traces: receivers: [otlp, zipkin] exporters: [zipkin, jaeger, logging, azuremonitor] processors: [batch] metrics: receivers: [otlp] processors: [batch] exporters: [logging, prometheus] Prometheus Config global: scrape_interval: 30s scrape_timeout: 10s scrape_configs: - job_name: "otel-prometheus" static_configs: - targets: ["otel-collector:8889"] Grafana Config datasources: - name: Prometheus access: proxy type: prometheus url: http://prometheus:9090 isDefault: true You can use the above manifests as a guide when deploying to k8s or docker compose and I recommend reading through the various options to understand how the OTEL Collector config and other push/pull exporters are composed together. Bonus Reading Have a look at how Dapr configures the OTEL Collector to capture telemetry and forwards it to a observability front end like Zipkin. Everything is setup to run in k8s. Finishing Up We looked at how to instrument a python app using MQTT and how to export telemetry via an OTEL Collector instance. Hopefully this serves as a starting point to help you orient yourself with the basic concepts of OTEL Signals and telemetry exporting. The code samples will be uploaded to https://github.com/dasiths/OpenTelemetryDistributedTracingSample/tree/master/python If you have any questions please reach out to me via twitter @dasiths. Happy coding.Going down the rabbit hole of EF Core and converting strings to dates2022-01-23T22:06:00+11:002022-01-23T22:06:00+11:00https://dasith.me/2022/01/23/ef-core-datetime-conversion-rabbit-hole<p>I am working on a greenfield project that uses EF Core 6 with AspNetCore 6 at the moment. The project involves exposing a set of legacy data through an API. Simple enough right?</p>
<p>The underlying data is stored in SQL Server 2019 but it is not very well designed. There are <code class="language-plaintext highlighter-rouge">varchar</code> columns for storing <code class="language-plaintext highlighter-rouge">boolean</code>, <code class="language-plaintext highlighter-rouge">numeric</code> and <code class="language-plaintext highlighter-rouge">date/time</code> values. It’s not uncommon to see these types of data stores though. As developers we have to deal with them often.</p>
<h2 id="dapper-or-ef-core">Dapper or EF Core</h2>
<p>When choosing the data access layer for the project I had the option to go with <a href="https://github.com/DapperLib/Dapper">Dapper</a> or EF Core. I choose to go with EF Core because this specific API had a lot of requirements around paging and sorting (See here for <a href="https://api.gov.au/standards/national_api_standards/">more</a>). You can easily implement paging and sorting with Dapper too. But I find constructing paging and sorting dynamically using EF Core <code class="language-plaintext highlighter-rouge">IQueryable</code> more appealing than manipulating strings in Dapper. I will do another post about dynamic paging and sorting using EF Core soon.</p>
<p>But this choice comes with trade offs as with any technical decision. While I don’t have to “construct” SQL with string manipulation, an ORM comes at a cost of not being able to execute the exact SQL I want if I’m using <code class="language-plaintext highlighter-rouge">IQueryable</code> to construct my LINQ query. This is a hot topic when it comes to designing your data access layer but that is a topic for another post.</p>
<h2 id="the-problem">The Problem</h2>
<p>Imagine the following schema for a table called <code class="language-plaintext highlighter-rouge">CustomerLease</code>.</p>
<table>
<thead>
<tr>
<th>Column</th>
<th>Data Type</th>
</tr>
</thead>
<tbody>
<tr>
<td>LeaseId</td>
<td>int</td>
</tr>
<tr>
<td>CustomerId</td>
<td>int</td>
</tr>
<tr>
<td>LeasedItem</td>
<td>nvarchar(2000) NULL</td>
</tr>
<tr>
<td>LeaseStart</td>
<td>nvarchar(10)</td>
</tr>
<tr>
<td>LeaseEnd</td>
<td>nvarchar(10) NULL</td>
</tr>
</tbody>
</table>
<p>We are required to find customer leases that started after a given date.</p>
<p>Now lets assume what we would do if the <code class="language-plaintext highlighter-rouge">LeaseStart</code> was <code class="language-plaintext highlighter-rouge">DateTime</code> .NET Type in my EF Core entity model for <code class="language-plaintext highlighter-rouge">CustomerLease</code>.</p>
<div class="language-c# highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">public</span> <span class="k">class</span> <span class="nc">CustomerLease</span>
<span class="p">{</span>
<span class="c1">//... other fields</span>
<span class="n">DateTime</span> <span class="n">LeaseStart</span> <span class="p">{</span><span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">MyRepo</span> <span class="p">{</span>
<span class="c1">// constructor and other properties will go here...</span>
<span class="c1">// example method to search within date periods</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">List</span><span class="p"><</span><span class="n">CustomerLease</span><span class="p">>></span> <span class="nf">GetCustomerLeases</span><span class="p">(</span><span class="n">SearchRequest</span> <span class="n">request</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">searchFrom</span> <span class="p">=</span> <span class="n">request</span><span class="p">.</span><span class="n">SearchFrom</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">query</span> <span class="p">=</span> <span class="n">MyDataContext</span><span class="p">.</span><span class="n">CustomerLeases</span>
<span class="p">.</span><span class="nf">Where</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="n">searchFrom</span> <span class="p"><=</span> <span class="n">c</span><span class="p">.</span><span class="n">LeaseStart</span><span class="p">);</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">query</span><span class="p">.</span><span class="nf">ToListAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><strong>This solution would work if my underlying DB type was DateTime BUT it is not.</strong></p>
<p>So my actual entity model looks like…</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">public</span> <span class="k">class</span> <span class="nc">CustomerLease</span>
<span class="p">{</span>
<span class="c1">//... other fields</span>
<span class="kt">string</span> <span class="n">LeaseStart</span> <span class="p">{</span><span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;}</span>
<span class="p">}</span>
</code></pre></div></div>
<h3 id="now-i-cant-write-my-linq-query-with-direct-comparison-to-searchfrom-what-are-my-alternatives">Now I can’t write my LINQ query with direct comparison to SearchFrom. What are my alternatives?</h3>
<ol>
<li>Try converting the <code class="language-plaintext highlighter-rouge">string</code> to a <code class="language-plaintext highlighter-rouge">DateTime</code> within the LINQ query.
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> DateTime.Parse(...)
// or
Convert.ToDateTime(...)
</code></pre></div> </div>
<p>This will work if our underlying <code class="language-plaintext highlighter-rouge">IQueryable</code> provider for SQL Server supported translating these functions to SQL. But unfortunately <a href="https://docs.microsoft.com/en-us/ef/core/providers/sql-server/functions">they aren’t</a>. So this approach is out of the question.</p>
</li>
<li>
<p>Using implicit conversion .</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> .Where(c => searchFrom <= (DateTime)(object)c.LeaseStart
</code></pre></div> </div>
<p>This technique generates the following SQL. “<code class="language-plaintext highlighter-rouge">CAST([S].[LeaseStart] as DateTime) >= @__searchFrom__</code>” This will work but word of caution. This double casting we have done in LINQ to trick the underlying provider to use CAST will only work for SQL Provider. It <strong>will not work</strong> for the In-Memory database provider if you’re using it for writing unit/integration tests.</p>
<p>The other drawback here is that it expects the dates to be in the default format of the current session language. (i.e. US English, British English etc). If you have a date there like <code class="language-plaintext highlighter-rouge">24/05/2021</code> and the the current language is US English then it will fail with a message like <code class="language-plaintext highlighter-rouge">"The conversion of a varchar data type to a datetime data type resulted in an out-of-range value".</code> I talk about this again below in option 3 and touch on some work arounds.</p>
</li>
<li>
<p>Using EF Core value converter.</p>
<p>With EF Core 5+ you can use <a href="https://docs.microsoft.com/en-us/ef/core/modeling/value-conversions?tabs=data-annotations#built-in-converters"><code class="language-plaintext highlighter-rouge">Value Converters</code></a> for this scenario and there are <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.storage.valueconversion.stringtodatetimeconverter?view=efcore-6.0">built in ones</a> for some common use cases.</p>
<p>Be mindful that ValueConverters work inside .NET and not SQL. So how do we get it to do a CAST on our <code class="language-plaintext highlighter-rouge">varchar</code> column?</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">OnModelCreating</span><span class="p">(</span><span class="n">ModelBuilder</span> <span class="n">modelBuilder</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// The column TextDate is the one that has date values but stored as text in the db</span>
<span class="n">modelBuilder</span>
<span class="p">.</span><span class="n">Entity</span><span class="p"><</span><span class="n">CustomerLease</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">Property</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="n">c</span><span class="p">.</span><span class="n">LeaseStart</span><span class="p">)</span>
<span class="p">.</span><span class="n">HasConversion</span><span class="p"><</span><span class="kt">string</span><span class="p">>();</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">CustomerLease</span>
<span class="p">{</span>
<span class="c1">//... other fields</span>
<span class="n">DateTime</span> <span class="n">LeaseStart</span> <span class="p">{</span><span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;}</span>
<span class="p">}</span>
</code></pre></div> </div>
<p>Then in LINQ simply do <code class="language-plaintext highlighter-rouge">.Where(e => e.LeaseStart >= startSearch)</code>.</p>
<p>Here is the kicker. For EF Core to generate the correct SQL statement, <strong>it will require <code class="language-plaintext highlighter-rouge">startSearch</code> parameter inside the LINQ query to be of type <code class="language-plaintext highlighter-rouge">DateTimeOffset</code></strong>.</p>
<p>It doesn’t use CAST if the parameter is <code class="language-plaintext highlighter-rouge">DateTime</code> as it simply converts your parameter to <code class="language-plaintext highlighter-rouge">varchar</code> and then compares. I made <a href="https://gist.github.com/dasiths/19b885c58442226d9fc8b89bc78511e4">this gist</a> to demo the behaviour.</p>
<p>This is more of a hack as we are relying on implicit conversion of <code class="language-plaintext highlighter-rouge">DateTime</code> from/to <code class="language-plaintext highlighter-rouge">DateTimeOffset</code> inside .NET and then letting the EFCORE SQL Provider do a CAST when comparing inside SQL.</p>
<p>The above LINQ will generate SQL like…</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">DECLARE</span> <span class="o">@</span><span class="n">__startSearch_0</span> <span class="n">datetimeoffset</span> <span class="o">=</span> <span class="s1">'2022-01-22T23:01:43.0090270+11:00'</span><span class="p">;</span>
<span class="o">#</span> <span class="k">and</span> <span class="n">query</span> <span class="k">like</span>
<span class="k">WHERE</span> <span class="p">((</span><span class="o">@</span><span class="n">__startSearch_0</span> <span class="o"><=</span> <span class="k">CAST</span><span class="p">([</span><span class="n">s</span><span class="p">].[</span><span class="n">LeaseStart</span><span class="p">])</span> <span class="k">AS</span> <span class="n">datetimeoffset</span><span class="p">))</span>
</code></pre></div> </div>
<p>The only good things about the ValueConverter here is that it simply allows us to have the Entity Model field type as a <code class="language-plaintext highlighter-rouge">DateTime</code> but doesn’t actually do anything when querying. You can remove the <code class="language-plaintext highlighter-rouge">.HasConversion<string>()</code> notation from the model builder and the logic for querying will still work regardless.</p>
<p>Again this has the same draw back as option 2 even though it does work with In-Memory DB. If you read the value converters documentation page linked above it says the DateTime/String converter uses “Invariant Culture”. Which means it uses <code class="language-plaintext highlighter-rouge">MM/dd/yyyy</code> by <a href="https://stackoverflow.com/questions/46778141/datetime-formats-used-in-invariantculture">default</a>. Which might not be ideal for non us based data.</p>
<p>Just like option 2 it uses <code class="language-plaintext highlighter-rouge">CAST</code> and is <strong>susceptible to the column having dates in a format that is different to the session’s</strong> <a href="https://docs.microsoft.com/en-us/sql/t-sql/statements/set-language-transact-sql?view=sql-server-ver15">language setting</a>.</p>
<p>For example if you have data in that text column in the form of <code class="language-plaintext highlighter-rouge">dd/MM/yyyy</code> then <code class="language-plaintext highlighter-rouge">SET LANGUAGE "British English"</code> before you execute your SQL query which has the CAST to avoid the <code class="language-plaintext highlighter-rouge">"The conversion of a varchar data type to a datetime data type resulted in an out-of-range value"</code> error. The default language can be set to the SQL login if you don’t want to execute the SET LANGUAGE command each time.</p>
</li>
<li>
<p>Using Custom <a href="https://docs.microsoft.com/en-us/ef/core/querying/user-defined-function-mapping">SQL Translation</a>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">ModelBuilderExtensions</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="n">DateTime</span><span class="p">?</span> <span class="nf">ToDateTime</span><span class="p">(</span><span class="k">this</span> <span class="kt">string</span> <span class="n">dateString</span><span class="p">,</span> <span class="kt">int</span> <span class="n">format</span><span class="p">)</span> <span class="p">=></span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">NotSupportedException</span><span class="p">();</span>
<span class="k">public</span> <span class="k">static</span> <span class="n">ModelBuilder</span> <span class="nf">AddSqlConvertFunction</span><span class="p">(</span><span class="k">this</span> <span class="n">ModelBuilder</span> <span class="n">modelBuilder</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">modelBuilder</span><span class="p">.</span><span class="nf">HasDbFunction</span><span class="p">(()</span> <span class="p">=></span> <span class="nf">ToDateTime</span><span class="p">(</span><span class="k">default</span><span class="p">,</span> <span class="k">default</span><span class="p">))</span>
<span class="p">.</span><span class="nf">HasTranslation</span><span class="p">(</span><span class="n">args</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">SqlFunctionExpression</span><span class="p">(</span>
<span class="n">functionName</span><span class="p">:</span> <span class="s">"CONVERT"</span><span class="p">,</span>
<span class="n">arguments</span><span class="p">:</span> <span class="n">args</span><span class="p">.</span><span class="nf">Prepend</span><span class="p">(</span><span class="k">new</span> <span class="nf">SqlFragmentExpression</span><span class="p">(</span><span class="s">"date"</span><span class="p">)),</span>
<span class="n">nullable</span><span class="p">:</span> <span class="k">true</span><span class="p">,</span>
<span class="n">argumentsPropagateNullability</span><span class="p">:</span> <span class="k">new</span><span class="p">[]</span> <span class="p">{</span> <span class="k">false</span><span class="p">,</span> <span class="k">true</span><span class="p">,</span> <span class="k">false</span> <span class="p">},</span>
<span class="n">type</span><span class="p">:</span> <span class="k">typeof</span><span class="p">(</span><span class="n">DateTime</span><span class="p">),</span>
<span class="n">typeMapping</span><span class="p">:</span> <span class="k">null</span><span class="p">));</span>
<span class="k">return</span> <span class="n">modelBuilder</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// then on model creating</span>
<span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">OnModelCreating</span><span class="p">(</span><span class="n">ModelBuilder</span> <span class="n">modelBuilder</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">Database</span><span class="p">.</span><span class="nf">IsSqlServer</span><span class="p">()){</span>
<span class="n">modelBuilder</span><span class="p">.</span><span class="nf">AddSqlConvertFunction</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// entity model</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">CustomerLease</span>
<span class="p">{</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">LeaseStart</span> <span class="p">{</span><span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;}</span>
<span class="p">}</span>
<span class="c1">// To query</span>
<span class="kt">var</span> <span class="n">dateFormat</span> <span class="p">=</span> <span class="m">103</span><span class="p">;</span> <span class="c1">// See all date formats here https://www.w3schools.com/sql/func_sqlserver_convert.asp</span>
<span class="kt">var</span> <span class="n">query</span> <span class="p">=</span> <span class="n">db</span><span class="p">.</span><span class="n">Set</span><span class="p"><</span><span class="n">CustomerLease</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">Where</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="n">c</span><span class="p">.</span><span class="n">LeaseStart</span><span class="p">.</span><span class="nf">ToDateTime</span><span class="p">(</span><span class="n">dateFormat</span><span class="p">)</span> <span class="p">>=</span> <span class="n">searchStart</span><span class="p">);</span>
</code></pre></div> </div>
<p>This will result in a SQL query like below..</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="p">((</span><span class="o">@</span><span class="n">__startSearch__</span> <span class="o"><=</span> <span class="k">CONVERT</span><span class="p">(</span><span class="nb">date</span><span class="p">,</span> <span class="p">[</span><span class="n">s</span><span class="p">].[</span><span class="n">LeaseStart</span><span class="p">],</span> <span class="mi">103</span><span class="p">);)</span>
</code></pre></div> </div>
<p>This is a much more precise solution as we explicitly define the date format we want for the conversion. One of the drawbacks with this approach for me was that I couldn’t get this to work with In-Memory DB provider which I used for unit/integration tests. Your mileage may vary.</p>
</li>
<li>
<p>Use the <code class="language-plaintext highlighter-rouge">EF.Functions.DateFromParts(year, month, day)</code> function.</p>
<p>Here you write the query using <code class="language-plaintext highlighter-rouge">EF.Functions.DateFromParts</code> function and pass the year, month and day in. This means you need to use <code class="language-plaintext highlighter-rouge">LeaseStart.substring(x,x)</code> to split extract each part and construct a proper date. I won’t write an example query here as the date formats will determine the substring start/end for each component.</p>
<p>The drawback from this approach is again that <code class="language-plaintext highlighter-rouge">EF.Functions.DateFromParts</code> has no translation in In-Memory DB.</p>
</li>
<li>
<p>Use the correct data type in SQL Server.</p>
<p>Simple isn’t it? You just add a new column and map the current column with a CAST and populate the new one. For scenarios where you can’t, maybe you create a new view with the desired data types. Yes it has performance implications but it is another option to consider nevertheless.</p>
</li>
</ol>
<h2 id="conclusion">Conclusion</h2>
<p>We learned that our data access layer tooling and abstractions come with trade offs. We also learnt that converting a string column type to date within a LINQ query is not trivial when it comes to EF Core SQL Provider.</p>
<p>Hopefully this gives you some options to try. While I can’t emphasise enough how important it is to have your underlying database column types represented in the correct data type sometimes we don’t have the option to change that. Not immediately anyway.</p>
<p>So I went back to the DBA and convinced them to change the underlying data type to reflect the correct type. This meant my entity model and LINQ query are much simpler and make sense in the domain.</p>
<p>Please let me know what you thought about this post and if you have other/better techniques to deal with this problem. Thanks for reading and have a great day.</p>
<h3 id="references">References</h3>
<ul>
<li>https://stackoverflow.com/questions/68728498/convert-string-to-datetime-in-linq-query-with-entity-framework-core</li>
<li>https://stackoverflow.com/questions/60969027/how-to-convert-string-to-datetime-in-c-sharp-ef-core-query</li>
<li>https://stackoverflow.com/questions/20838344/sql-the-conversion-of-a-varchar-data-type-to-a-datetime-data-type-resulted-in/40106812#40106812</li>
<li>https://docs.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql?view=sql-server-ver15</li>
<li>https://docs.microsoft.com/en-us/ef/core/providers/sql-server/functions</li>
<li>https://docs.microsoft.com/en-us/ef/core/modeling/value-conversions</li>
<li>https://docs.microsoft.com/en-us/sql/t-sql/statements/set-language-transact-sql?view=sql-server-ver15</li>
</ul>Dasith WijesiriwardenaI am working on a greenfield project that uses EF Core 6 with AspNetCore 6 at the moment. The project involves exposing a set of legacy data through an API. Simple enough right? The underlying data is stored in SQL Server 2019 but it is not very well designed. There are varchar columns for storing boolean, numeric and date/time values. It’s not uncommon to see these types of data stores though. As developers we have to deal with them often. Dapper or EF Core When choosing the data access layer for the project I had the option to go with Dapper or EF Core. I choose to go with EF Core because this specific API had a lot of requirements around paging and sorting (See here for more). You can easily implement paging and sorting with Dapper too. But I find constructing paging and sorting dynamically using EF Core IQueryable more appealing than manipulating strings in Dapper. I will do another post about dynamic paging and sorting using EF Core soon. But this choice comes with trade offs as with any technical decision. While I don’t have to “construct” SQL with string manipulation, an ORM comes at a cost of not being able to execute the exact SQL I want if I’m using IQueryable to construct my LINQ query. This is a hot topic when it comes to designing your data access layer but that is a topic for another post. The Problem Imagine the following schema for a table called CustomerLease. Column Data Type LeaseId int CustomerId int LeasedItem nvarchar(2000) NULL LeaseStart nvarchar(10) LeaseEnd nvarchar(10) NULL We are required to find customer leases that started after a given date. Now lets assume what we would do if the LeaseStart was DateTime .NET Type in my EF Core entity model for CustomerLease. public class CustomerLease { //... other fields DateTime LeaseStart {get; set;} } public class MyRepo { // constructor and other properties will go here... // example method to search within date periods public async Task<List<CustomerLease>> GetCustomerLeases(SearchRequest request) { var searchFrom = request.SearchFrom; var query = MyDataContext.CustomerLeases .Where(c => searchFrom <= c.LeaseStart); return await query.ToListAsync(); } } This solution would work if my underlying DB type was DateTime BUT it is not. So my actual entity model looks like… public class CustomerLease { //... other fields string LeaseStart {get; set;} } Now I can’t write my LINQ query with direct comparison to SearchFrom. What are my alternatives? Try converting the string to a DateTime within the LINQ query. DateTime.Parse(...) // or Convert.ToDateTime(...) This will work if our underlying IQueryable provider for SQL Server supported translating these functions to SQL. But unfortunately they aren’t. So this approach is out of the question. Using implicit conversion . .Where(c => searchFrom <= (DateTime)(object)c.LeaseStart This technique generates the following SQL. “CAST([S].[LeaseStart] as DateTime) >= @__searchFrom__” This will work but word of caution. This double casting we have done in LINQ to trick the underlying provider to use CAST will only work for SQL Provider. It will not work for the In-Memory database provider if you’re using it for writing unit/integration tests. The other drawback here is that it expects the dates to be in the default format of the current session language. (i.e. US English, British English etc). If you have a date there like 24/05/2021 and the the current language is US English then it will fail with a message like "The conversion of a varchar data type to a datetime data type resulted in an out-of-range value". I talk about this again below in option 3 and touch on some work arounds. Using EF Core value converter. With EF Core 5+ you can use Value Converters for this scenario and there are built in ones for some common use cases. Be mindful that ValueConverters work inside .NET and not SQL. So how do we get it to do a CAST on our varchar column? protected override void OnModelCreating(ModelBuilder modelBuilder) { // The column TextDate is the one that has date values but stored as text in the db modelBuilder .Entity<CustomerLease>() .Property(c => c.LeaseStart) .HasConversion<string>(); } public class CustomerLease { //... other fields DateTime LeaseStart {get; set;} } Then in LINQ simply do .Where(e => e.LeaseStart >= startSearch). Here is the kicker. For EF Core to generate the correct SQL statement, it will require startSearch parameter inside the LINQ query to be of type DateTimeOffset. It doesn’t use CAST if the parameter is DateTime as it simply converts your parameter to varchar and then compares. I made this gist to demo the behaviour. This is more of a hack as we are relying on implicit conversion of DateTime from/to DateTimeOffset inside .NET and then letting the EFCORE SQL Provider do a CAST when comparing inside SQL. The above LINQ will generate SQL like… DECLARE @__startSearch_0 datetimeoffset = '2022-01-22T23:01:43.0090270+11:00'; # and query like WHERE ((@__startSearch_0 <= CAST([s].[LeaseStart]) AS datetimeoffset)) The only good things about the ValueConverter here is that it simply allows us to have the Entity Model field type as a DateTime but doesn’t actually do anything when querying. You can remove the .HasConversion<string>() notation from the model builder and the logic for querying will still work regardless. Again this has the same draw back as option 2 even though it does work with In-Memory DB. If you read the value converters documentation page linked above it says the DateTime/String converter uses “Invariant Culture”. Which means it uses MM/dd/yyyy by default. Which might not be ideal for non us based data. Just like option 2 it uses CAST and is susceptible to the column having dates in a format that is different to the session’s language setting. For example if you have data in that text column in the form of dd/MM/yyyy then SET LANGUAGE "British English" before you execute your SQL query which has the CAST to avoid the "The conversion of a varchar data type to a datetime data type resulted in an out-of-range value" error. The default language can be set to the SQL login if you don’t want to execute the SET LANGUAGE command each time. Using Custom SQL Translation. public static class ModelBuilderExtensions { public static DateTime? ToDateTime(this string dateString, int format) => throw new NotSupportedException(); public static ModelBuilder AddSqlConvertFunction(this ModelBuilder modelBuilder) { modelBuilder.HasDbFunction(() => ToDateTime(default, default)) .HasTranslation(args => new SqlFunctionExpression( functionName: "CONVERT", arguments: args.Prepend(new SqlFragmentExpression("date")), nullable: true, argumentsPropagateNullability: new[] { false, true, false }, type: typeof(DateTime), typeMapping: null)); return modelBuilder; } } // then on model creating protected override void OnModelCreating(ModelBuilder modelBuilder) { if (Database.IsSqlServer()){ modelBuilder.AddSqlConvertFunction(); } } // entity model public class CustomerLease { public string LeaseStart {get; set;} } // To query var dateFormat = 103; // See all date formats here https://www.w3schools.com/sql/func_sqlserver_convert.asp var query = db.Set<CustomerLease>() .Where(c => c.LeaseStart.ToDateTime(dateFormat) >= searchStart); This will result in a SQL query like below.. ((@__startSearch__ <= CONVERT(date, [s].[LeaseStart], 103);) This is a much more precise solution as we explicitly define the date format we want for the conversion. One of the drawbacks with this approach for me was that I couldn’t get this to work with In-Memory DB provider which I used for unit/integration tests. Your mileage may vary. Use the EF.Functions.DateFromParts(year, month, day) function. Here you write the query using EF.Functions.DateFromParts function and pass the year, month and day in. This means you need to use LeaseStart.substring(x,x) to split extract each part and construct a proper date. I won’t write an example query here as the date formats will determine the substring start/end for each component. The drawback from this approach is again that EF.Functions.DateFromParts has no translation in In-Memory DB. Use the correct data type in SQL Server. Simple isn’t it? You just add a new column and map the current column with a CAST and populate the new one. For scenarios where you can’t, maybe you create a new view with the desired data types. Yes it has performance implications but it is another option to consider nevertheless. Conclusion We learned that our data access layer tooling and abstractions come with trade offs. We also learnt that converting a string column type to date within a LINQ query is not trivial when it comes to EF Core SQL Provider. Hopefully this gives you some options to try. While I can’t emphasise enough how important it is to have your underlying database column types represented in the correct data type sometimes we don’t have the option to change that. Not immediately anyway. So I went back to the DBA and convinced them to change the underlying data type to reflect the correct type. This meant my entity model and LINQ query are much simpler and make sense in the domain. Please let me know what you thought about this post and if you have other/better techniques to deal with this problem. Thanks for reading and have a great day. References https://stackoverflow.com/questions/68728498/convert-string-to-datetime-in-linq-query-with-entity-framework-core https://stackoverflow.com/questions/60969027/how-to-convert-string-to-datetime-in-c-sharp-ef-core-query https://stackoverflow.com/questions/20838344/sql-the-conversion-of-a-varchar-data-type-to-a-datetime-data-type-resulted-in/40106812#40106812 https://docs.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql?view=sql-server-ver15 https://docs.microsoft.com/en-us/ef/core/providers/sql-server/functions https://docs.microsoft.com/en-us/ef/core/modeling/value-conversions https://docs.microsoft.com/en-us/sql/t-sql/statements/set-language-transact-sql?view=sql-server-ver15Propagating context and tracing across your distributed process boundaries using OpenTelemetry - API Days Australia 20212022-01-23T22:06:00+11:002022-01-23T22:06:00+11:00https://dasith.me/2022/01/23/open-telemetry-apidays-australia-2021<p>I spoke at <a href="https://www.apidays.global/australia/">API Days Australia</a> about my experiences building distributed systems and some challenges I’ve faced.</p>
<p>We are amidst the 2nd wave of cloud migrations. This means it’s no longer enough just to have a presence on the web if you need a competitive advantage. You need to be able to thrive.</p>
<p>We are building more and more cloud native solutions with an emphasis on distributed systems more than any other time in the past. With cloud native distributed systems now the norm, tracing and tracking telemetry becomes a more pronounced problem for operations teams. What makes good teams stand out from the rest is how they tackle this “observability” aspect in my opinion.</p>
<p>This is the landscape where OpenTelemetry was born in. Telemetry data is needed to power observability products and traditionally, telemetry data has been provided by either open-source projects or commercial vendors but key the problem is lack of standardization.</p>
<p>OpenTelemetry was formed through a merger of the OpenTracing and OpenCensus projects which had similar motivations. The OpenTelemetry project solves these problems by providing a single, vendor-agnostic solution. The project has gained broad industry support and adoption from cloud providers, vendors and end users alike.</p>
<p>In this talk we will cover some of the modern challenges and some modern solutions to distributed tracing and see how all the paths lead to OpenTelemetry.</p>
<h2 id="about-api-days">About API Days</h2>
<p>This is the fourth time I spoke at API Days and I’ve made a little niche talking about modern approaches to develop distributed system and the kind of challenges they pose. It was great to be back and talking about distributed tracing this time around.</p>
<p>On the API Days website it says…</p>
<blockquote>
<p>It’s almost a cliché to say that the global pandemic has had profound effects on the way we do business and go about our lives. In Australia, organisations large and small have been forced to adapt to new business models and new channels as a way to survive and to provide business continuity. Others who already provided digital services were forced to expand their range and capacity to deal with higher levels of demand than previously imagined. This is the great digital acceleration of 2020-21. The digital genie is well and truly out of the bottle and in many cases we don’t want it to go back. Digital services, digital supply chains, digital ways of working – are all here to stay. <br /><br />At apidays Australia, we know this from direct experience. Last year, we too were forced to reimagine our conference as a digital experience that required us to rapidly develop new platforms and new ways to engage with our audience. This year, we’re back again and still digital. Join us in September to hear stories of new technologies and new ways of doing business from your peers – both local and international.</p>
</blockquote>
<p>The theme this year was “Accelerating Digital” and there were multiple tracks. My session was on the “Platform” stream along with some other technical presentations from various industry experts. There were also workshops and roundtable discussions as well. You can watch full <strong><a href="https://www.youtube.com/playlist?list=PLmEaqnTJ40OqWntvB5HacxMMoZSRPw58g">replays of talks here</a>.</strong></p>
<p><img src="/assets/images/apidays/apidays-australia-2021-lineup.jpg" alt="Speaker List" /></p>
<h2 id="propagating-context-and-tracing-across-your-distributed-process-boundaries-using-opentelemetry">Propagating context and tracing across your distributed process boundaries using OpenTelemetry</h2>
<p>The abstract is as follows.</p>
<blockquote>
<p>Everyone is building distributed systems these days. Some better than others. One thing the teams building and running distributed systems well have in common is they have very good observability of the components and services. Conversely, the teams that don’t have good observability struggle when things go wrong in a distributed system because it’s often terribly time consuming to put the pieces together to analyse the crime scene. The logs might sit in disparate log aggregation systems and even when in one place, leave you with having to do the hard work to correlate and visualize the system workflows yourself. <br /><br /> OpenTelemetry is an observability framework for cloud-native software which aims to solve some of these issues by having a common set of definitions of concepts around observability and exposing them to the tool of your choice. <br /><br /> In this talk, I examine how to propagate your tracing context across process boundaries and visualize the flow of requests through your distributed services (Microservices/Serverless/Other) easily using tools like Zipkin and Jaeger. We will see how to use already instrumented libraries and also how to propagate the trace information yourself. At the end of this talk you will know how to easily trace and observe distributed components of the systems you build.</p>
</blockquote>
<h2 id="recording--slide-deck">Recording & Slide deck</h2>
<iframe width="560" height="315" src="https://www.youtube.com/embed/5A3NIveTqOQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p><br /></p>
<iframe class="speakerdeck-iframe" frameborder="0" src="https://speakerdeck.com/player/f1da42d624cb4fb5afc7ea9beb6ce52a" title="Propagating context and tracing across your distributed process boundaries using OpenTelemetry" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true" style="border: 0px; background: padding-box padding-box rgba(0, 0, 0, 0.1); margin: 0px; padding: 0px; border-radius: 6px; box-shadow: rgba(0, 0, 0, 0.2) 0px 5px 40px; width: 560px; height: 314px;" data-ratio="1.78343949044586"></iframe>
<p><br /></p>
<p>I gave an extended version of the talk at Melbourne .NET user group meetup as well. The recording can be found below. <br /></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/nN9YSbnQXpY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p><br /><br /></p>
<p>If you have any thoughts or comments please leave them here. Thanks for taking the time to read this post.</p>Dasith WijesiriwardenaI spoke at API Days Australia about my experiences building distributed systems and some challenges I’ve faced. We are amidst the 2nd wave of cloud migrations. This means it’s no longer enough just to have a presence on the web if you need a competitive advantage. You need to be able to thrive. We are building more and more cloud native solutions with an emphasis on distributed systems more than any other time in the past. With cloud native distributed systems now the norm, tracing and tracking telemetry becomes a more pronounced problem for operations teams. What makes good teams stand out from the rest is how they tackle this “observability” aspect in my opinion. This is the landscape where OpenTelemetry was born in. Telemetry data is needed to power observability products and traditionally, telemetry data has been provided by either open-source projects or commercial vendors but key the problem is lack of standardization. OpenTelemetry was formed through a merger of the OpenTracing and OpenCensus projects which had similar motivations. The OpenTelemetry project solves these problems by providing a single, vendor-agnostic solution. The project has gained broad industry support and adoption from cloud providers, vendors and end users alike. In this talk we will cover some of the modern challenges and some modern solutions to distributed tracing and see how all the paths lead to OpenTelemetry. About API Days This is the fourth time I spoke at API Days and I’ve made a little niche talking about modern approaches to develop distributed system and the kind of challenges they pose. It was great to be back and talking about distributed tracing this time around. On the API Days website it says… It’s almost a cliché to say that the global pandemic has had profound effects on the way we do business and go about our lives. In Australia, organisations large and small have been forced to adapt to new business models and new channels as a way to survive and to provide business continuity. Others who already provided digital services were forced to expand their range and capacity to deal with higher levels of demand than previously imagined. This is the great digital acceleration of 2020-21. The digital genie is well and truly out of the bottle and in many cases we don’t want it to go back. Digital services, digital supply chains, digital ways of working – are all here to stay. At apidays Australia, we know this from direct experience. Last year, we too were forced to reimagine our conference as a digital experience that required us to rapidly develop new platforms and new ways to engage with our audience. This year, we’re back again and still digital. Join us in September to hear stories of new technologies and new ways of doing business from your peers – both local and international. The theme this year was “Accelerating Digital” and there were multiple tracks. My session was on the “Platform” stream along with some other technical presentations from various industry experts. There were also workshops and roundtable discussions as well. You can watch full replays of talks here. Propagating context and tracing across your distributed process boundaries using OpenTelemetry The abstract is as follows. Everyone is building distributed systems these days. Some better than others. One thing the teams building and running distributed systems well have in common is they have very good observability of the components and services. Conversely, the teams that don’t have good observability struggle when things go wrong in a distributed system because it’s often terribly time consuming to put the pieces together to analyse the crime scene. The logs might sit in disparate log aggregation systems and even when in one place, leave you with having to do the hard work to correlate and visualize the system workflows yourself. OpenTelemetry is an observability framework for cloud-native software which aims to solve some of these issues by having a common set of definitions of concepts around observability and exposing them to the tool of your choice. In this talk, I examine how to propagate your tracing context across process boundaries and visualize the flow of requests through your distributed services (Microservices/Serverless/Other) easily using tools like Zipkin and Jaeger. We will see how to use already instrumented libraries and also how to propagate the trace information yourself. At the end of this talk you will know how to easily trace and observe distributed components of the systems you build. Recording & Slide deck I gave an extended version of the talk at Melbourne .NET user group meetup as well. The recording can be found below. If you have any thoughts or comments please leave them here. Thanks for taking the time to read this post.The Shell Game Called Eventual Consistency - API Days Jakarta 20212021-03-09T22:06:00+11:002021-03-09T22:06:00+11:00https://dasith.me/2021/03/09/eventual-consistency-apidays-jakarta-2021<p>A few weeks ago I spoke at <a href="https://www.apidays.global/jakarta/">API Days Jakarta</a> about some of experiences building distributed systems.</p>
<p>As more and more companies take their businesses to the web, they are finding that their customers are demanding highly responsive and highly available systems. So developers are expected to build those responsive distributed systems more than anytime in the past. This means that in certain situations you as developers have to let go of strong consistency or distributed transactions. Even in other cases more and more software systems are embracing asynchronous workflows or messaging systems. In all of these scenarios your front end needs to deal with eventual consistent backend nodes/partitions.</p>
<p>The aim of the talk was to give the audience a cooks tour around the concept of eventual consistency and a few creative ways to deal with it.</p>
<p>I covered the following topics briefly…</p>
<ul>
<li>A primer to CAP theorem.</li>
<li>Comparing CP and CP systems and their strength/weaknesses.</li>
<li>Why you should embrace asynchronous business workflows.</li>
<li>How to deal with eventual consistency.</li>
</ul>
<p>If you or your development team are venturing into building distributed systems or messaging based architectures, this might give you some topics to research about. I recommend you read about CAP theorem and what you gain by going with a CP/AP system. This talk will be a good starting point.</p>
<h2 id="about-api-days">About API Days</h2>
<p>This is the third time I spoke at API Days but the first time doing so internationally. It was an awesome experience to participate and talk at the conference.</p>
<p>On the website it says</p>
<blockquote>
<p>The Covid-19 pandemic has pushed Indonesian companies to accelerate their adoption of digital tools and business models. Across retail, healthcare, financial services and logistics, connectivity enables companies to continue to serve customers. The enthusiasm of consumers and merchants for marketplaces and digital payments is building a new normal for e-commerce. Apidays is the leading industry tech and business series of conferences in APIs and the programmable economy.</p>
</blockquote>
<p>The theme this year was “Accelerating Digitization” and there were multiple tracks. There were also workshops and roundtable discussions as well. You can watch full <strong><a href="https://www.youtube.com/playlist?list=PLmEaqnTJ40Or4D_y4OtPPxb6zVINSBweS">replays of talks here</a>.</strong></p>
<p><img src="/assets/images/apidays/apidays-jakarta-lineup.webp" alt="Speaker List" /></p>
<h2 id="the-shell-game-called-eventual-consistency">The Shell Game Called Eventual Consistency</h2>
<p>The abstract is as follows.</p>
<blockquote>
<p>As we build distributed highly scalable systems the central data store and transactions are no longer a safety net we can afford. In the world of event sourcing and CQRS (Command Query Responsibility Segregation) we need to design clever systems that don’t show cracks and seams where eventual consistency is at play. We will tackle those unpleasant invariants and race conditions head on to investigate some technical and non technical smoke and mirror solutions that we can use to deliver a positive experience to end-users while finding the performance sweet spot. <br /><br />We are utilizing the various PaaS/Serverless solutions to build more and more distributed systems. Often these systems need to work together to produce a result. When performance and scalability is of high priority, consistency (CAP theorem) takes a back seat. We still need to find ways to shelter the end-user from these design realities. The aim of this talk is to find ways of doing it. Be it through changing the business process or by doing clever tricks on the front end while giving the backend has a heartbeat to catch up. There are countless ways to do it. My goal is to investigate a few of them and get the conversations happening.</p>
</blockquote>
<h2 id="recording--slide-deck">Recording & Slide deck</h2>
<iframe width="560" height="315" src="https://www.youtube.com/embed/uNVQxuGOLw8" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p><br /></p>
<script async="" class="speakerdeck-embed" data-id="2cc7089f971e4d348ef014fa56bf6db0" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<p>If you have any thoughts or comments please leave them here. If you’ve got an interesting way to deal with eventual consistency on the UI, I would love to hear about it too.</p>Dasith WijesiriwardenaA few weeks ago I spoke at API Days Jakarta about some of experiences building distributed systems. As more and more companies take their businesses to the web, they are finding that their customers are demanding highly responsive and highly available systems. So developers are expected to build those responsive distributed systems more than anytime in the past. This means that in certain situations you as developers have to let go of strong consistency or distributed transactions. Even in other cases more and more software systems are embracing asynchronous workflows or messaging systems. In all of these scenarios your front end needs to deal with eventual consistent backend nodes/partitions. The aim of the talk was to give the audience a cooks tour around the concept of eventual consistency and a few creative ways to deal with it. I covered the following topics briefly… A primer to CAP theorem. Comparing CP and CP systems and their strength/weaknesses. Why you should embrace asynchronous business workflows. How to deal with eventual consistency. If you or your development team are venturing into building distributed systems or messaging based architectures, this might give you some topics to research about. I recommend you read about CAP theorem and what you gain by going with a CP/AP system. This talk will be a good starting point. About API Days This is the third time I spoke at API Days but the first time doing so internationally. It was an awesome experience to participate and talk at the conference. On the website it says The Covid-19 pandemic has pushed Indonesian companies to accelerate their adoption of digital tools and business models. Across retail, healthcare, financial services and logistics, connectivity enables companies to continue to serve customers. The enthusiasm of consumers and merchants for marketplaces and digital payments is building a new normal for e-commerce. Apidays is the leading industry tech and business series of conferences in APIs and the programmable economy. The theme this year was “Accelerating Digitization” and there were multiple tracks. There were also workshops and roundtable discussions as well. You can watch full replays of talks here. The Shell Game Called Eventual Consistency The abstract is as follows. As we build distributed highly scalable systems the central data store and transactions are no longer a safety net we can afford. In the world of event sourcing and CQRS (Command Query Responsibility Segregation) we need to design clever systems that don’t show cracks and seams where eventual consistency is at play. We will tackle those unpleasant invariants and race conditions head on to investigate some technical and non technical smoke and mirror solutions that we can use to deliver a positive experience to end-users while finding the performance sweet spot. We are utilizing the various PaaS/Serverless solutions to build more and more distributed systems. Often these systems need to work together to produce a result. When performance and scalability is of high priority, consistency (CAP theorem) takes a back seat. We still need to find ways to shelter the end-user from these design realities. The aim of this talk is to find ways of doing it. Be it through changing the business process or by doing clever tricks on the front end while giving the backend has a heartbeat to catch up. There are countless ways to do it. My goal is to investigate a few of them and get the conversations happening. Recording & Slide deck If you have any thoughts or comments please leave them here. If you’ve got an interesting way to deal with eventual consistency on the UI, I would love to hear about it too.Object Capability Model And Mapping2021-03-09T22:06:00+11:002021-03-09T22:06:00+11:00https://dasith.me/2021/03/09/object-capability-model-and-mapping%20copy<p>Almost any type of software development work requires some sort mapping in the logic. Either from the persistence models to domain model or from the domain models to view models. In my experience dealing with a lot of brownfield projects, I’ve seen some practices that age poorly and make it difficult to understand intent. Most mapping approaches age gracefully like beer. Although my aim is not to kick a dead horse and <a href="https://cezarypiatek.github.io/post/why-i-dont-use-automapper/">rant about AutoMapper</a>, I agree with the sentiments provided in that linked post. I recommend you read it if you haven’t already. <strong>Pay attention to the comment thread</strong> there where Jimmy Bogard gives some decent pointers on when and when not to use it.</p>
<p><em>I haven’t been blogging as often as I would like to. I’ve recently started a new job and it’s been keeping me quite busy. I’m taking the latest snap lockdown in lieu of Covid19 in Victoria, Australia to write this. I wanted to do this before I reward myself with binging Netflix. This is a topic I’ve wanted to express my opinion on for a while.</em></p>
<p>In this post I’ll be expanding on some of the arguments made in that linked post.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tl;dr; I'm complaining about how your mapping choices affect maintainability, understandability and code navigation.
</code></pre></div></div>
<h2 id="object-capability-model">Object Capability Model</h2>
<p>Before we dive in, I want to introduce the <a href="https://en.wikipedia.org/wiki/Object-capability_model">Object Capability Model</a> to those of you who haven’t heard of it. (Most software developers understand it intuitively).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The object-capability model is a computer security model. A capability describes a transferable right to perform one (or more) operations on a given object. It can be obtained by the following combination:
- An unforgeable reference (in the sense of object references or protected pointers) that can be sent in messages.
- A message that specifies the operation to be performed.
</code></pre></div></div>
<p>To put it simply it’s a model which describes the capabilities of an object. The security implication is that, we should aim to give the <strong>least amount of privileges required to an object to able to perform its responsibilities</strong>.</p>
<p>In modern software development we give capabilities to an object through the references it holds. If you’re following inversion of control pattern in your design, this is often done through dependency injection via the constructor.</p>
<p>When I’m working on a brownfield project or a large project and trying to understand and navigate the code, I often look at the constructor of a class to understand the capabilities of it. This allows me to understand the what other area of the system can potentially impact this class as well. Now I know there are better ways to to do this i.e. Dependency Graph but it’s not something you can quickly access or visualise.</p>
<p>So what does this mean for mapping?</p>
<h2 id="the-problem-with-automapper">The Problem With AutoMapper</h2>
<p>The shortcomings of (or ways developers abuse) AutoMapper are well documented. I won’t repeat them again here. Please read the article linked above. My take is more targeted around how it affects the understandability of object capability.</p>
<p>Let’s take this class as an example:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IMapper</span> <span class="n">_mapper</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="nf">Foo</span><span class="p">(</span><span class="n">IMapper</span> <span class="n">mapper</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_mapper</span> <span class="p">=</span> <span class="n">mapper</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If we look at the constructor here, we know the class uses <em>some</em> mapping but it’s not clear what. I have to go through the code to understand the <code class="language-plaintext highlighter-rouge">from</code> and <code class="language-plaintext highlighter-rouge">to</code> models or indeed if there are multiples of them. Yes it’s not the end of the world. A software developer needing to read the code to understand it, the horror.</p>
<p>But this cognitive load adds up. Especially when you consider the fact that mapping is something done every time an object needs to be presented to/from a layer. (Lets rant about enterprise software development at a later time shall we?). This also poses a hurdle in easily understanding the impact this class has. It is very hard to navigate to the mapping logic and therefore <strong>hard to understand the capabilities this class will ultimately have</strong>. This impacts us in giving the class the least amount of privileges required to perform its actions.</p>
<h2 id="so-imappertfromtto-then">So, <code class="language-plaintext highlighter-rouge">IMapper<TFrom,TTo></code> then?</h2>
<p>This is another alternative to AutoMapper. Here we define an interface <code class="language-plaintext highlighter-rouge">IMapper<TFrom,TTo></code> like</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">interface</span> <span class="nc">IMapper</span><span class="p"><</span><span class="n">DomainModel</span><span class="p">,</span> <span class="n">ViewModel</span><span class="p">></span> <span class="p">{</span>
<span class="k">public</span> <span class="n">TTo</span> <span class="nf">Map</span><span class="p">(</span><span class="n">TFrom</span> <span class="n">model</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>and implement it</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">CustomerViewModelMapper</span> <span class="p">:</span> <span class="n">IMapper</span><span class="p"><</span><span class="n">CustomerDomainModel</span><span class="p">,</span> <span class="n">CustomerViewModel</span><span class="p">></span> <span class="p">{</span>
<span class="k">public</span> <span class="n">CustomerViewModel</span> <span class="nf">Map</span><span class="p">(</span><span class="n">TFrom</span> <span class="n">CustomerDomainModel</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// mapping logic here</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Using it is pretty easy</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IMapper</span><span class="p"><</span><span class="n">CustomerDomainModel</span><span class="p">,</span> <span class="n">CustomerViewModel</span><span class="p">></span> <span class="n">_mapper</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="nf">Foo</span><span class="p">(</span><span class="n">IMapper</span><span class="p"><</span><span class="n">CustomerDomainModel</span><span class="p">,</span> <span class="n">CustomerViewModel</span><span class="p">></span> <span class="n">mapper</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_mapper</span> <span class="p">=</span> <span class="n">mapper</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Something I like about this approach is whenever you see <code class="language-plaintext highlighter-rouge">IMapper<TFrom, TTo></code> <strong>you instantly know what capability it provides</strong> to the dependant class. You don’t have to rely on the name of the dependency. (We know how bad we are at naming things).</p>
<p>One of the other aspects that is appealing with this approach is that you can <strong>automatically scan your assemblies for implementations of <code class="language-plaintext highlighter-rouge">IMapper<TFrom,TTo></code> and register it in your dependency injection</strong> . This means you have one less thing to worry about.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">MapperModule</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">AddMappers</span><span class="p">(</span><span class="k">this</span> <span class="n">IServiceCollection</span> <span class="n">services</span><span class="p">,</span> <span class="n">IConfiguration</span> <span class="n">configuration</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">serviceType</span> <span class="p">=</span> <span class="k">typeof</span><span class="p">(</span><span class="n">IMapper</span><span class="p"><,>);</span>
<span class="nf">ScanServiceType</span><span class="p">(</span><span class="n">services</span><span class="p">,</span> <span class="n">serviceType</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">ScanServiceType</span><span class="p">(</span><span class="n">IServiceCollection</span> <span class="n">services</span><span class="p">,</span> <span class="n">Type</span> <span class="n">serviceType</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">mapperTypes</span> <span class="p">=</span> <span class="n">DependentAssemblies</span><span class="p">.</span><span class="n">All</span>
<span class="p">.</span><span class="nf">SelectMany</span><span class="p">(</span><span class="n">a</span> <span class="p">=></span> <span class="n">a</span><span class="p">.</span><span class="nf">GetTypes</span><span class="p">())</span>
<span class="p">.</span><span class="nf">Where</span><span class="p">(</span><span class="n">type</span> <span class="p">=></span> <span class="n">type</span><span class="p">.</span><span class="nf">GetInterfaces</span><span class="p">()</span> <span class="p">!=</span> <span class="k">null</span> <span class="p">&&</span>
<span class="n">type</span><span class="p">.</span><span class="nf">GetInterfaces</span><span class="p">().</span><span class="nf">Any</span><span class="p">(</span>
<span class="n">t</span> <span class="p">=></span> <span class="n">t</span><span class="p">.</span><span class="n">IsGenericType</span> <span class="p">&&</span>
<span class="n">t</span><span class="p">.</span><span class="nf">GetGenericTypeDefinition</span><span class="p">()</span> <span class="p">==</span> <span class="n">serviceType</span><span class="p">)</span> <span class="p">&&</span>
<span class="n">type</span><span class="p">.</span><span class="n">IsClass</span> <span class="p">&&</span>
<span class="p">!</span><span class="n">type</span><span class="p">.</span><span class="n">IsAbstract</span><span class="p">);</span>
<span class="k">foreach</span> <span class="p">(</span><span class="kt">var</span> <span class="n">mapperType</span> <span class="k">in</span> <span class="n">mapperTypes</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">@interfaces</span> <span class="p">=</span> <span class="n">mapperType</span><span class="p">.</span><span class="nf">GetInterfaces</span><span class="p">().</span><span class="nf">Where</span><span class="p">(</span><span class="n">t</span> <span class="p">=></span> <span class="n">t</span><span class="p">.</span><span class="n">IsGenericType</span> <span class="p">&&</span>
<span class="n">t</span><span class="p">.</span><span class="nf">GetGenericTypeDefinition</span><span class="p">()</span> <span class="p">==</span> <span class="n">serviceType</span><span class="p">);</span>
<span class="k">foreach</span> <span class="p">(</span><span class="kt">var</span> <span class="n">@interface</span> <span class="k">in</span> <span class="n">@interfaces</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">services</span><span class="p">.</span><span class="nf">AddScoped</span><span class="p">(</span><span class="n">@interface</span><span class="p">,</span> <span class="n">mapperType</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If you’ve been paying attention then you might have noticed something that is making me look like a hypocrite. We wanted to make code navigation easier, but now we have a dependency on <code class="language-plaintext highlighter-rouge">IMapper<CustomerDomainModel, CustomerViewModel></code> and it’s hard to navigate to the actual implementation because Visual Studio can’t detect implementations of generic types without the help of extensions/plugins. Sure it’s better from a “understanding object capability” perspective but unless you have Resharper installed we haven’t really made code navigation any easier. Also me being absent minded, have taken a dependency on <code class="language-plaintext highlighter-rouge">IMapper<TFrom, TTo></code> in a class only to realise at runtime that there is no implementation for it.</p>
<p>One could even argue that DI auto register magic is bad. So where does that leave us?</p>
<h2 id="simple-classes-that-only-map-who-would-have-thought">Simple classes that only map. Who would have thought?</h2>
<p>These days this is my go to option.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">CustomerViewModelMapper</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">AddressServiceMapper</span> <span class="n">addressServiceMapper</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">CustomerViewModelMapper</span><span class="p">(</span><span class="n">AddressServiceMapper</span> <span class="n">addressServiceMapper</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">addressServiceMapper</span> <span class="p">=</span> <span class="n">addressServiceMapper</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">CustomerViewModel</span> <span class="nf">Map</span><span class="p">(</span><span class="n">CustomerDomainModel</span> <span class="n">model</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">CustomerViewModel</span> <span class="p">{</span>
<span class="n">BusinessName</span> <span class="p">=</span> <span class="n">model</span><span class="p">.</span><span class="n">BusinessName</span><span class="p">,</span>
<span class="n">Address</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">addressServiceMapper</span><span class="p">.</span><span class="nf">Map</span><span class="p">(</span><span class="n">entity</span><span class="p">.</span><span class="n">Address</span><span class="p">)</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Using it is straight forward again.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">CustomerViewModelMapper</span> <span class="n">_mapper</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="nf">Foo</span><span class="p">(</span><span class="n">CustomerViewModelMapper</span> <span class="n">mapper</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_mapper</span> <span class="p">=</span> <span class="n">mapper</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Notice how I didn’t bother with an interface here. The truth is you don’t need one. I can’t see any reason to mock the mapper. If you have other dependencies inside the mapper that need mocking, then it’s a code smell. (i.e. Don’t call the database during a mapping operation etc).</p>
<h2 id="why-not-a-static-map-method">Why not a static map method?</h2>
<p>Can’t we just do away with dependency injection and just use a static <code class="language-plaintext highlighter-rouge">CustomerViewModelMapper</code> class with a static <code class="language-plaintext highlighter-rouge">Map()</code> method? IMO this defeats one of the aspects I wanted to achieve. Understandability of object capability easily. I made the point about how using statement, constructor signature and public settable properties are my go to place to understand the capabilities of the class. When you have calls the static methods sprinkled in code it becomes another cognitive load to understand object capability.</p>
<h2 id="bonus-revelations-from-functional-programming">Bonus: Revelations (from functional programming)</h2>
<p>Given the above options the best outcome here is to rely on a mapping function to be injected. I’ll wait for a proponent on a functional programming to give a more theoretical answer to why this is the most elegant answer.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">Func</span><span class="p"><</span><span class="n">CustomerDomainModel</span><span class="p">,</span> <span class="n">CustomerViewModel</span><span class="p">></span> <span class="n">_mapper</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="nf">Foo</span><span class="p">(</span><span class="n">Func</span><span class="p"><</span><span class="n">CustomerDomainModel</span><span class="p">,</span> <span class="n">CustomerViewModel</span><span class="p">></span> <span class="n">mapper</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_mapper</span> <span class="p">=</span> <span class="n">mapper</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here we don’t rely on a class. We rather rely on much more narrowly scoped capability via a Function signature (or delegate). But then again Visual Studio (or any IDE I know) does not support navigation from a <code class="language-plaintext highlighter-rouge">Func<CustomerDomainModel, CustomerViewModel></code> to the implementation easily.</p>
<h2 id="closing-remarks">Closing Remarks</h2>
<p>We explored how mapping relates of the object capability model and why easy to understand/navigate solution helps us with it. We also went through some mapping design patterns and discussed the pros and cons with them.</p>
<p>My advice is to keep the mapping logic simple as explained in the last few examples. You want the least amount of cognitive load when reading and navigation through your code base. Especially as it evolves and “matures” so that new developers to the team can start being productive faster and make less errors. You don’t want to design a pattern that lets developers shoot them selves in the foot easily with.</p>
<p>Mapping can be tedious the way I recommended. Yes I completely agree. Maybe consider using something like <a href="https://github.com/cezarypiatek/MappingGenerator">this</a>. The concept behind it is explained <a href="https://cezarypiatek.github.io/post/generate-mappings-on-build/">here</a>. With C# 9, we got <a href="https://devblogs.microsoft.com/dotnet/introducing-c-source-generators/">source generators</a> and we could have some potentially creative solution there as well.</p>
<p>I’m keen to learn about your own experiences as well. Please reach out to me via twitter @dasiths or post a comment here if you have some interesting opinion or view related to this. Happy coding.</p>Dasith WijesiriwardenaAlmost any type of software development work requires some sort mapping in the logic. Either from the persistence models to domain model or from the domain models to view models. In my experience dealing with a lot of brownfield projects, I’ve seen some practices that age poorly and make it difficult to understand intent. Most mapping approaches age gracefully like beer. Although my aim is not to kick a dead horse and rant about AutoMapper, I agree with the sentiments provided in that linked post. I recommend you read it if you haven’t already. Pay attention to the comment thread there where Jimmy Bogard gives some decent pointers on when and when not to use it. I haven’t been blogging as often as I would like to. I’ve recently started a new job and it’s been keeping me quite busy. I’m taking the latest snap lockdown in lieu of Covid19 in Victoria, Australia to write this. I wanted to do this before I reward myself with binging Netflix. This is a topic I’ve wanted to express my opinion on for a while. In this post I’ll be expanding on some of the arguments made in that linked post. tl;dr; I'm complaining about how your mapping choices affect maintainability, understandability and code navigation. Object Capability Model Before we dive in, I want to introduce the Object Capability Model to those of you who haven’t heard of it. (Most software developers understand it intuitively). The object-capability model is a computer security model. A capability describes a transferable right to perform one (or more) operations on a given object. It can be obtained by the following combination: - An unforgeable reference (in the sense of object references or protected pointers) that can be sent in messages. - A message that specifies the operation to be performed. To put it simply it’s a model which describes the capabilities of an object. The security implication is that, we should aim to give the least amount of privileges required to an object to able to perform its responsibilities. In modern software development we give capabilities to an object through the references it holds. If you’re following inversion of control pattern in your design, this is often done through dependency injection via the constructor. When I’m working on a brownfield project or a large project and trying to understand and navigate the code, I often look at the constructor of a class to understand the capabilities of it. This allows me to understand the what other area of the system can potentially impact this class as well. Now I know there are better ways to to do this i.e. Dependency Graph but it’s not something you can quickly access or visualise. So what does this mean for mapping? The Problem With AutoMapper The shortcomings of (or ways developers abuse) AutoMapper are well documented. I won’t repeat them again here. Please read the article linked above. My take is more targeted around how it affects the understandability of object capability. Let’s take this class as an example: public class Foo { private readonly IMapper _mapper { get; set; } public Foo(IMapper mapper) { _mapper = mapper; } ... } If we look at the constructor here, we know the class uses some mapping but it’s not clear what. I have to go through the code to understand the from and to models or indeed if there are multiples of them. Yes it’s not the end of the world. A software developer needing to read the code to understand it, the horror. But this cognitive load adds up. Especially when you consider the fact that mapping is something done every time an object needs to be presented to/from a layer. (Lets rant about enterprise software development at a later time shall we?). This also poses a hurdle in easily understanding the impact this class has. It is very hard to navigate to the mapping logic and therefore hard to understand the capabilities this class will ultimately have. This impacts us in giving the class the least amount of privileges required to perform its actions. So, IMapper<TFrom,TTo> then? This is another alternative to AutoMapper. Here we define an interface IMapper<TFrom,TTo> like public interface IMapper<DomainModel, ViewModel> { public TTo Map(TFrom model); } and implement it public class CustomerViewModelMapper : IMapper<CustomerDomainModel, CustomerViewModel> { public CustomerViewModel Map(TFrom CustomerDomainModel) { // mapping logic here } } Using it is pretty easy public class Foo { private readonly IMapper<CustomerDomainModel, CustomerViewModel> _mapper { get; set; } public Foo(IMapper<CustomerDomainModel, CustomerViewModel> mapper) { _mapper = mapper; } ... } Something I like about this approach is whenever you see IMapper<TFrom, TTo> you instantly know what capability it provides to the dependant class. You don’t have to rely on the name of the dependency. (We know how bad we are at naming things). One of the other aspects that is appealing with this approach is that you can automatically scan your assemblies for implementations of IMapper<TFrom,TTo> and register it in your dependency injection . This means you have one less thing to worry about. public static class MapperModule { public static void AddMappers(this IServiceCollection services, IConfiguration configuration) { var serviceType = typeof(IMapper<,>); ScanServiceType(services, serviceType); } private static void ScanServiceType(IServiceCollection services, Type serviceType) { var mapperTypes = DependentAssemblies.All .SelectMany(a => a.GetTypes()) .Where(type => type.GetInterfaces() != null && type.GetInterfaces().Any( t => t.IsGenericType && t.GetGenericTypeDefinition() == serviceType) && type.IsClass && !type.IsAbstract); foreach (var mapperType in mapperTypes) { var @interfaces = mapperType.GetInterfaces().Where(t => t.IsGenericType && t.GetGenericTypeDefinition() == serviceType); foreach (var @interface in @interfaces) { services.AddScoped(@interface, mapperType); } } } } If you’ve been paying attention then you might have noticed something that is making me look like a hypocrite. We wanted to make code navigation easier, but now we have a dependency on IMapper<CustomerDomainModel, CustomerViewModel> and it’s hard to navigate to the actual implementation because Visual Studio can’t detect implementations of generic types without the help of extensions/plugins. Sure it’s better from a “understanding object capability” perspective but unless you have Resharper installed we haven’t really made code navigation any easier. Also me being absent minded, have taken a dependency on IMapper<TFrom, TTo> in a class only to realise at runtime that there is no implementation for it. One could even argue that DI auto register magic is bad. So where does that leave us? Simple classes that only map. Who would have thought? These days this is my go to option. public class CustomerViewModelMapper { private readonly AddressServiceMapper addressServiceMapper; public CustomerViewModelMapper(AddressServiceMapper addressServiceMapper) { this.addressServiceMapper = addressServiceMapper; } public CustomerViewModel Map(CustomerDomainModel model) { return new CustomerViewModel { BusinessName = model.BusinessName, Address = this.addressServiceMapper.Map(entity.Address) }; } } Using it is straight forward again. public class Foo { private readonly CustomerViewModelMapper _mapper { get; set; } public Foo(CustomerViewModelMapper mapper) { _mapper = mapper; } ... } Notice how I didn’t bother with an interface here. The truth is you don’t need one. I can’t see any reason to mock the mapper. If you have other dependencies inside the mapper that need mocking, then it’s a code smell. (i.e. Don’t call the database during a mapping operation etc). Why not a static map method? Can’t we just do away with dependency injection and just use a static CustomerViewModelMapper class with a static Map() method? IMO this defeats one of the aspects I wanted to achieve. Understandability of object capability easily. I made the point about how using statement, constructor signature and public settable properties are my go to place to understand the capabilities of the class. When you have calls the static methods sprinkled in code it becomes another cognitive load to understand object capability. Bonus: Revelations (from functional programming) Given the above options the best outcome here is to rely on a mapping function to be injected. I’ll wait for a proponent on a functional programming to give a more theoretical answer to why this is the most elegant answer. public class Foo { private readonly Func<CustomerDomainModel, CustomerViewModel> _mapper { get; set; } public Foo(Func<CustomerDomainModel, CustomerViewModel> mapper) { _mapper = mapper; } ... } Here we don’t rely on a class. We rather rely on much more narrowly scoped capability via a Function signature (or delegate). But then again Visual Studio (or any IDE I know) does not support navigation from a Func<CustomerDomainModel, CustomerViewModel> to the implementation easily. Closing Remarks We explored how mapping relates of the object capability model and why easy to understand/navigate solution helps us with it. We also went through some mapping design patterns and discussed the pros and cons with them. My advice is to keep the mapping logic simple as explained in the last few examples. You want the least amount of cognitive load when reading and navigation through your code base. Especially as it evolves and “matures” so that new developers to the team can start being productive faster and make less errors. You don’t want to design a pattern that lets developers shoot them selves in the foot easily with. Mapping can be tedious the way I recommended. Yes I completely agree. Maybe consider using something like this. The concept behind it is explained here. With C# 9, we got source generators and we could have some potentially creative solution there as well. I’m keen to learn about your own experiences as well. Please reach out to me via twitter @dasiths or post a comment here if you have some interesting opinion or view related to this. Happy coding.Not All “Microservices Frameworks” Are Made The Same - NDC Sydney 20202021-03-08T22:06:00+11:002021-03-08T22:06:00+11:00https://dasith.me/2021/03/08/microservices-frameworks-ndc-sydney-2020<p>Last October I spoke at <a href="https://ndcsydney.com/">NDC Sydney</a> about the pitfalls associated with microservices frameworks and how you can still leverage libraries and runtimes to help with distributed system complexities. In my experience, microservices frameworks tend to give diminishing results as your project evolves and even makes your team less agile and productive. Microservices are an architectural style and frameworks tie you down to a lot of opinions that do not age well.</p>
<p>This is my first time speaking at NDC and due to the pandemic it was hosted in an online format. It had some awesome workshops as well as talks from industry experts sharing their knowledge.</p>
<p>I touched on the following topics during the talk…</p>
<ul>
<li>The definitions for frameworks, libraries and runtimes and why that matters.</li>
<li>How frameworks hinder agility and creative thinking among development teams.</li>
<li>Why you should still leverage runtimes to solve distributed system complexities.</li>
</ul>
<p>If your team is currently developing distributed systems or plan to develop microservices like architecture in the future I highly recommend you have a look and try to apply some of the concept to your own context. At least it might give you a shortlist of things to consider before making a jump to a framework.</p>
<h2 id="about-ndc-conferences">About NDC Conferences</h2>
<p>On the website it says</p>
<blockquote>
<p>Since its start-up in Oslo 2008, the Norwegian Developers Conference (NDC) quickly became one of Europe`s largest conferences for .NET & Agile development. Today NDC Conferences are 5-day events with 2 days of pre-conference workshops and 3 days of conference sessions. <br /><br /> The 5th NDC Sydney will be hosted online due to COVID-19. NDC Sydney is a five-day event with the same format as its European sisters NDC Oslo and NDC London. <br /><br /> The conference will feature 5 tracks covering topics such as:<br /><br /> .NET - Agile - C++ - Cloud - Database - Design - DevOps - Embedded - Front-End Framework - Fun - Functional Programming - Gadgets - Internet of Things - JavaScript - Microsoft - Mobile - People - Programming Languages - Security - Techniques - Testing - Tools - UX – Web and more.</p>
</blockquote>
<h2 id="building-distributed-systems-on-the-shoulders-of-giants">Building distributed systems on the shoulders of giants</h2>
<p>The abstract is as follows.</p>
<blockquote>
<p>Developing a distributed system is hard. Understanding the domain is harder. Therefore it makes sense not to reinvent the wheel and use ready made solutions to help build your microservices right? or does it? Turns out the there is a lot to consider here. We will look at how to leverage the right tools to help you achieve a good outcome and a sustainable growth path. We will also explore strategies to not get locked in “frameworks” that only give you short term wins. We will see how our choices can have a long lasting impact and what to consider when making them. <br /><br /> Many developers make the mistake to locking themselves into frameworks that give them an easy win upfront. But as the software solution evolves you end up developing around the quirks of the framework rather than let the domain dictate the architecture. This eventually leads to un sustainable systems that make the domain harder to understand and reason with, making feature development painfully slow. I want to discuss ways to avoid this by picking “frameworks” that don’t dictate how you solve business problems as oppose to being a tool or a layer which operates under the domain. Further I’ll touch on the bad aspects of vendor lock-in and how to pick the tools that gives you the freedom to run on any cloud vendor.</p>
</blockquote>
<h2 id="recording--slide-deck">Recording & Slide deck</h2>
<iframe width="560" height="315" src="https://www.youtube.com/embed/4oSDZI4oiAw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<script async="" class="speakerdeck-embed" data-id="966c93d11f88426fb7ad5e81c4a725e0" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<p>If you have any thoughts or comments please leave them here. I would love to hear about your experience using “microservices frameworks” to implement a microservices architecture.</p>Dasith WijesiriwardenaLast October I spoke at NDC Sydney about the pitfalls associated with microservices frameworks and how you can still leverage libraries and runtimes to help with distributed system complexities. In my experience, microservices frameworks tend to give diminishing results as your project evolves and even makes your team less agile and productive. Microservices are an architectural style and frameworks tie you down to a lot of opinions that do not age well. This is my first time speaking at NDC and due to the pandemic it was hosted in an online format. It had some awesome workshops as well as talks from industry experts sharing their knowledge. I touched on the following topics during the talk… The definitions for frameworks, libraries and runtimes and why that matters. How frameworks hinder agility and creative thinking among development teams. Why you should still leverage runtimes to solve distributed system complexities. If your team is currently developing distributed systems or plan to develop microservices like architecture in the future I highly recommend you have a look and try to apply some of the concept to your own context. At least it might give you a shortlist of things to consider before making a jump to a framework. About NDC Conferences On the website it says Since its start-up in Oslo 2008, the Norwegian Developers Conference (NDC) quickly became one of Europe`s largest conferences for .NET & Agile development. Today NDC Conferences are 5-day events with 2 days of pre-conference workshops and 3 days of conference sessions. The 5th NDC Sydney will be hosted online due to COVID-19. NDC Sydney is a five-day event with the same format as its European sisters NDC Oslo and NDC London. The conference will feature 5 tracks covering topics such as: .NET - Agile - C++ - Cloud - Database - Design - DevOps - Embedded - Front-End Framework - Fun - Functional Programming - Gadgets - Internet of Things - JavaScript - Microsoft - Mobile - People - Programming Languages - Security - Techniques - Testing - Tools - UX – Web and more. Building distributed systems on the shoulders of giants The abstract is as follows. Developing a distributed system is hard. Understanding the domain is harder. Therefore it makes sense not to reinvent the wheel and use ready made solutions to help build your microservices right? or does it? Turns out the there is a lot to consider here. We will look at how to leverage the right tools to help you achieve a good outcome and a sustainable growth path. We will also explore strategies to not get locked in “frameworks” that only give you short term wins. We will see how our choices can have a long lasting impact and what to consider when making them. Many developers make the mistake to locking themselves into frameworks that give them an easy win upfront. But as the software solution evolves you end up developing around the quirks of the framework rather than let the domain dictate the architecture. This eventually leads to un sustainable systems that make the domain harder to understand and reason with, making feature development painfully slow. I want to discuss ways to avoid this by picking “frameworks” that don’t dictate how you solve business problems as oppose to being a tool or a layer which operates under the domain. Further I’ll touch on the bad aspects of vendor lock-in and how to pick the tools that gives you the freedom to run on any cloud vendor. Recording & Slide deck If you have any thoughts or comments please leave them here. I would love to hear about your experience using “microservices frameworks” to implement a microservices architecture.Building Distributed Systems on the Shoulders of Giants - API Days Live, Australia 20202020-10-11T22:06:00+11:002020-10-11T22:06:00+11:00https://dasith.me/2020/10/11/building-ditributed-systems-apidays-2020<p>I recently spoke about building distributed systems at <a href="https://www.apidays.co/australia/">API Days Australia</a> conference. My talk is titled <code class="language-plaintext highlighter-rouge">Building distributed systems on the shoulders of giants</code>. It’s a reference to the <a href="https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants">popular metaphor</a> about leveraging the knowledge on people who have been here before us.</p>
<h2 id="about-api-days-live-australia">About API Days Live Australia</h2>
<p>On the website it says</p>
<blockquote>
<p>APIdays is the leading industry tech and business series of conferences in APIs and the programmable economy. As APIs become mainstream, our world becomes more connected, more automated and more intelligent. APIs are the gateway to data, services, devices and emerging technologies. APIs put power into the hands of developers, citizens and consumers.</p>
</blockquote>
<p>This is my 2nd time speaking at the event. Last year I spoke about <a href="https://dasith.me/2019/09/07/upcoming-talk-at-apidays/">Microservices</a>. The conference speakers consist of technology leaders from companies like Google, Amazon, Microsoft and Twitter alongside some local experts as well. The conference ran in a virtual format due to COVID19 restrictions but it was very well organised and saw 1000+ people attend.</p>
<h2 id="building-distributed-systems-on-the-shoulders-of-giants">Building distributed systems on the shoulders of giants</h2>
<p>The abstract is as follows.</p>
<blockquote>
<p>Developing a distributed system is one of the hardest things you will do as a software developer. You will end up having to deal with topics like network inconsistencies, load balancing and service discovery etc. Only to find that solving these problems require expert level knowledge. What if I told you there was a way to leverage the expertise of industry leaders like Google/Microsoft and benefit from their research so you don’t have to reinvent the wheel? Let’s investigate some tools you can use today to build modern, resilient and scalable microservices. Technologies like Service Fabric, Knative, Istio and DAPR can give you the right foundation to build on top of so you can concentrate on solving the business domain problems and be productive.</p>
</blockquote>
<p>Developing a distributed system requires expert knowledge that not every mortal software developer has. This results in most Microservices failing in production scenarios under load or intermittent network conditions. There is a better way than expecting every developer to do these complex calculations. We can leverage the knowledge from experts by using some modern distributed applications runtimes and tools that give us a good foundation. (Standing on the shoulders of giants) This frees us to focus on the domain and solve it better. In this talk I wanted to introduce (compare and contrast) developers to some of these choices.</p>
<h2 id="recording--slide-deck">Recording & Slide deck</h2>
<iframe width="560" height="315" src="https://www.youtube.com/embed/rctYpZqIT2Y" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p><br /></p>
<script async="" class="speakerdeck-embed" data-id="5bd3880097404aa18512ab9ba29ba6bd" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<p>If you have any thoughts or comments please leave them here. I would love to hear about your experiences with any of these technologies and how they helped or hindered your progress.</p>Dasith WijesiriwardenaI recently spoke about building distributed systems at API Days Australia conference. My talk is titled Building distributed systems on the shoulders of giants. It’s a reference to the popular metaphor about leveraging the knowledge on people who have been here before us. About API Days Live Australia On the website it says APIdays is the leading industry tech and business series of conferences in APIs and the programmable economy. As APIs become mainstream, our world becomes more connected, more automated and more intelligent. APIs are the gateway to data, services, devices and emerging technologies. APIs put power into the hands of developers, citizens and consumers. This is my 2nd time speaking at the event. Last year I spoke about Microservices. The conference speakers consist of technology leaders from companies like Google, Amazon, Microsoft and Twitter alongside some local experts as well. The conference ran in a virtual format due to COVID19 restrictions but it was very well organised and saw 1000+ people attend. Building distributed systems on the shoulders of giants The abstract is as follows. Developing a distributed system is one of the hardest things you will do as a software developer. You will end up having to deal with topics like network inconsistencies, load balancing and service discovery etc. Only to find that solving these problems require expert level knowledge. What if I told you there was a way to leverage the expertise of industry leaders like Google/Microsoft and benefit from their research so you don’t have to reinvent the wheel? Let’s investigate some tools you can use today to build modern, resilient and scalable microservices. Technologies like Service Fabric, Knative, Istio and DAPR can give you the right foundation to build on top of so you can concentrate on solving the business domain problems and be productive. Developing a distributed system requires expert knowledge that not every mortal software developer has. This results in most Microservices failing in production scenarios under load or intermittent network conditions. There is a better way than expecting every developer to do these complex calculations. We can leverage the knowledge from experts by using some modern distributed applications runtimes and tools that give us a good foundation. (Standing on the shoulders of giants) This frees us to focus on the domain and solve it better. In this talk I wanted to introduce (compare and contrast) developers to some of these choices. Recording & Slide deck If you have any thoughts or comments please leave them here. I would love to hear about your experiences with any of these technologies and how they helped or hindered your progress.Converting a Visual Studio database project to use DbUp migrations2020-06-08T12:00:00+10:002020-06-08T12:00:00+10:00https://dasith.me/2020/06/08/database-project-conversion-to-migrations<p>Database projects in Visual Studio have been a very popular way to develop and manage the database schema and logic in the past. With more and more applications now preferring to have the data tier logic and validation within the bounds of the application and use the database just for persistence, I’ve come across clients who want to convert their database projects to a more CI/CD friendly database migrations.</p>
<p>So I decided to share an approach used to convert existing database project to use database migrations using <a href="https://dbup.github.io/">DbUp</a> I’ve had success with.</p>
<h2 id="why-migrations">Why Migrations?</h2>
<p><strong>Because Transitions are easier to manage than States</strong>.</p>
<p>Visual Studio database projects rely on state comparison (called <code class="language-plaintext highlighter-rouge">diffs</code>) to compare your source and target schemas and come up with a deployment script. While these auto generated scripts are very good they aren’t perfect. The tooling can’t always create scripts for all the state transitions without causing inconsistencies (data loss etc). This makes it impossible to integrate them into an automated deployment pipeline with confidence.</p>
<p>Migrations on the other hand rely on transitions. Humans write code for the transitions. We can then have an automated process run the transitions to get the target database up to date.</p>
<p>The philosophy behind this is <a href="https://dbup.readthedocs.io/en/latest/philosophy-behind-dbup/">looked at in detail here</a>.</p>
<h2 id="why-dbup">Why DbUp?</h2>
<blockquote>
<p>DbUp is a .NET library that helps you to deploy changes to SQL Server databases. It tracks which SQL scripts have been run already, and runs the change scripts that are needed to get your database up to date.</p>
</blockquote>
<p>I’ve used tool like <a href="https://github.com/fluentmigrator/fluentmigrator"><code class="language-plaintext highlighter-rouge">FluentMigrator</code></a> to implement migrations before as well. But I like the simplicity and support <code class="language-plaintext highlighter-rouge">DbUp</code> provides. This at the end of the day is just personal preference.</p>
<h2 id="forward-only-migrations">Forward-only migrations</h2>
<p>DbUp doesn’t support down migrations though. IMO this is a good design decision. Down migrations are rarely tested and often only give a false sense of confidence of ability to rollback. If your up migration fails then the same set of data might cause your down migration to fail as well. To add to that not all migrations can have a paired down migration to consistently rollback changes. My preference is to have discipline around how you do migrations and not do destructive data changes in a single migration step.</p>
<p>As an example if I wanted to refactor a single <code class="language-plaintext highlighter-rouge">FullName</code> field into <code class="language-plaintext highlighter-rouge">FirstName</code> and <code class="language-plaintext highlighter-rouge">LastName</code> columns, I wouldn’t delete the <code class="language-plaintext highlighter-rouge">FullName</code> column in the same migration that I introduce the other two columns. In the first migration I would introduce the columns, copy the data over and then rename the <code class="language-plaintext highlighter-rouge">FullName</code> column to something like <code class="language-plaintext highlighter-rouge">FullName-Deprecated</code>. I would then let the system run for a while until we are confident that everything is working as intended. If not, you can then create subsequent migrations to rectify the problems as you still have data in the original column. After a certain period, I would then create another migration to get rid of the now deprecated column. This gives greater resiliency to the system and allows you to be more confident about changes you do.</p>
<h2 id="preparation">Preparation</h2>
<ol>
<li>
<p>We need to create our initial migration and the easiest way to do this is to consider your current state of the database as the initial state as far as the migrations are concerned.</p>
<p>This means exporting all your database objects to a script. You can do this from SQL Server Management Studio (SSMS). <a href="https://www.eversql.com/exporting-mysql-schema-structure-to-xml-using-mysql-clients/#sqlserver">Follow the steps here</a> to <code class="language-plaintext highlighter-rouge">Generate Scripts</code> for all database objects you require to be created as a part of the initial migration (Tables, Views, Stored Procs, Functions etc).</p>
<p><img src="/assets/images/export_sql_server_schema_step1-768x624.png" alt="Generate Scripts" /></p>
<p>Save this file as <code class="language-plaintext highlighter-rouge">Initial_Migration.sql</code>. Once exported open the file and remove any SQL statements at the begining that create the Database or Logins. Lets assume the migrations are run on an empty database that has already been created.</p>
</li>
<li>Now repeat the same steps as above with the following changes…
<ul>
<li>Only select Functions and Stored Procedures to export this time as we need them to be PR review friendly. More about this later on.</li>
<li>In the last window, go to <code class="language-plaintext highlighter-rouge">Advanced</code> -> Scripting Options -> set <code class="language-plaintext highlighter-rouge">Check for object existance</code> to <code class="language-plaintext highlighter-rouge">True</code>. Press OK.</li>
<li>
<p>Now choose the option to create a <code class="language-plaintext highlighter-rouge">Single file per object</code>. (Remember we exported everything in to one file file in the previous step, we are exporting a subset of those things again to their individual files.)</p>
</li>
<li>
<p>Save these in a folder called <code class="language-plaintext highlighter-rouge">PostDeployment</code>.</p>
<p><img src="/assets/images/export_sql_server_schema_per_object.png" alt="Generate Scripts" /></p>
<p><strong>Why are we duplicating the Stored Procedures in our all-in-one script and again in the individual files?</strong></p>
<p>Imagine that we only used migrations and no post deployment scripts. Unlike Table or View schemas you can’t alter a part of a Stored Procedure. So if you need to make a change you need to create a new migration with the full stored procedure definition. The person reviewing your change sees a new file and not just your change. It becomes very hard to review changes this way.</p>
<p>So we will use these one file per object scripts to create reviewable post deployment scripts in our migration project. The idea is you keep modifying these post deployment scripts (Stored Procedures and Functions) rather than create a new migration for them. So the reviewer sees only your change in the context of the larger set. This will be very helpful if you have large Stored Procedures in your database project.</p>
<p>There is more detail around the idea <a href="https://wengier.com/reviewable-sprocs/">here</a>.</p>
</li>
</ul>
</li>
<li>
<p>You should now have one big SQL file called <code class="language-plaintext highlighter-rouge">Initial_Migration.sql</code> and lot of smaller SQL files per Stored Procedure in the <code class="language-plaintext highlighter-rouge">PostDeployment</code> folder. Now we need to give the files inside that folder a sequence number. For example if your file is called <code class="language-plaintext highlighter-rouge">CreateCustomer.sql</code> we need to rename it to <code class="language-plaintext highlighter-rouge">000001 - CreateCustomer.sql</code> and anything that depends on this script must have a sequence/order number higher than <code class="language-plaintext highlighter-rouge">000001</code>. (This is because DbUp executes the scripts in their name order. We use the name as a way to enforce dependency order for post deployment scripts.)</p>
<p>If you have a lot of files, it will be time consuming to do this manually. So I used this <a href="https://www.bulkrenameutility.co.uk/">BulkRenaming</a> tool to put the sequence/order suffix. I set it so that it suffixed files in increments of 5 (So sequence is 1, 6, 11, 16 etc). Once this was done I could then manually set the dependent order (by renaming the suffix manually) as required without having to rename every subsequent file. This is because the increment by 5 meant I could squeeze 4 more files between two other scripts if required. You might need a bit of manual effort here depending on how interconnected your Stored Procedures are.</p>
</li>
</ol>
<h2 id="dbup-migration-runner">DbUp Migration Runner</h2>
<ol>
<li>
<p>The folder structure for the migration runner is as follows. This is a simple console app targeting .NET framework but it’s trivial to change this to work with .NET Core.</p>
<p><img src="/assets/images/dbup-project-structure-database-project.png" alt="Folder Structure" /></p>
<ul>
<li>
<p>As you can see I put the <code class="language-plaintext highlighter-rouge">Initial_Migration.sql</code> in the <code class="language-plaintext highlighter-rouge">Migrations</code> folder and prefixed it with <code class="language-plaintext highlighter-rouge">000001</code>. All subsequent migration will have the next sequence.</p>
</li>
<li>
<p>I also have <code class="language-plaintext highlighter-rouge">PostDeployment</code> and <code class="language-plaintext highlighter-rouge">PreDeployment</code> folders. I’ve organized Stored Procedures and Functions in their own respective folders but that is optional. This is where you need to put the individually exported SQL files (<code class="language-plaintext highlighter-rouge">PostDeployment</code> folder) from our preparation steps before.</p>
</li>
<li>
<p>There are environment specific folders which also have their own specific Pre and Post deployment folders. This gives you the ability to do customisations based on the environment if required. Once you see the migration runner code it will make more sense.</p>
</li>
<li>
<p>DbUp uses the fully qualified file name with the namespace to figure out the order of execution and we make use of the filename suffix or nested folders (Namespace) to enforce our desired order.</p>
</li>
</ul>
</li>
<li>
<p>I use config transforms to transform the <code class="language-plaintext highlighter-rouge">AppConfig</code> based on the build quality via the <a href="https://www.c-sharpcorner.com/article/transform-config-using-slow-cheetah/">SlowCheetah extension</a>.</p>
<p>App.config</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="cp"><?xml version="1.0" encoding="utf-8" ?></span>
<span class="nt"><configuration></span>
<span class="nt"><startup></span>
<span class="nt"><supportedRuntime</span> <span class="na">version=</span><span class="s">"v4.0"</span> <span class="na">sku=</span><span class="s">".NETFramework,Version=v4.7.2"</span> <span class="nt">/></span>
<span class="nt"></startup></span>
<span class="nt"><connectionStrings></span>
<span class="nt"><add</span> <span class="na">name=</span><span class="s">"MyConnectionString"</span> <span class="na">connectionString=</span><span class="s">"Integrated Security=SSPI;Initial Catalog=MyDatabase;Data Source=.;"</span> <span class="na">providerName=</span><span class="s">"System.Data.SqlClient"</span><span class="nt">/></span>
<span class="nt"></connectionStrings></span>
<span class="nt"><appSettings></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"AdditionalPreDeploymentNamespace"</span> <span class="na">value=</span><span class="s">""</span> <span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"AdditionalPostDeploymentNamespace"</span> <span class="na">value=</span><span class="s">""</span> <span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"Environment"</span> <span class="na">value=</span><span class="s">""</span> <span class="nt">/></span>
<span class="nt"></appSettings></span>
<span class="nt"></configuration></span>
</code></pre></div> </div>
<p>App.Debug.config</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="cp"><?xml version="1.0" encoding="utf-8"?></span>
<span class="nt"><configuration</span> <span class="na">xmlns:xdt=</span><span class="s">"http://schemas.microsoft.com/XML-Document-Transform"</span><span class="nt">></span>
<span class="nt"><appSettings></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"AdditionalPreDeploymentNamespace"</span> <span class="na">value=</span><span class="s">"Dev.PreDeployment"</span> <span class="na">xdt:Transform=</span><span class="s">"SetAttributes"</span> <span class="na">xdt:Locator=</span><span class="s">"Match(key)"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"AdditionalPostDeploymentNamespace"</span> <span class="na">value=</span><span class="s">"Dev.PostDeployment"</span> <span class="na">xdt:Transform=</span><span class="s">"SetAttributes"</span> <span class="na">xdt:Locator=</span><span class="s">"Match(key)"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"Environment"</span> <span class="na">value=</span><span class="s">"Dev"</span> <span class="na">xdt:Transform=</span><span class="s">"SetAttributes"</span> <span class="na">xdt:Locator=</span><span class="s">"Match(key)"</span><span class="nt">/></span>
<span class="nt"></appSettings></span>
<span class="nt"></configuration></span>
</code></pre></div> </div>
<p>App.Release.config</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="cp"><?xml version="1.0" encoding="utf-8"?></span>
<span class="nt"><configuration</span> <span class="na">xmlns:xdt=</span><span class="s">"http://schemas.microsoft.com/XML-Document-Transform"</span><span class="nt">></span>
<span class="nt"><connectionStrings></span>
<span class="nt"><add</span> <span class="na">name=</span><span class="s">"MyConnectionString"</span> <span class="na">connectionString=</span><span class="s">"#{databaseConnectionString}#"</span> <span class="na">providerName=</span><span class="s">"System.Data.SqlClient"</span> <span class="na">xdt:Transform=</span><span class="s">"SetAttributes"</span> <span class="na">xdt:Locator=</span><span class="s">"Match(name)"</span> <span class="nt">/></span>
<span class="nt"></connectionStrings></span>
<span class="nt"><appSettings></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"AdditionalPreDeploymentNamespace"</span> <span class="na">value=</span><span class="s">"#{releaseEnvironment}#.PreDeployment"</span> <span class="na">xdt:Transform=</span><span class="s">"SetAttributes"</span> <span class="na">xdt:Locator=</span><span class="s">"Match(key)"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"AdditionalPostDeploymentNamespace"</span> <span class="na">value=</span><span class="s">"#{releaseEnvironment}#.PostDeployment"</span> <span class="na">xdt:Transform=</span><span class="s">"SetAttributes"</span> <span class="na">xdt:Locator=</span><span class="s">"Match(key)"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"Environment"</span> <span class="na">value=</span><span class="s">"#{releaseEnvironment}#"</span> <span class="na">xdt:Transform=</span><span class="s">"SetAttributes"</span> <span class="na">xdt:Locator=</span><span class="s">"Match(key)"</span><span class="nt">/></span>
<span class="nt"></appSettings></span>
<span class="nt"></configuration></span>
</code></pre></div> </div>
<p>My release pipeline injects the <code class="language-plaintext highlighter-rouge">databaseConnectionString</code> and <code class="language-plaintext highlighter-rouge">releaseEnvironment</code> variables in to the <code class="language-plaintext highlighter-rouge">App.Release.config</code> / <code class="language-plaintext highlighter-rouge">AppName.exe.Config</code> file upon building/deploying the app. That is why you see tokens like <code class="language-plaintext highlighter-rouge">#{databaseConnectionString}#</code>. I use <a href="https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens">this extension</a> in Azure DevOps to do it.</p>
<p>When running locally with the <code class="language-plaintext highlighter-rouge">Debug</code> build quality <code class="language-plaintext highlighter-rouge">MSBuild</code> uses the <code class="language-plaintext highlighter-rouge">App.Debug.Config</code> to transform your original config file. So you can easily run and test the migration runner locally. This makes your local development exprience consistent among the team. No more complex database update scripts to run.</p>
</li>
<li>
<p>Now let’s look at my migration runner (<code class="language-plaintext highlighter-rouge">Program.cs</code>). You obviously need a reference to the <code class="language-plaintext highlighter-rouge">DbUp-SqlServer</code> NuGet as we use it to do the heavy lifting.</p>
</li>
</ol>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="k">class</span> <span class="nc">Program</span>
<span class="p">{</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">Main</span><span class="p">(</span><span class="kt">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">noWait</span> <span class="p">=</span> <span class="k">false</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">Length</span> <span class="p">==</span> <span class="m">1</span> <span class="p">&&</span> <span class="n">args</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p">==</span> <span class="s">"--nowait"</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">noWait</span> <span class="p">=</span> <span class="k">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="n">exitCode</span> <span class="p">=</span> <span class="m">0</span><span class="p">;</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">$"Database Migration Runner. "</span> <span class="p">+</span>
<span class="s">$"Version=</span><span class="p">{</span><span class="k">typeof</span><span class="p">(</span><span class="n">IEnvironmentsFolderMarker</span><span class="p">).</span><span class="n">Assembly</span><span class="p">.</span><span class="nf">GetName</span><span class="p">().</span><span class="n">Version</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="c1">// NOTE: PLEASE MAKE SURE YOUR SCRIPT IS MARKED AS EMBEDDED</span>
<span class="c1">// https://www.c-sharpcorner.com/uploadfile/40e97e/saving-an-embedded-file-in-C-Sharp/</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">"\nIMPORTANT: Please ensure your scripts are EMBEDDED in the executable."</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">baseNamespace</span> <span class="p">=</span> <span class="k">typeof</span><span class="p">(</span><span class="n">Program</span><span class="p">).</span><span class="n">Namespace</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">baseEnvironmentsNamespace</span> <span class="p">=</span> <span class="k">typeof</span><span class="p">(</span><span class="n">IEnvironmentsFolderMarker</span><span class="p">).</span><span class="n">Namespace</span><span class="p">;</span>
<span class="c1">// You can use IConfiguration (Microsoft.Extensions.Configuration) to </span>
<span class="c1">// achieve the same thing in a .NET Core project as shown here </span>
<span class="c1">// https://stackoverflow.com/questions/38114761/asp-net-core-configuration-for-net-core-console-application</span>
<span class="kt">var</span> <span class="n">additionalPreDeploymentNamespace</span> <span class="p">=</span>
<span class="n">ConfigurationManager</span><span class="p">.</span><span class="n">AppSettings</span><span class="p">[</span><span class="s">"AdditionalPreDeploymentNamespace"</span><span class="p">];</span>
<span class="kt">var</span> <span class="n">additionalPostDeploymentNamespace</span> <span class="p">=</span>
<span class="n">ConfigurationManager</span><span class="p">.</span><span class="n">AppSettings</span><span class="p">[</span><span class="s">"AdditionalPostDeploymentNamespace"</span><span class="p">];</span>
<span class="kt">var</span> <span class="n">connectionString</span> <span class="p">=</span>
<span class="n">ConfigurationManager</span><span class="p">.</span><span class="n">ConnectionStrings</span><span class="p">[</span><span class="s">"MyConnectionString"</span><span class="p">];</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">"\nListing variables...\n"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">variables</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Dictionary</span><span class="p"><</span><span class="kt">string</span><span class="p">,</span> <span class="kt">string</span><span class="p">>();</span>
<span class="k">foreach</span> <span class="p">(</span><span class="kt">var</span> <span class="n">k</span> <span class="k">in</span> <span class="n">ConfigurationManager</span><span class="p">.</span><span class="n">AppSettings</span><span class="p">.</span><span class="n">AllKeys</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">variables</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">ConfigurationManager</span><span class="p">.</span><span class="n">AppSettings</span><span class="p">[</span><span class="n">k</span><span class="p">]);</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">$"$</span><span class="p">{</span><span class="n">k</span><span class="p">}</span><span class="s">$ = \"</span><span class="p">{</span><span class="n">ConfigurationManager</span><span class="p">.</span><span class="n">AppSettings</span><span class="p">[</span><span class="n">k</span><span class="p">]}</span><span class="s">\""</span><span class="p">);</span>
<span class="c1">// See how to use variables in your scripts: </span>
<span class="c1">// https://dbup.readthedocs.io/en/latest/more-info/variable-substitution/</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(!</span><span class="kt">string</span><span class="p">.</span><span class="nf">IsNullOrWhiteSpace</span><span class="p">(</span><span class="n">additionalPreDeploymentNamespace</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">additionalPreDeploymentNamespace</span> <span class="p">=</span> <span class="n">baseEnvironmentsNamespace</span> <span class="p">+</span>
<span class="s">"."</span> <span class="p">+</span> <span class="n">additionalPreDeploymentNamespace</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(!</span><span class="kt">string</span><span class="p">.</span><span class="nf">IsNullOrWhiteSpace</span><span class="p">(</span><span class="n">additionalPostDeploymentNamespace</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">additionalPostDeploymentNamespace</span> <span class="p">=</span> <span class="n">baseEnvironmentsNamespace</span> <span class="p">+</span>
<span class="s">"."</span> <span class="p">+</span> <span class="n">additionalPostDeploymentNamespace</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">noWait</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="s">"\nPress return to run scripts..."</span><span class="p">);</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">ReadLine</span><span class="p">();</span>
<span class="p">}</span>
<span class="c1">// Pre deployments</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">"Start executing predeployment scripts..."</span><span class="p">);</span>
<span class="kt">string</span> <span class="n">preDeploymentScriptsPath</span> <span class="p">=</span> <span class="n">baseNamespace</span> <span class="p">+</span> <span class="s">".PreDeployment"</span><span class="p">;</span>
<span class="nf">RunMigrations</span><span class="p">(</span><span class="n">connectionString</span><span class="p">,</span>
<span class="n">preDeploymentScriptsPath</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="k">true</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(!</span><span class="kt">string</span><span class="p">.</span><span class="nf">IsNullOrWhiteSpace</span><span class="p">(</span><span class="n">additionalPreDeploymentNamespace</span><span class="p">))</span>
<span class="p">{</span>
<span class="nf">RunMigrations</span><span class="p">(</span><span class="n">connectionString</span><span class="p">,</span>
<span class="n">additionalPreDeploymentNamespace</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="k">true</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// Migrations</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">"Start executing migration scripts..."</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">migrationScriptsPath</span> <span class="p">=</span> <span class="n">baseNamespace</span> <span class="p">+</span> <span class="s">".Migrations"</span><span class="p">;</span>
<span class="nf">RunMigrations</span><span class="p">(</span><span class="n">connectionString</span><span class="p">,</span>
<span class="n">migrationScriptsPath</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="k">false</span><span class="p">);</span>
<span class="c1">// Post deployments</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">"Start executing postdeployment scripts..."</span><span class="p">);</span>
<span class="kt">string</span> <span class="n">postdeploymentScriptsPath</span> <span class="p">=</span> <span class="n">baseNamespace</span> <span class="p">+</span> <span class="s">".PostDeployment"</span><span class="p">;</span>
<span class="nf">RunMigrations</span><span class="p">(</span><span class="n">connectionString</span><span class="p">,</span>
<span class="n">postdeploymentScriptsPath</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="k">true</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(!</span><span class="kt">string</span><span class="p">.</span><span class="nf">IsNullOrWhiteSpace</span><span class="p">(</span><span class="n">additionalPostDeploymentNamespace</span><span class="p">))</span>
<span class="p">{</span>
<span class="nf">RunMigrations</span><span class="p">(</span><span class="n">connectionString</span><span class="p">,</span>
<span class="n">additionalPostDeploymentNamespace</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="k">true</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">Exception</span> <span class="n">e</span><span class="p">)</span>
<span class="p">{</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">Message</span><span class="p">,</span> <span class="n">ConsoleColor</span><span class="p">.</span><span class="n">Red</span><span class="p">);</span>
<span class="n">exitCode</span> <span class="p">=</span> <span class="p">-</span><span class="m">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">noWait</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="s">"Press return key to exit..."</span><span class="p">);</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">ResetColor</span><span class="p">();</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">ReadKey</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">exitCode</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">static</span> <span class="kt">int</span> <span class="nf">RunMigrations</span><span class="p">(</span><span class="kt">string</span> <span class="n">connectionString</span><span class="p">,</span>
<span class="kt">string</span> <span class="n">@namespace</span><span class="p">,</span>
<span class="n">Dictionary</span><span class="p"><</span><span class="kt">string</span><span class="p">,</span> <span class="kt">string</span><span class="p">></span> <span class="n">variables</span><span class="p">,</span>
<span class="kt">bool</span> <span class="n">alwaysRun</span> <span class="p">=</span> <span class="k">false</span><span class="p">)</span>
<span class="p">{</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">$"Executing scripts in </span><span class="p">{</span><span class="n">@namespace</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">builder</span> <span class="p">=</span> <span class="n">DeployChanges</span><span class="p">.</span><span class="n">To</span>
<span class="p">.</span><span class="nf">SqlDatabase</span><span class="p">(</span><span class="n">connectionString</span><span class="p">)</span>
<span class="p">.</span><span class="nf">WithVariables</span><span class="p">(</span><span class="n">variables</span><span class="p">)</span>
<span class="p">.</span><span class="nf">WithScriptsEmbeddedInAssembly</span><span class="p">(</span>
<span class="n">Assembly</span><span class="p">.</span><span class="nf">GetExecutingAssembly</span><span class="p">(),</span> <span class="n">file</span> <span class="p">=></span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">file</span>
<span class="p">.</span><span class="nf">ToLower</span><span class="p">()</span>
<span class="p">.</span><span class="nf">StartsWith</span><span class="p">(</span><span class="n">@namespace</span><span class="p">.</span><span class="nf">ToLower</span><span class="p">());</span>
<span class="p">})</span>
<span class="p">.</span><span class="nf">LogToConsole</span><span class="p">();</span>
<span class="n">builder</span> <span class="p">=</span> <span class="n">alwaysRun</span> <span class="p">?</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">JournalTo</span><span class="p">(</span><span class="k">new</span> <span class="nf">NullJournal</span><span class="p">())</span> <span class="p">:</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">JournalToSqlTable</span><span class="p">(</span><span class="s">"dbo"</span><span class="p">,</span> <span class="s">"DatabaseMigrations"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">executor</span> <span class="p">=</span> <span class="n">builder</span><span class="p">.</span><span class="nf">Build</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">result</span> <span class="p">=</span> <span class="n">executor</span><span class="p">.</span><span class="nf">PerformUpgrade</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">result</span><span class="p">.</span><span class="n">Successful</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">Exception</span><span class="p">(</span><span class="n">result</span><span class="p">.</span><span class="n">Error</span><span class="p">.</span><span class="nf">ToString</span><span class="p">());</span>
<span class="p">}</span>
<span class="nf">ShowSuccess</span><span class="p">();</span>
<span class="k">return</span> <span class="m">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">ShowSuccess</span><span class="p">()</span>
<span class="p">{</span>
<span class="nf">WriteToConsole</span><span class="p">(</span><span class="s">"Success!"</span><span class="p">,</span> <span class="n">ConsoleColor</span><span class="p">.</span><span class="n">Green</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">WriteToConsole</span><span class="p">(</span><span class="kt">string</span> <span class="n">msg</span><span class="p">,</span>
<span class="n">ConsoleColor</span> <span class="n">color</span> <span class="p">=</span> <span class="n">ConsoleColor</span><span class="p">.</span><span class="n">Green</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="n">ForegroundColor</span> <span class="p">=</span> <span class="n">color</span><span class="p">;</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="n">msg</span><span class="p">);</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">ResetColor</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="running-the-migration-runner">Running the migration runner</h2>
<p>You can run it manually (From Visual Studio or Opening the executable file) or run it as a part of your release pipeline with the <code class="language-plaintext highlighter-rouge">--nowait</code> argument.</p>
<p>Before you run it for the first time make sure to drop the existing database and create an empty database with the same name in the connection string.</p>
<p>That’s it. You now have all the pieces required to convert your database project to one that uses database migrations instead.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We looked at why database migrations offer us a better alternative when it comes to maintaining and deploying database changes, how to export your existing database as migrations and post deployment scripts, how to structure your migration runner and how it can work in an automated release pipeline.</p>
<p>Most of the steps we took are no different to how you would structure any other database migrations project. The interesting bit was how we leveraged the concept of post deployment scripts to put our Stored Procedures in and made the PR review process much more easier. This will make the process of conversion a much more acceptable prospect for a team that has been working with database projects and Stored Procedures in the past. I’ve had good traction with this approach at a couple of clients I’ve consulted for.</p>
<p>Please leave any comments or feedback you have here.</p>Dasith WijesiriwardenaDatabase projects in Visual Studio have been a very popular way to develop and manage the database schema and logic in the past. With more and more applications now preferring to have the data tier logic and validation within the bounds of the application and use the database just for persistence, I’ve come across clients who want to convert their database projects to a more CI/CD friendly database migrations. So I decided to share an approach used to convert existing database project to use database migrations using DbUp I’ve had success with. Why Migrations? Because Transitions are easier to manage than States. Visual Studio database projects rely on state comparison (called diffs) to compare your source and target schemas and come up with a deployment script. While these auto generated scripts are very good they aren’t perfect. The tooling can’t always create scripts for all the state transitions without causing inconsistencies (data loss etc). This makes it impossible to integrate them into an automated deployment pipeline with confidence. Migrations on the other hand rely on transitions. Humans write code for the transitions. We can then have an automated process run the transitions to get the target database up to date. The philosophy behind this is looked at in detail here. Why DbUp? DbUp is a .NET library that helps you to deploy changes to SQL Server databases. It tracks which SQL scripts have been run already, and runs the change scripts that are needed to get your database up to date. I’ve used tool like FluentMigrator to implement migrations before as well. But I like the simplicity and support DbUp provides. This at the end of the day is just personal preference. Forward-only migrations DbUp doesn’t support down migrations though. IMO this is a good design decision. Down migrations are rarely tested and often only give a false sense of confidence of ability to rollback. If your up migration fails then the same set of data might cause your down migration to fail as well. To add to that not all migrations can have a paired down migration to consistently rollback changes. My preference is to have discipline around how you do migrations and not do destructive data changes in a single migration step. As an example if I wanted to refactor a single FullName field into FirstName and LastName columns, I wouldn’t delete the FullName column in the same migration that I introduce the other two columns. In the first migration I would introduce the columns, copy the data over and then rename the FullName column to something like FullName-Deprecated. I would then let the system run for a while until we are confident that everything is working as intended. If not, you can then create subsequent migrations to rectify the problems as you still have data in the original column. After a certain period, I would then create another migration to get rid of the now deprecated column. This gives greater resiliency to the system and allows you to be more confident about changes you do. Preparation We need to create our initial migration and the easiest way to do this is to consider your current state of the database as the initial state as far as the migrations are concerned. This means exporting all your database objects to a script. You can do this from SQL Server Management Studio (SSMS). Follow the steps here to Generate Scripts for all database objects you require to be created as a part of the initial migration (Tables, Views, Stored Procs, Functions etc). Save this file as Initial_Migration.sql. Once exported open the file and remove any SQL statements at the begining that create the Database or Logins. Lets assume the migrations are run on an empty database that has already been created. Now repeat the same steps as above with the following changes… Only select Functions and Stored Procedures to export this time as we need them to be PR review friendly. More about this later on. In the last window, go to Advanced -> Scripting Options -> set Check for object existance to True. Press OK. Now choose the option to create a Single file per object. (Remember we exported everything in to one file file in the previous step, we are exporting a subset of those things again to their individual files.) Save these in a folder called PostDeployment. Why are we duplicating the Stored Procedures in our all-in-one script and again in the individual files? Imagine that we only used migrations and no post deployment scripts. Unlike Table or View schemas you can’t alter a part of a Stored Procedure. So if you need to make a change you need to create a new migration with the full stored procedure definition. The person reviewing your change sees a new file and not just your change. It becomes very hard to review changes this way. So we will use these one file per object scripts to create reviewable post deployment scripts in our migration project. The idea is you keep modifying these post deployment scripts (Stored Procedures and Functions) rather than create a new migration for them. So the reviewer sees only your change in the context of the larger set. This will be very helpful if you have large Stored Procedures in your database project. There is more detail around the idea here. You should now have one big SQL file called Initial_Migration.sql and lot of smaller SQL files per Stored Procedure in the PostDeployment folder. Now we need to give the files inside that folder a sequence number. For example if your file is called CreateCustomer.sql we need to rename it to 000001 - CreateCustomer.sql and anything that depends on this script must have a sequence/order number higher than 000001. (This is because DbUp executes the scripts in their name order. We use the name as a way to enforce dependency order for post deployment scripts.) If you have a lot of files, it will be time consuming to do this manually. So I used this BulkRenaming tool to put the sequence/order suffix. I set it so that it suffixed files in increments of 5 (So sequence is 1, 6, 11, 16 etc). Once this was done I could then manually set the dependent order (by renaming the suffix manually) as required without having to rename every subsequent file. This is because the increment by 5 meant I could squeeze 4 more files between two other scripts if required. You might need a bit of manual effort here depending on how interconnected your Stored Procedures are. DbUp Migration Runner The folder structure for the migration runner is as follows. This is a simple console app targeting .NET framework but it’s trivial to change this to work with .NET Core. As you can see I put the Initial_Migration.sql in the Migrations folder and prefixed it with 000001. All subsequent migration will have the next sequence. I also have PostDeployment and PreDeployment folders. I’ve organized Stored Procedures and Functions in their own respective folders but that is optional. This is where you need to put the individually exported SQL files (PostDeployment folder) from our preparation steps before. There are environment specific folders which also have their own specific Pre and Post deployment folders. This gives you the ability to do customisations based on the environment if required. Once you see the migration runner code it will make more sense. DbUp uses the fully qualified file name with the namespace to figure out the order of execution and we make use of the filename suffix or nested folders (Namespace) to enforce our desired order. I use config transforms to transform the AppConfig based on the build quality via the SlowCheetah extension. App.config <?xml version="1.0" encoding="utf-8" ?> <configuration> <startup> <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.2" /> </startup> <connectionStrings> <add name="MyConnectionString" connectionString="Integrated Security=SSPI;Initial Catalog=MyDatabase;Data Source=.;" providerName="System.Data.SqlClient"/> </connectionStrings> <appSettings> <add key="AdditionalPreDeploymentNamespace" value="" /> <add key="AdditionalPostDeploymentNamespace" value="" /> <add key="Environment" value="" /> </appSettings> </configuration> App.Debug.config <?xml version="1.0" encoding="utf-8"?> <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform"> <appSettings> <add key="AdditionalPreDeploymentNamespace" value="Dev.PreDeployment" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/> <add key="AdditionalPostDeploymentNamespace" value="Dev.PostDeployment" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/> <add key="Environment" value="Dev" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/> </appSettings> </configuration> App.Release.config <?xml version="1.0" encoding="utf-8"?> <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform"> <connectionStrings> <add name="MyConnectionString" connectionString="#{databaseConnectionString}#" providerName="System.Data.SqlClient" xdt:Transform="SetAttributes" xdt:Locator="Match(name)" /> </connectionStrings> <appSettings> <add key="AdditionalPreDeploymentNamespace" value="#{releaseEnvironment}#.PreDeployment" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/> <add key="AdditionalPostDeploymentNamespace" value="#{releaseEnvironment}#.PostDeployment" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/> <add key="Environment" value="#{releaseEnvironment}#" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/> </appSettings> </configuration> My release pipeline injects the databaseConnectionString and releaseEnvironment variables in to the App.Release.config / AppName.exe.Config file upon building/deploying the app. That is why you see tokens like #{databaseConnectionString}#. I use this extension in Azure DevOps to do it. When running locally with the Debug build quality MSBuild uses the App.Debug.Config to transform your original config file. So you can easily run and test the migration runner locally. This makes your local development exprience consistent among the team. No more complex database update scripts to run. Now let’s look at my migration runner (Program.cs). You obviously need a reference to the DbUp-SqlServer NuGet as we use it to do the heavy lifting. static class Program { static int Main(string[] args) { var noWait = false; if (args.Length == 1 && args[0] == "--nowait") { noWait = true; } int exitCode = 0; try { WriteToConsole($"Database Migration Runner. " + $"Version={typeof(IEnvironmentsFolderMarker).Assembly.GetName().Version}"); // NOTE: PLEASE MAKE SURE YOUR SCRIPT IS MARKED AS EMBEDDED // https://www.c-sharpcorner.com/uploadfile/40e97e/saving-an-embedded-file-in-C-Sharp/ WriteToConsole("\nIMPORTANT: Please ensure your scripts are EMBEDDED in the executable."); var baseNamespace = typeof(Program).Namespace; var baseEnvironmentsNamespace = typeof(IEnvironmentsFolderMarker).Namespace; // You can use IConfiguration (Microsoft.Extensions.Configuration) to // achieve the same thing in a .NET Core project as shown here // https://stackoverflow.com/questions/38114761/asp-net-core-configuration-for-net-core-console-application var additionalPreDeploymentNamespace = ConfigurationManager.AppSettings["AdditionalPreDeploymentNamespace"]; var additionalPostDeploymentNamespace = ConfigurationManager.AppSettings["AdditionalPostDeploymentNamespace"]; var connectionString = ConfigurationManager.ConnectionStrings["MyConnectionString"]; WriteToConsole("\nListing variables...\n"); var variables = new Dictionary<string, string>(); foreach (var k in ConfigurationManager.AppSettings.AllKeys) { variables.Add(k, ConfigurationManager.AppSettings[k]); WriteToConsole($"${k}$ = \"{ConfigurationManager.AppSettings[k]}\""); // See how to use variables in your scripts: // https://dbup.readthedocs.io/en/latest/more-info/variable-substitution/ } if (!string.IsNullOrWhiteSpace(additionalPreDeploymentNamespace)) { additionalPreDeploymentNamespace = baseEnvironmentsNamespace + "." + additionalPreDeploymentNamespace; } if (!string.IsNullOrWhiteSpace(additionalPostDeploymentNamespace)) { additionalPostDeploymentNamespace = baseEnvironmentsNamespace + "." + additionalPostDeploymentNamespace; } if (!noWait) { Console.Write("\nPress return to run scripts..."); Console.ReadLine(); } // Pre deployments WriteToConsole("Start executing predeployment scripts..."); string preDeploymentScriptsPath = baseNamespace + ".PreDeployment"; RunMigrations(connectionString, preDeploymentScriptsPath, variables, true); if (!string.IsNullOrWhiteSpace(additionalPreDeploymentNamespace)) { RunMigrations(connectionString, additionalPreDeploymentNamespace, variables, true); } // Migrations WriteToConsole("Start executing migration scripts..."); var migrationScriptsPath = baseNamespace + ".Migrations"; RunMigrations(connectionString, migrationScriptsPath, variables, false); // Post deployments WriteToConsole("Start executing postdeployment scripts..."); string postdeploymentScriptsPath = baseNamespace + ".PostDeployment"; RunMigrations(connectionString, postdeploymentScriptsPath, variables, true); if (!string.IsNullOrWhiteSpace(additionalPostDeploymentNamespace)) { RunMigrations(connectionString, additionalPostDeploymentNamespace, variables, true); } } catch (Exception e) { WriteToConsole(e.Message, ConsoleColor.Red); exitCode = -1; } if (!noWait) { Console.Write("Press return key to exit..."); Console.ResetColor(); Console.ReadKey(); } return exitCode; } private static int RunMigrations(string connectionString, string @namespace, Dictionary<string, string> variables, bool alwaysRun = false) { WriteToConsole($"Executing scripts in {@namespace}"); var builder = DeployChanges.To .SqlDatabase(connectionString) .WithVariables(variables) .WithScriptsEmbeddedInAssembly( Assembly.GetExecutingAssembly(), file => { return file .ToLower() .StartsWith(@namespace.ToLower()); }) .LogToConsole(); builder = alwaysRun ? builder.JournalTo(new NullJournal()) : builder.JournalToSqlTable("dbo", "DatabaseMigrations"); var executor = builder.Build(); var result = executor.PerformUpgrade(); if (!result.Successful) { throw new Exception(result.Error.ToString()); } ShowSuccess(); return 0; } private static void ShowSuccess() { WriteToConsole("Success!", ConsoleColor.Green); } private static void WriteToConsole(string msg, ConsoleColor color = ConsoleColor.Green) { Console.ForegroundColor = color; Console.WriteLine(msg); Console.ResetColor(); } } Running the migration runner You can run it manually (From Visual Studio or Opening the executable file) or run it as a part of your release pipeline with the --nowait argument. Before you run it for the first time make sure to drop the existing database and create an empty database with the same name in the connection string. That’s it. You now have all the pieces required to convert your database project to one that uses database migrations instead. Conclusion We looked at why database migrations offer us a better alternative when it comes to maintaining and deploying database changes, how to export your existing database as migrations and post deployment scripts, how to structure your migration runner and how it can work in an automated release pipeline. Most of the steps we took are no different to how you would structure any other database migrations project. The interesting bit was how we leveraged the concept of post deployment scripts to put our Stored Procedures in and made the PR review process much more easier. This will make the process of conversion a much more acceptable prospect for a team that has been working with database projects and Stored Procedures in the past. I’ve had good traction with this approach at a couple of clients I’ve consulted for. Please leave any comments or feedback you have here.