AlexandreHTRB's blog

Periodic table of computing

2026-06-23T00:00:00Z

A few days ago, I was thinking about the different file extensions of programming languages and realized that they look like symbols of chemical elements. Then I thought: "we could make a periodic table of computing".

The best HTML periodic table I found on the Internet was the one from Adrian Roselli. It's responsive, works on vertical and horizontal, covers all the accessibility needs, supports dark mode, works offline and you can even print it!

I tried to make a table including everything I consider important on computing as a whole, in a broader sense. There was no space left for things like architecture and design patterns. Maybe that would be better in a periodic table of software, instead of computing.

Click here to view the table.

Making a holiday calendar with functional programming

2026-01-06T00:00:00Z

One of my personal projects is the website Feriados do Brasil, which is a calendar that displays national, state and municipal holidays of many cities in Brazil. While I was developing this project, I found an interesting use case to apply functional programming and wanted to share it in this article, showing how FP can provide unique and elegant solutions.

Fixed and moveable holidays

There are two kinds of holidays: fixed feasts, whose dates always happen in the same day of the same month, and moveable feasts, whose date varies every year. The most common moveable feasts in Brazil are Holy Friday, Carnival and Corpus Christi.

Although less common, there are many other moveable holidays, each with its own determinative rule. The table below shows some of them.

Moveable holiday	Date
Holy Friday	calculated via mathematical formula
Carnival Tuesday*	47 days before Easter
Corpus Christi	60 days after Easter
Pentecost	50 days after Easter
Sacred Heart of Jesus	8 days after Corpus Christi
Commerce Day on Rio de Janeiro	Third Monday of October
State holidays on Acre**	If within Tuesday and Thursday, they get transferred to Friday
State holidays on Santa Catarina	If on work day, they get transferred to Sunday

* Carnival is not an official holiday, although celebrated in almost the entire country.
** Except for the State Anniversary, on June 15th.

The question is: how can we store holidays in a database, considering that some are fixed and some are moveable? Furthermore, each moveable feast has its own rule?

Solution with functional programming

In this project, the listing and calculation of holidays are done entirely on the frontend, in JavaScript. Basically, a city's holidays are the union between national, state and municipal holidays. In the code, each set of holidays is represented by an array of functions.

const nationalHolidays = [
  ff(JANUARY, 1, "New Year"),
  mf(calculateCarnivalTuesday, "Carnival"),
  mf(calculateHolyFriday, "Holy Friday"),
  ff(APRIL, 21, "Tiradentes"),
  ff(MAY, 1, "Worker's Day"),
  ff(SEPTEMBER, 7, "Independency of Brazil"),
  ff(OCTOBER, 12, "Our Lady the Appeared"),
  ff(OCTOBER, 28, "Public Worker Day"),
  ff(NOVEMBER, 2, "All Souls' Day"),
  ff(NOVEMBER, 15, "Proclamation of the Republic"),
  ff(NOVEMBER, 20, "Black Consciousness Day"),
  ff(DECEMBER, 25, "Christmas")
];

ff and mf respectively represent fixed feast and moveable feast. These functions receive either the day and month or a function to calculate the date, and also a description. They return another function, which receives a numerical year and returns a holiday object, with the description and the exact date (JavaScript Date).

// fixed feast
function ff(month, day, desc) {
  return function (year) {
    return { date: new Date(year, month, day), description: desc };
  }
}

// moveable feast
function mf(funcCalcDate, desc) {
  return function (year) {
    return { date: funcCalcDate(year), description: desc };
  }
}

Here we use two functional programming concepts: higher-order functions and currying.

Higher-order functions are functions that receive other functions as input parameters or that return functions as outputs (or even both things at the same time). mf() receives funcCalcDate as an argument, which will receive a numerical year and return a Date object. Both mf() and ff() return functions.

Currying is a strategy of nesting functions inside other functions. In our case, ff and mf nest the internal function (year) {...}. Check out this other article to learn more about currying.

To obtain the holidays of a year, all we need is to iterate each function in the array, passing the year as argument:

let year = 2026;
let holidays = nationalHolidays.map((func) => func(year));

// returns:
[
  { date: new Date(2026, 1, 1), description: "New Year" },
  { date: new Date(2026, 2, 17), description: "Carnival" },
  { date: new Date(2026, 4, 3), description: "Holy Friday" },
  // ...
]

The website's code is public and is available on GitHub. Comments and variables' names are written in Portuguese.

Interesting reads

Excel as a frontend

2025-09-29T00:00:00Z

Corporate and personal finances. Governmental and management reports. Account statements. Inventory. Projections. Simulations. Lists of people. 90% of the global GDP. What do all those things have in common?

They go through Excel.

On the projects I have worked on, one of the most common features requested by clients is exporting data to Excel, especially for back-office operations, so analysts can more easily find information, make reports and calculate sums. Not just back-office — field agents, support and sales people also use it.

Some real-life examples:

Internal reports of transactions per individual;
Reports of sales grouped by business partners;
A salesman that needs to see which offers are available for a customer;
An economist that needs the inflation rates of previous months;
An HR analyst checking employees' worked hours;
An investment manager analysing the historical valuation of a product.

Excel is a great tool for visualizing data and is very intuitive, easily used by anyone.

Considering that many times it is the final data format, here is the question: instead of having a web app, backend and frontend, hosting, servers, etc., can't we just do everything on Excel? Wouldn't it be easier? In this article, let's explore this idea.

Connecting to external data sources

Excel can connect to external sources and download data from them. Many integrations are available, such as:

XMLs from the internet
XML, CSV, TXT and XLSX files
Flat files
Databases, like SQL Server, PostgreSQL, MySQL
Azure Blob Storage
HTTP endpoints
And others.

For example, let's create a local XML file with the content below:

<yearly_inflation>
<inflation><year>2014</year><rate>1.6%</rate></inflation>
<inflation><year>2015</year><rate>0.1%</rate></inflation>
<inflation><year>2016</year><rate>1.3%</rate></inflation>
<inflation><year>2017</year><rate>2.1%</rate></inflation>
<inflation><year>2018</year><rate>2.4%</rate></inflation>
<inflation><year>2019</year><rate>1.8%</rate></inflation>
<inflation><year>2020</year><rate>1.2%</rate></inflation>
<inflation><year>2021</year><rate>4.7%</rate></inflation>
<inflation><year>2022</year><rate>8.0%</rate></inflation>
<inflation><year>2023</year><rate>4.1%</rate></inflation>
<inflation><year>2024</year><rate>2.9%</rate></inflation>
</yearly_inflation>

On Excel, on the Data tab, let's click on New Query > From File > From XML, and then select our file.

After importing, the result will be:

Whenever you wish to pull data, you can go on Data > Refresh All, and the table will be updated with the latest remote data.

Advantages

Having Excel pull remote data has many advantages.

One of them is that we don't need to spend effort, time or infrastructure building a website to make that information available; all we need is the data source and a spreadsheet.

Even information security is solved for us:

If the data source is a remote file, we just need to control access permission on it.
If the data source is a database or HTTP endpoint, we can provide a personalized credential with permissions only for specific queries.
Windows Authentication is an option for the cases above.

Another advantage is that once downloaded, the data is available for usage even if there is no internet connection.

Last, having individual spreadsheets gives users a great degree of freedom, because they can customize them however they want.

Write operations

Can we use Excel to insert, delete and modify remote data? The answer is yes. UserForms and macros are the way to achieve that.

UserForms are custom dialogs for user data input. Macros on the other hand are the code behind, responsible for data validations and connecting to the database and server.

Userform example on Excel.

Keep in mind that whenever a change is required on those write integrations, users will need to download updated versions of the spreadsheet.

I recommend the following userform tutorials: one from Excel Easy and another from Wise Owl Training.

Final thoughts

For internal services within a business, or those that heavily rely on mathematics and reports, Excel can definitely be a good frontend solution.

When I discovered that Excel can connect to external data sources, I thought "how come I didn't know about this earlier?". It makes a lot of sense. Since it's a software not widely used by IT professionals, interesting features like this may go unnoticed by us, even though they could help us a lot.

What is currying?

2025-09-04T00:00:00Z

Currying is a strategy for sharing logic in programming. With it, functions are nested inside others, each one responsible for a parameter of the whole logic.

JavaScript example:

// curried function
function multiplyAndSum(a) {
  return function(b) {
    return function(x) {
      return a*x + b;
    }
  }
}

const f = multiplyAndSum(2)(1);
// f(x) = 2x + 1

console.log(f(4)); // 9

const g = multiplyAndSum(3);
// g(x) = 3x + b
// b unspecified

console.log(g(2)(1)); // 5, b = 2

With currying, derived functions can pre-fill arguments of a parent function and inherit the declaration of remaining parameters.

Curried functions can be imagined as matryoshka Russian dolls: one inside another, inside another.

Image source: AliExpress.

This style comes from functional programming and is very useful for mathematics.

Another example, in F#:

let volumePyramid(baseAreaFormula)(w)(l)(h) = 
  h * (baseAreaFormula(w)(l))/3.0

let areaRectangle(w)(l) = w * l

let areaTriangle(w)(l) = (w * l)/2.0

let areaCircle(_)(r) = Math.PI * r**2.0

let volumeRectangularBasePyramid =
  volumePyramid(areaRectangle)

let volumeTriangularBasePyramid =
  volumePyramid(areaTriangle)

let volumeCone =
  volumePyramid(areaCircle)(0)

// usage
let v1 = volumeRectangularBasePyramid(7.0)(4.0)(11.0)
let vc = volumeCone(9.0)(12.0) // radius 9, height 12

The less a parameter varies across functions, the further to the left it should be in a curried declaration.

Containers should be an operating system responsibility

2025-06-10T00:00:00Z

Not before 2018, gradually, my colleagues at work started talking about Docker, a technology for setting up environments for applications on servers. In less than two years, Docker had become the standard and new backend apps included Dockerfiles and YAMLs for Kubernetes.

Even though I understood what containers were for, part of me said that they were an overweight solution, and the problems solved by them should actually be solved by operating systems. My opinion still stands.

In this article, let's cover some alternatives to containers and propose a solution that could be implemented by operating systems.

Beginning from the end

After all, why do we use containers? The majority of the answers will be: "To run my app in the cloud".

Docker solved two problems: environment setup and safe execution.

Docker images include everything needed to run the app, which usually is the app runtime, its dependencies and the user space of an operating system. This can be sensitive, especially for Linux, because the dependency tree may be fragile and break when an inner dependency is updated. Having a rigid and reproducible dependency tree ensures environment stability.

Safe execution results from the container being a process with an isolated file system and network, reducing the risk of attacks and infections on the host machine.

Environment setup, without containers

There are many ways to setup an environment without resorting to containers.

One is actually installing the required dependencies on the host machine.

Another is self-contained deployment, where the compilation includes the runtime alongside or inside the program. Thus, the target machine does not require the runtime to be installed to run the app.

Some languages offer ahead-of-time compilation (AOT), which compiles into native machine code. This allows program execution without runtime, and lowers the number of required external dependencies. As a positive side effect, AOT compilation also produces apps with better performance and reduced memory usage.

An operating system's Docker image corresponds to its user space, which is the operating system minus its kernel. This includes essential programs, libraries, services and some other files.

Loading an entire operating system's user space for each container instance wastes memory and disk space. Ubuntu's Docker image, for instance, uses 188MB, which is a lot. There are ultra-light images that leave only what is strictly necessary; for example, Alpine Linux image uses only 5MB. Nevertheless, it's still an additional weight.

The solutions above show how to mitigate the risk of a fragile dependency tree.

Safe execution, without containers

This is a bit more complicated. Here, we want to limit the access to the host machine's file system and network.

On most operating systems, file system access control is done at user-level. In order to restrict a program's access to specific files and directories, we need to create a user (or user group) with those rules and ensure the program always runs under that user.

Network restriction, on the other hand, is done via firewall, with user and program-scoped rules.

Proposal: Execution manifests

A suggestion to be implemented by operating systems would be execution manifests, that clearly define how a program is executed and its system permissions.

The execution manifest could also define which devices and peripherals a program can interact with. This would make it useful for desktop applications too.

The manifest file could be signed with a public-private key pair for authentication. A hash would verify that the executable file corresponds to the one expected in the manifest.

manifestVersion: v1
author: myCompany
name: myApp
run: ./myapp
hash: <executable_file_hash> # optional

allowedDirectories:
  - name: appDir
    path: ./**/ # all subdirectories
    permission: read
  - name: sharedFolder
    path: \\EnvGeoServer\Repository\Tools\admin\
    permission: readWrite

allowedNetwork:
  - name: localhost
    direction: inbound
    ip: 127.0.0.1
    protocol: tcp
    port: 8080

allowedDevices:
  - mouse
  - keyboard
  - audioOut
  - display

Sources and interesting reads

Structured logging in .NET with NativeAOT

2025-04-15T00:00:00Z

NativeAOT and trimming are two new .NET compiler options that came up over the last years, resulting in applications with smaller sizes, faster execution and lower memory usage. In a short explanation, NativeAOT compiles to native machine code and trimming removes unused code.

To apply these options, all code paths must be statically determinable, i.e., the program cannot use reflection, under the risk of having removed important parts of code and having undesirable behaviours during execution.

Currently, most logging libraries do use reflection, mainly to support structured logging. In this article, we will see how to make a custom structured logger, compatible with NativeAOT and trimming.

Structured logging

In a structured log, messages are formatted in a key-value structure, like JSON or XML. This format allows easier processing by databases and logging engines.

Consider the code below:

string city = "Bento Gonçalves / RS";

WeatherForecast forecast = new(
    minTemp: 16,
    maxTemp: 21,
    rainMm: 32.5,
    windKph: 6);

using (logger.BeginScope("{@Forecast}", forecast))
{
    logger.LogInformation(
        "Weather forecast for {City} on {Date}",
        city, DateTime.Today);
}

JSON structured log:

{
  "Timestamp": "2025-04-12T20:25:57.8532799Z",
  "Level": "Information",
  "Template": "Weather forecast for {City} on {Date}",
  "City": "Bento Gonçalves / RS",
  "Date": "2025-04-13",
  "@Forecast": {
    "MinTemperatureInC": 16,
    "MaxTemperatureInC": 21,
    "RainInMm": 32.5,
    "WindInKph": 6
  }
}

When a parameter between braces is prefixed with an '@' sign (arroba), it is treated as a complex object and its internal properties are read as key-value pairs. Without this sign, the parameter is treated as a simple value, like a number, a Guid or a string.

Scopes are complementary information, that may be relevant.

`ILogger` interface in .NET

Modern .NET makes extensive usage of the Microsoft.Extensions.Logging.ILogger interface (source code):

namespace Microsoft.Extensions.Logging;

public interface ILogger
{
    void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter);

    bool IsEnabled(LogLevel logLevel);

    IDisposable? BeginScope<TState>(TState state) where TState : notnull;
}

Although not explicit in the interface, TState virtually always is a type inheriting from IReadOnlyList<KeyValuePair<string, object?>>, this is verifiable by looking at .NET runtime source code: LoggerExtensions and FormattedLogValues classes.

TState contains key-value pairs, including the message template and parameters' values. This is valid for both Log and BeginScope methods.

Reflectionless logger

The question here is how to format a log message in a generic way and without resorting to reflection.

As a solution, we will make a JSON logger with the help of source generators. On the JsonSerializerContext, we'll need to include all types of object that may come as log parameters.

Things to consider

Our logger will implement the ILogger interface.
We will consider that TState always inherits from IReadOnlyList<KeyValuePair<string, object?>>.
Let's filter messages to ignore the ones that are not relevant, and also to not need to include many object types on our JsonSerializerContext. This is important especially for ASP.NET projects.
In this example, log lines will be printed on the console.
We'll use the CLEF format.
The scopeProvider is responsible for reading the captured scopes. We'll use the LoggerExternalScopeProvider default implementation.

Code

On the HostBuilder / WebApplicationBuilder setup (Program.cs):

webAppBuilder.Logging
    .ClearProviders()
    .AddProvider(new ClefLoggerProvider());

Logger provider:

public sealed class ClefLoggerProvider : ILoggerProvider
{
    public ILogger CreateLogger(string categoryName) =>
        new ClefLogger(categoryName, new LoggerExternalScopeProvider());

    public void Dispose() { }
}

JsonSerializerContext:

[JsonSerializable(typeof(Dictionary<string, object?>))]
[JsonSerializable(typeof(double))]
[JsonSerializable(typeof(long))]
// add types here, like:
[JsonSerializable(typeof(WeatherForecast))]
// JSON options
[JsonSourceGenerationOptions(
    WriteIndented = false,
    PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase,
    DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)]
public partial class AppJsonSrcGenContext : JsonSerializerContext
{
}

Logger:

public sealed class ClefLogger(
    string categoryName,
    IExternalScopeProvider? scopeProvider) : ILogger
{
    private static readonly string[] logsToIgnore = [
        "Request starting",
        "Executing",
        "Setting HTTP status code",
        "Executed",
        "Writing value",
        "Write content",
        "Sending file",
        "Request reached the end of the middleware pipeline without being handled by application code."
    ];

    public IDisposable? BeginScope<TState>(TState state)
        where TState : notnull =>
        scopeProvider?.Push(state) ?? default;

    public bool IsEnabled(LogLevel logLevel) => true;

    private static bool FilterProperties(KeyValuePair<string, object?> kv) =>
        kv.Key != "{OriginalFormat}"
        && kv.Value != null
        && (kv.Value is string s ? !string.IsNullOrEmpty(s) : true);

    private static bool ShouldIgnoreLog(string? originalFormat) =>
        originalFormat is null || logsToIgnore.Any(originalFormat.StartsWith);

    public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
    {
        if (state is not IEnumerable<KeyValuePair<string, object?>> msgProps)
            return;

        string? originalFormat = (string?)msgProps.FirstOrDefault(kv => kv.Key == "{OriginalFormat}").Value;

        if (ShouldIgnoreLog(originalFormat))
            return;

        List<IEnumerable<KeyValuePair<string, object?>>>? scopes = new();
        scopeProvider?.ForEachScope((scope, st) =>
        {
            if (scope is IEnumerable<KeyValuePair<string, object?>> scopeItems)
            {
                scopes.Add(scopeItems.Where(FilterProperties));
            }
        }, state);

        IEnumerable<KeyValuePair<string, object?>> standardProps =
        [
            new("@i", eventId.Name),
            new("@t", DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ss.fffffffZ")),
            new("@c", categoryName),
            new("@l", Enum.GetName(logLevel)),
            //new("@m", formatter(state, exception)),
            new("@mt", originalFormat),
            new("@x", exception?.ToString())
        ];

        var msg = standardProps
            .Concat(msgProps.Where(FilterProperties))
            .Concat(scopes.SelectMany(x => x))
            // avoiding repeated keys
            .DistinctBy(kv => kv.Key)
            // removing null values for JSON
            .Where(x => x.Value is not null)
            .ToDictionary();

        try
        {
            WriteLogLine(msg);
        }
        catch (Exception jsonEx)
        {
            msg = standardProps
                .Append(new("WARNING", "Could not serialize original message to JSON."))
                .Append(new("JsonException", jsonEx.ToString()))
                // removing null values for JSON
                .Where(x => x.Value is not null)
                .ToDictionary();

            WriteLogLine(msg);
        }
    }

    // TODO: Use queue to avoid concurrency
    // Change output if you want: file, logging engine, etc.
    private void WriteLogLine(Dictionary<string, object?> msg)
    {
        string json = JsonSerializer.Serialize(msg, AppJsonSrcGenContext.Default.DictionaryStringObject);
        Console.WriteLine(json);
    }
}

The logger above works with NativeAOT and trimming enabled.

If there is any JSON serialization error, the catch clause formats a new log message without extra properties and scopes (that may be responsible for the error) and inserts the JsonException related to the problem.

Example of generated log

{
  "@t": "2025-04-15T12:09:58.4990143Z",
  "@c": "Microsoft.AspNetCore.Hosting.Diagnostics",
  "@l": "Information",
  "@mt": "Request finished {Protocol} {Method} {Scheme}://{Host}{PathBase}{Path}{QueryString} - {StatusCode} {ContentLength} {ContentType} {ElapsedMilliseconds}ms",
  "ElapsedMilliseconds": 124.2172,
  "StatusCode": 201,
  "ContentType": "application/json; charset=utf-8",
  "Protocol": "HTTP/2",
  "Method": "POST",
  "Scheme": "https",
  "Host": "localhost:5001",
  "Path": "/",
  "RequestId": "0HNBS2ORC5B08:00000001",
  "RequestPath": "/"
}

Sources and interesting reads

Least squares fitting with functional programming

2025-03-07T00:00:00Z

Let's suppose you have a dataset of numerical values and you would like to make a future estimate based on them. Some real-life applications include:

Product pricing;
Historical production and demand;
Financial assets valuation;
Calibration of a machine through time;
Concentration of chemical reagents and products;
Disease transmission rates;
Water reservoir levels;
Electricity demand;
Fuel consumption.

The least squares fitting is a mathematical tool that generates an equation close to existing measurements. Many types of curves can be used, such as linear, polynomial, exponential, logarithmic, and many others.

Consider the graph below, sales per month. The blue line is the linear approximation and the red line corresponds to the real values.

---
config:
    xyChart:
        width: 540
    themeVariables:
        xyChart:
            backgroundColor: "#00000000"
            plotColorPalette: "#C00, #22F"
---
xychart-beta
    title "Sales per month ($)"
    x-axis "month" [1,2,3,4,5,6,7,8,9,10,11,12]
    y-axis 0 --> 50000
    line [17000, 24000, 16000, 30000, 35000, 26000, 29000, 40000, 44000, 35000, 41000, 47000]
	line [18461.538, 20923.076, 23384.614, 25846.152, 28307.690, 30769.228, 33230.766, 35692.304, 38153.842, 40615.380, 43076.918, 45538.456]
	%% =2461,538 * x + 16000

Theory

Consider f(x) the equation we want to generate and (x_i, y_i) the measurements we have. The best equation is the one where the differences between approximations and measurements are the smallest possible; in other words:

|f(x_i) − y_i| is the distance between approximation and reality;
Σ is the summation.
Σ|f(x_i) − y_i| should be the closest to zero.

Let's name SSE the sum of squared errors: SSE = Σ(f(x_i) − y_i)². As our objective is to measure the distance between approximation and reality, it's easier to simply power square the difference, instead of applying the absolute function.

We need to apply an absolute or power square operation for distance measurement because a simple subtraction could result in a negative value, which isn't valid for distances.

The minimum of SSE depends on the type of desired approximation:

Linear: f(x) = ax + b
Exponential: f(x) = a ⋅ e^bx
Power law: f(x) = ax^b
Logarithmic: f(x) = a + b ⋅ ln(x)

Substituting f(x) by the functions above in SSE, we'll see that it depends on variables a and b. In the case of linear approximation: SSE(a,b) = Σ(ax_i + b − y_i)². x_i and y_i are constants, due to being known values.

The minimum of a function happens on the point where its derivative (slope) equals zero:

---
config:
    xyChart:
        width: 540
    themeVariables:
        xyChart:
            backgroundColor: "#00000000"
            plotColorPalette: "#CC0000, #f59316, #f59316, #f59316"
---
xychart-beta
    title "y = x²"
    x-axis "x" -6 --> 6
    y-axis "y" 0 --> 40
    line [36,25,16,9,4,1,0,1,4,9,16,25,36]
	line [27,21,15,9,3,-3,-9,-15,-21,-27,-33,-39,-45]
    line [0,0,0,0,0,0,0,0,0,0,0]
    line [-45,-39,-33,-27,-21,-15,-9,-3,3,9,15,21,27]

In red: y = x². In orange: the derivatives (slopes) for x = −3, 0 and 3.

SSE will have its minimum point where the partial derivatives relative to a and b are zero, solving the following system:

∂SQE(a,b) / ∂a = 0

∂SQE(a,b) / ∂b = 0

This is the basics for the least squares fitting method. If you are interested and would like to know more, there are links in the end of this page with additional explanations on this topic. Particularly, I recommend this lesson by Stan Brown.

Functional programming

Functional programming is a paradigm that focuses on data transformations: like in a mathematical function, a value x is transformed by f(x). In functional code, transformations are often chained, e.g., f(g(h(x))).

In this article, we will use the F# language, that uses the .NET runtime.

This post explains how to set up your environment and contains tips on the language syntax.

Linear approximation

The equation will be f(x) = ax + b.

From a set of values (x_i, y_i), a and b are determined by:

The F# code below calculates the linear approximation.

x and y are the input values.
Seq.sumBy sums the results of a function applied over elements of a sequence:
- x = [1, 2, 3]
- x |> Seq.sumBy(fun i -> 2 * i) = 12
- (2*1 + 2*2 + 2*3 = 12)
Seq.zip makes a sequence of pairs coming from two other sequences:
- x = [1, 2, 3]
- y = [9, 8, 7]
- Seq.zip(x)(y) = [ (1, 9), (2, 8), (3, 7) ]

// s = sum
let s(v: double[]): double =
    v |> Seq.sum

// sm = sum of multiplications
let sm(v1: double[])(v2: double[]): double =
    Seq.zip(v1)(v2)
    |> Seq.sumBy (fun (x, y) -> x * y)

let linearApproximation(x: double[])(y: double[]): (double * double) =
    let n = x.Length |> double
    let sx = s(x)
    let sx2 = sm(x)(x)
    let sy = s(y)
    let sxy = sm(x)(y)
    let d = n*sx2 - sx**2
    let a = (n*sxy - sx*sy)/d
    let b = (sx2*sy - sxy*sx)/d
    (a,b)

Exponential approximation

The equation will be f(x) = A ⋅ e^bx.

e is the natural base, approximately equal to 2.71828.

From a set of values (x_i, y_i), a and b are determined by:

The F# code below calculates the exponential approximation.

ln is the natural logarithm, on base e.
A = e^a.
Seq.zip3 follows the same logic as Seq.zip, but forms a sequence of groups of three, coming from three other sequences.
Important: the values of y need to be all greater than 0, because there is no logarithm of zero or negative numbers.

// s = sum
let s(v: double[]): double =
    v |> Seq.sum

// sm = sum of multiplications
let sm(v1: double[])(v2: double[]): double =
    Seq.zip(v1)(v2)
    |> Seq.sumBy (fun (x, y) -> x * y)

let ln = System.Math.Log

// smm = sum(v1 * v2 * v3)
let smm(v1: double[])(v2: double[])(v3: double[]): double =
    Seq.zip3(v1)(v2)(v3)
    |> Seq.sumBy (fun (a,b,c) -> a * b * c)

// smln = sum(v1 * ln(v2))
let smln(v1: double[])(v2: double[]): double =
    Seq.zip(v1)(v2)
    |> Seq.sumBy (fun (a,b) -> a * ln(b))

// smmln = sum(v1 * v2 * ln(v3))
let smmln(v1: double[])(v2: double[])(v3: double[]): double =
    Seq.zip3(v1)(v2)(v3)
    |> Seq.sumBy (fun (a,b,c) -> a * b * ln(c))

// exponential: y = A * e^(b*x)
let exponentialApproximation(x: double[])(y: double[]): (double * double) =
    let sx2y = smm(x)(x)(y)
    let sylny = smln(y)(y)
    let sxy = sm(x)(y)
    let sxylny = smmln(x)(y)(y)
    let sy = s(y)
    let d = sy*sx2y - sxy**2
    let a = (sx2y*sylny - sxy*sxylny)/d
    let b = (sy*sxylny - sxy*sylny)/d
    let A = System.Math.Exp(a)
    (A, b)

Case study: Silver mining

Pirquitas mine, Jujuy, Argentina.

Let's extract data from the Our World in Data website on world silver mining production, between 1900 and 2023, and let's make the linear and exponential approximations.

CSV data available here.

Results:

f(x) is the mining production in metric tonnes, x is the year.
Linear: f(x) = 162.897x − 308090.816
Exponential: f(x) = 0.000000002257 ⋅ e^0.01485166x

---
config:
    xyChart:
        width: 540
    themeVariables:
        xyChart:
            backgroundColor: "#00000000"
            plotColorPalette: "#CC0000, #2222FF, #00CC00"
---
xychart-beta
    title "Silver mining production (metric tonnes)"
    x-axis "Year" 1900 --> 2023
    y-axis 0 --> 30000
    line [ 5400, 5380, 5060, 5220, 5110, 5360, 5130, 5730, 6320, 6600, 6900, 7040, 6980, 7010, 5240, 5730, 5250, 5420, 6140, 5490, 5390, 5330, 6530, 7650, 7450, 7650, 7890, 7900, 8020, 8120, 7740, 6080, 5130, 5340, 5990, 6890, 7920, 8640, 8320, 8300, 8570, 8140, 7780, 6380, 5740, 5040, 3970, 5220, 5440, 5570, 6320, 6210, 6700, 6900, 6670, 7000, 7020, 7190, 7430, 6910, 7320, 7370, 7650, 7780, 7730, 8010, 8300, 8030, 8560, 9200, 9360, 9170, 9380, 9700, 9260, 9430, 9840, 10300, 10700, 10800, 10700, 11200, 11500, 12100, 13100, 13100, 13000, 14000, 15500, 16400, 16600, 15600, 14900, 14100, 14000, 14900, 15100, 16500, 17200, 17600, 18100, 18700, 18800, 18800, 20000, 20800, 20100, 20800, 21300, 22300, 23300, 23300, 24300, 26700, 28000, 27600, 28600, 26500, 25900, 25800, 23500, 25000, 25600, 26000 ]
	line [ 1413.484, 1576.381, 1739.278, 1902.175, 2065.072, 2227.969, 2390.866, 2553.763, 2716.66, 2879.557, 3042.454, 3205.351, 3368.248, 3531.145, 3694.042, 3856.939, 4019.836, 4182.733, 4345.63, 4508.527, 4671.424, 4834.321, 4997.218, 5160.115, 5323.012, 5485.909, 5648.806, 5811.703, 5974.6, 6137.497, 6300.394, 6463.291, 6626.188, 6789.085, 6951.982, 7114.879, 7277.776, 7440.673, 7603.57, 7766.467, 7929.364, 8092.261, 8255.158, 8418.055, 8580.952, 8743.849, 8906.746, 9069.643, 9232.54, 9395.437, 9558.334, 9721.231, 9884.128, 10047.025, 10209.922, 10372.819, 10535.716, 10698.613, 10861.51, 11024.407, 11187.304, 11350.201, 11513.098, 11675.995, 11838.892, 12001.789, 12164.686, 12327.583, 12490.48, 12653.377, 12816.274, 12979.171, 13142.068, 13304.965, 13467.862, 13630.759, 13793.656, 13956.553, 14119.45, 14282.347, 14445.244, 14608.141, 14771.038, 14933.935, 15096.832, 15259.729, 15422.626, 15585.523, 15748.42, 15911.317, 16074.214, 16237.111, 16400.008, 16562.905, 16725.802, 16888.699, 17051.596, 17214.493, 17377.39, 17540.287, 17703.184, 17866.081, 18028.978, 18191.875, 18354.772, 18517.669, 18680.566, 18843.463, 19006.36, 19169.257, 19332.154, 19495.051, 19657.948, 19820.845, 19983.742, 20146.639, 20309.536, 20472.433, 20635.33, 20798.227, 20961.124, 21124.021, 21286.918, 21449.815 ]
	line [ 4059.94, 4120.69, 4182.34, 4244.92, 4308.44, 4372.90, 4438.33, 4504.74, 4572.14, 4640.55, 4709.99, 4780.46, 4851.99, 4924.58, 4998.27, 5073.06, 5148.96, 5226.00, 5304.20, 5383.56, 5464.11, 5545.87, 5628.85, 5713.07, 5798.55, 5885.31, 5973.37, 6062.75, 6153.46, 6245.53, 6338.98, 6433.83, 6530.10, 6627.80, 6726.97, 6827.62, 6929.78, 7033.47, 7138.71, 7245.52, 7353.93, 7463.96, 7575.64, 7689.00, 7804.04, 7920.81, 8039.32, 8159.61, 8281.70, 8405.62, 8531.39, 8659.04, 8788.60, 8920.10, 9053.56, 9189.03, 9326.52, 9466.07, 9607.70, 9751.46, 9897.36, 10045.45, 10195.76, 10348.31, 10503.15, 10660.30, 10819.81, 10981.70, 11146.01, 11312.78, 11482.05, 11653.85, 11828.22, 12005.20, 12184.83, 12367.15, 12552.19, 12740.00, 12930.62, 13124.10, 13320.47, 13519.78, 13722.07, 13927.38, 14135.77, 14347.28, 14561.95, 14779.83, 15000.97, 15225.43, 15453.24, 15684.46, 15919.13, 16157.32, 16399.08, 16644.45, 16893.49, 17146.26, 17402.81, 17663.20, 17927.49, 18195.73, 18467.98, 18744.31, 19024.77, 19309.43, 19598.34, 19891.58, 20189.21, 20491.29, 20797.89, 21109.08, 21424.93, 21745.50, 22070.86, 22401.10, 22736.28, 23076.47, 23421.75, 23772.20, 24127.89, 24488.90, 24855.32, 25227.21 ]
	%% linear: 162,897 * year - 308090,816
	%% exponential: =(0,000000002257)*e^(year*0,01485166)

Red line: year production; blue line: linear approximation; green line: exponential approximation.

Sources and interesting reads

Interfaces as a principle of engineering

2025-02-20T00:00:00Z

An interface is how two different things communicate with each other.

In the field of engineering, interfaces are very useful for abstracting the macro logic of a system. With them, components can focus on specific tasks, and interfaces define how they connect to each other.

flowchart LR
  A
  B
  subgraph S[S]
    direction RL
    a
    b
  end
  a-.->A  
  b-.->B
  classDef abstract stroke:#f66,stroke-width:2px,stroke-dasharray: 5 5
  class a,b abstract;

In the diagram above, the system S delegates tasks a and b to components A and B, respectively. Arrows represent interfaces.

Let's analyse here some benefits of interfaces.

Reuse

Interfaces enable a component to be designed to support multiple systems.

Repurposing

It's similar to reuse, but reuse is about designing a component to support many use cases; repurposing is about using an existing component, instead of creating a new one.

Interchangeability

Parts can be replaced with equivalent ones — from a different supplier, for example.

Work parallelism

Different teams can work in parallel on each component, resulting in a faster final product delivery.

Risk mitigation

It's related to work parallelism. Problems on one work front do not affect other work fronts.

Machinery costs

The production of a part requires machines, that cost money. Delegating this production to other teams or companies avoids this cost.

Specialization

Teams and companies with specific knowledge of a type of component produce them with higher quality.

Unit tests

Interfaces enable isolated testing of a component, such that external dependencies are mocked, i.e., replaced with simulations or imitations. Because these are individualized tests, there is a higher level of attention and focus on the component itself, and it's not necessary for the entire system to be finished in order to test the component.

This also contributes to shift-left testing, which is the anticipation of quality assurance on the workflow — unit tests highlight problems earlier in the process.

Comparison between shift-left and traditional testing approaches.

Examples of interfaces in engineering

Computers

The hardware of a computer is extremely modular, making it easy to swap parts during repairs or upgrades.

Diagram of a computer's motherboard.

Any processor with the same socket as the motherboard can be used;
Any RAM memory stick with the same socket (e.g. DIMM, SO-DIMM) and the same class (e.g. DDR3, DDR4, DDR5) as the motherboard is accepted;
Any SATA device is accepted: hard drives, CD/DVD drives.
Any power supply can be used, as long as it provides power through the motherboard's corresponding connector (e.g. ATX);
And last but not least, a motherboard can be changed while keeping other parts.

Plumbing and electrical systems

Plumbing and electrical systems are also highly modularized.

Circuit breakers and valves enable subsytems to be isolated and repaired without affecting others.
Parts can be easily replaced.
- For electrical: outlets, lamps, switches, circuit breakers; parts just need to comply with voltage and maximum current specifications.
- For plumbing: siphons, faucets, shower heads; parts just need to comply with the plumbing diameter.
The exception is repairs on conduits and pipes, usually located inside walls and under floors, but issues with those parts tend to be rarer.

Possible drawbacks

Energy and material efficiency

Connections consume energy and materials during construction and operation phases. In condensed or directly linked components, these losses are mitigated or even eliminated.

System cohesion

In every connection, there is the risk of component mismatch. Reducing the number of intermediaries makes the system more cohesive.

Freedom of innovation

Interfaces defined by third parties may not fulfill a use case in the best possible way. In these situations, it may be beneficial to deviate from existing standards and create your own interfaces.

Case study: Aviation fuel in airports

Airports need aviation fuel (usually kerosene) to refill aircrafts' tanks.

At small and medium-sized airports, kerosene is delivered by tanker trucks to storage tanks. At large airports with high fuel demand, however, kerosene is supplied via pipelines, directly from refineries.

These pipelines are expensive to build, but the high demand justifies their cost, and the operation is cheaper and more efficient than relying on tanker trucks.

During the 2018 Brazil truck drivers' strike, airports that relied on tanker trucks suffered from fuel shortages and some of them had their operations paralysed. On the other hand, airports served by pipelines (Cumbica and Galeão) kept their operations as usual.

Conclusion

The concept of interfaces originates from the idea of interchangeable parts, and their use results in better, more reliable systems, that are cheaper, easier to maintain, and quicker to build. Moreover, interfaces contribute to the reuse of parts, preventing the need to replace entire systems.

In some cases, nevertheless, creating your own interfaces, or "cutting out the middleman", can make your system more cohesive and efficient.

Sources and interesting reads

Introduction to FFmpeg

2025-01-13T00:00:00Z

Introduction

Do you know that funny video on the Internet? Or the movie in the theatre? If you've watched any digital video over the last 15 years, there's a good chance that FFmpeg was involved at some point, whether for editing, playback, or distribution.

FFmpeg is a video and audio transformation software that is used by multimedia companies like YouTube, TikTok, Vimeo, and many others.

In this article, we'll learn how to use it through practical examples.

Index

FFmpeg installation

Linux

Debian / Ubuntu:

sudo apt update
sudo apt install ffmpeg
ffmpeg -version

Fedora:

sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf upgrade --refresh
sudo dnf install ffmpeg --allowerasing
sudo dnf install ffmpeg-devel --allowerasing
ffmpeg -version

Arch Linux:

sudo pacman -Sy 
sudo pacman -S ffmpeg 
ffmpeg -version

macOS

Install Brew.
Execute on Terminal:

brew update
brew install ffmpeg
ffmpeg -version

Windows

You can install via Winget (available for Windows 11, Windows Server 2025):

winget install --id=Gyan.FFmpeg -e

Or install manually:

Download the latest FFMpeg for Windows release (link). The essentials version is the most appropriate for most users.
Unzip the file.
Add the bin folder to PATH. For that:
- On Windows Explorer, mouse right-click on This Computer > Properties.
- On the next window, click on Advanced system configurations.
- On System Properties, click on Environment Variables....
- On User Variables for X, select the line with Path and click on Edit....
- Add the bin folder path to the list. It will be something like: C:\Users\user\Downloads\ffmpeg-7.0.1-essentials_build\bin (change for the path in your machine).
- Click OK on the windows to save the changes.

How to use

FFmpeg is used via Terminal, Command Prompt, or PowerShell. In your computer, open one of those programs.
In the examples below, the file paths are specified without their folders, which implies the current folder in the command line. To change the current directory:

cd C:\Users\user\Videos\

If the files are (or will be generated) in other folders, the full file path is required:

ffmpeg -i "C:\Videos\input.mov" -c copy "C:\Videos\output.mp4"

Operations

Change file format

Video file formats usually are just transport containers, whereas their content is determined by the codecs. Some video formats: MP4, AVI, MKV, MOV, WebM.

The most popular video codecs are:

H.264, it's one of the oldest and is widely supported.
H.265, more recent and has up to 50% smaller file sizes than H.264. However, some browsers and players do not support it.
AV1 is a newer codec. It has a royalty-free license and promises better compression than H.265, but FFmpeg doesn't support it very well yet (01/06/2025).
VP9, used by YouTube and in .webm videos.

The main audio codecs are:

MP3, widely supported, lossy, good compression, good quality.
AAC, used in MP4, compression and quality a bit better than MP3.
Opus, lossy with good quality.
FLAC, lossless, but with compression.

File conversion keeping the codecs:

ffmpeg -i input.mov -c copy output.mp4

File and codec conversion (transcoding):

-c:v: video codec
-c:a: audio codec
-b:a: audio bitrate

ffmpeg -i input.mov -c:v h264 -c:a libopus -b:a 96k output.mp4

Extract audio from a video

Full extraction, keeping the codec:

MP4 -> AAC or M4A
MKV -> many codecs
WebM -> Opus or Ogg Vorbis
MOV -> ALAC

ffmpeg -i input.mp4 -vn -c:a copy output.m4a

Extraction with transcoding. -q:a is the quality, from 0 to 9; lower means better.

ffmpeg -i input.mp4 -vn -c:a libmp3lame -q:a 2 output.mp3

Audio extraction from part of the video. -ss is the beginning, -to is the end.

ffmpeg -i input.mp4 -ss 00:00:07 -to 00:00:10.7 -vn -c:a libmp3lame -q:a 2 output.mp3

Reduce file size

There are many ways to reduce the file size of a video. In this section, we'll cover compression.

To fully understand how it works, I recommend reading the codec official docs (such as: H.264 and H.265).

Limit with CRF. The -crf flag, Constant Rate Factor, controls quality. Lower value means less compression and higher quality. The scale goes from 0 to 51, default value is 23.

ffmpeg -i input.mp4 -c:v libx264 -crf 23 -pix_fmt yuv420p -c:a copy output.mp4

-pix_fmt yuv420p is necessary for compatibility with Apple QuickTime and some other players.

Limit bitrate. You can limit the bitrate, making the file size more predictable. This option can be combined with CRF. In the example below, the encoder aims to limit the maximum bitrate in 1Mbit/s.

ffmpeg -i input.mp4 -c:v libx264 -crf 23 -maxrate 1M -bufsize 2M -pix_fmt yuv420p -c:a copy output.mp4

Transcode to more modern codecs. H.265, for example, generates lower files sizes than H.264. CRF scale goes from 0 to 51, default value is 28.

Keep in mind that not every player and browser supports H.265.

ffmpeg -i input.mp4 -c:v libx265 -crf 28 -tag:v hvc1 -c:a copy output.mp4

-tag:v hvc1 is for compatibility with Apple QuickTime.

Cut part of a video

ffmpeg -i input.mp4 -ss 00:00:19.4 -to 00:03:50 -c copy output.mp4

The time parameters can be placed before or after the input parameter.

When placed before (-ss -to -i), the operation is faster because the cut is made on the input, which is processed afterwards; however, the cut may be imprecise, and there may dyssynchrony between audio and video.

When placed after (-i -ss -to), the cut is more precise and there is no risk of dyssynchrony, but the operation is slower because the cut is made on the output, after the video is processed.

When in doubt, you can first try the input cut (first option) and check the result. If the result is not good, then try the output cut (second option).

Increase and decrease audio volume

The example below increases the audio volume by 2.4 times.

To reduce it by half, use 0.5.

ffmpeg -i input.mp4 -af "volume=2.4" output.mp4

Resize frames

Fixed dimensions, width x height:

ffmpeg -i input.mp4 -lavfi "scale='720:480'" output.mp4

One fixed dimension and the other proportional, to maintain the aspect ratio:

ffmpeg -i input.mp4 -lavfi "scale='720:-2'" output.mp4

Proportional reduction:

ffmpeg -i input.mp4 -lavfi "scale='floor(iw/4)*2:floor(ih/4)*2'" output.mp4

In the example above, the function floor(d/4) * 2 was used to calculate half, because some codecs require dimensions to always be even numbers (divisible by 2).

Insert subtitles

Optional subtitles

Also known as soft subtitles, they are visible on the player with mouse right-click and selecting the subtitle.

More than one subtitle can be added, for example, for more than one language.

ffmpeg -i input.mp4 -i subtitles.srt -c copy -c:s mov_text -metadata:s:s:0 language=eng output.mp4

Burned subtitles

Hard or burned subtitles are imprinted over the video.

subtitles.srt is the subtitles file. Alignment=2 means bottom center position.

ffmpeg -i input.mp4 -lavfi "subtitles=subtitles.srt:force_style='Alignment=2,OutlineColour=&H100000000,BorderStyle=3,Outline=1,Shadow=0,Fontsize=18,MarginL=5,MarginV=25'" -crf 1 -c:a copy output.mp4

Example of result:

Add text on top of the video

We will use pad and drawtext.

pad adds a layer around the video; in the example below, we'll make it black above the video. For that, we'll extend the frame height by 100 pixels and draw the layer between 0 and 100 on the vertical direction:

# ih = input height
"pad=width=iw:height=(ih+100):x=0:y=100:color=black"

drawtext renders the text on the layer. Our text will have two lines, so we'll use it twice.

x=(w-text_w)/2 centers the text horizontally. The y coordinate determines the line's vertical position.

"drawtext=:font='Arial':text='Men′s Volleyball 2016 Olympics':fontsize=24:fontcolor=white:x=(w-text_w)/2:y=25"

"drawtext=:font='Arial':text='BRA 3 - 0 ITA':fontsize=24:fontcolor=white:x=(w-text_w)/2:y=54"

Command:

ffmpeg -i input.mp4 -lavfi "format=yuv420p,pad=width=iw:height=(ih+100):x=0:y=100:color=black, drawtext=:font='Arial':text='Men′s Volleyball 2016 Olympics':fontsize=24:fontcolor=white:x=(w-text_w)/2:y=25, drawtext=:font='Arial':text='BRA 3 - 0 ITA':fontsize=24:fontcolor=white:x=(w-text_w)/2:y=54" -c:v libx264 -crf 25 output.mp4

format=yuv420p is necessary for compatibility with Apple QuickTime and some other players.

Example of result:

Inspect metadata

Details which codecs were used; video and audio resolution and bitrate; and other information.

ffprobe video.mp4

Sources and interesting reads

Special properties in .NET projects

2024-12-17T00:00:00Z

.NET project files

Certain configurations in .NET project files modify how a project is compiled, and the performance and size of published apps.

These configurations reside in .csproj (C#), .fsproj (F#) and .vbproj (VB.NET) files; they can also be passed via command-line arguments, when running dotnet run, dotnet build and dotnet publish.

In project files

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

</Project>

Via command-line arguments

dotnet publish `
  --configuration Release `
  --self-contained true `
  --runtime "linux-x64" `
  -p:PublishSingleFile=true `
  --output "./out/"

Properties

TargetFramework

Specifies which .NET versions your project should compile against. Usually, only one TFM (target framework moniker) is specified, but, if your project needs to support multiple .NET versions, you can specify a list of TFMs, separated by semi-colon (';').

<PropertyGroup>

  <!-- only one -->
  <TargetFramework>net8.0</TargetFramework>

  <!-- multiple. plural used. -->
  <TargetFrameworks>netstandard1.4;net40;net45</TargetFrameworks>

</PropertyGroup>

Nullable

When present with the value true, warns whenever a variable that can be null is used in a place that doesn't accept null.

Consider the code below. There is a warning that a null value may be passed to the int.Parse() method, which does not accept null strings.

There are four ways to solve this warning:

Modify the method signature. When a variable has a question mark after its type (string?), it means that it accepts null values; abscence of question mark (string) means only non-null values are accepted.

public int ConvertToInt(string numericString) =>
    int.Parse(numericString);

Protect your code against null values.

public int ConvertToInt(string? numericString)
{
    if (numericString == null)
        throw new ArgumentNullException(nameof(numericString));
    else
        return int.Parse(numericString);
}

Apply an exclamation mark. It indicates that the value is safe.

public int ConvertToInt(string? numericString) =>
    int.Parse(numericString!);

Disable nullable verification via preprocessor directive.

#nullable disable warnings

    public int ConvertToInt(string? numericString) =>
        int.Parse(numericString);

#nullable restore warnings

ImplicitUsings

Implicitly adds to all code files (.cs, .fs, .vb) some commonly used namespaces, so they don't need to be declared.

The list of namespaces varies according to the project type.

For console apps and class libraries:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

For Web projects, the ones above, plus:

using System.Net.Http.Json;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

For Workers, the first list, and:

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

TreatWarningsAsErrors

If there are any warnings in the project, the compilation will fail.

<PropertyGroup>
  <TreatWarningsAsErrors>True</TreatWarningsAsErrors>
</PropertyGroup>

This is an interesting option to improve code quality, as it forces developers to pay attention to warnings, which indicate possible weaknesses and problems. An example is combining this option with Nullable.

Author's comment: this alone will not make your code better. The warnings can be circumvented, through #pragma warning disable or by ignoring nullability checks with exclamation marks (see above). A good quality code results from good programming practices and peer revisions.

Configuration

Determines the compilation mode, Debug or Release.

In Debug mode, the compilation produces symbol files (.pdb) that are used by the IDE for debugging and to set breakpoints. Also, the code is not optimized by the compiler.

In Release mode, the compilation applies optimizations to the code and no symbol files are produced. When a program is compiled for deployment, Release config should be used.

This option is passed by flags.

SelfContained

Embeds the .NET runtime in the published app. This means that the target machine (where the app will run) won't need to have it installed.

This option increases the app size (for having the .NET runtime inside).

PublishSingleFile

Condenses the program and its DLLs into a single executable file, reducing the number of published files. This is interesting for portable programs.

Interop DLLs in C/C++ are unaffected by this option and will appear in separate files.

<PropertyGroup>
  <PublishSingleFile>True</PublishSingleFile>
</PropertyGroup>

EnableCompressionInSingleFile

When used with the option above, compresses the condensed executable file, decreasing its size.

A side-effect is it makes the program initialization slower, because the content needs to be extracted before execution. (The extraction is in memory; no files are created.)

<PropertyGroup>
  <EnableCompressionInSingleFile>True</EnableCompressionInSingleFile>
</PropertyGroup>

PublishReadyToRun

This is a special compiler optimization that makes the program startup faster, predicting how it will run in the target machine and replacing parts of intermediary code (IL, intermediate language) with native machine code. This is interesting for APIs, because they take less time to become operational.

This option increases the app size and the compilation takes a longer time (when publishing).

<PropertyGroup>
  <PublishReadyToRun>True</PublishReadyToRun>
</PropertyGroup>

Trimming

Trims away unused parts of the code and of inner dependencies, greatly reducing the size of the published app.

This option must be used carefully, because it may remove important parts of the code and cause errors during execution. To avoid these problems, the compiler emits warnings on lines susceptible to removal, for the developer to find alternatives or mark the code to not be removed.

The trimming may be partial, only on specified assemblies; or full, over the entire app.

This is an extensive subject and requires a thorough read, so I recommend checking the docs.

Example of partial trimming:

<PropertyGroup>
  <PublishTrimmed>true</PublishTrimmed>
  <TrimMode>partial</TrimMode>
</PropertyGroup>

<!-- list of assemblies that will be trimmed -->
<ItemGroup>
  <TrimmableAssembly Include="MyAssembly" />
</ItemGroup>

NativeAOT

NativeAOT is a special type of compilation straight to native machine code, improving app performance.

In traditional compilation, the final executable is composed of intermediate language code (IL) that is interpreted by the .NET runtime (CoreCLR). During execution, the runtime translates each IL instruction into native machine code, and that's how the program runs.

In NativeAOT compilation, the runtime is no longer needed and many processor architecture and operating system optimizations can be applied, causing:

faster execution,
less memory usage,
and lower app size.

This compilation option is not so easy to use, though. It requires that all code paths can be statically analysed, in other words, reflection cannot be used in the code, which affects for example JSON and XML (de)serialization.

For JSON (de)serialization, a solution is System.Text.Json source generators, that write and read JSONs through compile-time generated code.

WPF and Windows Forms are heavily reflection-based and because of that they don't support NativeAOT. However, console and ASP.NET minimal APIs do have support for NativeAOT.

Just like trimming, using this feature requires caution and the official docs should be read.

<PropertyGroup>
  <PublishAot>true</PublishAot>
</PropertyGroup>

Version

Sets the application, assembly or file version.

<PropertyGroup>
  <Version>3.7.1</Version>
  <!-- Version value is reused below -->
  <FileVersion>$(Version)</FileVersion>
  <AssemblyVersion>$(Version)</AssemblyVersion>
</PropertyGroup>

To retrieve the version by code:

string versionName = Assembly.GetExecutingAssembly()
                             .GetName()
                             .Version?
                             .ToString(3);
// versionName = "3.7.1"

Important: the versioning of NuGet packages is done through other properties. Read more in the docs.

ApplicationIcon

Sets an icon on the program for Windows Explorer.

<PropertyGroup>
  <ApplicationIcon>my_program.ico</ApplicationIcon>
</PropertyGroup>

Sources and interesting reads

The fathers of quality, Deming and Shewhart

2024-10-14T00:00:00Z

In the 1980s, there was a prevailing thought in popular culture that Japan would take over the world — not by force, but by its vigorous economy, which was the second largest in the world between 1990 and 2010. We can see this in many movies, such as Back To The Future Part II, where Marty McFly has a Japanese boss in the future; and in Die Hard, where John McClane's wife works for a Japanese company. Her boss is Mr. Takagi, in the picture below.

The keyword that summarizes the Japanese economic miracle is quality. By focusing on better products, with fewer defects and more efficient processes, their industry gained the attention of consumers and space in the global economy.

This transformation began with two men, whose works changed Japan and companies around the whole world.

Deming and Shewhart

William Edwards Deming (1900-1993) was an American statistician, engineer, economist and consultant. For his work, he is known as the father of quality.

Deming graduated in electrical engineering and later specialized in mathematical physics. In 1927, he started working in the United States Department of Agriculture, and there, through colleagues, he met Walter Andrew Shewhart (1891-1967), who is known as the father of statistical quality control. Shewhart was an engineer of Bell Telephone Company and used statistics to improve the reliability and voice clarity of telephone lines.

Shewhart taught Deming many tools that served as the basis for his future works, among them, control charts and PDCA cycles.

Control chart

Shows how much something stays within a margin of tolerance over time.

Some real-life examples: the size of produced mechanical parts; the amount of active ingredient in each tablet; the travel time between two cities.

Below, we see a process that is initially out of control; later, it stays within the tolerance zone; and finally, within a tighter zone, indicating an improvement on the quality of the process.

With a control chart we can detect anomalies in a production line, or estimate when a machine will need maintenance.

Second World War and post-war

During World War II, Deming was part of the U.S. Emergency Technical Committee, focused on improving the wartime production. Statistical quality control was used on factories and reduced the number of failures in American weaponry.

After the war, Deming was convoked to prepare the 1951 Japanese census and his effort and involvement in Japanese society caught attention, to the point that the Japanese Union of Scientists and Engineers (JUSE) invited him to teach them about statistics and quality. His lessons were a tremendous success and were transcribed into books with thousands of copies sold.

In a lecture to Japanese executives in 1950, Deming told them that implementing statistical quality control on their companies would lower costs and waste, increase productivity and attract consumers, allured by better products. The executives applied these ideas into their businesses and had very positive results.

In the following decades, Japan witnessed its economy rise from the ashes and become one of the largest in the world, largely due to the implementation of Deming's ideas. The GDP growth was around 8-10% yearly.

Deming, at the right, visiting Japan.

Case study: Mazda-Ford transmissions

In the 80s, Ford partnered with Mazda to produce automatic transmissions for Ford automobiles.

Both companies produced this transmission and in both of them the products had full specification conformance. Despite that, Ford transmissions had a higher number of customer complaints and incurred higher repair costs; not just that, customers came over to Ford dealerships asking for their new cars to have Japanese transmissions, because they had a smoother driving experience.

Ford engineers investigated and noted that Ford transmissions had a variance of 70% within the margin of tolerance, while Mazda transmissions had a variance of 27%. For example, if the specification of a part was 300mm ± 3mm, Ford's parts size varied by 2.1mm; while Mazda's varied by 1.15mm.

Mazda's production cost was slightly more expensive, but compensated by having less warranty claims.

After this study, the Ford factory improved the quality of its transmissions, that became better than Mazda's.

Case study: Sony TVs

Also in the 1980s, Sony had two TV production plants, one in Japan and one in San Diego, California.

One of the quality factors for a TV was the colour density, with a margin of tolerance between 9 and 14. TVs made in San Diego all were within tolerance, while 1.3 for every 1000 of the ones made in Japan were out. Regardless of that, consumers preferred Japanese TVs, because the image on the screen was more homogenous and with a better resolution, due to a lower variance of colour density.

Recognition in the U.S.

Ironically, Deming's ideas fell into disuse in the United States after the war and were not widely accepted by American executives, who believed that quality was a cost, rather than an investment.

In the following decades, Deming extended his work for beyond statistics and addressed how to improve the efficiency and organization of companies, summarizing his ideas in 14 points for management.

In 1980, NBC aired a documentary comparing American and Japanese companies. Deming was interviewed in this documentary, and after his appearance on TV, many corporations requested Deming's consultancy services. One of the companies that hired him was Ford.

At the beginning of the 1980s, Ford was facing a major crisis, with losses exceeding $3 billion. Deming was called and said that the management was responsible for 85% of the problems in developing better cars. Ford executives took this seriously and adopted Deming's ideas: by 1986, Ford became the most profitable American auto company, surpassing General Motors for the first time since the 1920s. Revenue grew consistently over the following years.

The Ford Taurus assembly line in 1986. The car was a major success and saved the company from bankruptcy.

14 points for management

In his book Out of the Crisis, Deming presents the following points for transforming business effectiveness:

Create constancy of purpose toward improvement of product and service, with the aim to become competitive, to stay in business and to provide jobs.
Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.
Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.
End the practice of awarding business on the basis of a price tag. Instead, minimize total cost. Move towards a single supplier for any one item, on a long-term relationship of loyalty and trust.
Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
Institute training on the job.
Institute leadership. The aim of supervision should be to help people and machines and gadgets do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.
Drive out fear, so that everyone may work effectively for the company.
Break down barriers between departments. People in research, design, sales, and production must work as a team, to foresee problems of production and usage that may be encountered with the product or service.
Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.

a) Eliminate work standards (quotas) on the factory floor. Substitute with leadership.

b) Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership.
Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.
Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means, inter alia, abolishment of the annual or merit rating and of management by objectives.
Institute a vigorous program of education and self-improvement.
Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job.

Sources and interesting reads

Collation and encoding in databases

2024-08-29T00:00:00Z

Introduction

Every information in a computer is stored and processed in its binary number representation.

But what about texts? A text is a sequence of characters, a character being a letter or a symbol. Each character is represented by a binary number and the character-number mapping is called encoding.

The table below is for Windows-1252 encoding, which addresses 255 possible characters, each represented by one byte. You can see this table on Windows Character Map (Win+R, charmap).

The collation is the ordering and comparison rule for texts. It also determines which encoding is used for storage.

On Microsoft SQL Server, in the collation's name, CI / CS means case (in)sensitive; AI / AS, accent (in)sensitive. With a case-insensitive (CI) collation, for example, searching for either silva or SILVA yields the same results; with an accent-insensitive (AI) collation, Acucar and Açúcar are considered equal for comparisons.

Accent-insensitive is insensitive actually for all the diacritics of a language, like cedillas (ç, ş) and accents (é, ã, ô, etc). Some older collations do not cover all diacritics on comparisons (link), so newer collations should be preferred.

SELECT
  [Name],
  COLLATIONPROPERTY( [Name], 'LCID' ) AS [LCID],
  COLLATIONPROPERTY( [Name], 'CodePage' ) AS [CodePage]
FROM sys.fn_helpcollations()
ORDER BY [Name];

Name	Encoding	Compares casing	Compares accents
Latin1_General_ CI_AS	Windows-1252	No	Yes
Latin1_General_ 100_CI_AS_ KS_SC_UTF8	UTF-8 (65001)	No	Yes
Latin1_General_ 100_CS_AS_ KS_SC_UTF8	UTF-8 (65001)	Yes	Yes

Unicode, UTF-8 and UTF-16

Unicode is a table that defines a number for each character, covering symbols, digits and letters from many languages. The number attributed to a character is called code point.

UTF-8 and UTF-16 are encodings that follow Unicode. Basically, they are ways of storing those numbers into bytes.

UTF-16 uses 2 bytes for most chars and 4 for those above the standard range. This encoding is used in the strings of most programming languages.

UTF-8 uses a variable number of bytes, starting at 1 and up to 4 for a char. It's the main encoding on the Internet.

Unicode range	Groups	Bytes per char, UTF-8	Bytes per char, UTF-16
0x0000 - 0x007F	Basic latin alphabet, arabic digits (0-9), basic keyboard symbols	1	2
0x0080 - 0x07FF	Extended latin alphabet, greek, cyrillic, arabic, hebrew	2	2
0x0800 - 0xFFFF	Japanese and chinese ideograms; varied symbols; math operators	3	2
0x010000 - 0x10FFFF	Ancient writing pictograms (e.g. egyptian hieroglyphs); emojis; musical symbols	4	4

The choice of encoding directly affects the size of text storage. If most characters lie in the basic latin range, UTF-8 is better, because it uses fewer bytes than UTF-16; however, if it's an asian text, UTF-16 is the best, because each character occupies 2 bytes, instead of 3 on UTF-8.

The table below shows how an Unicode number is converted to UTF-8 or UTF-16, for each range above.

Example character	Code point, in binary	In UTF-8	In UTF-16
P (0x0050)	00110010	00110010	00000000 00110010
Ω (0x03A9)	00000011 10101001	11001110 10101001	00000011 10101001
€ (0x20AC)	00100000 10101100	11100010 10000010 10101100	00100000 10101100
🐎 (0x1F40E)	00000001 11110100 00001110	11110000 10011111 10010000 10001110	11011000 00111101 11011100 00001110

The logic for UTF-16 code points above 0x010000 is:

U = code point
W1 = 2 upper bytes
W2 = 2 lower bytes

W = U - 0x10000 
W = yyyyyyyyyyxxxxxxxxxx (20 binary digits)
W1 = 110110yy yyyyyyyy
W2 = 110111xx xxxxxxxx

-> there is no risk of W1 and W2 being mistaken for 
other characters because the possible interval for them 
is protected on the Unicode table.

Texts in SQL databases

CHAR and NCHAR store fixed-size texts; VARCHAR and NVARCHAR store variable-sized texts.

NCHAR and NVARCHAR are types present on SQL Server and the 'N' indicates that they store text in UTF-16 encoding. CHAR and VARCHAR, on the other hand, store according to the encoding of the database's collation.

The storage size is specified on the column type declaration, such as NVARCHAR(n). Many people think that n is the number of characters, but that is not true. For CHAR and VARCHAR, n defines the size in bytes; for NCHAR and NVARCHAR, n is the size in byte-pairs (x2).

Practical example

Let's have two databases, one with the Latin1_General_CI_AS collation (Windows-1252 encoding) and another with Latin1_General_100_CI_AS_KS_SC_UTF8 collation (UTF-8 encoding). For each of them, we will compare the storage sizes between VARCHAR and NVARCHAR, for texts in basic and extended latin alphabet, greek, japanese and emojis. Below, the script to run:

CREATE TABLE [dbo].[Person] (
  [Name] VARCHAR(24) NOT NULL,
  [NameUtf16] NVARCHAR(24) NOT NULL
);

INSERT INTO [dbo].[Person] VALUES
('Pericles','Pericles'), -- latin without accent
('Péricles','Péricles'), -- latin with accent
(N'Περικλῆς',N'Περικλῆς'), -- greek
(N'美しいキモノ',N'美しいキモノ'), -- japanese katakana
(N'Santa Claus 🎅',N'Santa Claus 🎅'); -- with emoji
-- the N prefix is necessary for unicode strings

SELECT
  [Name], DATALENGTH([Name]) AS [SizeInBytes],
  [NameUtf16], DATALENGTH([NameUtf16]) AS [SizeInBytes]
FROM [dbo].[Person];

DROP TABLE [dbo].[Person];

Latin1 General CI AS

VARCHAR name	Size in bytes	NVARCHAR name	Size in bytes
Pericles	8	Pericles	16
Péricles	8	Péricles	16
?e??????	8	Περικλῆς	16
??????	6	美しいキモノ	12
Santa Claus ??	14	Santa Claus 🎅	28

Note that Windows-1252 encoding does not support greek and japanese characters, nor emojis, that are replaced by '?'. Despite that, it handles very well latin words, with only 1 byte per letter, even on those with accents or cedillas.

Latin1 General 100 CI AS KS SC UTF8

VARCHAR name	Size in bytes	NVARCHAR name	Size in bytes
Pericles	8	Pericles	16
Péricles	9	Péricles	16
Περικλῆς	17	Περικλῆς	16
美しいキモノ	18	美しいキモノ	12
Santa Claus 🎅	16	Santa Claus 🎅	28

With an UTF-8 collation, the VARCHAR field successfully supported all characters and had a higher efficiency for most cases. The third name, Περικλῆς, needed 17 bytes because the letter ῆ (unicode 0x1FC6) is from ancient greek and requires 3 bytes in UTF-8 encoding. For japanese katakana, UTF-16 proved more efficient.

Sources and interesting reads

SQL iceberg

2024-08-14T00:00:00Z

Introduction

The SQL language, Structured Query Language, first emerged in 1974 and is one of the leading languages to this day, when it comes to databases.

SQL evolved a lot throughout the years and especially over the last two decades, with modern data organization techniques, that came up as a response to new competitors (NoSQL) and new needs (analytics, data warehouse, cloud computing).

This article is a SQL cheatsheet, covering some of the most common commands to those that are rare and have interesting uses. This list tries to be easy to understand for beginners and for those who already have some familiarity with the language.

The examples here are for Microsoft SQL Server, but most of them apply in a similar or equal way for other databases, such as PostgreSQL, MySQL and Oracle.

Before we begin

Reads and queries

Data changes

Table structuring

Special

Final conclusions

Before we begin

Installation

To test the examples of this article, I recommend installing SQL Server (download) and SQL Server Management Studio (SSMS) (download).

After the installations, enter SSMS and login in your instance:

Windows Authentication
Server name: . (dot)
Check Trust server certificate

These options may differ according to the configuration during setup.

Then, choose a database in the left panel; if there are none, click with the mouse right button on Databases and create one.

To write a SQL script, click on New query, in the top menu. To run, press F5 or click on Execute.

Sample tables

Let's start with two tables that will be used in our examples: tables Fruit and Family.

Table [dbo].[Fruit]:

Id	Name	IdFamily	Calories*
1	Coconut	1	354
2	Date	1	282
3	Strawberry	3	32
4	Watermelon	2	30
5	Passionfruit	4	97
6	Cherry	3	64
7	Pineapple	5	48
8	Apple	3	52

* Calories per 100g. Source: Mundo Boa Forma.

Table [dbo].[Family]:

Id	Family
1	Arecaceae
2	Cucurbitaceae
3	Rosaceae
4	Passifloraceae
5	Bromeliaceae
6	Malvaceae

To create them, run the following SQL script in your database.

CREATE TABLE [dbo].[Family](
  [Id] INT NOT NULL IDENTITY(1,1) PRIMARY KEY CLUSTERED,
  [Name] NVARCHAR(64) NOT NULL)
GO

INSERT INTO [dbo].[Family] VALUES
  ('Arecaceae'), ('Cucurbitaceae'), ('Rosaceae'),
  ('Passifloraceae'), ('Bromeliaceae'), ('Malvaceae');
GO

CREATE TABLE [dbo].[Fruit](
  [Id] INT NOT NULL IDENTITY(1,1) PRIMARY KEY CLUSTERED,
  [Name] NVARCHAR(64) NOT NULL,
  [IdFamily] INT NOT NULL FOREIGN KEY REFERENCES [dbo].[Family](Id),
  [Calories] INT NOT NULL)
GO

INSERT INTO [dbo].[Fruit] VALUES 
  ('Coconut', 1, 354), ('Date', 1, 282), ('Strawberry', 3, 32),
  ('Watermelon', 2, 30), ('Passionfruit', 4, 97), ('Cherry', 3, 64),
  ('Pineapple', 5, 48), ('Apple', 3, 52);
GO

Reads and queries

SELECT

Gets data in a tabular structure.

SELECT [Name], [Calories]
FROM [dbo].[Fruit]
WHERE [Calories] < 50

Returns:

Name	Calories
Strawberry	32
Watermelon	30
Pineapple	48

* means all columns:

SELECT *
FROM [dbo].[Fruit]
WHERE [Calories] < 50

Id	Name	IdFamily	Calories
3	Strawberry	3	32
4	Watermelon	2	30
7	Pineapple	5	48

LEFT, RIGHT, INNER JOINS

Interlaces two or more tables by reference columns; this interlacing can be used for queries, insertions, updates and deletions.

Here we'll focus only on INNER JOIN and LEFT JOIN.

INNER JOIN

INNER JOIN is a perfect intersecction, that is, the interlaced values must exist in both tables. If it exists in a line of table X, but with no corresponding line in another table Y, the line of table X is not considered. Example:

Table [dbo].[Fruit]:

Id	Name	IdFamily	Calories
3	Strawberry	3	32
4	Watermelon	2	30
7	Pineapple	5	48

Table [dbo].[Family]:

Id	Family
2	Cucurbitaceae
3	Rosaceae
5	Bromeliaceae

SQL query:

SELECT
    fr.[Name] AS [Name],
    fa.[Name] AS [Family]
FROM [dbo].[Fruit] fr
INNER JOIN [dbo].[Family] fa ON fr.[IdFamily] = fa.[Id]
WHERE fr.[Calories] < 50

Returns:

Name	Family
Strawberry	Rosaceae
Watermelon	Cucurbitaceae
Pineapple	Bromeliaceae

LEFT / RIGHT JOIN

LEFT JOIN and RIGHT JOIN are imperfect intersecctions, such that the correspondence value is allowed to not exist in one of the tables; if it doesn't exist, then values from the table with no correspondence will come null.

In the tables above, the Malvaceae family (id = 6) has no registered fruit.

SELECT
    fa.[Name] AS [Family],
    fr.[Name] AS [Fruit]
FROM [dbo].[Family] fa
LEFT JOIN [dbo].[Fruit] fr ON fr.[IdFamily] = fa.[Id]

Family	Fruit
Arecaceae	Coconut
Arecaceae	Date
Cucurbitaceae	Watermelon
Rosaceae	Strawberry
Rosaceae	Cherry
Rosaceae	Apple
Passifloraceae	Passionfruit
Bromeliaceae	Pineapple
Malvaceae	`NULL`

VIEWS

VIEWS are encapsulated queries: instead of always typing a complex query, you can "alias" this query with a view, which simplifies the call. This technique is good for abstracting logic and to control access permissions — an user can call the view, but be blocked from querying the source tables.

View creation:

CREATE OR ALTER VIEW [dbo].[ViewGetFruits] AS

  SELECT
    fr.[Name] AS [Fruit],
    fa.[Name] AS [Family]
  FROM [dbo].[Fruit] fr
  INNER JOIN [dbo].[Family] fa ON fr.[IdFamily] = fa.[Id]

GO

To run:

-- the view behaves like a table
SELECT * FROM [dbo].[ViewGetFruits]

GROUP BY, HAVING

GROUP BY is a grouping instruction and HAVING is a filter applied over groups.

Columns returned by a SELECT with grouping must be specified on the GROUP BY, or be in aggregation functions, like counts, sums, averages, etc.

Example:

SELECT fr.[IdFamily], COUNT(fr.[Id]) AS NumberOfFruits
FROM [dbo].[Fruit] fr
GROUP BY fr.[IdFamily]

IdFamily	NumberOfFruits
1	2
2	1
3	3
4	1
5	1

With inner join and filter over the groups:

SELECT fa.[Name] AS [Family], COUNT(fr.[Id]) AS NumberOfFruits
FROM [dbo].[Fruit] fr
INNER JOIN [dbo].[Family] fa ON fr.[IdFamily] = fa.[Id]
GROUP BY fa.[Name]
HAVING COUNT(fr.[Id]) >= 2 -- family w/ 2 or more fruits

Family	NumberOfFruits
Arecaceae	2
Rosaceae	3

UNION, INTERSECT, EXCEPT

These are set operators (see set theory in mathematics). They are different from the interlacing that happens in JOINs — while in JOINs the interlacing is between columns, with sets the interlacing is between lines. The examples below show this visually.

UNION: Returns elements that belong to one set or another.
INTERSECT: Returns elements that belong to one set and another, at the same time.
EXCEPT: Returns elements that belong to one set, but not to another.

DECLARE @a TABLE (X INT);
INSERT INTO @a VALUES (1),(2),(3),(4),(5);

DECLARE @b TABLE (X INT);
INSERT INTO @b VALUES (2),(4),(6);

SELECT * FROM @a

UNION -- INTERSECT, EXCEPT

SELECT * FROM @b

A	B	UNION	INTERSECT	EXCEPT
1	2	1	2	1
2	4	2	4	3
3	6	3	-	5
4	-	4	-	-
5	-	5	-	-
-	-	6	-	-

Why use an UNION when I can just write WHERE criterion1 OR criterion2 ?

Sometimes we want to unite lines from different tables:

SELECT * FROM Table1 WHERE criterion1
UNION
SELECT * FROM Table2 WHERE criterion2

What is UNION ALL?

In set theory, a set cannot have repeated elements. However, in SQL queries, sometimes we want to allow duplications in a union, for that, we use UNION ALL, instead of just UNION.

SUBQUERIES

These are queries that run inside others. They can be useful in more complex operations, however, whenever possible, it is best to pick joins over subqueries, for better performance.

In the example below, the fruit family's name is obtained through a subquery:

SELECT
    fr.[Name] AS [Name],
    (SELECT fa.[Name] FROM [dbo].[Family] fa WHERE fa.[Id] = fr.[IdFamily]) AS [Family]
FROM [dbo].[Fruit] fr
WHERE fr.[Calories] < 50

SELF JOIN

Is a join of table with itself. Useful for tree operations.

Below, we have a Employee table and an employee can be someone else's boss.

DECLARE @employee TABLE (Id INT, [Name] NVARCHAR(32), IdBoss INT);
INSERT INTO @employee VALUES
(1, 'Mark', 4),
(2, 'John', NULL),
(3, 'Michael', 2),
(4, 'Rachel', 2),
(5, 'Jessica', 4);

SELECT
  f1.[Name] AS [Employee],
  f2.[Name] AS [Boss]
FROM @employee f1
LEFT JOIN @employee f2 ON f1.[IdBoss] = f2.[Id]
ORDER BY f2.[Id] ASC

Employee	Boss
John	`NULL`
Michael	John
Rachel	John
Jessica	Rachel
Mark	Rachel

CROSS JOIN

It is the cartesian product between tables, basically, it makes all possible combinations of elements of one table to another.

Consider two tables, IceCream and Topping, and we want to know what combinations we can have.

DECLARE @icecream TABLE (Id INT, [Name] NVARCHAR(32));
INSERT INTO @icecream VALUES
(1, 'Strawberry'), (2, 'Flake'), (3, 'Chocolate');

DECLARE @topping TABLE (Id INT, [Name] NVARCHAR(32));
INSERT INTO @topping VALUES
(1, 'Chocolate syrup'), (2, 'Nuts');

SELECT i.[Name] AS IceCream, t.[Name] AS Topping
FROM @icecream i CROSS JOIN @topping t
ORDER BY i.[Id] ASC, t.[Id] ASC

IceCream	Topping
Strawberry	Chocolate syrup
Strawberry	Nuts
Flake	Chocolate syrup
Flake	Nuts
Chocolate	Chocolate syrup
Chocolate	Nuts

PIVOT

The PIVOT instruction is very interesting. It transforms lines into columns, by rotating part of the table in 90º. It is very good for crossing two or more columns of a table.

In the example below, we have a table Sales with each line being the quantity of a product sold on a certain day. We want to visualize this in a better way, with date columns, and for that we will use PIVOT.

DECLARE @sales TABLE (IdProduct INT, Quantity INT, SalesDate DATE);
INSERT INTO @sales VALUES
(5, 52, '2024-08-05'),
(3, 902, '2024-08-05'),
(1, 196, '2024-08-06'),
(4, 212, '2024-08-06'),
(3, 488, '2024-08-07'),
(4, 384, '2024-08-07'),
(1, 1121, '2024-08-08'),
(2, 49, '2024-08-08'),
(3, 88, '2024-08-09'),
(2, 12, '2024-08-09'),
(5, 98, '2024-08-09');

SELECT * FROM
(
  SELECT [SalesDate], [IdProduct], [Quantity]
  FROM @sales
) t
PIVOT(
  SUM([Quantity]) FOR [SalesDate] IN ([2024-08-05],[2024-08-06],[2024-08-07],[2024-08-08],[2024-08-09])
) AS pvt;

IdProduct	2024-08-05	2024-08-06	2024-08-07	2024-08-08	2024-08-09
1	NULL	196	NULL	1121	NULL
2	NULL	NULL	NULL	49	12
3	902	NULL	488	NULL	88
4	NULL	212	384	NULL	NULL
5	52	NULL	NULL	NULL	98

What if I want to generate columns dynamically, with no fixed values for them?

You can generate the command's text at runtime and then run it. This article shows how to do it inside a procedure:

CREATE PROCEDURE dbo.GenerateStudentsReportCards
  @ColumnToPivot  NVARCHAR(255),
  @ListToPivot   NVARCHAR(255)
AS
BEGIN
 
  DECLARE @SqlStatement NVARCHAR(MAX)
  SET @SqlStatement = N'
    SELECT * FROM (
      SELECT [Student], [Subject], [Marks]
      FROM [Grades]
    ) StudentResults
    PIVOT (
      SUM([Marks])
      FOR ['+@ColumnToPivot+']
      IN ('+@ListToPivot+')
    ) AS PivotTable'; 
  EXEC(@SqlStatement)
 
END

To run:

EXEC dbo.GenerateStudentsReportCards
  N'Subject'
  ,N'[Mathematics],[Science],[Geography]'

ROLLUP, CUBE, GROUPING SETS

These are types of groupings that also analyze subgroups.

DECLARE @sales TABLE (IdProduct INT, Quantity INT, SalesDate DATE);
INSERT INTO @sales VALUES
(5, 52, '2024-08-05'),
(3, 902, '2024-08-05'),
(1, 196, '2024-08-06'),
(4, 212, '2024-08-06'),
(3, 488, '2024-08-07'),
(4, 384, '2024-08-07');

SELECT [SalesDate], [IdProduct], SUM([Quantity]) AS [Quantity]
FROM @sales
GROUP BY ROLLUP([SalesDate],[IdProduct]) -- CUBE, GROUPING SETS

ROLLUP

Analyzes subgroups on a pyramid, from the right to left in the declaration. GROUP BY ROLLUP([ColA], [ColB]) will check:

[ColA], [ColB] (subgroup of A and B)
[ColA], NULL (subgroup of A for all values of B)
NULL, NULL (general group)

SalesDate	IdProduct	Quantity
2024-08-05	3	902
2024-08-05	5	52
2024-08-05	NULL	954
2024-08-06	1	196
2024-08-06	4	212
2024-08-06	NULL	408
2024-08-07	3	488
2024-08-07	4	384
2024-08-07	NULL	872
NULL	NULL	2234

CUBE

Analyzes all possible subgroups. GROUP BY CUBE([ColA], [ColB]) will check:

[ColA], [ColB] (subgroup of A and B)
[ColA], NULL (subgroup of A for all values of B)
NULL, [ColB] (subgroup of B for all values of A)
NULL, NULL (general group)

SalesDate	IdProduct	Quantity
2024-08-06	1	196
NULL	1	196
2024-08-05	3	902
2024-08-07	3	488
NULL	3	1390
2024-08-06	4	212
2024-08-07	4	384
NULL	4	596
2024-08-05	5	52
NULL	5	52
NULL	NULL	2234
2024-08-05	NULL	954
2024-08-06	NULL	408
2024-08-07	NULL	872

GROUPING SETS

Analyzes the subgroups of each column, but without crossing them. GROUP BY GROUPING SETS([ColA], [ColB]) checks:

[ColA], NULL (subgroup of A for all values of B)
NULL, [ColB] (subgroup of B for all values of A)

SalesDate	IdProduct	Quantity
NULL	1	196
NULL	3	1390
NULL	4	596
NULL	5	52
2024-08-05	NULL	954
2024-08-06	NULL	408
2024-08-07	NULL	872

WINDOW FUNCTION

They are functions that generate values looking over slices (or windows) of a table. They are very useful for pagination, like product searches.

This is a lengthy subject and with many possibilities, which is why I will not cover it here, but, I recommend this article that explains very well how window functions work.

NOLOCK

NOLOCK, or READ UNCOMMITTED, is a query hint that, as the name suggests, ignores locks.

A regular query uses the standard isolation level, READ COMMITTED, that reads only committed data from a table. In a table with heavy traffic (many simultaneous operations), a query may face concurrency problems with inserts, updates and deletes.

With the NOLOCK hint, the read does not compete with those operations. However, this reading mode includes uncommitted records, in such a way that inconsistent data may appear in the results. Generally speaking, NOLOCK should be used only if there actually is a concurrency problem, and if eventual inconsistencies are tolerable. An example of usage is the extraction of analytical reports.

SELECT *
FROM [dbo].[Fruit] WITH (NOLOCK)
WHERE [Calories] < 50

Data changes

INSERT, UPDATE, DELETE

Adds, modifies and excludes lines from a table.

DECLARE @person TABLE (Id INT, [Name] NVARCHAR(32));

-- inserting with VALUES
INSERT INTO @person VALUES
(1, 'Mario'), (2, 'Caroline');

-- inserting with SELECT
INSERT INTO @person SELECT 3, 'James';

-- changing name
UPDATE @person
SET [Name] = 'Maria'
WHERE [Id] = 1;

-- deleting person
DELETE FROM @person
WHERE [Name] = 'James';

-- final result
SELECT * FROM @person

STORED PROCEDURES

They are scripts that accept parameters and execute actions in the database; they can return table-valued results and numerical values.

The procedure below can be used to add fruits (considering the example from the beginning).

CREATE OR ALTER PROCEDURE [dbo].[ProcAddFruit]
  @fruit NVARCHAR(32),
  @family NVARCHAR(32),
  @calories INT
AS
BEGIN

  -- checks if the fruit is already registered
  IF EXISTS (SELECT 1 FROM [dbo].[Fruit] WHERE [Name] = @fruit)
  BEGIN
    RAISERROR('Fruit already registered!', 16, 1)
    RETURN 1; -- stop execution
  END

  -- adds fruit family if it doesn't exist
  IF NOT EXISTS (SELECT 1 FROM [dbo].[Family] WHERE [Name] = @family)
  BEGIN
    INSERT INTO [dbo].[Family] VALUES (@family);
  END

  -- get family id
  DECLARE @idFamily INT = (SELECT [Id] FROM [dbo].[Family] WHERE [Name] = @family)

  -- adds fruit
  INSERT INTO [dbo].[Fruit] VALUES (@fruit, @idFamily, @calories);

  RETURN 0; -- everything OK

END
GO

To run:

EXECUTE [dbo].[ProcAddFruit]
  @fruit = 'Orange',
  @family = 'Rutaceae',
  @calories = 37

MERGE

Allows simultaneous execution of insertions, updates and deletions, comparing a source table to a target table. This instruction is used for sincronizing different data sources. If the sincronizations are always of the same type (insertions, updates, deletes), then using these specific operations offers better performance.

I will not enter into details here, but, I recommend this article for those interested.

MERGE target_table USING source_table
ON merge_condition
WHEN MATCHED THEN update_statement
WHEN NOT MATCHED THEN insert_statement
WHEN NOT MATCHED BY SOURCE THEN DELETE;

BULK INSERT

BULK INSERT is a mass insertion of data coming from a file, usually in CSV or XML formats, or a flat file.

This is a strategy to move large volumes of data from one place to another, especially if flat files are chosen, because they are very compact and have very high compression rates (example: a 900MB flat file, when zipped, can be reduced to 30MB, depending on the situation).

Consider the flat file below. Note that the starting positions of each field are always the same on a line — fruit name (0..16), family name (16..32), calories (32..35):

Graviola        Annonaceae      66 
Mamão Papaya    Caricaceae      46 
Cambucá         Myrtaceae       66 
Bacuri          Clusiaceae      105
Cupuaçu         Malvaceae       49

To import a file using BULK INSERT, I recommend 1) using a format file and 2) first insert into an intermediary table, because many times we need to polish data, by trimming whitespaces or converting dates. The following is a XML format file.

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<RECORD>
		<FIELD ID="F1" xsi:type="CharFixed" LENGTH="16"/><!-- Fruit name -->
		<FIELD ID="F2" xsi:type="CharFixed" LENGTH="16"/><!-- Family name -->
		<FIELD ID="F3" xsi:type="CharFixed" LENGTH="3"/><!-- Calories -->
		<FIELD ID="F4" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="6"/><!-- Line break -->
	</RECORD>
	<ROW>
		<COLUMN SOURCE="F1" NAME="FruitName" xsi:type="SQLVARYCHAR" LENGTH="16" NULLABLE="NO"/>
		<COLUMN SOURCE="F2" NAME="FamilyName" xsi:type="SQLVARYCHAR" LENGTH="16" NULLABLE="NO"/>
		<COLUMN SOURCE="F3" NAME="Calories" xsi:type="SQLINT"/>
	</ROW>
</BCPFORMAT>

To run a BULK INSERT:

CREATE TABLE #temp1
(Fruit NVARCHAR(16), Family NVARCHAR(16), Calories INT);

BULK INSERT #temp1 
FROM 'C:\MyFolder\flat_file.txt' 
WITH 
(
  CODEPAGE = '65001', -- UTF-8 with BOM encoding
  FORMATFILE = 'C:\MyFolder\format_file.xml', 
  BATCHSIZE = 4000
);

-- polishing
UPDATE #temp1
SET [Fruit] = TRIM([Fruit]),
    [Family] = TRIM([Family])

SELECT * FROM #temp1

DROP TABLE #temp1

Exporting flat files

Unfortunately, there are no easy ways of generating files straight from the database; for SQL Server, there is only the Import / Export Wizard, which is a graphical interface inside SSMS.

However, a simple console application could read lines from the database and output a flat file from them.

Table structuring

PRIMARY KEY, FOREIGN KEY

The primary key uniquely identifies each row of a table and determines what is the main search column, structuring the table in a B-tree; this makes reading it much faster.

The foreign key is a link between the column of one table to the primary key of another. This increases performance of joins between those tables and is also a form of validation, because a value is only valid if it exists in the foreign table.

Taking the initial example:

CREATE TABLE [dbo].[Fruit](
  [Id] INT NOT NULL IDENTITY(1,1) PRIMARY KEY CLUSTERED,
  [Name] NVARCHAR(64) NOT NULL,
  [IdFamily] INT NOT NULL FOREIGN KEY REFERENCES [dbo].[Family](Id),
  [Calories] INT NOT NULL)

INDEXES

They are additional search rails on a table that run parallel to the primary key. They are useful for tables with more than one search criteria.

Imagine a Person table, with columns Id, NationalIdentifier and Name: it's common to search a person by its national identifier or name, but, these columns are not the primary key, which makes searching by them slower. In these cases, the search is done individually line by line, over the whole table; moreover, it's a textual comparison, because NationalIdentifier and Name are texts. Textual comparisons are slower than numerical comparisons, which is the case of Id.

To speed the search, we can create indexes. The most common type is the hash index, which attributes a random number to each indexed value and saves this number next to the record. Whenever a search comes to use this criterion, the comparison will be numerical and with better performance than checking the field's value on every line.

Index creation:

-- one column
CREATE INDEX IX_Person_NationalIdentifier ON [dbo].[Person]([NationalIdentifier]);

-- more than one column
CREATE INDEX IX_Address_CityStateCountry
ON [dbo].[Address]([City],[State],[Country]);

UNIQUE

It's a special type of constraint that blocks repeated values in a column. In most databases, creating an unique constraint also creates an index for this column.

CREATE TABLE [dbo].[Person] (  
  Id INT NOT NULL PRIMARY KEY CLUSTERED,
  NationalIdentifier CHAR(11) NOT NULL,
  [Name] NVARCHAR(64) NOT NULL,
  CONSTRAINT CT_Unique_Person_NationalIdentifier UNIQUE([NationalIdentifier])
);

SPARSE

Sparse is a column modifier that optimizes storage when there are many null values.

In traditional storage, a field has a reserved amount of bytes for it in the row even when the value is null, which is a waste.

With sparse columns, a null value occupies zero bytes on the row and a non-null value occupies the original size plus 4 bytes.

Practical example, NVARCHAR(16) column needs 32 bytes on each row. With sparse, if null, 0 bytes will be used; if not null, 36 bytes.

Percentage of nulls	Column size difference with SPARSE
0%	+12.5%
25%	-15.6%
75%	-71.9%

Formula: 1 - ((T+4)/T) * (1-N)

T : original field size
N : percentage of nulls

If there are not many nulls, there is no advantage in using sparse columns, but they save space if the percentage of nullability is high.

CREATE TABLE [dbo].[Person] (  
  Id INT NOT NULL PRIMARY KEY CLUSTERED,
  NationalIdentifier CHAR(11) NOT NULL,
  [Name] NVARCHAR(64) NOT NULL,
  FavouriteColour NVARCHAR(32) SPARSE NULL);

COLUMNSTORE

Columnstore is a storage strategy.

The default storage mode of a table is linear, with rows laid inside pages. A, B and C are columns:

flowchart TB
  subgraph page[Page]
    direction LR
    subgraph row1[Row 1]
      direction TB
      A1
      B1
      C1
    end
    subgraph row2[Row 2]
      direction TB
      A2
      B2
      C2
    end    
    subgraph row3[Row 3]
      direction TB
      A3
      B3
      C3
    end
end

With columnar storage, a page is exclusive for a column:

flowchart TB
  subgraph pageC[Page]
    subgraph columnC[Column C]
      direction LR
      C1
      C2
      C3
    end
  end
  subgraph pageB[Page]
    subgraph columnB[Column B]
      direction LR
      B1
      B2
      B3
    end
  end  
  subgraph pageA[Page]
    subgraph columnA[Column A]
      direction LR
      A1
      A2
      A3
    end
  end

What is the advantage of this approach? Columnar storage has compression by default and if values frequently repeat in columns, the storage size can be greatly reduced.

See below how columnar compression works:

Uncompressed column	Compressed column
A	5 x A
A	2 x B
A	3 x A
A
A
B
B
A
A
A

According to SQL Server docs, the storage size can be up to 10x smaller and the reading speed up to 10x higher, compared to a linear storage.

When to use columnstore

High number of repetitions of values in vertical directions.
Low cardinality (possible variations of values).
High volume of data (over 500k lines).
Analytical processes: sums, averages, counts.

When not to use columnstore

Records that have many updates and deletes, because compression is lost when there are changes.
Records that are entities, because columnstore tables have no primary keys.
If this table is used for many joins.

Clustered vs nonclustered

A clustered columnstore index is when the table uses columnar storage. A nonclustered columnstore index is when the table uses linear storage and columnstore is an auxiliary index; this is an option when you want the benefits of columnstore (analytical performance) without losing the benefits of linear tables (primary keys).

Table creation

CREATE TABLE [dbo].[EntryFruitFile](
	Fruit NVARCHAR(16) NOT NULL,
	Family NVARCHAR(16) NOT NULL,
	Calories INT NOT NULL,
	INDEX CCI_EntryFruitFile CLUSTERED COLUMNSTORE);

Special

IMPORT DLLs

Yes, it is possible to import DLLs to execute on SQL Server.

Custom DLLs are interesting for use cases where there is no native similar, like: read and write files; HTTP calls; generate and validate hashes.

DLLs must be .NET Framework source code and signed with a strong name.

The process is a bit complex and the steps here are based on this article from C-Sharp Corner.:

Create a Class Library project in .NET Framework
C# code example below:

using System;

namespace MyDotnetSqlProj
{
    public static class Class1
    {
        public static double CalculateHypotenuse(double side1, double side2) =>
            Math.Sqrt(Math.Pow(side1, 2) + Math.Pow(side2, 2));
    }
}

On Visual Studio, click with the mouse right button on the project, Properties, Signing, check Sign the assembly. Choose or make a new strong name key file. If you are creating, you can uncheck Protect my key file with a password.
Build the project.
Now, we must import the assymetric signing key, create a login in the master database and concede UNSAFE ASSEMBLY permission to it:

USE [master]
GO

CREATE ASYMMETRIC KEY AssymetricKey1
FROM EXECUTABLE FILE = 'C:\Projects\MyDotnetSqlProj\bin\Debug\MyDotnetSqlProj.dll';
GO

CREATE LOGIN LoginAssymetricKey1
FROM ASYMMETRIC KEY AssymetricKey1;
GO

GRANT UNSAFE ASSEMBLY TO LoginAssymetricKey1;
GO

In the application database, create an user for the recently created login, import the assembly and enable CLR execution:

USE [YourDB];
GO

CREATE USER UserLoginAssymetricKey1
FOR LOGIN LoginAssymetricKey1;
GO

CREATE ASSEMBLY AssemblyMyLib
FROM 'C:\Projects\MyDotnetSqlProj\bin\Debug\MyDotnetSqlProj.dll'
WITH PERMISSION_SET = SAFE;
GO

sp_configure 'clr enabled', 1;
GO
RECONFIGURE;
GO

Let's create a FUNCTION that calls the DLL's code:

USE [YourDB];
GO

-- for string, use NVARCHAR parameters
CREATE FUNCTION [dbo].[CalculateHypotenuse] (@side1 FLOAT, @side2 FLOAT)
RETURNS FLOAT
AS
  EXTERNAL NAME [AssemblyMyLib].[MyDotnetSqlProj.Class1].[CalculateHypotenuse];
GO

To execute the function:

-- returns 5
SELECT [dbo].[CalculateHypotenuse] (3.0, 4.0) as [Hypotenuse];

-- returns 13
SELECT [dbo].[CalculateHypotenuse] (5.0, 12.0) as [Hypotenuse];

JSON

To handle JSONs inside the database.

DECLARE @json NVARCHAR(100) =
'{"cities": ["Aracaju", "João Pessoa", "Maceió"]}';

-- returns 3 lines with the cities
SELECT * FROM OPENJSON(@json, '$.cities')

-- returns 1 line with the selected city
SELECT JSON_VALUE(@json, '$.cities[1]')

Check more in the docs.

APPLOCKS

They are application locks inside the database. They are different from row locks.

Can be used for concurrency control, such as for reading messages with multiple simultaneous consumers.

The SQL Server functions are sp_getapplock and sp_releaseapplock.

Learn more in the article about asynchronous architecture without queues.

VIEW EXECUTION PLAN

The execution plan describes which steps the database engine needs to perform to execute a command or query.

On SSMS, enabling the Include execution plan on the top menu will include the plan along with query results.

By analyzing the cost of each step, you can understand which performance bottlenecks exist and from there improve your query, by creating indexes, rewriting WHERE and JOIN criteria, and others.

Final conclusions

SQL is an extremely powerful and complete tool, and acquired new capabilities over the years. It is no wonder that remains being the mainstream for databases up to today; it is the basis for complex systems.

This article covered common and less known SQL features, and still there are many others that could be addressed here with real world applications, therefore, it is worth it to explore the documentation of your database, because for many times it has the answer for a challenge you might be facing.

Shiki and Mermaid in 11ty

2024-07-16T00:00:00Z

11ty, Shiki and Mermaid

11ty (Eleventy) is a static-site generator tool that uses templating engines to generate websites. Eleventy has a pipeline execution that makes it easy to add plugins and transformations to HTML, JS and CSS files.

Shiki is a npm package that renders HTML code blocks with syntax highlighting for a specified language. Among its interesting features, it allows dual theming (light and dark switching), code focusing and diffing notations, and custom themes.

While I was making this blog, which uses 11ty, I wanted to have support for Mermaid diagrams in my blog posts, alongside code blocks. In this post, let's see how to integrate Shiki and Mermaid into your 11ty project.

Shiki integration

Add Shiki to your project

npm install -D shiki

Shiki custom plug-in

Create a Javascript file to configure Shiki for your project, for example, at ./shiki.config.mjs (let's make it using ES Modules). The highlighter ideally should be used as a singleton.

import { createHighlighter } from "shiki";

export default function (eleventyConfig, options) {
  // empty call to notify 11ty that we use this feature
  // eslint-disable-next-line no-empty-function
  eleventyConfig.amendLibrary('md', () => { });

  eleventyConfig.on('eleventy.before', async () => {

    const highlighter = await createHighlighter(
    {
      themes: ["light-plus", "dark-plus"],
      langs: ['html', 'sql', 'xml', 'javascript']
    });

    eleventyConfig.amendLibrary('md', (mdLib) =>
      mdLib.set({
        highlight: (code, lang) => {
          return highlighter.codeToHtml(code,
          {
            lang: lang,
            themes: {
              light: "light-plus",
              dark: "dark-plus"
            }
          });
        }
      })
    );
  });
};

Call Shiki custom plug-in in `eleventy.config.mjs`

import shikiPlugin from "./shiki.config.mjs"; 

export default function (eleventyConfig) {
  // ...
  // replace 11ty default syntax highlight plugin
  eleventyConfig.addPlugin(syntaxHighlight) 
  eleventyConfig.addPlugin(shikiPlugin); 
  // ...
}

Mermaid integration

If you want to use Shiki and also want to render Mermaid diagrams, you need to change the Shiki plug-in to render Mermaid as HTML divs, instead of code blocks.

Add htmlencode to your project

npm install -D htmlencode

Modify Shiki plug-in

import { createHighlighter } from "shiki";
import htmlencode from "htmlencode"; 

export default function (eleventyConfig, options) {
  // empty call to notify 11ty that we use this feature
  // eslint-disable-next-line no-empty-function
  eleventyConfig.amendLibrary('md', () => { });

  eleventyConfig.on('eleventy.before', async () => {

    const highlighter = await createHighlighter(
    {
      themes: ["light-plus", "dark-plus"],
      langs: ['html', 'sql', 'xml', 'javascript']
    });

    eleventyConfig.amendLibrary('md', (mdLib) =>
      mdLib.set({
        highlight: (code, lang) => {
          if (lang === "mermaid") { 
            const extra_classes = options?.extra_classes ? ' ' + options.extra_classes : ''; 
            return `<div class="mermaid${extra_classes}">${htmlencode.htmlEncode(code)}</div>`; 
          } 
          else { 
            return highlighter.codeToHtml(code,
            {
              lang: lang,
              themes: {
                light: "light-plus",
                dark: "dark-plus"
              }
            });
          } 
        }
      })
    );
  });
};

Mermaid custom plug-in

Create a Javascript file to configure Mermaid for your project, for example, at ./mermaid.config.mjs (let's make it using ES Modules):

export default (eleventyConfig, options) => {
  let mermaid_config = {
    startOnLoad: false,
    theme: "default",
    loadOnSave: true
  };
  let src = options?.mermaid_js_src || "https://unpkg.com/mermaid/dist/mermaid.esm.min.mjs";

  eleventyConfig.addShortcode("mermaid_js_scripts", () => {
    return `<script type="module" async>
              import mermaid from "${src}";
              const config = ${JSON.stringify(mermaid_config)};
              mermaid.initialize(config);
              await mermaid.run({
                querySelector: '.mermaid'
              });
            </script>`
  });
  return {}
};

Call Mermaid custom plug-in in `eleventy.config.mjs`

import shikiPlugin from "./shiki.config.mjs";
import mermaidPlugin from "./mermaid.config.mjs"; 

export default function (eleventyConfig) {
  // ...
  eleventyConfig.addPlugin(shikiPlugin);
  eleventyConfig.addPlugin(mermaidPlugin); 
  // ...
}

Include Mermaid script in pages

Add the mermaid_js_scripts shortcode to pages and templates that will have diagrams:

{% mermaid_js_scripts %}

Final results

Let's see it working.

SQL code block with syntax highlight:

SELECT * INTO dbo.NewProducts
FROM Production.Product
WHERE ListPrice > 25
AND ListPrice < 100;

Mermaid flowchart:

flowchart TD
    A[Christmas] -->|Get money| B(Go shopping)
    B --> C{Let me think}
    C -->|One| D[Laptop]
    C -->|Two| E[iPhone]
    C -->|Three| F[fa:fa-car Car]

flowchart TD
    A[Christmas] -->|Get money| B(Go shopping)
    B --> C{Let me think}
    C -->|One| D[Laptop]
    C -->|Two| E[iPhone]
    C -->|Three| F[fa:fa-car Car]

If you want to check a full source code example for this blog project, visit our GitHub repo. It includes switching between light and dark themes for both Shiki and Mermaid.

Copy code buttons

Shiki doesn't provide copy code buttons by default, but we can add them through a client-side script. I use a slightly modified version of the script shown in this blog post.

Reference the script in templates

Add the <script> tag below to templates and HTML files:

<script src="/assets/scripts/addCopyCodeButtons.js"></script>

Add script to insert copy code buttons

Create a Javascript file for a client-side script that will find all code blocks and add copy code buttons to them. The icons SVGs are declared inline in this example.

For example, create your file at src/scripts/addCopyCodeButtons.js:

let blocks = document.querySelectorAll("pre.shiki");
const copyCodeIconSvg = '<svg height="16" viewBox="0 0 16 16" version="1.1" width="16"><path fill="#666666" d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path fill="#666666" d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg>';
const codeCopiedIconSvg = '<svg height="16" viewBox="0 0 16 16" version="1.1" width="16"><path fill="#28D751" d="M13.78 4.22a.75.75 0 0 1 0 1.06l-7.25 7.25a.75.75 0 0 1-1.06 0L2.22 9.28a.751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018L6 10.94l6.72-6.72a.75.75 0 0 1 1.06 0Z"></path></svg>';

blocks.forEach((block) => {
    if (!navigator.clipboard) {
        return;
    }

    let button = document.createElement("button");
    button.className = "button-copy-code";
    button.ariaLabel = button.title = "Copy";
    button.innerHTML = copyCodeIconSvg;
    block.appendChild(button);

    button.addEventListener("click", async () => {
        await copyCode(button, block);
    });
});

async function copyCode(button, block) {
    let copiedCode = block.cloneNode(true);
    copiedCode.removeChild(copiedCode.querySelector("button.button-copy-code"));

    const html = copiedCode.outerHTML.replace(/<[^>]*>?/gm, "");

    block.querySelector("button.button-copy-code").innerHTML = codeCopiedIconSvg;
    button.ariaLabel = button.title = "Copied!";
    setTimeout(function () {
        block.querySelector("button.button-copy-code").innerHTML = copyCodeIconSvg;
        button.ariaLabel = button.title = "Copy";
    }, 2000);

    const parsedHTML = htmlDecode(html);

    await navigator.clipboard.writeText(parsedHTML);
}

function htmlDecode(input) {
    const doc = new DOMParser().parseFromString(input, "text/html");
    return doc.documentElement.textContent;
}

CSS for copy code buttons

Add the CSS code below to style the button position and visibility:

pre[class*="shiki"] {
  position: relative;
  margin: 5px 0;
  padding: 1.75rem 0 1.75rem 1rem;
}

pre > .button-copy-code {
  position: absolute;
  top: 32px;
  right: 16px;
}

@media only screen and (max-device-width: 480px) {
  pre > .button-copy-code {
    display: none;
  }
}

References

Asynchronous architecture without queues

2024-06-24T00:00:00Z

Asynchronous architecture

A common strategy in complex systems is asynchronism, which is leaving heavier tasks running on background and getting a return from them only when they are finished. The idea is to deliver an immediate response to the user, allowing him / her to do other activities in the meanwhile. This approach exists both in programming languages, by the use of the async / await keywords, and in systems architectures, with an event or message oriented processing.

There are dedicated queues that are very popular today, among them, Apache Kafka and RabbitMQ, that are used for asynchronous processing. They can handle high volumes of messages received and processed in parallel, taking care of concurrency and post-processing confirmation.

Despite the popularity of those queues, other lighter and more practical alternatives can be used in substitution, solving the same problems in a simpler and even cheaper way. Let's see them.

Messages in databases

Here, a database stores the messages. There is an application that inserts messages in the database (publisher) and another that reads them (consumer).

In this case, messages are persisted and there is no risk of losing them due to reinitializations. Both SQL and NoSQL databases can be used.

For example, we can have a SQL table that contains the messages:

| Id |      Status      | Content  |
|----|------------------|----------|
|  1 | Processed        |   msg1   |
|  2 | Processed        |   msg2   |
|  3 | ProcessingFailed |   msg3   |
|  4 | InProcess        |   msg4   |
|  5 | Waiting          |   msg5   |
|  6 | Waiting          |   msg6   |

The Status column describes which message already were processed and which are waiting. At least four status need to exist:

Waiting processing
In process
Processing failed
Processed

The ProcessingFailed status is very important, because it allows developers to analyse the failed messages for possible bugs at the processing, and to identify messages that come invalid or malformatted.

Here, if there is more than one simultaneous consumer, we must prepare against concurrency issues, to avoid that a message be processed more than once. The solution for this is a lock.

Database locks

A lock (mutex) restrains access for only the transaction that detains this lock.

When a consumer goes to take the next message for processing, it should lock for reading, to block other consumers from taking this same message at the same time. If a consumer cannot open the lock, it is because another consumer arrived earlier, which means that it should wait until the lock is released.

During the locking, the message is retrieved and marked as InProcess, and after that, the reading is unlocked for other consumers.

This technique exists in many databases:

SQL Server: sp_getapplock with sp_releaseapplock
PostgreSQL: pg_advisory_xact_lock() with SET LOCAL lock_timeout = '4s'
MongoDB: db.coll.findAndModify()

The code below is a SQL Server stored procedure that takes the next message to be consumed using an app lock.

CREATE OR ALTER PROCEDURE TakeMessageToProcess
    @status VARCHAR(30)
AS
BEGIN TRANSACTION
    DECLARE @res INT
    -- locking for reading
    EXEC @res = sp_getapplock                 
         @Resource = 'TAKE_MESSAGE_LOCK',
         @LockMode = 'Exclusive',
         @LockOwner = 'Transaction',
         @LockTimeout = 60000,
         @DbPrincipal = 'public';
    -- verifying that the lock was obtained
    IF @res NOT IN (0, 1)
    BEGIN
        RAISERROR ('Unable to get lock.', 16, 1)
    END 
    ELSE
    BEGIN    
      -- taking a message with the desired status
      DECLARE @msgId INT = (SELECT TOP 1 [Id] FROM [dbo].[Messages] WHERE [Status] = @status);
      -- marking message status as 'InProcess'
      UPDATE [dbo].[Messages] SET [Status] = 'InProcess' WHERE [Id] = @msgId;
      -- returning the message for consumer
      SELECT * FROM [dbo].[Messages] WHERE [Id] = @msgId;
      -- releasing the lock after reading
      EXEC @res = sp_releaseapplock 
           @Resource = 'TAKE_MESSAGE_LOCK',
           @DbPrincipal = 'public',
           @LockOwner = 'Transaction';
    END    
COMMIT
GO

The consumer must execute the SQL command:

EXECUTE TakeMessageToProcess @status = 'Waiting'

Invocation

The database cannot proactively invoke a consumer, alerting it that there is a new message to be processed. Therefore, other triggers must be used, such as the consumer periodically checking for new messages; or cron jobs, that schedule the consumer for execution on certain times of the day.

Messages in distributed caches

Distributed caches, like Redis, can store messages, holding them inside key-value pairs, queues, or other data structures.

This implementation can be considered a variation of the one above.

An advantage is that Redis already has native queues, saving development effort.

However, a disadvantage is that messages can be lost in case of a reinitialization of the cache, since there is no persistence (due to being a volatile storage).

Example of a queue using Redis commands:

# inserts at the end of the queue
LPUSH my_queue "Task1"

# inserts at the end of the queue
LPUSH my_queue "Task2"

# pops the head of the queue: "Task1"
RPOP my_queue

# pops the head of the line: "Task2"
RPOP my_queue

Distributed locks

Distributed caches can provide locks to prevent read concurrencies. Redis documentation details how to implement a distributed lock in a safe way and considering multiple consumers.

Parallel threads

This option is very practical when you want to condense the processing in a single application — it produces and consumes the message internally, in separate threads.

The communication between production and consumption threads is through thread-safe concurrent queues, like channels, in .NET, or ConcurrentLinkedQueue, in Java. The consumption thread must run indefinitely in background, processing messages as they arrive in the queue.

Some real life scenarios in which this option may be appropriate:

User registration: an API receives a call for user registration, which involves many substeps, such as identity verification, anti-fraud prevention and sending a confirmation email. These substeps may be light enough to not need to be run in a separate application.
Logs and metrics: in most programs, logs and metrics are sent to their servers in background threads.

We must take into account that this processing implies higher CPU and RAM usage on the application and may compete with other tasks for resources. Solutions for that are vertical scaling (increase available CPU and RAM) and horizontal scaling (distribute the load among other instances).

Practical example

The code below is an ASP.NET minimal API that uses channels to process messages, that are received on an HTTP endpoint and handled by a MyMessageProcessor.

To run the program below:

Have the .NET SDK installed on your machine
Create a minimal API project:

# command-line
mkdir MyChannelsAPI
cd ./MyChannelsAPI/
dotnet new web

Copy the code below and paste it in the Program.cs file
Run the project, dotnet run

using System.Threading.Channels;
using Microsoft.AspNetCore.Mvc;

var builder = WebApplication.CreateBuilder(args);

// creating the channel and the processor
var channel = Channel.CreateUnbounded<MyMessage>();
builder.Services.AddSingleton<ChannelWriter<MyMessage>>(channel.Writer);
builder.Services.AddSingleton<ChannelReader<MyMessage>>(channel.Reader);
builder.Services.AddSingleton<MyMessageProcessor>();

var app = builder.Build();

app.MapGet("/", () => "Hello World!");
// endpoint for receiving messages:
// GET /message?content=my_message_here
app.MapGet("/message",
    async
    (
        [FromQuery(Name = "content")] string? content,
        [FromServices] ILogger<MyMessage> logger,
        [FromServices] ChannelWriter<MyMessage> channelWriter
    ) => 
    {
        logger.LogInformation("Message received: {content}", content);
        await channelWriter.WriteAsync(new(content));
        return Results.Ok();
    });

// required to begin processing
var msgProcessor = app.Services.GetService<MyMessageProcessor>();
msgProcessor!.StartProcessing();

app.Run();

// message class
public sealed record MyMessage(string? Content);

// processor class
public sealed class MyMessageProcessor
{
    private readonly ILogger<MyMessageProcessor> logger;
    private readonly ChannelReader<MyMessage> channelReader;

    public MyMessageProcessor(ILogger<MyMessageProcessor> logger,
                              ChannelReader<MyMessage> channelReader)
    {
        this.logger = logger;
        this.channelReader = channelReader;
    }

    public void StartProcessing()
    {
        // starts background thread
        // to process messages
        Task.Factory.StartNew(async () =>
        {
            await foreach (var msg in channelReader.ReadAllAsync())
            {
                await ProcessMessageAsync(msg);
            }
        }, TaskCreationOptions.LongRunning);
    }

    private async Task ProcessMessageAsync(MyMessage msg)
    {
        logger.LogInformation("Processing message: {content}", msg.Content);
        // operations here
        await Task.Delay(3000);
        logger.LogInformation("Message processed: {content}", msg.Content);
    }
}

To test, open a browser and go to the URL: http://localhost:{port}/message?content=my_message_here

Criteria analysis

To decide if an asynchronous architecture suits your case, consider:

Does the user need an immediate response?
Does an individual message take a long time to process?
Does an individual message use a lot of CPU and RAM to process?
Can the messages be handled in batches (many at once)?
Is the volume of messages expected to be high?

Many positive answers above indicate that it is adequate.

To choose which implementation to use, ask yourself:

Does the processing need to be started immediately?
Does the order of processing matter?
Is there a priority of messages?
Can there be more than one consumer at the same time?
Would the eventual loss of a message be tolerable?
If a message is lost, how to recover it?
If a message is invalid or unprocessable, how to treat it?
If a message was partially processed, would a retry cause troubles?

The questions above can help you to decide which asynchronous strategy is the best for your scenario.

Sources and interesting reads

Introduction to functional programming

2024-06-05T00:00:00Z

Paradigms

A paradigm is a manner of thinking, observing and describing something; it's a mental model, through which we understand the things around us. Each person has its own way of seeing the world and this reflects on the things he / she does and makes.

In programming, a code can be written in many different ways with equivalent results — which means that the concept of paradigm also applies.

There are some programming paradigms that are the most popular, among them:

Imperative programming: Here, the program is organized around orders: "do this", "do that", "if this, then do this other thing".
Object-oriented programming (OOP): The focus is on who (or what) performs an action. Each object is an actor responsible for an action, and can call other objects to execute other actions.
Functional programming (FP): In this paradigm, the model revolves around how actions interconnect. They are joined together like in an assembly line, where the end of a step arrives at the beginning of the next one, and in the end there is a finished product.

A little bit about FP

Functional programming origins are on mathematical functions.

Mathematical functions can be composed, such that the output of one is the input of another:

f(x) = 3x + 1

g(x) = 2x

// compositions

f ○ g (x) = f(g(x)) = 3(2x) + 1 = 6x + 1

g ○ f (x) = g(f(x)) = 2(3x + 1) = 6x + 2

The same rationale applies for functional programming: the return of a method is frequently used as the input of other method. The concrete example below shows function composition in F#.

module Program =

    let f(x: int) = 3*x + 1

    let g(x: int) = 2*x

    let fg = g >> f // f(g(x)), output of g is input of f

    let gf = f >> g // g(f(x)), output of f is input of g

    let [<EntryPoint>] main _ =
        printfn "f(1) = %d" (f(1))
        printfn "g(1) = %d" (g(1))
        printfn "fg(1) = %d" (fg(1))
        printfn "gf(1) = %d" (gf(1))
        0

    // will print:
    // f(1) = 4
    // g(1) = 2
    // fg(1) = 7
    // gf(1) = 8

When to use functional programming

FP excels in mathematical-related tasks. Think on engineering calculus, accounting, taxes, finances, calendars and statistics. It also handles very well categorizations and pattern detections.

Artificial intelligence and machine learning need heavy mathematics and therefore benefit from functional programming.

Can I call a functional code from a OOP or imperative project?

Yes, you can! Some object-oriented languages allow invoking functional languages, such as C# code interacting with F#, or Java code interacting with Scala.

When not to use functional programming

For tasks like sending and receiving messages, reading and writing informations in a database, or publishing in a queue, there are not many advantages in favor of FP, because most actions do not interlace to each other. In these cases, a OOP or imperative language is usually better.

F# language

In this article, we will use the F# functional language for a case study. F# uses .NET runtime.

To setup your machine for F# development, you will need:

.NET SDK
Visual Studio or
VS Code with F# language server updated extension (the Ionide extension is no good)

To create an F# project, enter on the command line:

mkdir MyFSharpProject
cd ./MyFSharpProject/
dotnet new console --language F#

Before we begin, some syntax tips:

Function declaration:

// note that each parameter has its own pair of parentheses.
// the last line is what the function returns.
let sum(x: int)(y: int) =
  x + y

Anonymous function / lambda:

fun x -> x * 2

Discard operator _

let myTuple = (1, 2)
// only the first part is relevant.
// the second is discarded `_`
let (x, _) = myTuple

Pipelining |>

let numbers = [ 1; 2; 3 ]
let doble(x: int) = x * 2
let increment(x: int) = x + 1

(*
|> means the result of last function is argument 
for the last parameter of the next function.

numbers |> Seq.map(increment)

is the same as:

Seq.map(increment)(numbers)
*)

let doubledIncrementedNumbers =
  numbers // 1, 2, 3
  |> Seq.map(increment) // 2, 3, 4
  |> Seq.map(doble) // 4, 6, 8

Case study: poker

In poker, a player has a hand with five deck cards, each card defined by a suit with a rank. There are four suits (diamonds ♦️, clubs ♣️, hearts ♥️ and spades ♠️) and thirteen ranks (2 to 10, jack J, queen Q, king K, ace A). The cards of a hand can form special combinations and the player that has the rarest combination wins the round.

Combination	Example	Description	Chance
Royal Straight Flush	A️♦️, K♦️️️, Q♦️️️, J♦️️️, 10♦️️	Consecutive sequence of a same suit, from 10 to ace.	0,000154%
Straight Flush	5♣️, 6♣️,️ 7♣️,️ 8♣️, 9♣️	Consecutive sequence of a same suit (except the royal, above).	0,00139%
Four of a kind	J♣️, J♠️,️ J♦️️️,️ J♥️️️, 2♣️️	Four cards with the same rank.	0,02401%
Full House	4♣️, 4♦️️️,️ 10♦️️, 10♣️️, 10♠️	Three cards of a rank and two of another rank.	0,1441%
Flush	4♥️️️, 7♥️️️,️ 8♥️️️, J♥️️️, A♥️️️	All cards with the same suit, but no consecutive sequence.	0,1965%
Straight	A♥️️️, 2♣️,️ 3♥️️️, 4♦️️️, 5♠️	Consecutive sequence of ranks, but cards with different suits.	0,3925%
Three of a kind	2♦️️, K♣️,️ 7♦️️️, 7♣️, 7♠️	Three cards with the same rank.	2,1128%
Two pair	8♥️️️,️ 8♦️️️, 9♥️️️, 9♠️, 7♥️️	One pair of a rank and another pair of a different rank.	4,7539%
Pair	10♦️️,️ K♠️, K♥️️️, 2♠️, 5♠️	Two cards with the same rank.	42,2569%
High card	6♦️️, 3♦️️, 10♥️️,️ 7♦️️, 2♦️️	When the combination does not match one of the above.	50,1177%

Domain code for a poker game

namespace Poker.Domain

module PokerGame =

    // this is a discriminated union
    type Suit = Spades | Clubs | Hearts | Diamonds

    // this is an enum
    type Rank =
    | Two = 2 | Three = 3 | Four = 4 | Five = 5 | Six = 6 | Seven = 7 | Eight = 8
    | Nine = 9 | Ten = 10 | Jack = 11 | Queen = 12 | King = 13 | Ace = 14

    // this is a tuple
    type Card = Suit * Rank

    type Hand = Card list

    type HandRanking =
    | RoyalStraightFlush = 10
    | StraightFlush = 9
    | FourOfAKind = 8
    | FullHouse = 7
    | Flush = 6
    | Straight = 5
    | ThreeOfAKind = 4
    | TwoPairs = 3
    | OnePair = 2
    | HighCard = 1

    let hasStraight(hand: Hand) =
        hand
        |> Seq.sortBy(fun (_,rank) -> rank)
        |> Seq.pairwise
        |> Seq.forall(fun ((_,rank1), (_,rank2)) ->
            (int rank2) - (int rank1) = 1 || (rank2 = Rank.Ace && rank1 = Rank.Five))

    let hasFlush(hand: Hand) =
        let (someSuit, someRank) = Seq.head hand
        hand
        |> Seq.map (fun (suit,_) -> suit)
        |> Seq.forall ((=)someSuit)

    let hasKingAndAce(hand: Hand) =
        let hasKing = Seq.exists(fun (_,rank) -> rank = Rank.King)(hand)
        let hasAce = Seq.exists(fun (_,rank) -> rank = Rank.Ace)(hand)
        hasKing && hasAce

    let getSortedQuantitiesPerRank(hand: Hand) =
        hand
        |> Seq.countBy(fun (_, rank) -> rank)
        |> Seq.sortBy(fun (_, qtyForRank) -> qtyForRank)
        |> Seq.map(fun (_, qtyForRank) -> qtyForRank)
        |> Seq.toList

    let getHandRanking(hand: Hand) =
        match getSortedQuantitiesPerRank(hand) with
        | [ 1; 4 ] -> HandRanking.FourOfAKind
        | [ 2; 3 ] -> HandRanking.FullHouse
        | [ 1; 1; 3 ] -> HandRanking.ThreeOfAKind
        | [ 1; 2; 2 ] -> HandRanking.TwoPairs
        | [ 1; 1; 1; 2 ] -> HandRanking.OnePair
        | _ -> match (hasStraight(hand), hasFlush(hand), hasKingAndAce(hand)) with
               | (true, true, true) -> HandRanking.RoyalStraightFlush
               | (true, true, false) -> HandRanking.StraightFlush
               | (true, false, _) -> HandRanking.Straight
               | (false, true, _) -> HandRanking.Flush
               | _ -> HandRanking.HighCard

In only one file, we were able to describe the logic of the problem that needs to be resolved. Even if the language syntax is not familiar for those seeing it for the first time, there is a logical consistency of transformations and comparisons in the methods.

The last method, getHandRanking, compares the quantities of cards by rank to know if there is a combination by quantity (four-of-a-kind, full house, three-of-a-kind, two pairs, pair). If there are none, then analyses if the combination is sequential and/or by suits (straight, flush, straight flush and royal straight flush). In the end, if there are no matches, then it is a high card ranking.

I confess that when I first used a functional language I found it difficult, a bit because of the syntax which is very unique, and because many code examples on the internet use implicit typing, which is less obvious to understand:

// implicit typing (inference)
let sum x y =
  x + y

// most explicit typing available
let sum(x: int)(y: int): int =
  x + y

Nevertheless, if you have a situation where a functional approach is better, I say that it's worth to persist and learn, starting with simpler and smaller examples, and advancing as you feel more confident.

The code above is available on GitHub, with unit tests and showing how to integrate a F# project with a C# project.

fs files for compilation

If you are using VS Code, whenever you add a new .fs file to your project, remember to include it in your .fsproj:

<ItemGroup>
  <Compile Include="Library.fs" />
  <Compile Include="Module2.fs" />
</ItemGroup>

Sources and interesting reads

Pipelines for .NET

2024-04-29T00:00:00Z

Pipelines

Pipelines are sequences of commands that are executed to ensure the integrity of the code and to produce a final artifact, such as an executable program or a library.

Having a pipeline means having a consistent process that mitigates the risk of human errors in the final product and saves the programmer's time, because he/she can take care of other tasks while the code is compiled, verified and packed.

Steps

flowchart TD
    checkout -->
    clean --> 
    restore -->
    audit -->
    sbom -->
    build -->
    unittests[unit tests]
    unittests --if program--> publish --with docker--> dockerk8s[docker / kubernetes]
    publish --no docker--> zipinstaller[zip / installer]
    unittests --if nuget package--> pack --> nugetpush[nuget push]

checkout

Fetches the code from the Git branch. If it's an integration (CI) process, the branch is the one that intends to be merged; if it's a deployment (CD), the code is from the release branch, such as develop, master or release_candidate.

Command line: git clone

clean

Cleans the bin and obj folders, to guarantee that everything will start from zero, with no interferences from previous compilations.

Command line: dotnet clean

restore

Guarantees that the references between projects in the solution are correct and downloads the required NuGet packages.

Command line: dotnet restore

audit

Verifies if there are any NuGet packages in the project with security problems, checking in the CVE (Common Vulnerabilities and Exposures) and GHSA (GitHub Advisory Database) lists.

Command line: dotnet list package --vulnerable --include-transitive

sbom

SBOM, software bill of materials, is a document that informs which components were used to produce a program or a library.

This document is of utmost importance for critical software, because with it, organizations can easily know which of its applications are in danger when a vulnerability in a library is reported. After the 2020 cyberattack on the USA government, SBOMs became endorsed by the White House.

I recommend the CycloneDX format, for being more succinct and easier to read.

Command line:

In CycloneDX format: dotnet CycloneDX
In SPDX format: sbom-tool generate

build

Compiles the solution code.

Command line: dotnet build

unit tests

Runs the solution's unit tests to ensure that they are passing.

In this step, we can produce a report that shows the coverage level of the unit tests against the code, revealing which classes, methods and lines were covered by the tests. ReportGenerator is the main tool for these reports in .NET projects.

Command line:

Unit tests: dotnet test
Coverage report: reportgenerator

publish

Generates the final program for execution. This step differs from build because here certain compilation options can be specified, like the target runtime, self-contained, single-file, and others.

Command line: dotnet publish

pack

Makes a NuGet package, in case of a code that is meant to be a library.

Command line: dotnet pack

nuget push

Uploads the package to a NuGet server, private or public, so it can be used by other people.

Command line: dotnet nuget push

Pipeline engines

There are many pipeline engines available, such as GitHub Actions, GitLab CI, Jenkins, Azure Pipelines, CircleCI and many others.

You can also have your pipeline as a script, to run locally in your machine. This is a good practice for being a safeguard when your remote pipeline is unavailable or offline and because you can test modifications before commiting them.

I personally recommend using PowerShell scripts for local pipelines, because it's a multiplatform and friendly language, with easy interaction with XML and JSON. Nevertheless, you can use other scripting languages, like Batch, Shell, Python and others you like.

GitHub Actions example for .NET program

name: Publish console / API / desktop program

on:
  workflow_dispatch: # manual trigger
    inputs:
      version:
        required: true
        type: string
      rid:
        required: true
        default: linux-x64 # where the program will run
        type: string
        # https://learn.microsoft.com/en-us/dotnet/core/rid-catalog

jobs:
  generate_program:

    runs-on: ubuntu-latest

    env:
      OUTPUT_FOLDER: ${{ format('./out/{0}/', inputs.rid) }}
      VERSION_NAME: ${{ inputs.version }}
      RID: ${{ inputs.rid }}

    steps:
    - name: Checkout
      uses: actions/checkout@v4
      with:
        fetch-depth: 1

    - name: Install .NET SDK
      uses: actions/setup-dotnet@v4
      with:
        dotnet-version: 8.x # .NET version here

    - name: Install CycloneDX .NET
      run: dotnet tool install --global CycloneDX

    - name: Clean solution
      run: dotnet clean --nologo --verbosity quiet
    
    - name: Restore solution
      run: dotnet restore --nologo --verbosity quiet

    - name: Audit solution
      shell: pwsh
      run: |
        $projectPath = "./src/MyProject.Console/MyProject.Console.csproj"
        $jsonObj = (dotnet list $projectPath package --vulnerable --include-transitive --format json) | ConvertFrom-Json;
        $hasAnyVulnerability = ($jsonObj.projects[0].frameworks -ne $null);
        if ($hasAnyVulnerability) {
          dotnet list package --vulnerable --include-transitive;
          exit 1;
        }

    - name: Build solution
      run: dotnet build --no-restore --configuration Release --nologo --verbosity quiet

    - name: Run unit tests
      run: dotnet test --no-build --configuration Release --nologo --verbosity quiet --collect:"XPlat Code Coverage" --results-directory ./TestResults/

    - name: Unit tests coverage report
      uses: danielpalme/ReportGenerator-GitHub-Action@5.2.4
      with:
        reports: TestResults/**/coverage.cobertura.xml
        targetdir: TestResults
        reporttypes: JsonSummary;Html

    - name: Generate SBOM
      shell: pwsh
      run: dotnet CycloneDX ./src/MyProject.Console/MyProject.Console.csproj -o $env:OUTPUT_FOLDER -f sbom.json -sv $env:VERSION_NAME --json

    - name: Publish program
      shell: pwsh
      run: |
        dotnet publish ./src/MyProject.Console/MyProject.Console.csproj `
        --verbosity quiet `
        --nologo `
        --configuration Release `
        -p:PublishSingleFile=true `
        -p:Version=${env:VERSION_NAME} `
        --self-contained true `
        --runtime ${env:RID} `
        --output ${env:OUTPUT_FOLDER};

    - name: Set execution attributes (UNIX only)
      if: ${{ startsWith(inputs.rid, 'linux') || startsWith(inputs.rid, 'osx') }}
      shell: pwsh
      run: chmod +x "${env:OUTPUT_FOLDER}/MyProject.Console"

    - name: Pack program
      shell: pwsh
      run: |
        $zipName = "MyProject.Console_${env:VERSION_NAME}_${env:RID}.zip";
        # if Linux or MacOSX, we should use zip instead of Compress-Archive,
        # to preserve the Unix file attributes.
        if ($IsWindows) {
          Compress-Archive -CompressionLevel Optimal -Path $env:OUTPUT_FOLDER -DestinationPath "./out/${zipName}"
        } else {
          cd $env:OUTPUT_FOLDER
          zip -9 -r ../${zipName} *
          cd ../..
        }
        Remove-Item $env:OUTPUT_FOLDER -Force -Recurse -ErrorAction Ignore
        echo "OUTPUT_FILE_NAME=${zipName}" | Out-File -FilePath $Env:GITHUB_ENV -Encoding utf8 -Append

    - name: Upload program to workflow results
      uses: actions/upload-artifact@v4
      with:
        compression-level: 0 # previous step already compresses
        name: ${{ env.OUTPUT_FILE_NAME }}
        path: ${{ format('./out/{0}', env.OUTPUT_FILE_NAME) }}
    
    - name: Upload SBOM to workflow results
      uses: actions/upload-artifact@v4
      with:
        name: sbom.json
        path: ./out/sbom.json
    
    - name: Upload coverage report to workflow results
      uses: actions/upload-artifact@v4
      with:
        name: coverage_report
        path: TestResults
    
    # other subsequent steps can be added here,
    # like docker and kubernetes,
    # or installer generation, in case of desktop programs.

GitHub Actions example for NuGet package

name: Publish NuGet package

on:
  workflow_dispatch: # manual trigger

jobs:
  generate_nuget_package:

    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v4
      with:
        fetch-depth: 1

    - name: Install .NET SDK
      uses: actions/setup-dotnet@v4
      with:
        dotnet-version: 8.x # .NET version here

    - name: Install CycloneDX .NET
      run: dotnet tool install --global CycloneDX

    - name: Clean solution
      run: dotnet clean --nologo --verbosity quiet
    
    - name: Restore solution
      run: dotnet restore --nologo --verbosity quiet

    - name: Audit solution
      shell: pwsh
      run: |
        $projectPath = "./src/MyProject.Console/MyProject.Console.csproj"
        $jsonObj = (dotnet list $projectPath package --vulnerable --include-transitive --format json) | ConvertFrom-Json;
        $hasAnyVulnerability = ($jsonObj.projects[0].frameworks -ne $null);
        if ($hasAnyVulnerability) {
          dotnet list package --vulnerable --include-transitive;
          exit 1;
        }

    - name: Build solution
      run: dotnet build --no-restore --configuration Release --nologo --verbosity quiet

    - name: Run unit tests
      run: dotnet test --no-build --configuration Release --nologo --verbosity quiet --collect:"XPlat Code Coverage" --results-directory ./TestResults/

    - name: Unit tests coverage report
      uses: danielpalme/ReportGenerator-GitHub-Action@5.2.4
      with:
        reports: TestResults/**/coverage.cobertura.xml
        targetdir: TestResults
        reporttypes: JsonSummary;Html

    - name: Read package version
      shell: pwsh
      run: |
        # PackageVersion needs to be declared in .csproj
        ([XML]$nugetCsprojXml = Get-Content ./src/MyProject.Library/MyProject.Library.csproj)
        $versionName = $nugetCsprojXml.Project.PropertyGroup.PackageVersion
        # adds to workflow environment variables
        echo "VERSION_NAME=${versionName}" | Out-File -FilePath $Env:GITHUB_ENV -Encoding utf8 -Append

    - name: Generate SBOM
      shell: pwsh
      run: dotnet CycloneDX ./src/MyProject.Library/MyProject.Library.csproj -o ./out/ -f sbom_MyProject_Library.json -sv $env:VERSION_NAME --json

    - name: Generate package
      run: dotnet pack ./src/MyProject.Library/MyProject.Library.csproj --nologo --verbosity quiet --configuration Release
    
    - name: Upload package to NuGet server
      shell: pwsh
      run: |
        $filePath = "./src/MyProject.Library/bin/Release/MyProject.Library.${env:VERSION_NAME}.nupkg"
        dotnet nuget push $filePath --api-key $env:NUGET_API_KEY --source https://api.nuget.org/v3/index.json
        # if it's a private NuGet, specify other source.
        # web portal for testing NuGet uploads: https://int.nugettest.org
        # source for testing: https://apiint.nugettest.org/v3/index.json
      env:
        NUGET_API_KEY: ${{ secrets.MY_NUGET_API_KEY }}

    - name: Upload package to workflow results
      uses: actions/upload-artifact@v4
      with:
        compression-level: 0 # .nupkg already is a compressed zip
        name: ${{ format('MyProject.Library.{0}.nupkg', env.VERSION_NAME) }}
        path: ${{ format('./src/MyProject.Library/bin/Release/MyProject.Library.{0}.nupkg', env.VERSION_NAME) }}

    - name: Upload SBOM to workflow results
      uses: actions/upload-artifact@v4
      with:
        name: sbom_MyProject_Library.json
        path: ./out/sbom_MyProject_Library.json
    
    - name: Upload coverage report to workflow results
      uses: actions/upload-artifact@v4
      with:
        name: coverage_report
        path: TestResults

Image source

https://dyno.co.nz/products/telescopic-and-expandable-conveyors/telescopic-conveyor/

HTTP/2 and HTTP/3 explained

2024-03-11T00:00:00Z

In the beginning of the 1990s, Tim Berners-Lee and his team at CERN worked together to elaborate the basis of the World Wide Web, defining four building blocks for the Internet:

A document format for hypertext (HTML)
A data transmission protocol (HTTP)
A web browser to view hypertext (the first browser, WorldWideWeb)
A server to transmit the data (an early version of httpd)

HTTP reused for data transport the existing TCP/IP protocols, with HTTP message bytes residing in the application layer, light blue in the image below.

HTTP/0.9

It was the first HTTP draft. The only existing method was GET; there were no headers nor status codes; and the only data format available was HTML. Just like in HTTP/1.0 and HTTP/1.1, the HTTP messages were in an ASCII text structure.

Example of a HTTP/0.9 request:

GET /mypage.html

Response example:

<html>
  A very simple HTML page
</html>

HTTP/1.0

This version gave to HTTP its current structure, similar to a memorandum, also introducing new methods (HEAD and POST), MIME types, status codes and protocol versioning.

Example of a HTTP/1.0 request:

GET /mypage.html HTTP/1.0
User-Agent: NCSA_Mosaic/2.0 (Windows 3.1)

Response example:

200 OK
Date: Tue, 15 Nov 1994 08:12:31 GMT
Server: CERN/3.0 libwww/2.17
Content-Type: text/html

<HTML>
A page with an image
  <IMG SRC="/myimage.gif">
</HTML>

HTTP/1.1

This version came up in early 1997, few months after its predecessor. The main changes were:

Persistent TCP connections (keep-alive), saving machine and network resources. In the previous version, a new TCP connection was opened for each request and closed after the response.
Host header, allowing more than one server under the same IP.
Header conventions for encoding, cache, language and MIME type.

Example of a HTTP/1.1 request:

GET /api/fruit/orange HTTP/1.1
Host: www.fruityvice.com
Accept-Encoding: gzip, deflate, br

Response example:

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Sun, 10 Mar 2024 20:44:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-store, must-revalidate, no-cache, max-age=0
Pragma: no-cache
X-Frame-Options: DENY
Content-Type: application/json
Expires: 0

{"name":"Orange","id":2,"family":"Rutaceae","order":"Sapindales","genus":"Citrus","nutritions":{"calories":43,"fat":0.2,"sugar":8.2,"carbohydrates":8.3,"protein":1.0}}

HTTP/2

In 2015, after many years of observation and studies on the performance of the Internet, the HTTP/2 was proposed and created, based on Google's SPDY.

Among its differences, were the multiplexing of many messages in a single TCP packet; binary format of the messages; and HPACK compression for headers.

In HTTP/1.1, two requests cannot ride together the same TCP connection — it is necessary that the first one ends for the subsequent to begin. This is called head-of-line blocking. In the diagram below, request 2 cannot be sent until response 1 arrives, considering that only one TCP connection is used.

sequenceDiagram
    Client->>+Server: req 1
    Server-->>-Client: res 1
    Client->>+Server: req 2
    Server-->>-Client: res 2

With HTTP/2, this problem is solved with streams, each stream corresponds to a message. Many streams can be interleaved in a single TCP packet. If a stream can't emit its data for some reason, other streams can take its place in the TCP packet.

HTTP/2 streams are divided in frames, each one containing: the frame type, the stream that it belongs to, and the length in bytes. In the diagram below, a coloured rectangle is a TCP packet and a ✉ is a HTTP/2 frame inside it. The first and third TCP packets carry frames of different streams.

sequenceDiagram
    rect rgb(239, 190, 125)
        Client->>+Server: req1: #9993;1/1<br>+<br>req2: #9993;1/1
    end
    rect rgb(197, 234, 189)
        Server-->>Client: res1: #9993;1/2
    end
    rect rgb(197, 234, 189)
        Server-->>-Client: res1: #9993;2/2<br>+<br>res2: #9993;1/1
    end

The image below shows how frames go inside a TCP packet. Stream 1 carries a HTTP response for a JavaScript file and stream 2 carries a HTTP response for a CSS file.

HTTP/3

HTTP/3 was born from a new transport protocol, QUIC, created by Google in 2012. QUIC is encapsulated inside UDP, and compared to TCP, it proposes:

fewer packet roundtrips to establish connection and TLS authentication;
more resilient connections regarding packet losses;
to solve the head-of-line blocking that exists in TCP and TLS.

HTTP/2 solves the HTTP head-of-line blocking, but, this problem also happens with TCP and TLS. TCP understands that the data it needs to send is a contiguous sequence of packets, and if any packet is lost, it must be resent, in order to preserve information integrity. With TCP, subsequent packets cannot be sent until the lost packet successfully arrives to the destination.

The diagram below explains visually how this happens in HTTP/2. The second packet only had frames of response 1, but its loss delays both response 1 and response 2 — that means that in this case, there is no parallelism.

sequenceDiagram
    rect rgb(239, 190, 125)
        Client->>+Server: req1: #9993;1/1<br>+<br>req2: #9993;1/1
    end
    rect rgb(197, 234, 189)
        Server--xClient: res1: #9993;1/2
    end
    Note over Client,Server: lost TCP packet<br>must be resent.<br>delays both res1 and res2
    rect rgb(197, 234, 189)
        Server-->>Client: res1: #9993;1/2
    end
    rect rgb(197, 234, 189)
        Server-->>-Client: res1: #9993;2/2<br>+<br>res2: #9993;1/1
    end

To solve TCP's head-of-line blocking, QUIC decided to use UDP for its transport protocol, because UDP does not care for guarantees of arrival. The responsibility of data integrity, that in TCP is part of the transport layer, is moved in QUIC to the application layer, and the frames of a message can arrive out of order, without blocking unrelated streams.

sequenceDiagram
    rect rgb(255, 179, 217)
        Client->>Server: req1: #9993;1/1<br>+<br>req2: #9993;1/1
    end
    rect rgb(179, 205, 230)
        Server--xClient: res1: #9993;1/2
    end
    Note over Client,Server: lost QUIC packet<br>doesn't block sending<br>other packets
    rect rgb(179, 205, 230)
        Server-->>Client: res1: #9993;2/2<br>+<br>res2: #9993;1/1
    end
    Note over Client,Server: resending lost packet.<br>res2 wasn't delayed
    rect rgb(179, 205, 230)
        Server-->>Client: res1: #9993;1/2
    end

The head-of-line blocking related to TLS (SSL) happens on TCP because the cryptography is usually applied over the entire message content, meaning that all data (all packets) needs to be received for the decryption to happen. With QUIC, the cryptography is individual for each QUIC packet, that is decrypted on arrival, without having to receive all packets beforehand.

TLS with TCP:

Input data: A+B+C
Encrypted data: crypt(A+B+C) = D+E+F
Packets: D, E, F
Received: decrypt(D+E+F)
A+B+C

TLS with QUIC:

Input data: A+B+C
Encrypted data: crypt(A) = X, crypt(B) = Y, crypt(C) = Z
Packets: X, Y, Z
Received: decrypt(X) + decrypt(Y) + decrypt(Z)
A+B+C

Comparison table

	HTTP/1.1	HTTP/2	HTTP/3
Transport protocol	TCP, persistent connection	TCP, persistent connection	UDP, persistent connection
Head-of-line blocking	HTTP/1.x HOL TCP HOL TLS HOL	TCP HOL TLS HOL	-
Message format	ASCII text	binary	binary
Header compression	-	HPACK	QPACK
Number of roundtrips before start (handshakes)	3 1 from TCP +2 from TLS 1.2*	2 1 from TCP +1 from TLS 1.3*	0 0 from UDP +0 from TLS 1.3 with 0-RTT*
Connection identification	source IP and port	source IP and port	connection ID**, resistant to IP changes
Cryptography	optional; applied over the entire message	optional; applied over the entire message	embedded TLS 1.3; applied over each QUIC packet

* TLS 1.2 requires 2 roundtrips for cryptographic handshake and TLS 1.3 requires only 1, with the option for 0-RTT (zero roundtrip time resumption), where there is no need of previous handshake. However, 0-RTT enables replay attacks and therefore is unsafe. This is an optional feature, that can be disabled.

** QUIC's connection ID can be used for fingerprinting, posing a risk to user privacy, according to a research.

Which is the best version?

The two best versions currently are HTTP/2 and HTTP/3.

HTTP/3 was designed for unstable connections, such as cellphone and satellite networks. To counter network instabilities, QUIC has a great degree of independence between the data streams and good resilience if packets are lost. Nevertheless, HTTP/3 has performance penalties, mainly for 1) the UDP protocol wasn't optimized by routers and operating systems over the last decades due to its low usage, making it comparatively slower than TCP; and 2) the packet-by-packet cryptography used by QUIC requires a greater number of mathematical operations, becoming less efficient than the entire message cryptography used in TCP. Also, there is the issue that the UDP protocol is restricted in some networks to protect against attacks like UDP flood attack and DNS amplification attack.

On reliable and stable connections, HTTP/2 many times offers better performance than HTTP/3.

To avoid HOL blocking with HTTP/1.x, many web browsers and HTTP clients open multiple TCP connections, so the requests go in parallel. In scenarios with many heavy parallel requests and responses, this technique may make HTTP/1.x offer a better throughput than HTTP/2 or HTTP/3, however, it is a less efficient way to solve the problem. An alternative solution is having multiple HTTP/2-3 connections at the same time (C# example).

Generally speaking, it's recommended to run compatibility and performance tests to determine which version is the most appropriate, and furthermore, a server can accept both HTTP/2 and HTTP/3 connections, leaving to the client the decision of which version to use.

Testing tools

I recommend Pororoca (made by myself 🙂).

Bibliography

Modern image formats: JXL and AVIF

2024-01-28T00:00:00Z

Computers, photography and images

The visualization of images is fundamental in many areas, such as arts, professional photography, cartography, astrophotography, diagnostic medicine, historic preservation and movies. The quality of the images is very important to highlight details, like an artistic touch, an imprecision, or an element that is imperceptible at first sight.

On a computer screen, an image is represented by thousands of pixels (picture element), each pixel corresponding to a colour and its intensity. Consider an image of 1,024px x 768px = 768,432 pixels; if each pixel is RGBA and has 4 bytes, 1 for red, 1 for green, 1 for blue and 1 for transparency, this image will have a size of 3.1MB, which is quite large. However, by choosing proper compression techniques and file formats, we can have this image with the same quality and with a much lower file size.

Newer formats

Around 2019, two new file formats, JXL and AVIF, came to market promising lower disk usage and better preservation of details.

Both share as characteristics:

Support for lossless and lossy compression at more efficient rates than previous formats, like JPEG and PNG, and with better quality retention.
Support for animations and transparencies.
Open-source and royalty-free licenses.

JXL

JXL, or JPEG-XL, began in 2017 with a call for papers for a next-generation image format. The final proposal was the junction of two others: PIK, from Google, and FLIF, from Cloudinary.

AVIF

AVIF is a format created by the Alliance for Open Media, an industry-consortium of companies like Netflix, Meta and Google, with the objective of forming standards for multimedia delivery. AVIF's first specification is from 2018.

When should I use JXL?

JXL is better for:

Images with high and extremely high resolution and level of details.
When processing speed is important. JXL encoding is 5 to 10 times faster than AVIF's.
Very high colour precision.
Progressive decoding: the image can be rendered as it is downloaded.
Images with very high dimensions, because it supports sizes of over 1 billion x 1 billion pixels. AVIF supports images at most of 8K resolution (8193px x 4320px).

When should I use AVIF?

AVIF is better for:

Images in Web pages, because most of the browsers support AVIF (94%), while only few browsers support JXL (15%). Data obtained from caniuse (05/08/2025).
Videos and animations, because it has excellent compression rates for them, over 90% in some cases. It's a great substitute for GIFs.

Compression and performance tests

Examples:

The original image below is a JPEG of 8192x5464, file size 11MB. In JXL format (q=90), the size is 7.3MB. In AVIF (q=90), is 8.3MB.

The original GIF below has a file size of 3.1MB. In JXL format (q=90), the size is 2.5MB. In AVIF (q=90), is 79kB. Impressive!

This GitHub repo executes tests to compare compression rates and speeds of JXL and AVIF. The comparison is not qualitative, nor scientific. The input images include:

Animated GIFs of high, medium and low complexity.
JPEG photographies of medium, high and very high resolution.
PNG and JPEG infographics, to evaluate compression of images with texts on them.

Tools

There are websites on the Internet that can convert to AVIF and JXL. I personally recommend ezgif.

Most browsers support opening .avif files.

To view JXL, there is GIMP, Irfanview and other programs.

Conversion to `.jxl`

To convert an image to JXL, using the official tool:

cjxl input_image.jpg output_image.jxl -q 90 --lossless_jpeg=0

Conversion to `.avif`

There are two main tools: cavif-rs and ffmpeg. cavif-rs currently doesn't support GIFs, so we need to use ffmpeg for them.

To convert a static image to AVIF, using cavif-rs:

cavif input_image.jpg -o output_image.avif -Q 90 -f

To convert a GIF to AVIF using ffmpeg, you will also need libavif.

# sudo apt install ffmpeg
# sudo apt install libavif-bin

# first, from .gif to .y4m
ffmpeg -i input.gif -pix_fmt yuv420p -f yuv4mpegpipe temp.y4m

# then, from .y4m to .avif
avifenc temp.y4m output.avif

HTML rendering with fallback

If you want to use these new formats, but also want to guarantee compatibility with older web browsers, you can use the HTML <picture> tag.

<html>
  <head>
    <title>Animated AVIF Example</title>
  </head>
  <body>
    <picture>
      <source type="image/avif" srcset="my_animation.avif" />
      <img src="my_animation.gif" />
    </picture>
  </body>
</html>

If the browser supports, it will first try to load the picture in AVIF format; else, a GIF will be loaded. The animation below is inside a <picture> tag — you can verify which format is it by saving the picture.

Due to the reduced file size of the images, the web pages will load faster.

JPEG XL vs AVIF: A Comparison (WebArchive)

Image formats comparison

What’s the best lossless image format? Comparing PNG, WebP, AVIF, and JPEG XL (WebArchive)

How to compare images fairly (WebArchive)

How JPEG XL Compares to Other Image Codecs (WebArchive)

The Current Role of Image Compression Standards in Medical Imaging (WebArchive)

Serving AVIF images codelab

Images used in this article

All taken from Unsplash:

Symlinks for multi-repos

2024-01-19T00:00:00Z

Repeated files

Tell me: does the file organization below look familiar?

> tree C:\MyProjects\ /F

C:\MyProjects\
│
├──── Repo1\
│          src\
│          .editorconfig
│          lint.xml
│          build.sh
│          runtestcoverage.ps1
│          ...
│
├──── Repo2\
│          src\
│          .editorconfig
│          lint.xml
│          build.sh
│          runtestcoverage.ps1
│          ...
│
│ ...

Notice that there are files with the same name and content in each project directory. Some examples of repeated files are:

Code style files:
- .editorconfig (many languages)
- checkstyle.xml (Java, Android)
- .eslintrc (JavaScript, TypeScript)
Scripts:
- build.sh
- runtestcoverage.ps1 (general scripts)
- CMakeLists.txt
- Makefile (for C/C++ compilation)
- Jenkinsfile
Secrets and app configuration files:
- appsettings.json (.NET)
- Web.config (ASP NET Framework)
- .env (npm)
- others

If you have many repositories with repeated files, it becomes tiresome to change them, because the effort needs to be done for each repository.

Pointers?

In programming, there is the concept of pointers — something that points to something else. Usually, we think on memory pointers, that point to a value or structure in RAM memory. However, this concept also exists in file systems: it's called symbolic links, symlinks, files that point to other files.

Symbolic links are a mature technology, that showed as early as in 1960's computers. They are fully supported on Linux, Mac OSX and Windows, even in older versions.

Links

In a file system, each element, file or directory, is represented by an inode.

An inode of a file is composed by metadata and address in the hard drive. Directory inodes are lists of other inodes.

There are two types of links: hardlinks and symlinks.

A hardlink is when a file points to the same inode as other file. Both access the same content in the disk. When one of these files is deleted, the content of the other file is not lost.

A symlink is when a file points to another file. In this case, the pointer file depends on the target file to access its content. If the target file is deleted or moved, the symlink becomes invalid.

Symlinks can have absolute or relative paths.

Example of an absolute path: /home/alexandre/Projects/file.txt

Example of a relative path: ../file.txt

graph TD;
    I(inode)
    A(file.txt)
    AHL(hardlinked_file.txt)
    ASL(symlinked_file.txt)
    A==>I
    AHL==hardlink==>I
    ASL-.symlink.->A
    style I fill:#f9f,stroke:#333,stroke-width:4px,color:#000

Symlinks for organization of files

Considering the scenario in the beginning of the article, we can unify files of many projects through the use of symlinks.

Inside each repository, let's create links that will point to files in a Configs folder, located one level above. The Configs folder can become a repository itself, separate from Repo1 and Repo2.

The diagram below shows better. --> indicates a symlink.

C:\MyProjects\
│
├──── Configs\
│          .editorconfig
│          build.sh
│
├──── Repo1\
│          .editorconfig --> ../Configs/.editorconfig
│          build.sh --> ../Configs/build.sh
│          ...
│
├──── Repo2\
│          .editorconfig --> ../Configs/.editorconfig
│          build.sh --> ../Configs/build.sh
│          ...
│
│ ...

On VS Code, symlinks have an arrow icon to the right of the file name.

The command to create symlinks on Linux and Mac OSX is:

ln -s ../Configs/original_file.txt symlinked_file.txt

On Windows command line:

REM requires admin elevation
mklink symlinked_file.txt ..\Configs\original_file.txt

On PowerShell:

# requires admin elevation
New-Item -ItemType SymbolicLink -Path symlinked_file.txt -Value ../Configs/original_file.txt

Git

Git supports symlink files committed in repositories.

On GitHub, they appear with an arrow and name coloured in blue.

On online navigation and in pull requests, the file content appears as the symlink's path, either absolute or relative.

When should I use this approach?

Organizing files with symlinks is good in situations with many projects that share many common files.

If there are few files in common, or if there are not too many repos, it's OK to have repeated files!

In some cases, like pipelines, it's better to choose other reusability approaches. GitHub Actions, for example, suggests to adopt reusable workflows.

Symbolic directories can be a good alternative to Git submodules, in some cases.

Important details

If you are using Windows, you need to enable symbolic links during the Git installation:

Some file systems do not support symlinks, such as FAT32 and exFAT. These two are commonly found in pen-drives and SD cards.
Symlinks are different from Windows shortcuts. Windows shortcuts are files with the .lnk extension that can only be opened by Windows Explorer and Desktop.