I used to very proactively describe it as “immediate mode”, but recently I am starting to use the term “declarative” instead, because it deviates from several “de facto standard” conventions about what “immediate mode” means (or implies).

For one thing, it’s not a C++ library, and it’s not meant to be used with game engines.

In fact, we don’t even use GPU for rendering — at least at the time of this writing.

More importantly though, the API we expose is not about “drawing” things to the screen “immediately”.

Rather, what you do “immediately” is create a container tree that describes what the UI should look like at the end of this rendering cycle.

The containers do not even have their sizes and positions “resolved” when you create them; that gets resolved later by the core engine.

In a way, it’s similar to the React model, at least on a superficial level.

I still think it qualifies as “immediate mode”, because the way you use the API, your code does not keep widget objects around to pass to the API.

If you have a complex data structure that you want to reflect in the UI, you do not need to create and maintain a mirror widget structure. You just write procedural code to traverse or iterate your data structure.

The UI is not created by gluing together a bunch of widget objects.

The UI is built instead by running a view function that builds a container tree, and then having the system run the layout algorithm on that tree and turning it to a graphical interface to display on the screen.

Procedurally building a container tree

The biggest difference between Shirei view functions and React style functions, is the function signature.

A react style function takes one argument: a “props”, and returns a JSX Element, which is essentially a description of what a “subtree” of the dom is supposed to look like.

type MyComponentProps = {
    ...
}

function MyComponent(props: MyComponentProps): JSX.Element {
    return <div>
        ...
    </div>
}

Shirei, on the other hand, does not pass around container trees as objects. Instead, you build the container tree procedurally: opening a new container, setting attributes on it, and creating child containers. The child containers are also built the same way: procedurally.

So, if you have an html element tree like this:

    <div style="background: hsl(0, 10%, 90%); padding: 10px">
        <div style="width: 100px; height: 80px; background: hsl(0, 50%, 50%)">
        </div>
    </div>

We build it procedurally this way (conceptually):

open container
    set attribute: background=/__u/hasen.substack.com/(0, 10, 90, 1)
    set attribute: padding = (10, 10, 10, 10)
    open container
        set attribute: min width = 100
        set attribute: max width = 100
        set attribute: min height = 80
        set attribute: max height = 80
        set attribute: background=/__u/hasen.substack.com/(0, 50, 50, 1)
    close container
close container

This requires there to be an invisible “builder” object that is keeping track of the tree structure as it is being built.

This is another aspect of being “immediate”: you’re manipulating a data structure, you are issuing commands to a system.

Building the tree procedurally allows us to freely change the steps being performed based on current conditions. For example, if we are being hovered, we can change the background.

open container
    ...
    open container
        ...
        set attribute: background=/__u/hasen.substack.com/(0, 50, 50, 1)
        if hovered() {
            set attribute: background=/__u/hasen.substack.com/(0, 50, 70, 1)
        }
    close container
close container

In Shirei, the actual code looks like this:

Container(Attrs(Background(0, 10, 90, 1), Pad(10)), func() {
    Container(Attrs(FixSize(100, 80), Background(0, 50, 50, 1)), func() {
        if IsHovered() {
            ModAttrs(Background(0, 50, 70, 1))
        }
    })
})

The function Container takes as a parameter the initial attribute set and the “builder” function. Implicitly, the open container happens at the start of the builder function, and the close container happens at the end of it.

Attrs is a function that allows specifying the attributes ergonomically using a set of helper functions. It’s only there for “aesthetic” reasons, to avoid the code looking like this:

Container(Attributes{
    Background: Vec4{0, 10, 90, 1},
    Padding: Vec4{10, 10, 10, 10},
}, ...)

FixSize is a helper function that sets both min size and max size at the same time. Here is the actual, full implementation of it:

func FixSize(w, h float32) AttrsFn {
	return func(a *AttrSet) {
		a.MaxSize = Vec2{w, h}
		a.MinSize = Vec2{w, h}
	}
}

Most of these helpers don’t do much of “computational work” at runtime; they are there to serve aesthetic purposes.

Opening a container and setting the attributes are usually the same operation, but changing the attributes has to happen inside the builder function, because we cannot check if the current container is hovered unless we first “open” it.

Container Types

Unlike the DOM, there is only one container type: Container.

We do have a function to draw a button. It’s basically Button(icon, label).

But Button is not a container type. It’s just a function that builds out a container tree that, when rendered to the screen, looks like a button. The function also handles input and changes the appearance of the button based on the user interaction: highlighting it when hovered, pressing it when pressed.

This is not a “quirk” in the framework; it’s a deliberate architecture design decision.

By treating all containers as the same, we greatly simplify the code that performs container layout. We just walk through the container tree and run the same logic on every level. We never had to check the container type to apply special “rules” to it. All the “rules” are present in the attribute set.

Whether we are a button or a text input or a text label or a scrollbar - is a concern of the UI builder code.

Once the container tree is built, Shirei’s layout engine takes over to size and position all the elements, route events, retain identities, etc.

Now, the astute reader might have noticed the function IsHovered() and wondering what it means and how it works.

If the ui building code only declares the shape the container tree, not the sizing or positioning, how does it know whether the current container is being hovered?

The answer is that it’s based on where this container was positioned in the previous frame.

But this assumes we know where this container was on the previous frame, which assumes that we know the “identity” of the current container, relative to the containers created on the previous frame.

But how does that work? Doesn’t immediate mode mean we build the UI, render it to the screen, then throw away all the data?

Well, yes and no. We do throw away the container tree, but we do keep around a parallel tree, which we use to resolve container identities and remember their states.

Container identity

A Container has a continuous identity across frames based on its position in the tree.

How can this work?

Let’s step out of Shirei for a bit and think more abstractly. Imagine a node tree where each node has a “key” that identifies its type.

Imagine two node trees, one produced at frame N, and the other produced at frame N+1

Tree at frame N:

- Root
    - A
    - B
    - C

Tree at frame N+1:

- Root
    - A
    - B
    - D
    - C

We can easily see that a container like “Root -> A” is the same: the node has the same name, and its parent is also the same.

“Root -> C” is also easily identifiable as the same container. Even though its absolute position relative to its parent changes (at frame N it’s the third child, but at frame N+1 it’s the fourth child).

How do we know it’s the same? Because we use the node type as part of positioning; in both frames, it’s the first C node relative to its parent.

I said earlier that containers do not have “types”, but we can identify the container by its location in the codebase. We refer to this in an abstract way as an “implicit type”.

Now, if we render containers in a loop, then all containers will have the same “implicit” type, which might create problems if the ordering of the loop can change across frames.

In cases like this, we can use an explicit key per container, which is the same solution React resorts to:

for idx := range list {
    item := &list[idx]
    ContainerWithKey(item.id, Attrs(...), func() {
        ....
    })
}

So, let’s consider the following snippet

attrs := Attrs(Expand, Pad(10), Corners(6), Background(145, 25, 92, 1))
if options.ShowEmail {
    Container(attrs, func() {
        Label(contact.Email)
    })
}
if options.ShowPhone {
    Container(attrs, func() {
        Label(contact.Phone)
    })
}

We have two Container calls, guarded by an if statement. If the first conditional changed from true to false, the position of the phone container, relative to its parent, is going to change, but, that container has a distinct implicit key from the email container, so the implicit id is easy to resolve correctly.

Retained state

We use the retained id in order to retain the layout information: the size and position of this container, so we can query it at later frames.

More than that, we allow retaining arbitrary “state” related to the current component.

This allows components to embody complex behavior without the caller having to retain any state themselves.

The “API” presented appears “immediate”, but behind the scenes there is retained state.

Here’s a demo of a “special label” component. It features “two” retained state variables:

• Has this component ever been hovered?

• What is the requested text size?

It “hides” the label until the component has been hovered at least once. It shows small text control buttons that increase or decrease the text size.

func SpecialLabel(text string) {
	Container(Attrs(Corners(4), Gap(10), Pad(20), BorderColor(0, 50, 50, 1), BorderWidth(1)), func() {
		type MyState struct {
			hoveredOnce bool
			sizeInc     int
		}

		// retained component state
		s := Use[MyState]("my-state")

		// ephemeral frame state
		var hoveredNow = false

		if IsHovered() {
			hoveredNow = true
			s.hoveredOnce = true
		}

		textColor := Vec4{0, 0, 0, 0}
		if s.hoveredOnce {
			textColor[3] = 1
		}

		if hoveredNow {
			Container(Attrs(Row, Float(2, 2), Gap(4)), func() {
				if CtrlButton(SymPlus, "", true) {
					s.sizeInc += 5
				}
				if CtrlButton(SymMinus, "", true) {
					s.sizeInc -= 5
				}
			})
		}

		Label(text, FontSize(20+float32(s.sizeInc)), TextColorVec(textColor))
	})
}

func root() {
	ModAttrs(Spacing(20))

	SpecialLabel("Hello")
	SpecialLabel("World")
}

Multi-pass layout rendering

Getting the position data from the retained state has a caveat: on the first frame, there is no size or position data, as the element has not had its position and size determined yet. To work around this, we can run the UI building code several times even within the same frame cycle. This means, the resolved size and position of this container will be available on the second pass.

A second layout pass can also be triggered when an element moves.

See, the rendering cycle goes roughly through the following stages:

1. Collect input data from platform layer

2. Run user provided UI builder code to produce the container tree

3. Resolve all sizes and positions of containers in the tree

4. Produce rendering primitives (Surfaces)

5. Render Surface list onto platform provided window surface buffer

Steps 2 and 3 are what can run “multiple times”.

The catch is that we limit how many times we do this. Currently we only do it once, meaning we can run a second pass, but we will not run a third pass.

So, even though the UI builder code will be dealing with data from the previous frame, in practice, most of the time, there will be no noticeable glitch.

Hovering mechanics

Since I used IsHovered() as the hook for this article, I feel the need to clarify one important aspect of hovering detection logic: it’s not just about whether the mouse is over the container’s rectangle on the screen; it also requires understanding that containers can overlap, and we need to know which containers are in front of other containers.

So at the end of each frame, we record the list of hoverable containers in paint order: if two or more containers overlap, the latest one to be painted is the one most in front.

Then, at the start of the next frame, we walk that list backwards, and the first rect to contain the current mouse position “wins” as the container that is being directly hovered. All of its parent containers are then also considered as “hovered”, though indirectly.

During UI building, when we call IsHovered() we do not yet have the full picture of what the situation regarding container ordering will look like. We need to wait for the UI building to finish for this frame, and for sizing, positioning, and ordering to all be resolved.

This is why we must use the data from the previous frame, and why this mechanism cannot workunless we do retain enough state so that we can identify containers across frames.

Over three years ago, I wrote a post about the lack of tools to make self-hosting easy without sysadmin skills.

The intention was always to work on a project to solve this problem! Things did not go to plan, and the project got derailed several times for various reasons, until about 4 months ago when I started working in earnest on releasing this first version.

This first iteration focuses on static content. While the intention was always to make a website where you can interact with your visitors, static content was the lowest hanging fruit and I decided to go for it as a starting point.

I am now using this program to publish my own static website: Judi Systems

Here’s a one minute demo of publishing a website to a domain name on the internet, on a freshly provisioned VPS linux box from Vultr

Head over here to learn more https://judi.systems/sprouts/

If you give it a try, I would love to hear your feedback!

Thank you for your attention to this matter!

It might seem like an intractable problem, but it’s actually not.

It’s solvable, but you have to tackle it on several different fronts:

• Application Specs

• Code Architecture

• QA Automation

• Development Environment

Most bugs happen because:

• Specs are not well understood

• Architecture is confusing

Most bugs can't get fixed because:

• Manual testing is near impossible

(1) Application Specs

If no one knows what the program is supposed to do, forget it. Bugs will never get fixed, and QA will never get automated. This is what you should focus on first.

Specs should be clear but not verbose. No one wants to read half a page when a sentence would suffice.

Do not let business people write them! The chief architect or tech lead should be writing them.

Specification documents are an artifact that engineering is responsible for.

The specs will derive both the code architecture and QA automation.

(2) Code Architecture

The ground truth about computers is they can only do two things: (a) Process data (b) Move it around (I/O)

Code Architecture should reflect this. No crazy abstractions. No service providers. No business domain.

Do not model the problem. Model the solution.

Programmers need to understand how data is flowing through the system. It should be crystal clear. No obfuscation.

Every little thing you add to obfuscate data flow will be a time bomb waiting to explode into an intractable bug.

(3) QA Automation

Forget about unit tests. They are mostly a waste of time.

Allocate time and resource for an automated end-to-end suite.

How do you know whether the tests are good? Here are some heuristics:

• They simulate accurately how a user interacts with the program/system

• They do not deal with system state. Only inputs that a user can give and outputs that a user can see.

• Zero mocks! The test executes all the relevant code paths in the system in exactly the same way that would happen in production.

• Never once do you need to change the test as a result of refactoring the system internals. Tests only change when you make changes to the user interactions (inputs/outputs).

Given the above, it's easy to fix bugs with confidence.

• You can check the spec to see what's the expected behavior of these other things

• You can run the test suite to see if anything broke as a result of your fix

But there’s one last piece to the puzzle!

(4) Development Environment

All the above is almost useless if it's not easy for anyone on the team to run the QA test suite with one button (or one simple command).

Local development should be seamless. Every single programmer on the team should be able to run the entire system with one simple command. They should have a quick edit/run/debug cycle.

QA should also be trivially easy to run on staging too.

When you don't have all these points taken care of, an endless stream of bugs is just par for the course. There's nothing you can do about it unless you fix the root cause.

This is Episode 7 of HandCraftedForum.

In the past few episodes we talked about the project's basic structured and discussed some technical points.

In this episode I want to start prototyping the conversation view.

I would like to try out some ideas that allows us to have threaded conversations but still have a linear view of all the replies to a post.

Something like Twitter/X, where when you open a post, you see its parents, and its descendants (replies).

X is not very good at showing you all the replies, specially when they are threaded. You have to keep drilling. Instead I think we could show all the replies to a post, flattened.

I don't like threaded views line Reddit and Hacker News, because replies to top comments get shown before replies that would otherwise rank higher; whether you are sorting by date or by some kind of score.

However, I do like having comments that are replies to other comments, show a link to the comment they are replying to. I think Discourse does that. I think it's a good idea and we will steal it.

Data Model

To keep track of parent-child relationships between posts and their replies, we'll add a ParentPostId to the Post struct, and we'll use an index to keep track of replies.

Since we have no real data yet, we don't need to migrate anything; we just add a migration process to reset the posts bucket, and now we can change the packing function for Posts without incrementing the version.

To keep track of parent child relationships, we declare an index from int to int.

This index can be used to get both all the ancestors of a post and all of its descendants, iteratively instead of recursively.

Remember that the index is a two way multi-map between terms and targets: If the term points to the direct parent id, then the target will point to the direct children ids. Since the term points to all its ancestors, then the target will point to all of its descendants.

To explain a bit further (feel free to skip this paragraph if you already understand):
- The parent id is always on the post
- When we first post a comment A without a parent, there's nothing for it in the index.
- When we then post a reply B to A, we store [A] as the term for B.
- When we post C as a reply to B, we get the list of ancestors of B, which is [A], and add [B] to it, and store it as the list of parents for C.
- When we post D as a reply to C, we get the list of ancestors of C, which is [A, B], and we add C to it and store it as the list of ancestors for D. Storing a list of terms for a target basically adds entries of the form (term, priority, target) for each of the terms. If we ignore the priority and the ordering, you can think of the index this way:

Think of it as a table that we iterate and filter. It's clear that given key 'B' we can find all the ancestors and descendants by just iterating and filtering.

So to implement writing to this index, we look at the parent id, and if it's set, we retrieve the list of the parents parents from the index, add the direct parent id to the list, then write that as the set of terms for the post id.

populating the post replies index

We will see below how to query this index for the ancestors and replies.

Importing Data

Now, in order to start prototyping the UI, we need to have some conversation data. I don't want to create the conversations manually, nor do I want to generate them.

I did a bit of research to see if we can find a dump of some conversation data from HN or some mailing list. Turns out, HN has an API to get conversation data:

https://github.com/HackerNews/API

I wrote some code to "import" a thread recursively by the id of the root post. The code does the import in two stages: first stage downloads all posts from the API, second stage populates our DB with all the downloaded posts.

We keep the post ids from HN. Since this is temporary data, we don't care about maintaining an auto increment id for the posts. This also allows us to import multiple times; for example, if we fix our import code, or our index updating code, we don't have to do anything to the posts themselves.

The only other note worthy thing item here is that I also download posts locally so that we don't touch the API more than needed. If we already downloaded a post, we don't have to download it again.

The importer is an executable package forum/hn_importer. The command takes one argument: the post to download.

go run forum/hn_importer 24649786

API / UI to view a single post

Having imported some posts, let's verify it by implementing a simple view page.

On the server side, we'll define a function that retrieves a single post by id

type PostQuery struct {
    PostId int
}

type PostResponse struct {
    Post Post
}

var PostNotFound = errors.New("PostNotFound")

func GetPost(ctx *vbeam.Context, req PostQuery) (resp PostResponse, err error) {
    if !vbolt.Read(ctx.Tx, PostsBkt, req.PostId, &resp.Post) {
        err = PostNotFound
    }
    return
}

On the client side we add a router entry so that navigating to “/item/:id” shows a page with the post's content.

We'll define two functions: one to fetch the post, one to view it.

export const itemHandler = {
    fetch: fetchPostItem,
    view: viewPostItem
}

async function fetchPostItem(route: string, prefix: string) {
    const postId = vlens.intUrlArg(route, prefix)
    return server.GetPost({PostId: postId})
}

const clsPostPlain = vlens.cssClass("post-plain", {
    // .. snipped
})

function viewPostItem(route: string, prefix: string, data: server.PostResponse) {
    return <div class={clsPostPlain}>
        <p>
            {data.Post.Content}
        </p>
        <a class="permalink" href=/__u/hasen.substack.com/%7B%22/item/%22 + data.Post.Id}>Permalink: {data.Post.Id}</a>
    </div>
}

Now we can navigate to “/item/<id>” for some id to check how the content looks like.

Unfortunately I did not create a screenshot of what that would look like, so I can't really show it. I could show you the API response but you can already imagine what it would look like.

Originally I had intended to stream myself programming this. The idea was that I'd later cut things from the stream and share them here. However, the stream was not setup properly: half the screen was out of view, and it got cut abruptly half way through.

At any rate, the above is just the basic skeleton. We want to expand it so that the page shows not only the post itself, but all of its ancestors and all of its descendants.

API/UI to view all parents and replies

One of the major points I want to impart on you in this article is how we design the API response.

Take a look at the very first sketch in this article and consider what kind of data we need to implement it.

What we have is a list of posts. Depending on what we decide to add to the UI, we'll need some additional metadata about the post. For now, I want to display the username and the number of replies so the user can decide whether they want to "zoom in" on the specific comment or not.

Here's my API design:

Notice the response is just a container for basic types we already have.

This is in contrast to what most people would do, which is design a post object modeled around the UI.

They will think: the UI to view the post needs to know the username and the number of replies so let's create this response model this way:

Notice the comment in scary uppercase telling you that this is a bad idea.

Why is it a bad idea? Because it's creating additional work for yourself, with no return on investment. You have to write code on the server side to transform a regular Post object into a UIPostModel. This is not a useful data transformation. The data is already available in the Post object. Creating this parallel object that is very similar but also subtly different serves no purpose. The data provided is the same; it's just arranged in a different way.

What's the purpose of this re-arrangement? Nothing.

So many people hold in their minds a bunch of bogus notions about the problem domain and how your object model should reflect the business rules. Just pure non-sense.

The UI needs to display a list of posts in a specific order, so just provide the order in a list, and then provide the id -> post mapping.

The post has a user id and you need to get the username? Just slap another map. You need the number of replies to each post? Just slap another map.

Now the implementation is pretty straight forward. The list of posts is as follows:

• Parent posts

• Self (the requested post id)

• Replies

So we start by reading these post ids off the database in that order (the post replies index)

Now resp.PostIds is the list of post ids we need to load. So we just load them! We already saw in a previous episode that we have a helper function that does just that.

Now we use the counting feature from the index to get the number of replies.

Now we load the users. This is the "trickiest" part of the code because it tries to avoid loading a user that's already been loaded.

This gives us all the data we need for the UI as we are prototyping.

Here's what the prototype UI looks like

The UI code is pretty straight forward and I don't think there's anything about it that is new or remarkable to make it worthy of showing here.

Refer to the code attached to this episode.

Here's a demo of importing posts from HN and viewing it in the UI

The perils of Clean Code adjacent paradigms

What do you notice about our implementation?

The same "Post" type is used for the storage layer, the application code, the API, the UI.

This should not come off as remarkable, it's just straight forward common sense.

However, a lot of "programming education" material out there teaches people a very different thing.

They teach people to completely isolate the different components of the system, and to separate the data types across the different components, so each component has its own representation of the data. They also teach you to use a "Data Transfer Object" to move data between layers.

So you would have a "Repository" layer that has its own representation of "Post" that it stores and loads from the database, and a "PostModel" object that it exposes to the outside world.

You would also have a "Domain" layer with its own representation of "Post". It would use the "repository.PostModel" to communicate with the Repository layer, and use another "api.PostModel" to communicate with the outside world through the API.

The UI will also have its own representation of the post, perhaps even several representations depending on context: PostViewModel, PostEditForm. It will have a convertor for each such type to and from the api PostModel.

If you follow these ideas, you will have to write tons of code. All of it noisy boilerplate that happens to be a great minefield for bugs to hide.

We're having none of it. This alone means we can be several times more productive than someone who programs using such paradigms. At least 3x more productive, as a conservative estimate.

Download the code: EP007.zip

View the code online: HandCraftedForum/tree/EP007

In the previous episode (EP005), I pointed out that we have a problem with code duplication when it comes to fetching posts by hashtag and by userid.

This again was not one of the planned topics for the series but I think it's worth a dedicated episode.

We have two almost identical code paths that are only different in a couple of small details.

We have two features:

• Find posts by hashtag

• Find posts by userid

For each we use a special Index object. The hash tag index uses the hashtag (string) as the key, while the user posts index uses the userid (int) as the key.

Because the index object is different, we have two different RPCs to call:

• PostsByHashtag

• PostsByUser

They both return the same thing, but have different inputs.

The input for PostsByUser looks like this:

struct {
    UserId int
    Cursor []byte
}

The input for PostsByHashtag looks like this:

struct {
    Hashtag string
    Cursor  []byte
}

As you can see, each has two parameters: cursor, and the input to the index. Each chooses to call the input by a different name, and of course, the input has a different type: for one it's a string, for the other it's an int.

The functions themselves are almost identical though. They only differ in the name of the input parameters, the index they use, and the query term they pass to the index.

Now, this divergence on the server side will cause a divergence on the client side.

The frontend code has two divergent but almost identical blocks of code to implement the fetching of posts either by userid or by hashtag.

Fortunately, the response from both RPCs is the same, allowing us to use the same state object to track the post listing itself:

But the fetching of more posts is duplicated in two almost identical functions

The fragment where we display posts and show the "More" button is also almost identical but divergent

The starting point for this divergence was using two different index objects to answer each query, and each index using a different type for the query term. This resulted in two different server side procedures to perform the query, each takes a different request object that specifies the query term using a differently named field using a different type. This results in the client side having to implement "fetch more" in two separate functions, one for each backend procedure. Finally culminating in the post list and "more" button fragment to also diverge in two place, each one binding itself to its corresponding callback handler.

Naive approach

The naive approach is to notice the duplicate pattern, extract helper functions that take parameters with "holes" in them, and the caller fills the hole with the appropriate information.

In other words, create parameterized versions of all the above duplicated code fragments.

Something like this:

function viewPostsWithFetchMore<P>(form: Form, param: P, fetchFn: (p: P, form: Form) => any) {
    return <>
        {viewPosts(form.posts)}
        {form.cursor && <button disabled={form.sending}
            onClick={vlens.cachePartial(fetchFn, param, form)}>More</button>}
    </>
}

If you do this, the call sites can transform this way:

// Inside the hashtag search view function
viewPostsWithFetchMore(form, hashtag, fetchMoreByHashtag)

// Inside the user posts view function
viewPostsWithFetchMore(form, userId, fetchMoreUserPosts)

Now, the process can be repeated for the fetch function itself. We extract the common bits to a function with holes, and we supply the holes

async function fetchMoreUserPosts(userId: number, form: Form) {
    return fetchMoreFromServer(form, { UserId: userId, Cursor: form.cursor }, server.PostsByUser)
}

async function fetchMoreByHashtag(hashtag: string, form: Form) {
    return fetchMoreFromServer(form, { Hashtag: hashtag, Cursor: form.cursor }, server.PostsByHashtag)
}

async function fetchMoreFromServer<P>(form: Form, params: P, remoteFn: (p: P) => Promise<rpc.Response<server.Posts>>, ) {
    form.sending = true
    let [resp, err] = await remoteFn(params)
    form.sending = false
    vlens.scheduleRedraw()
    if (resp) {
        form.posts.push(...resp.Posts)
        form.cursor = resp.Cursor
    } else {
        form.error = err
    }
}

The same thing can be done on the server side: extract the common parts of the duplicated code into a function that takes parameters for the different parts that you fill at the call site with the appropriate values:

type ByUserReq struct {
    UserId int
    Cursor []byte
}

func PostsByUser(ctx *vbeam.Context, req ByUserReq) (resp Posts, err error) {
    return PostsBy(ctx, req.Cursor, req.UserId, UserPostsIdx)
}

type ByHashtagReq struct {
    Hashtag string
    Cursor  []byte
}

func PostsByHashtag(ctx *vbeam.Context, req ByHashtagReq) (resp Posts, err error) {
    return PostsBy(ctx, req.Cursor, req.Hashtag, HashTagsIdx)
}

const Limit = 2
func PostsBy[T comparable](ctx *vbeam.Context, cursor []byte, term T, index *vbolt.IndexInfo[int, T, time.Time]) (resp Posts, err error) {
    var window = vbolt.Window{
        Limit:     Limit,
        Direction: vbolt.IterateReverse,
        Cursor:    cursor,
    }
    var postIds []int
    resp.Cursor = vbolt.ReadTermTargets(
        ctx.Tx,   // the transaction
        index,    // the index
        term,     // the query term
        &postIds, // slice to store matching targets
        window,   // query windowing
    )
    vbolt.ReadSlice(ctx.Tx, PostsBkt, postIds, &resp.Posts)

    generic.EnsureSliceNotNil(&resp.Posts)
    generic.EnsureSliceNotNil(&resp.Cursor)
    return	
}

You could go even crazier with the object oriented approach, create a base class and inherit into sub classes. I hate to even think about what that would look like.

Anyway, this approach, of extracting common code parts into a function with holes, is not very good actually. On one hand, yes, we are reducing code duplication, but on the other hand, we are creating new concepts, and now, someone trying to read the code has to understand why we have these generic higher order functions that take other functions and parameters as inputs.

Before, we had two parallel code paths. After, we still effectively have two code paths, but they are heavily intertwined.

Simple approach

We can notice the reason for the code paths divergence is us having two different index objects, using a different type for the query term.

This can all go away if we just use the same index for both types of queries. After all, there's nothing special about querying by user id that makes it different from querying by hashtag. The fact that user ids are numeric is largely irrelevant, because we will serialize both to a byte buffer before we pass it to the index. The only thing we need to worry about is avoiding overlap between the two types of queries, and this is trivial to achieve by using a prefix. This allows us to use different types of prefixes for different types of queries that we will add in the future.

For instance, we can use '#' for hashtags and '@' for user ids, or we can use 't:' for hashtags and 'u:' for user ids. It does not really matter what we use. It just matters that we are consistent.

So, we start by unifying the two index objects into one:

Then, when it comes time to create the post and update the index, we combine the tags terms and the user term into one list and set them on the new index:

terms := make([]string, 0, len(post.Tags)+1)
generic.Append(&terms, fmt.Sprintf("u:%d", post.UserId))
for _, tag := range post.Tags {
    generic.Append(&terms, "t:"+tag)
}
priority := post.CreatedAt
vbolt.SetTargetTermsUniform(
    ctx.Tx,   // transaction
    PostsIdx, // index reference
    post.Id,  // target
    terms,    // terms (slice)
    priority, // priority (same for all terms)
)

Since we want to collapse the code paths, we also update the querying procedure, along with its inputs and outputs, in a way that helps the client code be more uniform.

type PostsQuery struct {
    Query  string
    Cursor []byte
}

type PostsResponse struct {
    Posts      []Post
    NextParams PostsQuery
}

Now the posts query is basically the same as the previous two functions, except it does not care at all what type of terms you're looking for: the operation is all the same.

func QueryPosts(ctx *vbeam.Context, req PostsQuery) (resp PostsResponse, err error) {
    var window = vbolt.Window{
        Limit:     Limit,
        Direction: vbolt.IterateReverse,
        Cursor:    req.Cursor,
    }
    var postIds []int
    resp.NextParams = req
    resp.NextParams.Cursor = vbolt.ReadTermTargets(
        ctx.Tx,    // the transaction
        PostsIdx,  // the index
        req.Query, // the query term
        &postIds,  // slice to store matching targets
        window,    // query windowing
    )
    vbolt.ReadSlice(ctx.Tx, PostsBkt, postIds, &resp.Posts)
    generic.EnsureSliceNotNil(&resp.Posts)
    generic.EnsureSliceNotNil(&resp.NextParams.Cursor)
    return
}

Next, we update the client side code to use 't:' and 'u:' prefixes on initial page load

export async function fetchUserPosts(route: string, prefix: string) {
    const params = vlens.urlParams(route);
    const userId = vlens.intParam(params, "user_id", 0);
    return server.QueryPosts({ Query: 'u:' + userId, Cursor: "" })
}

export async function fetchByHashtag(route: string, prefix: string) {
    const params = vlens.urlParams(route);
    const hashtag = params.get("hashtag") ?? "";
    return server.QueryPosts({ Query: 't:' + hashtag, Cursor: "" });
}

Now because the response contains `NextParams`, the client side "fetch more" can be really simple. It doesn't need to care about the search type, or what page we are currently on.

It just stores the NextParams on initialization, and keeps storing it after every "fetch more" operation

The callback function for the "More" button can be really simple now

The fragment to display the list of posts followed by the "More" button is now also greatly simplified.

Now we've not only eliminated duplicate code; we've collapsed the code paths. The divergent part is very small: just the fetching function for different pages: fetchUserPosts and fetchHashtagPosts. Everything else after that is uniform.

This way of collapsing code paths is really powerful but it is hardly taught anywhere. It allows us to add more features without adding any complication.

Suppose, for instance, that we wanted to also index posts by year and year-month combinations. With this approach to collapsing code paths, we can easily just add more terms this way:

    generic.Append(&terms, fmt.Sprintf("y:%d", post.CreatedAt.Year()))
    generic.Append(&terms, fmt.Sprintf("m:%s", post.CreatedAt.Format("2006.01")))

Where as, if we had taken the naive approach described above, where we painstakingly create generic functions with holes to be filled in by the caller, it would be more complicated: we'd have to create a new index (or two), new functions to query by dates, new "fetch more" client side functions to query by dates, etc. We'd just have a lot more boilerplate code to write. Which means increased friction towards anything we want to do to improve the querying function.

For example, we might consider supporting queries by "range". Since the Index is backed by a B-Tree, it's not difficult to see how we can query by a range of query terms, for example from 'm:2024.05' to 'm:2024.10'.

We can, for example, change the PostsQuery to be more like this:

We'd interpret `To` being unset to mean it's not a query range but a normal query. And because the response object will include the `NextParams`, the client code for "fetch more" does not have to change at all.

Having just one index object and one function to query it, means we can add features of this type without a lot of trouble.

Had we kept around many index objects and many querying functions, it would be a lot more effort to support this for all of them. We'd have more functions to change, more "query input" types to change, and more UI code to change.

Next, imagine if we wanted to support querying by multiple terms, some of which could be a range, and some of which would be just normal terms. For example, let's say we combine 'u:2' with 'm:2024.12' to get all the posts by the given user in the given month.

Again, we'd just have one type to change and one function to update. The input parameters could be change to look like this:

type QueryTerm struct {
    From string
    To   string
}

type PostsQuery struct {
    Terms  []QueryTerm
    Cursor []byte
}

The function logic would be somewhat involved, but it would be tractable, because it only deals with one index.

Had we gone the naive approach and kept several index objects and several query param structs for each query type, it would be a lot more complicated. It would probably take a Herculean effort to to implement a function that lets you query by multiple query terms, because each query term would hit a different index, and you'd have to keep track of the type of each query parameter and the index associated with it.

Maybe something crazy like this:

type HashtagQuery struct {
    Hashtag string
}

type ByUserQuery struct {
    UserId int
}

type ByDateQuery struct {
    From time.Time
    To   *time.Time
}

type PostQuery interface {
    IsPostQuery() bool
    CountItems(tx *bolt.Tx) int
    PerformQuery(tx *bolt.Tx, cursor []byte, limit int) []int
    // TODO: add whatever other methods are needed
    // for the implementation of QueryPosts to work
    // with a list of PostQuery items
}

// TODO: implement PostQuery for HashtagQuery, ByUserQuery, ByDateQuery

func ParseQueryString(query string) []PostQuery {
    // somehow parse a complex query string into a list 
    // of query items that implement the interface
}

func QueryPosts(tx *vbolt.Tx, query []PostQuery) []Post {
    // perform the query
}

While this example is hypothetical, it's not mere speculation.

This is actually the kind of programming that happens in the wild. This is what senior engineers in most web companies spend their time doing. A team of ~6 engineers would spend two~three months to make this kind of query work.

As a side note: the interface PostQuery exists as a placeholder (a template with holes to be filled), and the content of the interface depends on the implementation details of QueryPosts as it attempts to combine multiple queries. I put `CountItems` in it because in my head I imagine it as part of the implementation, but maybe what we would need is something else.

Now, I'm not saying that implementing "query by multiple query terms" would otherwise be easy. It might be easy or it might be difficult. But either way, implementing it on one index object with one unified query type, is for sure going to be an order of magnitude simpler than implementing it on multiple indexes and multiple types of queries.

With our uniform structure, we only have to solve the basic problem: if we can do it on one index with one type of query term, we can trivially do it for all type combinations.

Where as, with the naive approach, we first have to figure out how to combine multiple query terms for one index, and then after we implement it and test it, we have to figure out how to generalize it to querying multiple indexes.

Should we actually try to implement this? I haven't thought about this problem that deeply, but since it's an interesting challenge, I'll give it some thought. We'll need to have a good deal of sample posts to work with, because it'd be difficult to assess the effectiveness of our implementation when we only have ~20 posts or so.

Originally my plan was to delay the implementation of "Search" as far back as possible, but something is urging me to focus on it sooner. Maybe I'll try it for the next episode! No promises, but stay tuned!

Migration

We changed how the index stores and queries data, but the data we've already saved and indexed is not using this new index, so if we just run the local server now and try to fetch posts by users or by hashtags, we'd see nothing, because the index has no data.

We have two ways to solve this:

(1) We can delete the database file and let the server create a new one when it starts up. At this stage we don't have a lot of data, so we can just manually recreate some fake sample data like we did before.

(2) We can run a migration process on startup to re-index all the existing posts.

I chose the latter because I have a system for it. I add some code to the OpenDB function to "apply db processes". I give each migration/process a unique label prefixed with 'year-monthday-'. The migration system uses the label to ensure the migration process is only ever executed once on the database, so this function is safe to keep around for a long time. I'll try to cover the migration system a bit more in depth at a later time.

    vbolt.ApplyDBProcess(db, "2024-1217-unify-post-index", func() {
        // delete old indexes
        vbolt.WithWriteTx(db, func(tx *vbolt.Tx) {
            tx.DeleteBucket([]byte("user-posts"))
            tx.DeleteBucket([]byte("hashtags"))
            tx.Commit()
        })
        log.Println("Deleted old indexes")
        // populate the new index
        vbolt.TxWriteBatches(db, PostsBkt, BatchSize, func(tx *vbolt.Tx, batch []Post) {
            for _, post := range batch {
                UpdatePostIndex(tx, post)
            }
            vbolt.TxCommit(tx)
        })
    })

Demo

With that, here's a demo of the application after our changes. You can see from the network tab that we are using the new unified procedure.

Note: in the process of making this work, I had to fix a few bugs in vbolt. Make sure to run `go mod download` to pull in the dependencies.

Download the code: EP006.zip

View the code online: HandCraftedForum/tree/EP006

In these first few episodes we're doing an introduction to our homegrown mini framework and data storage layer that we'll be using through out the project.

In the previous episode, we wrote the backend code for creating users and posts and indexing posts by hashtag, but we did not implement the UI.

In between the last episode and this episode I implemented the basic UI and fixed a minor bug along the way.

There isn't anything special in the UI code that we didn't previously cover so I did not think it's worth much to explain in detail. The code will be attached at the end of the article as usual, so make sure to check it out. Here's a quick demo of the UI.

• Creating a user account

• Creating a post on behalf of the user

• Finding posts by hashtags

• Finding posts by users

One small change I did to the backend code is to read entries off the index in reverse. We want to see newest posts first, but the index sorts by the creation time in ascending order.

To make this happen, I had to update VBolt to support iterating the index in reverse order.

When the client creates a new post, it just pushes it to the top of the list:

The topic of this episode was not planned, but I think it's a neat little thing that just comes out of using B-Trees for indexing, and being able to directly make use of its properties, and would probably be quite painful to mimic using higher level concepts like relational tables.

In the previous episode I did a quick overview of the concept behind the "Index" storage and explained that it stores tuples in a list that gets sorted by the underlying B-Tree.

Here's one way to visualize how the sorted term -> target mapping is stored in the B-Tree. It's basically a list of `[]byte` keys, each key is internally composed of a three-tuple: (term, priority, target). They are arranged such that when the B-Tree sorts these keys in byte order, it's equivalent to sorting the list of tuples in order.

To find targets for term 'A', we iterate the B-Tree on keys that start with "A" by seeking to the first key that has the prefix (A ...), then iterating one by one until we find an entry that does not have that prefix.

We can stop the iteration after N steps, and ask the B-Tree to return the full byte representation of the key it stopped at.

This will serve as a "cursor". We can encode it to base64 (or any ascii representation really) and use it later to continue the iteration where it stopped.

Compared to pagination by passing a page number, this has some advantages and some disadvantages. The advantage is that it performs better if you have thousands of elements to skip; it's more work for the computer to read hundreds of entries from the B-Tree only to drop them.

The B-Tree is really good at jumping to a particular key, but it's not very good at jumping N keys ahead. It has to move there one step at a time.

On the other hand, using a cursor means that as an end-user, we do not have random access: we can only grab the next page. If for some reason the user wants to skip to page 20, they have to hit "next page" 20 times, which is a lot more wasteful than the computer skipping 20 pages when reading the B-Tree.

Another advantage of the cursor is that it's more stable. It's like you left a bookmark where you were reading so you can come grab it again next time to continue where you left off. Even if items got added or deleted somewhere else in the list, you will still continue exactly where you left off.

If you are interested in the topic, you can lookup "cursor pagination vs offset pagination". Here's a Grok summary:

https://x.com/i/grok/share/6pA5jISvoRiKDFrdwQerQkiNV

The most suitable problem for cursor navigation is "infinite scrolling".

Now, here's how we can incorporate cursor pagination into our UI:

• Add a cursor field to the response and the request `Cursor: []byte`

• Pass the cursor to form the request to the query function `vbolt.ReadTermTargets` and pass output cursor back to the response.

• UI retains the cursor value

• "fetch more" button sends the cursor value

Here are the changes to the backend code:

We also update the PostsByHashtag function in the same way (not shown for brevity).

Here are the changes to the UI code:

Notice how the callbacks for fetching by userid and fetching by hashtags are almost identical.

This is not very good. We'll discuss how to collapse these duplicate functions in the next episode, which although was not planned, is still a very good topic that deserves its own independent treatment, and not be shoved as a side quest in an article about cursor pagination.

Now, if you watch closely, there's a bug: fetching more does not seem to be working for hashtags! Even though the response does contain the correct data!

What gives?

A little bit of debugging reveals that the code for listing the posts was using `data.Posts`, which is the posts from the initial page fetch, when it should have been using `form.posts`, which is the list we retain in the UI state.

This is one of those problems that happens when you have mostly duplicate code doing the same things with slight variations! As you keep making changes, you have to remember to keep updating both places, and it's easy to forget things. Things are still small now so the mistakes are harmless and easy to find, but you can imagine how things would get out of hand as the combinations of code path increases exponentially.

In the next episode, we'll show how we collapse the code paths without resorting to OOP like abstractions. The solution will be so much better and so much simpler! Stay tuned!

Here's the code for today's episode.

Note: I updated vbeam and vbolt. If you are pulling from github, make sure to run `go mod tidy` after pulling.

Download the code:

EP005.zip

View the code online:

HandCraftedForum/tree/EP005

Last time I showed how to persist simple user account data to the database. This time I want to demonstrate indexing and querying, and since this is a "forum" project, let's do this with basic posts.

We'll do a very barebones kind of post. Think a "Tweet". Each post has nothing but the id of the user who posted, when he posted it, and the content of the post.

We'll use two indexes: one to keep track of posts by user id, and one to enable querying posts by hashtags, like the old Twitter.

We don't have proper sessions yet, so we'll make the username part of the data we submit, and we will not validate the authorization.

On the user list page, we'll turn each user to a link, when you click on it, it takes you to a page where you can post on that user's behalf. The page will have a list of recent posts by the user, and a textbox on the top to create a new post.

Then for each post, we'll extract hash tags, and create a link for each. When you click it, you see a page listing all posts that use this hashtag

We have basically two new pages, each one needs its own RPC to fetch data:

• List latest posts by user id

• Find posts by hash tag

We'll also need one procedure to create a post.

We'll start with the backend code first, because it's surprisingly straight forward.

First, define the post struct

type Post struct {
    Id        int
    UserId    int
    CreatedAt time.Time

    Content string
}

We also define a method to extract hash tags from content. You can checkout the implementation from the codebase attached at the end of this article.

Now we define three vbolt objects:

• A bucket to hold the posts themselves

• An index to group posts by user id

• An index to query posts by hash tag

var PostsBkt = vbolt.Bucket(&dbInfo, "posts", vpack.FInt, PackPost)

// UserPostsIdx term: user id. priority: timestamp. target: post id
var UserPostsIdx = vbolt.IndexExt(&dbInfo, "user-posts", vpack.FInt, vpack.UnixTimeKey, vpack.FInt)

// HashTagsIdx term: hashtag, priority: timestamp, term: post id
var HashTagsIdx = vbolt.IndexExt(&dbInfo, "hashtags", vpack.StringZ, vpack.UnixTimeKey, vpack.FInt)

One way to think of an index is as a bidirectional multi-map. You want to use a query term to find matching targets. Thus we use the terminology:

• Term: the search term you want to use to query for data

• Target: the matching item that you're targeting for queries

It's a multi-map, because each term can have multiple matching targets.

It's bidirectional, because you can go from term to targets, or from target to terms.

Structurally, it's a collection of entries, sorted in ascending order, each entry is composed of three elements: (term, priority, target). For each such entry, there's a mirror entry with (target, term, priority).

We can think of an index as an accelerator structure to help us quickly find items matching a query term, or we can think of it as a way to group elements under a group key. If we choose to think of it as a collection, the grouping key would be the term.

(NOTE: I have recently introduced the concept of "collection" into vbolt, but it's still in flux, and it overlaps very much with Index, so it might go away, or the API might get merged with the Indexing API, so I choose for now to not talk about it much).

Now, when we create a new Post, we save it to the posts bucket, and we also add its entry in the user posts index and the hashtags index.

In both indexes, we use the creation timestamp as the priority index. This means when we iterate matching posts, they will be ordered by creation time.

Updating the index happens by setting all the terms for a target, along with their priorities. This is the basic concept, but we have a few different functions for doing that, depending on the use case:

• We can set several terms, associating a different priority to each

• We can set several terms, with the same priority

We also have helpers to set one term (so we don't have to create a slice when calling the function), and we can ignore the priority parameter (using the zero value).

Here's the actual code to write the data to the bucket and indexes:

vbolt.Write(ctx.Tx, PostsBkt, post.Id, &post)

vbolt.SetTargetSingleTermExt(
    ctx.Tx,         // transaction
    UserPostsIdx,   // index reference
    post.Id,        // target
    post.CreatedAt, // priority
    post.UserId,    // term (single)
)

tags := ExtractHashTags(post.Content)
vbolt.SetTargetTermsUniform(
    ctx.Tx,         // transaction
    HashTagsIdx,    // index reference
    post.Id,        // target
    tags,           // terms (slice)
    post.CreatedAt, // priority (same for all terms)
)

I agree the API is not very clean or consistent; hopefully it will be cleaned up in a new future release.

Now, to find posts by user id, we do the following:

const Limit = 100
var window = vbolt.Window{Limit: Limit}
var postIds []int
vbolt.ReadTermTargets(
    ctx.Tx,       // the transaction
    UserPostsIdx, // the index
    req.UserId,   // the query term
    &postIds,     // slice to store matching targets
    window,       // query windowing
)
vbolt.ReadSlice(ctx.Tx, PostsBkt, postIds, &resp.Posts)

We read the matching post ids to a list of numbers, then use this list of numbers to read the list of posts to a slice of []Post.

Querying the index is basically the same:

var postIds []int
vbolt.ReadTermTargets(
    ctx.Tx,      // the transaction
    HashTagsIdx, // the index
    req.Hashtag, // the query term
    &postIds,    // slice to store matching targets
    window,      // query windowing
)
vbolt.ReadSlice(ctx.Tx, PostsBkt, postIds, &resp.Posts)

This is all the code we need to index and query data.

There's no "SQL" layer that the code has to go through. There's no "ORM". We don't need an ORM. We don't need to create a "Repository" interface; vbolt is already a suitable programmatic interface, with reasonably designed building blocks. We don't need to mock the database for testing.

If we want to do some automated testing, we can write test code that uses a temporary file as the database, and then let it run the code. Then we can verify the inputs and outputs of the system as a whole.

In fact, let's do just that. (I was not going to do it until I wrote the above paragraph).

First we setup the temporary test database.

In case you are wondering what that OpenDB function is, I simply extracted the few lines we had inside MakeApplication

Back to the test, we specify a few test cases and a set of expected outputs:

Then we execute the inputs and check the outputs

We can run the tests, and they pass!!

Now, you may wonder if it's really executing the tests! OK, we can easily add some logging and run the test command with `-v`

Then the output will be:

Now that we have an automated test that confirms the code is working, we can start coding the frontend.

It should be pretty obvious what we need to do. There's nothing new here so I'm not sure how much explanation is required for the frontend code.

Just like last time, we define a route, we fetch data for said route, and we display the data.

We'll do that next episode, and we will add some basic session token management as well.

Hope you enjoyed this episode and found it useful.

Download the code: EP004.zip

View the code online: HandCraftedForum/tree/EP004

In the previous episode I walked you through setting up an empty project and creating a simple RPC on the server the the client can call, as well as basic UI interaction and state management.

In this episode I will show you how we do data persistence by continuing the basic auth implementation.

The purpose of this exercise is to demonstrate the basic structure of applications written in this framework. Once we are done this demonstration, we will begin building out the project at a faster pace, and I will not necessarily be showing all the steps in their tiny details, as writing up about everything is going to take considerable effort and slow down the project work itself.

So in these first few episodes, I want to give you the tools to understand the basic way we do things, so that you can follow along as we proceed further.

Native data persistence, indexing, and querying

Unlike most web frameworks, we are not going to be using an external database server that we connect to, instead, our database engine is a library that is embedded into the program itself. Sort of like SQLite, but not.

Also unlike most web frameworks, our data model is not relational, and our querying model is not SQL. Instead, we just call regular functions to persist data, persist indexing information, and load/query it.

In the SQL model, you send commands in a textual query language, which you must construct first, and send to the storage engine. Even in SQLite, that's what you do. The SQL engine then parses the query you sent and turns it into an execution plan, and executes it, usually in some kind of byte code virtual machine; at least, that seems to be what SQLite does.

Our data persistence models is completely different. We use a B-Tree based storage engine (BoltDB) to persist data in buckets, indexes, and collections.

You can think of a bucket as a persisted map: you put in the object Id, you get back a copy of the object.

An index is like a persisted bidirectional multi-map, but with a sorting key, and the ability to set all the search terms (keys) for a particular target.

A collection is a way to group a list of keys under a parent key, with an ordering key.

We have regular function to interact with these storage blocks:

• Buckets: Store/Load/Delete items by id

• Collection: Add/remove (key, order) by id

• Index: Set target terms, and iterate matching targets for a search term.

A few other utility functions exist, but the above constitutes the core of our persistence API, and it suffices for almost everything you want to do in a web application.

To allow data to be persisted, it needs to be serialized. We define a serialization function using VPack.

This is what a typical serialization function would look like:

type User struct {
    Id       int
    Username string
    Email    string
    IsAdmin  bool
}

func PackUser(self *User, buf *vpack.Buffer) {
    vpack.Version(1, buf)
    vpack.Int(&self.Id, buf)
    vpack.String(&self.Username, buf)
    vpack.String(&self.Email, buf)
    vpack.Bool(&self.IsAdmin, buf)
}

The PackUser function is used for both Serialization and Deserialization. You do not need to define these two separately.

Now that we have a type and a serialization function, we can define a bucket:

var dbInfo vbolt.Info

var UsersBkt = vbolt.Bucket(&dbInfo, "users", vpack.FInt, PackUser)

The first argument is an object that collects information about the database file: the list of buckets, indexes, and collections. This information will be used to initialize the database. We will see how in a bit.

The second parameter is a name for the bucket. You will almost never use this directly, but we still choose to give a short but meaningful name. The only actual requirement is for it to be unique.

The third and fourth parameters define the types of the "key" and "value" objects, and their serialization functions. vpack.FInt is the serialization function for integers that uses fixed width (8 bytes), as opposed to the regular vpack.Int which uses a special scheme that allows using the minimum number of bytes to store small number values.

The Bucket function takes packing functions, not types, as its parameters. The types for Keys and Values are derived from the packing functions.

Now, let's say we want to define an RPC to create a user with a username, email, and password.

We need a few more things in addition to the above:

We need a bucket store the password hash. Note: we do not put the hash in the User struct; we don't want that data to be sent to the browser when we read stuff from the UsersBkt.

var PasswdBkt = vbolt.Bucket(&dbInfo, "passwd", vpack.FInt, vpack.ByteSlice)

We also need to keep track of usernames that are taken so we can prevent duplicate usernames, and we also want to be able to retrieve the user id for a given user name.

var UsernameBkt = vbolt.Bucket(&dbInfo, "username", vpack.StringZ, vpack.Int)

Now there's an important step when you define a set of buckets: we must create them if they don't exist on program startup.

Without this bit of initialization code, any attempt to read or write to buckets would panic at runtime.

Note: we don't return errors for reads or writes to the database; we only panic. It's the programming equivalent of reading/writing to a null or invalid pointer, or to outside of array bounds.

When they panic, the response handler for the RPC returns a special "Server Error" response.

Now we can implement AddUser and GetUsers in terms of the buckets.

Here's a utility function to fetch all the users from the UsersBkt (without pagination)

func fetchUsers(tx *vbolt.Tx) (users []User) {
    vbolt.IterateAll(tx, UsersBkt, func(key int, value User) bool {
        generic.Append(&users, value)
        return true
    })
    return
}

We'll use this at the end of both the AddUser and ListUsers.

Now, let's take a look at how I implement adding a user. It's a bit difficult to explain everything in a blog format, so I took a screenshot of the code and annotated it.

We can make a few changes to the frontend code, they should be easy to guess, and we have an interface to add users:

I hope this serves as a good introduction to my style of programming with database: it's just a set of building blocks and functions you call to manipulate data on buckets, indexes, and collections. We haven't seen how to use indexes yet, but when the time comes and you see it, hopefully you will not be surprised.

There's a lot to talk about in terms of what the storage layer can do and how to use it. I'll introduce more aspects of it gradually as we move along.

Download the code: EP003.zip

View the code online: HandCraftedForum/tree/EP003

The core feature of a forum is the post. If you cannot read posts and reply to them, you don't have a forum.

Can you have a forum without categories? You can.

Can you have a forum without user auth? Yes you can. Everyone can be anonymous. If they can read and participate in discussions, it's a forum.

Can you have a forum without search? Yes you can. People can just share links, or external sites can index the content and expose search feature.

So normally I'd say: ignore everything and start with the discussion. I'm even trying to work out a UI design:

However, I was to start by introducing the basic features of our mini framework:

• RPC Wiring

• UI State Management

• Data persistence & Indexing

So we'll work on something simple that allows us to showcase how our framework works.

To this end, I decided to start with a basic, bare bones, user registration system.

Features of a basic auth system

• Form to create an account; nothing but username, email, and password

• Form to login to account; nothing but username and password

• Session management using tokens

• Distinguishing "Admin" accounts

• Page to list all accounts

• Persisting account and session data

• [DEBUG] Button to login as any account (if you are admin)

Things we will not do now:

• Email confirmation

• Pagination for the user list page

• User Profile & Bio

Stage 1: Volatile anonymous "accounts"

We start with the most basic thing: an input box an a submit button. You can create as many accounts as you want, and the server keeps track of them.

We start by creating a procedure on the server side that is exposed to the client. The input is the new username, the output is a list of all current usernames.

// global (but volatile) list of usernames
var usernames []string

type AddUserRequest struct {
    Username string
}

type UserListResponse struct {
    AllUsernames []string
}

func AddUser(ctx *vbeam.Context, req AddUserRequest) (resp UserListResponse, err error) {
    usernames = append(usernames, req.Username)
    resp.AllUsernames = usernames
    return
}

This procedure does not do much other than appending the given name to a list and returning the list.

• Input and Output params must be structs

• First input param is *vbeam.Context

• Second output param is error

• The content is regular code that does not care about HTTP or JSON.

Now, to expose this code to the client, we call vbeam.RegisterProc(app, AddUser) inside the MakeApplication function:

func MakeApplication() *vbeam.Application {
    vbeam.RunBackServer(cfg.Backport)
    db := vbolt.Open(cfg.DBPath)
    var app = vbeam.NewApplication("HandCraftedForum", db)
    vbeam.RegisterProc(app, AddUser)    //    <<====   Added!
    return app
}

Great, now, how can we test that this works? Do we need to build the UI first?

Actually no. There's a much simpler way.

Inside main.tsx add a line to import the generated server module, and expose it on window

import * as vlens from "vlens";
import * as server from "@app/server"     //   <==== added

async function main() {
    vlens.initRoutes([
        vlens.routeHandler("/", () => import("@app/home")),
    ]);
}

main();

(window as any).server = server           //   <==== added

Now, we can call it from the browser console!

Just like the Go function returns a response and an error, so does the generated binding function. It returns a response and an error, except the error is a string type. When there's an error, the response is set to null and the error is the returned error message.

Since it goes over the network, it needs to be awaited

typescript

let [resp, err] = await server.AddUser({Username: "admin"})

This is the basic pattern for communication between the client and the server.

Now let's create the client side code. This will be a special page so let's create a new route, for instance, `/users`

The idea is to show the current list of users, let you add new ones, and then "select" a user to login as.

We start by adding the route, and note that we have to add it before the '/' route, because we match like Go's http routes, by prefix, and the first route that matches as a prefix for the current location will be picked.

The initial content of users.tsx is to just list the current user names

import * as preact from "preact"

import * as server from "@app/server";

export async function fetch(route: string, prefix: string) {
    return server.ListUsers({})
}

export function view(route: string, prefix: string, data: server.UserListResponse): preact.ComponentChild {
    return <div>
        <h3>Users</h3>
        {data.AllUsernames.map(name => <div key={name}>{name}</div>)}
    </div>
}

The fetch function now calls ListUsers, which I haven't shown, but you can easily imagine how we added it.

The view function takes that initial response and just renders the list of names.

NOTE: We have to restart the server after defining new RPC functions in order to generate the typescript bindings.

When we run this we face a problem: because the server has restarted, the username list is cleared, but it was initialized to nil, so the response list will be null.

This is a weakness of the binding system: it does not set the type of slices to nullable. This is on purpose: I do not want to litter the code will nullable array types. It's borderline retarded. Instead, I just do a bit of extra work to ensure all lists in responses are not null. Sometimes I can miss some cases, but I see it as low risk. Ideally the JSON encoder never encodes a nil slice as a JSON null, but that day is not today.

For now I will just init usernames to an actual empty slice.

That should do it, but it does not change the fact that the list is empty.

We can again just add names from the dev console. When we do it and reload the page, we see those names, indicating the TSX code shown above works!

Now we want to show a form where you add a new name and the list is auto refreshed. To do this we will need to store some state:

• The current name input

• The latest known user list

This state must be "stable" across rendering cycles, but we are:

• Not using class components

• Not using React Hooks

So how do we make a "stable" state object across rendering cycles? Caching

vlens ships with a simple caching system that lets you specify:

• The cache key (usually a list of things)

• The computation function if no hit for key

There happens to be no limit to how many keys you can cache. We don't have an LRU. We just expect you not to fill the cache with tons of things. Which is arguably not a great cache API, but it lets us rely on the cache to allow for creating "stable" objects that are "hooked" to other "stable" objects.

Here's our type definition for the state object:

type Form = {
    data: server.UserListResponse
    name: string
    error: string
}

Now, we can create a function that retrieves a stable reference to it this way:

function getForm(data: server.UserListResponse): Form {
    function create(): Form {
        return {
            data, name: "", error: "",
        }        
    }
    const key = [getForm, vlens.cacheById(data)]
    return vlens.cacheGet(key, create)
}

When this is called for the first time with the given parameters, there will be nothing that matches the given key, so the create inner function will be called to create the instance, and the reference to that instance will be stored in the cache.

Next time we call getForm with the same reference to data, it will not create a new instance of Form, instead, it will return a reference to the instance previously created.

This effectively creates a hook, but not in the React style. It hooks one object reference B to another object reference A. As long as the reference to A is stable, the reference to B will also be stable.

Now, since this pattern is so desirable, we have a specialized interface for it: declareHook. It takes a function that receives parameters, and hooks the output to those parameters.

const useForm = vlens.declareHook((data: server.UserListResponse): Form => ({
    data, name: "", error: ""
}))

This is functionally the same as the previous code: it create a function (we called it useForm instead of getForm here) that returns a reference hooked to its input.

Our plan is that when we submit the new name and get the update user list, we update the list in the Form object, and leave the original `data` object untouched, because it's what we got from the initial page load, and we would like to treat it as effectively immutable.

So now we add the input form, and this is a chance to introduce another powerful feature of vlens: binding input to fields without explicitly writing callbacks

<input type="text" {...events.inputAttrs(vlens.ref(form, "name"))} />

There are two features in this snippet:

• vlens.ref: an object that acts like a pointer to a field.

• events.inputAttrs: set the value and onInput attributes to bind the input field to the given ref

The ref is, in reality, a poorly implemented 'pointer' from C. What we really want to do is say something like:

input({ .ref = &form.name })

It's just a bit too verbose because of the language and the underlying DOM API.

Now, just having this, we want a simple way to check that the code does indeed bind the input field to the form.name field, so we just print it:

And here we confirm that it works:

Next, we need to add a button that, when clicked, calls the AddUser proc on the server, and replace the username list with the one we get from the response.

Well, we can easily define a function that does exactly that:

async function onAddUserClicked(form: Form) {
    let [resp, err] = await server.AddUser({Username: form.name})
    if (resp) {
        form.name = ""
        form.data = resp
        form.error = ""
    } else {
        form.error = err
    }
}

But this function takes a form as a parameter; we can't just pass it to an onClick .. we'd have to use a closure 🤔

The problem with closures in JSX attributes is they are not stable references, and to make good use of the virtual dom, we would really like to keep all the callback references stable.

vlens has just the right tool for that, again using the caching module. We have a utility that lets us return a stable closure reference!!

So instead of doing this:

<button onClick={() => onAddUserClicked(form)}>Add</button>

We do this:

<button onClick={vlens.cachePartial(onAddUserClicked, form)}>Add</button>

vlens.cachePartial(fn, a, b, c) returns a stable reference to what you would get from calling fn.partial(a, b, c). However, for this to work properly, the input function itself must not be a closure that depends on variables in its environment with an unstable reference.

Now, one last thing to know about: because none of the variables we're working with has any special status in the context of the UI library, we have to explicitly tell the UI to refresh after we are done with the response:

`vlens.scheduleRedraw()` does not cause an immediate re-rendering. It just tells the system to re-render in the next animation frame. It's safe to call this as many time as you want per rendering-cycle. It's idempotent.

Here's testing the UI:

We can try adding some error checking for fun, but it's too trivial to cover in great detail here, and we can leave it as an exercise to the reader. For instance, we can check for duplicate entries, and refuse an entry if it already exists. We can enforce only alphanumeric characters and underscore, dash, and dot.

Now, more importantly, the next step is to persist the data to a database that survives server restarts.

But, I think we've covered enough ground in this episode already, so we'll save data persistence to the next episode.

Download the source code: EP002.zip

View the source code online: HandCraftedForum/tree/EP002

This is the first article about the development of "HandCraftedForum"

Repository URL: https://github.com/hasenj/HandCraftedForum

HandCraftedForum is a web project that will use a mini framework I have developed over the course of the last couple of years.

As such, the base/skeleton project setup will reflect the framework.

The framework overall does not have a particular name, but it's composed of the following components:

VBolt: a storage layer built on top of BoltDB and VPack, a serialization library.

VBeam: the server side web framework. It mostly consists of an RPC system that lets the client side call functions on the server side without thinking about ReST or HTTP.

VLense: a client side framework, consist mostly of routing and some utilities to help create UIs (p)react style without callbacks or "hooks". (We provide our own version of what hooks are meant to do; more on that later).

The overall theme is straight forward programming with data and procedures. The server side code uses VBolt to store and retrieve data. VBeam provides an interface to the client to communicate with the server side code. The client code renders the UI using the data it obtained from the server + transformation applied via user interaction.

The skeleton of the project will let us run a web server locally and opens a mostly empty page with a welcome message rendered via client side code.

License

Before we start, we should choose a license. This is mostly a technicality but has implications.

I'm generally in favor of open source for libraries and code snippets, but not for final products. I believe software products should be sold for money.

I asked Claude if there's a license I can use that allows people to freely use the source code and study from it while prohibiting people from taking the product as-is and rebranding it as their own. I didn't think there was such an option, but to my surprise, turns out there is: the

commons clause

combined with any other license.

So I combined the commons clause with the MIT license.

The license does not comply with the OSI definition of "open source", but we don't care.

The license lets you use the code freely in all the way that matters. It just does not late you take the final product and make money from it.

Check the linked website for the FAQ regarding the license.

Server side code

First we create a module using go mod init forum

module forum

go 1.22.1

Then we create app.go:

go

package forum

import (
    "forum/cfg"

    "go.hasen.dev/vbeam"
    "go.hasen.dev/vbolt"
)

func MakeApplication() *vbeam.Application {
    vbeam.RunBackServer(cfg.Backport)
    db := vbolt.Open(cfg.DBPath)
    var app = vbeam.NewApplication("HandCraftedForum", db)
    return app
}

This creates a mostly empty application.

The MakeApplication function is mostly a convention I use. Note that this package is not the main package, so there will be no main function here. Instead, we will have two different main packages: one that we run locally, and one that we run in production.

The local and production version differ in the following ways:

• The local server also bundles the frontend code and performs type checking on it

• It generates the server side bindings for the client (more on that later)

• It uses different paths for database file and static folder

• The local server serves the frontend code from the local file system, while the production server embeds the generated frontend bundle and serves it from RAM. In fact, the code is written to expect the frontend bundle to have been already generated in a prior step.

Let's look at this function line by line:

go

vbeam.RunBackServer(cfg.Backport)

This innocent looking line does something very important: it tells any previously running instance of this server to shutdown.

A "back server" is a private backdoor that lets us control the server. The way we do that is by using a specific port number that no other program uses. If you have multiple servers running and they all use this framework, you need to pick these port numbers such that they do not conflict.

The first thing a back server does is send a "shutdown" command to the given port number. If a server is already listening there, it will shutdown.

This allows nearly instant deployment with zero shut down time.

The next line create a db instance. For now we have nothing on the db, so it doesn't matter what we do with it. It's just that the db is required for the next line: the vbeam function that creates a new application and sets up the RPC system.

Again this is the empty skeleton application so we will just return it as-is.

We also create a cfg package to store some code level configuration variables that vary between local and production. In this case, we see cfg.DBPath.

There's also a variable that does not vary between local and production: the backport number. I just chose this number arbitrarily.

cfg/cfg.go

package cfg

const Backport = 12832

To distinguish local from production, we use build tags.

cfg/local.go

//go:build !release

package cfg

const IsRelease = false
const DBPath = ".serve/data/db.bolt"
const StaticDir = ".serve/static/"

cfg/release.go

//go:build release

package cfg

const IsRelease = true
const DBPath = "data/db.bolt"
const StaticDir = "static/"

Next we create the command that runs the local development server. This will mostly follow a template.

First we create a directory local and create local.go and name the package main so it can be executed:

local/local.go

package main

func main() {

}

Now, before writing out the full content, I want to first show the function that launches the server:

import (
    "fmt"
    "forum"
    "net/http"
    "os"

    core_server "go.hasen.dev/core_server/lib"

    "go.hasen.dev/vbeam"

    "forum/cfg"
)

const Port = 5212
const Domain = "forum.localhost"
const FEDist = ".serve/frontend"

func StartLocalServer() {
    defer vbeam.NiceStackTraceOnPanic()

    app := forum.MakeApplication()
    app.Frontend = os.DirFS(FEDist)
    app.StaticData = os.DirFS(cfg.StaticDir)
    vbeam.GenerateTSBindings(app, "frontend/server.ts")

    var addr = fmt.Sprintf(":%d", Port)
    var appServer = &http.Server{Addr: addr, Handler: app}

    core_server.AnnounceForwardTarget(Domain, Port)
    appServer.ListenAndServe()
}

Notice that after we call MakeApplication that we created above, we set the Frontend and StaticData fields on it.

In the production main, we will set those variables differently. We will see when the time comes.

Next we generate the client side bindings. We don't have "RPC" yet, but when we make one, a binding module will be generated automatically.

This assumes that we will put all the frontend code in the frontend directory.

Next we prepare an http server from the Go standard library and set the handler to the app we just created, because the vbeam.Application object implements the http server interface.

The next line is important for allowing us to open the website using a domain on port 80 instead of localhost:5212

core_server.AnnounceForwardTarget(Domain, Port)

This assumes we have core_server running.

Core server is a very small reverse proxy that is configured only via UDP messages. This is the most important message: it maps a domain to a port. Meaning, when a request comes for the given domain, it reverse proxies to the given port on localhost.

The core server can be installed this way:

go install go.hasen.dev/core_server@latest

Once installed, run it this way:

nohup core_server & disown

You don't need to worry about running it multiple times; it's idempotent. If you run it again, it will shutdown the previous instance before starting.

Keep in mind: this is entirely optional. If you don't have core_server, you can always access the server using the localhost:port combination.

Next we show the main function for the local server:

var FEOpts = esbuilder.FEBuildOptions{
    FERoot: "frontend",
    EntryTS: []string{
        "main.tsx",
    },
    EntryHTML: []string{"index.html"},
    CopyItems: []string{
        "images",
    },
    Outdir: FEDist,
    Define: map[string]string{
        "BROWSER": "true",
        "DEBUG":   "true",
        "VERBOSE": "false",
    },
}

var FEWatchDirs = []string{
    "frontend",
    "frontend/images",
}

func main() {
    os.Mkdir(".serve", 0644)
    os.Mkdir(".serve/static", 0644)
    os.Mkdir(".serve/frontend", 0644)

    var args local_ui.LocalServerArgs
    args.Domain = Domain
    args.Port = Port
    args.FEOpts = FEOpts
    args.FEWatchDirs = FEWatchDirs
    args.StartServer = StartLocalServer

    local_ui.LaunchUI(args)
}

This setups the frontend builder configuration as well as the list of directories to watch for changes in order to typecheck and rebuild.

This won't make sense without talking about the frontend first.

The frontend

The frontend consists of an index.html file that loads the entry typescript file, which will start a client side router, and then render the current page depending on the URL.

frontend/index.html

<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>HandCraftedForum</title>
</head>

<body>
    <script src="/__u/hasen.substack.com/main.tsx" type="module"></script>
</body>

</html>

Nothing special here, except the fact that we are referring to /main.tsx. The frontend bundler will replace this with the path that resulted from the bundling operation.

We need to first setup the dependencies

npm install --save preact vlens
npm install --save-dev typescript

We also setup a basic tsconfig.json file and configure it to use preact for jsx.

json

{
    "compilerOptions": {
        "module": "es2020",
        "target": "es2020",
        "lib": ["es2020", "dom"],
        "moduleResolution": "node",
        "esModuleInterop": true,
        "skipLibCheck": true,
        "noEmit": true,
        "strict": true,
        "jsx": "react",
        "jsxFactory": "preact.h",
        "jsxFragmentFactory": "preact.Fragment",
        "baseUrl": "frontend",
        "noImplicitAny": true,
        "paths": {
            "@app/*": ["./*"]
        }
    },
}

Notice I've set it up so that we can use the prefix @app to refer to our own codebase.

Now we can setup a skeleton main.tsx:

import * as vlens from "vlens";

async function main() {
    vlens.initRoutes([
        vlens.routeHandler("/", () => import("@app/home")),
    ]);
}

main();

This initialize the app and sets up a single route mapping.

Client side routing works by matching a route with a fetching function and view function.

The fetch function must return a promise for some data. This data will be stored and passed to the view function. The view function always takes this data as input. The reference to this piece of data will be stable across rendering cycles.

This is a crucial feature of this mini framework: there's no state management. There's just data. Data can be used to reference other data. The "entry point" to all the data you need is the object returned from the fetch function above.

You can associate "side" data ("hooked" data) by relying on the stability of the reference. We will show how later in the project.

The routeHandler is a helper function that sets the route by dynamically importing a module and looking for "magic" names fetch and view.

Actually the second argument does not need to be a module. It can be any function that returns a promise for an object satisfying the interface:

export type RouteHandler<Data = any> = {
    fetch: (route: string, prefix: string) => Promise<rpc.Response<Data>>;
    view: (route: string, prefix: string, data: Data) => preact.ComponentChild;
}

You can just as easily do something more complicated than just loading a module, but for now we will simplify our life by just creating a module for each route with a fetch and view functions.

For the home route, we will again just make the bare minimum empty fetch and view functions:

import * as preact from "preact"
import * as rpc from "vlens/rpc";

type Data = {}

export async function fetch(route: string, prefix: string) {
    return rpc.ok<Data>({})
}

export function view(route: string, prefix: string, data: Data): preact.ComponentChild {
    return <div>
        <h2>Hand Creafted Forum</h2>
        <img src="/__u/hasen.substack.com/images/framework.png" />
    </div>
}

The fetch function fetches no data per se; it just returns an empty object as if it was an "ok" response from the server. While the empty object is generally useless, it's not entirely useless: it has a reference id, and that reference id is stable.

We will not use that yet, but it's worth keeping in mind.

Our view function just returns basically static html (as jsx).

In the jsx, we refer to an image. We create an images directory inside frontend and place the image file there.

We are now ready to run our local server. Here's a video of me doing that:

Here's the resulting page:

We have successfully created an empty skeleton project using our framework.

Note: in the video you can see me edit the jsx code and then reload the page to see the results. This is not "hot module reload". You still have to reload manually. We are just saving you the bundling step by doing it automatically when you save the code file.

Code base for Episode 001:

Web view: HandCraftedForum/tree/EP001

Zip download: EP001.zip

Follow along

Download the code base, see if you can run it locally.

If you face any problem, report it to me, either by opening issues on github or replying here on substack.

2024.12.05

I've been meaning for a while to build a web project in public.

After some deliberation, I'm settling on a forum. For various reasons:

• There does not seem to exist a very good one, or at least, not many choices are available, so if the one or two famous ones do not fit your needs, you are out of luck.

• It's familiar enough that most people know what to expect.

• It's simple enough on the surface, but more advanced features can contain quite a bit of complexities.

Until we settle on a product/brand name, I'm going to call it "Hand Crafted Forum" (because "hand made" is taken!)

I want to use it to demonstrate a few things I care about in web development:

• Straight forward programming. Just data and functions. This means there's not a hint of "OOP". No adapters, no providers, no services, no repositories. It also means we take into account what the computer has to do to accomplish the requested task: we will try to minimize waste, to a point, without requiring deep expertise in low level programming. This implies the data model and transformations are guided by the desired product features, not by some idealized version of what the data is supposed to represent.

• Simplified development environment & deployment process. This means not much to install or configure other than language compiler(s). Most things will be embedded libraries instead of external programs. If something *must* be an external program (say, ffmpeg), it will be zero configuration, or we put all the configuration necessary in our own code base. If you have the codebase and the compiler, you can start the local server in one command, and you can deploy to staging/production with one command.

• Inside out iteration process We start with the core functionality and add side features and ornamentations later. The UI design will visually suck at first, and gradually get better over time. We will dedicate a lot of time to the UX, because the UX is the product.

Overview of Scrum

Scrum is both a process and a meta process.

It's a process because it has strongly defined roles and activities: the scrum master "SM", the product owner "PO", the daily scrum, the retrospective, the backlog, etc.

It's a meta process because it doesn't define the actually important roles well enough and instead tell you to define your own:

• Who is responsible for architecture?

• What are the different roles for senior, intermediate, and junior programmers?

• What happens if there's disagreement about important architecture decisions?

These are the actually important questions that you need a framework in place in order to answer. "Talking about it as a team" is not a framework.

Scrum pretends that the answers to these questions is a mystery and you can only discover it by doing the Scrum process. But the answer is not a mystery at all, and by avoiding the answer to these important questions, Scrum provides no value at all.

Scrum's idea for project management starts with the "PO" creating a backlog of tickets. On what basis are the tickets created? That's left up to them!

The PO then talks to their team of programmers to select tickets to complete in a "Sprint", which is a time-box where the team "commits" to a specific "goal".

After the sprint is complete, the teams stops to evaluate the situation and adjust accordingly.

Starting on the wrong foot

Quote from the scrum guide:

1. A Product Owner orders the work for a complex problem into a Product Backlog.

Scrum's very first step is already a mistake. You can't just have one non-technical person competently break down the product into a series of tickets.

The PO, according to scrum rules, should not be a technical person. As such, they don't really know what's the best way to break down the problem into a list of tickets. Whatever they come up with is probably not of good quality.

By the time the programmers get to have a look at the backlog, it's too late to fix.

The PO has already been put in too much effort into creating all these elaborate user stories, and they are not going to fundamentally change how the tickets are structured.

The implicit assumption in Scrum is that programmers are incapable of performing the work of breaking down a large goal into a series of small steps, because it's a kind of soft skill that programmers don't have.

This is of course misguided. Many programmers have those skills. The better the programmer is, the more likely it is that they have a deeper understanding of how to best break down a project into a series of steps.

Phases and Milestones

You can't actually start at the level of an individual ticket, or even a user story.

You have to start at a somewhat higher level: breaking it down into a series of phases, each encompassing a meaningful set of features that can be released, either to the public, or to a select group of test users. It should be something that can be completed in a few months.

Each phase is then broken further down into a series of milestones that can be reasonably completed within a few weeks.

You may notice that "few weeks" is vague. This is on purpose. You need some amount of flexibility in terms of scope and time. This will be explained further on the section titled "The Sprint".

Arriving at meaningful phases and milestones requires collaboration between technical leaders and stakeholders.

A tech leader can come up with milestones on their own, but these milestones might not make business sense. Similarly, the business owner can come up with arbitrary milestones, but they might not make sense from a technical perspective. That's why it should be a collaboration between the two sides.

Now, to reduce communication overhead, one or two people from each side would usually suffice. Adding more people to the mix does not usually result in a better understanding of the problem. Instead it would just increase the communication overhead. It would also increase the probability of personal conflicts, as each member tries to prove how valuable they are to the process by suggesting new ideas and thinking of new potential obstacles.

The goal is not to arrive at the perfect plan. We just need something reasonable; a rough plan that allows us to start moving forward.

How to break the project down into a series of milestones and how to prioritize the work is itself an important topic that needs its own treatment.

You have to start by building a shared understanding (between the business side and the tech side) of the high level goals and the vision behind the project.

Based on these goals, you think of a smaller version of the product that aligns with the ultimate goals and vision.

In each phase, you increase the scope of the project so that it comes closer and closer to fulfilling all of its goals.

One common mistake I've seen some managers make, specially when operating under the tacit assumptions of scrum, is the following: they break down the project by slicing it down vertically; for example: their plan is to develop the product one page at a time. Each time, they aim to get all the details for that page fleshed out in terms of graphic design and implementation. If the page is too large to develop in one sprint, they break it further down (again vertically) into sections, each of which they expect to be developed fully in a sprint.

It's as if you plan the construction of a house by building one room at a time, each time everything about that room is fleshed out: all the electric sockets, the windows, the water pipes, etc, but only for that room! And then you break that down to one wall at a time, again with all the windows and electricity sockets and water pipes for that wall!

This is not a good way to manage the project's milestones.

Instead, start with a broad sketch; a skeleton. Then flesh out the details gradually and iteratively.

Start with a rough sketch of the core feature. Use fake data to compensate for lack of real data.

For example, if the the current phase of the project is mainly about booking and managing appointments, start with the calendar page. It doesn't have to look good. It doesn't even have to look correct; it's fine at first to pretend that all months just have 30 days.

What matters at this stage is nailing the feel and interaction.

Since you haven't yet built anything to input the data, you will have to fake it: fake users, fake sessions, fake appointments, etc.

You try various ideas for how to interact with the calendar until you find something that seems to work well enough. Then you can start filling out the other details: getting the month display to be correct, working out the graphic design, the colors, the shadows, etc.

If you have more programmers, you can let them work on sketching out the other more peripheral pages, or do various types of independent UX experiments, etc.

Think of how you would go about putting in screws on a table. You don't put it one screw at a time. You put in all the screws first but keep them loose, then you gradually start tightening them up, one by one, but all together: never tightening one screw too much while leaving others too loose.

Planning the sprint

Quote from the scrum guide:

2. The Scrum Team turns a selection of the work into an Increment of value during a Sprint.

The second step in the scrum process is also deeply misguided. It assumes that a PO, having created a backlog of tickets without much engineering considerations, can just share the top items from the list with his team of programmers, and they can just grab a bunch of tickets that fit into a sprint and just start working on them!

That's not how software development works. You cannot just pickup a ticket and "implement it" as if it's an isolated unit.

A lot of upfront preparatory work is usually required, specially if we assume that most tickets are created such that each can be completed by a single engineer in a couple of days.

At the very least, you need to define the interface between the different components in the system, but more broadly, you need to define the data model and outline the list of processes that need to be implemented.

This is especially required if you have a team of multiple people that are expected to work in tandem on different components of the same feature or related set of features.

Scrum provides no guidance here. Instead, it's often a hinderance, as it primes the PO into thinking that the programmers just need to break down the tickets into new, smaller tickets if necessary, and that's it!

As if features can just be built onto the air, with no foundation in place.

When the programmers manage to convince the PO that they do in fact need to build some foundation first, the PO and the SM would ask the programmers to build the minimum amount of foundation necessary for the next sprint, and nothing more!

To see how absurd this is, imagine a construction crew being asked to build a house in sprints, each sprint they build just one wall, and each sprint they are only allowed to build the foundation for that wall only and nothing else!

Building software is not like building physical buildings, but the analogy here stands: you need to build the lower layers first, you need some foundation to build on, and there are good and bad ways of breaking down the work into stages. Building a house one wall at a time is nonsensical, but if you've never seen how houses are built, you wouldn't know.

How do you to take a series of abstract user requirements, and produce an actionable plan that would bring them to reality? This is one of most important problems in project management, but Scrum provides no guidance here what so ever. It is left as an exercise for the reader.

The PO just creates a list of backlog items. The team just makes a plan to implement an increment for the next sprint.

If you are looking into scrum as a solution to project management, this should be the biggest red flag!

Quote from the Scrum Guide

Topic Three: How will the chosen work get done?

For each selected Product Backlog item, the Developers plan the work necessary to create an Increment that meets the Definition of Done. This is often done by decomposing Product Backlog items into smaller work items of one day or less. How this is done is at the sole discretion of the Developers. No one else tells them how to turn Product Backlog items into Increments of value.

Notice the strong implication that developer's planning stage mostly involves breaking down backlog tickets into small tickets that can be completed in a day or two.

It assumes the creation of the backlog tickets was already done in a way that is appropriate from an engineering standpoint.

The programmers, according to scrum, do not get to tell the PO that the current way the work is broken down makes no sense from a technical point of view.

That is not their job!

They are just to figure out how to make it work! If the PO says we need to build the west facing wall with a window in it that can be opened and closed, you are not allowed to tell the PO that this is not how houses are built. Your job is to figure out how to just build the west facing wall with the window on it. You can break the work down into two stages: first you build the west facing wall with a hole for the window, and then you attach the window with the handles.

The missing Architect

There's no one in Scrum who plays the role of an Architect.

Quote from the Scrum Guide:

Scrum Team

The fundamental unit of Scrum is a small team of people, a Scrum Team. The Scrum Team consists of one Scrum Master, one Product Owner, and Developers. Within a Scrum Team, there are no sub-teams or hierarchies. It is a cohesive unit of professionals focused on one objective at a time, the Product Goal.

Scrum Teams are cross-functional, meaning the members have all the skills necessary to create value each Sprint. They are also self-managing, meaning they internally decide who does what, when, and how.

Scrum purports to be a software development process, yet it has no place and makes no room for an architect. It does not even pretend that it addresses the question of how to handle architectural decisions.

This is by far the worst aspect of this process: the diffusion of leadership roles - other than the SM and the PO.

This can be very appealing to business people, but it's disastrous for a software project.

A well functioning engineering team needs a well defined hierarchy. The senior engineers and junior engineers need to understand their role within the team. Responsibilities must be assigned individually, and individual incentives must be aligned.

Scrum diffuses responsibility, and thus destroys incentives.

Under Scrum, the responsibility for architecture is assigned to "the team". There's might be a de-facto architect: a team member who demonstrates more knowledge and experience to his peers, and maybe gets some tacit recognition as such; but never formal recognition, even if they end up doing the majority of architectural work.

The PO and the SM have well defined roles, so they receive visibility and recognition, while individual developers, even those making critical decisions or making important contributions, often end up underappreciated. This can lead to misaligned incentives, and may open the door for unintentional sabotage amongst team members.

A well functioning company would have a structure where the senior engineers, the architects and the tech leads get the visibility and the recognition they need to align their incentives.

The importance of hierarchy

Leadership roles within a software development team should be clearly defined. If there's no leader, infighting and bitterness ensues, as everyone tries to be a leader.

There has to be a leader who makes decision and everyone follows along. This is not to say that team members have no say in the matter. It's of course important that people can state their opinions and objections, and the leader does have a duty to listen to them, but at the end of the day, the buck stops with him; he makes the call.

Without leadership roles, politics take over: people who want power and can tolerate disagreement and social conflict will often get their way, not because of any merit, but because other people give in to keep the peace.

Without leadership roles, a team composed of mostly junior programmers will block the senior architect from making the required decisions because they disagree with him or don't understand the reasoning behind his decisions.

It gets worse if there are multiple senior or intermediate engineers with overlapping responsibilities, as they can disagree about how to do things or second guess each other's decisions.

Someone has to be responsible for making the decision at the end of the day, and when that role is well defined, most engineers would have no problem deferring to him even if they disagree with the final decision.

Engineering Roles

This is a rough outline of what the engineering roles ought to be and what the responsibilities and limits of each role are.

• Tech Lead / Architect

  • The most senior engineer on the team.

  • Makes technical decisions after consulting with seniors.

  • Produces design documents as needed to ensure everyone understands how the system is supposed to work.

  • Responsible for ensuring a coherent technical design for the product as a whole.

• Senior Engineers

  • Each responsible for a relatively large sub-system within the project.

  • Work with the tech lead to design the system as a whole and the sub-system they are responsible for in particular.

  • Implement the core and most critical features of their respective sub-system.

  • Mentor the junior members

• Junior Engineers

  • Each assigned to a Senior

  • Learn how the system is supposed to work

  • Implement all the peripheral and non-critical features

  • Handle the mundane details to ensure their Seniors can focus on the important aspects of the system

  • Aim to improve at their craft and eventually attain the requisite skill level to be recognized as a senior.

Committing and Reviewing Code

Seniors commit/merge to the master branch without waiting for code review, unless they have doubts and need some help.

Juniors at first must get approval from their Senior before merging their changes into the master branch. After a while they get permission to merge without review.

Code reviews can occur after the merging has already been done. For example, Senior engineers can hold code review sessions with their juniors if they notice a drop in the quality of code being committed.

The Sprint

Scrum requires programmers to commit to completing a fixed amount of work in a fixed amount of time. This is the "core" of scrum. It appeals to managers because the consistent delivery of small improvements in fixed cycles creates the impression of consistent progress. But it's mostly an illusion.

The scrum sprint is very strict. You have to make 100% commitment to completing the work. If you end up only completing 95% of it, Scrum would view this as a failure! During the "Retrospective", the PO and the SM will tell the programmers that they failed to deliver the promised work this sprint, and they need to work on improving their delivery, their estimates, or their ticket refinements. Sometimes, an entire hour, or even two, would be spent discussing why the sprint failed, and the programmers would be required to criticize their own efforts and add new rules to the process to make sure such a travesty (of having completed only 95% of the work instead of 100%) does not happen again.

The reality is, unless you set to yourself very "easy" goals, it's impossible to consistently deliver on the estimated time.

An estimate and a commitment are not the same thing. When you estimate that a thing can be done in three days, you cannot commit to completing it in three days. If you have to commit, you need to add in some slack.

When you do scrum, your team are always spending a week to finish work that could be basically completed in two days, because if they don't, they will have to suffer through the next retrospective again!

Setting fuzzy goals

If one wanted to actually be agile, they would not fix the scope nor the time. To be agile, you need flexibility. You must allow some "fuzziness" into the process: we will aim to get roughly this much work done in roughly this much time. Maybe we'll end up getting 90% of the work done in 90% of the time, but the rest would take much longer than expected, so we stop here.

This is good and normal. A healthy work environment would see it as positive progress towards the goal.

Fuzziness is important because you can't actually know how far ahead you would progress without actually doing the work. By the time you get near the end of milestone, you'll have a better idea about what can be reasonably completed in the remaining time, what would take a bit more time but can still be completed soon, and what would take a lot more than originally anticipated.

You can then decide whether it's worth to extend the milestone a little bit in order to give the engineers the time needed to get more work done.

This kind of flexibility is made impossible in Scrum, but it's essential for agility.

Imagine a predator chasing its prey, but at the very start they set a timer for themselves: I will catch it in 10 seconds! If 10 seconds pass and they haven't caught the prey yet, they halt the chase! Full stop! Even if they were just two seconds away from actually catching the prey! This cannot be described as "agile". In the real world, an agile predator would only halt the chase if they determined it to be futile because the prey has outpaced them and ran too far for them to catch up to.

Closing words

We've outlined some problems with Scrum and introduced a set of alternative ideas for project management. I'm not aware of any name for these set of ideas. They're just the obvious way to produce software, and I've seen them practiced informally in several companies I've worked with.

A part of me feels a little bit of unease about trying to outline these ideas. They feel so natural and obvious.

However, we live in a world were bad ideas like Scrum are being touted as "agile" and being adopted by companies trying to figure out how to manage their software projects, so we have no choice to counter act this except to expose its problems and outline the obvious better way.

A web server is just a program that listens to incoming tcp connections on port 80 (or 443, if you want to support https, which you should).

The only problem with this statement is that you can only have one program listening on the port.

Now, mitigating this would be easy if the OS would let you listen on a domain:port combination.

The OS provides a mechanism for different programs to listen on different ports, and even on different IP:port combinations, however, it does not provide a mechanism for different web servers to listen on different domains.

This is because the TCP/IP protocol does not know about domain names. They are separate systems.

The HTTP protocol is aware of domain names though; via the Host header. When you visit example.com, the browser sets the `Host: example.com` header. But that information is at the http level, not at the tcp level, and as far as I understand, the Linux kernel does not have builtin http support.

So while it is possible in principle to multiplex incoming requests to different programs based on the requested domain, the OS does not provide a standard mechanism for this.

Instead, you need a user-land process to fill this role: it would listen on the http and https ports, handles the TLS handshake, parses the request header, and then forwards the request to another program, based on the Host header.

This kind of program is called a reverse-proxy. A regular proxy (in internet parlance) is used by a client to hide their address: your server sees the reqeust coming from the proxy, instead of the end user. But here we’re doing the opposite: the user sends a request to your server, but unbeknownst to them, your server process is behind a proxy.

The most popular program that does this is nginx. It’s very versatile and can do many more things than just act as a reverse proxy. It’s just not very easy to configure programmatically: you have to find out where and how it manages its configuration files, and then edit them to add an entry to reverse proxy to your program, and then send it a signal to reload the configuration files.

Another popular tool in recent years appears to be Traefik. I have no experience with it, but their official tutorial reveals that they depend on Docker, and require configurations with weird and obscure options.

A recent addition is Caddy. It’s particularly good because it handles https and uses Let’s Encrypt to automatically issue certificates. It’s written in Go and has an command line interface as well as an http json api. However, it’s not obvious how to do it programmatically via the json API. Essentially, just like nginx, it does many more things than just reverse-proxying.

The problem is actually very simple, and the solution does not need to be that complicated. At least, not complicated to setup and use.

Internally it might involve some complications: handling https, issuing certificates, forwarding requests.

The usage pattern should be very simple:

• Launch it via the command line

• Safe to launch multiple times (e.g. new version)

  • Previous process gracefully shuts down

  • Configuration is carried forward by the new process

• Exposes a command that accepts two parameters: the domain and the port

  • Easy to call this command programmatically from any programming language

• Does not output anything to the command line

• Log filepath is well defined and easy to discover

I’m releasing a Go program that does just that. Check it out at go.hasen.dev/core_server

It’s only a few hundred lines of code. The way to configure the mapping is to send the following kind of message to the UDP port 40608

add example.com 5555

The structure of the message is simple:

• the word add

• the domain name

• the port number

That’s it.

If you use it, I’d love to hear your feedback.

P.S. I asked an LLM to provide the implementation of a simple function that sends a string message over a udp port, and here’s the output:

Python:

import socket

def send_udp_message(port, message):
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.sendto(message.encode(), ('localhost', port))
    sock.close()

Ruby:

require 'socket'

def send_udp_message(port, message)
  sock = TCPSocket.new('localhost', port)
  sock.send(message, 0)
  sock.close
end

PHP:

function send_udp_message($port, $message) {
    $sock = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
    socket_sendto($sock, $message, strlen($message), 0, 'localhost', $port);
    socket_close($sock);
}

Node.JS (with modifications; perhaps needs further cleanup)

const dgram = require('dgram');

async function sendUDPMessage(port, message) {
  return new Promise(resolve => {
    const client = dgram.createSocket('udp4');
    client.send(message, port, 'localhost', (err) => {
      client.close();
      resolve(err)
    });
  })
}

And here’s my Go implementation:

func SendUDP(port int, cmd string) error {
	var udpAddress = net.UDPAddr{Port: port}
	conn, err := net.DialUDP("udp", nil, &udpAddress)
	if err != nil {
		return err
	}
	defer conn.Close()
	_, err = io.WriteString(conn, cmd)
	return err
}

—

Background

We are building a website where people make "posts" and see other people's posts. People can "follow" certain accounts and some kind of algorithm "recommends" posts for accounts.

Problem statement

We want to see how many views our posts are getting. We also want to distinguish between total impressions vs reach.

How many unique people saw this post? If people come to the post again and again, that's valuable information, but we do want to distinguish between impressions and reach.

We also want to see a graph of the growth of impressions and reach over time.

How to model the data

In previous designed documents, I just presented the data model as a given. This time I want to try something different. I want to walk through the problem solving process.

Views

How do we know the number of views? Well, everytime someone views the post, we increment a counter. So that should be pretty easy.

Now the problem might be that of a definition: how do we decide that someone has viewed the post? If the only way to view a post was by opening the page that holds it, that would be easy. But what if the post shows up on their news feed? Does that count as a view? What if we sent the post to the browser, but it was the 10th post in the feed and the user never scrolled down to it! Do we count it when we send it, or do we wait for a signal from the client that the user has scrolled past it?

What if the post is long and the thing that appears in the feed is just a summary, do we count that as a view, or do we keep separate counts of views of the summary vs views of the details?

We can create two counters: summary_view and detail_view

What if the post is short, and the summary and the detail are the same thing? Well we should still count the detail_view separately because it does have significance: expanding the post details is not just about viewing its content, it's also about posting a reply or seeing what replies other people have left.

So, the solution is, for each post, have two counts:

• summaryViews

• detailViews

Simply increment them when there's a viewing event.

Now we need to answer the question: how do we count a view event? I think we can just let the client tell us when the post is brought into view.

This of course leaves the possibility that a client can just spam view requests to artificially inflate the view count on a post, hoping that this would increase its odds of being recommended to more people by the algorithm.

Perhaps we don't care because that kind of spamming would actually decrease the odds of the post being recommended as it decreases the engagement rate. But then this opens the door for the opposite kind of attack: artificially decreasing the engagement rate for a post you don't like by spamming view requests to increase its total view count.

At first we probably don't care, but if we do care, perhaps several months down the road, we can guard against this kind of behavior by adding another check before counting a view: only count a view from the same user if the previous view from that user was more than 10 minutes ago.

How do we know when was the last time a user viewed a post? Well we could maintain a mapping where the key is the tuple (user_id, post_id) and the value is (view_timestamp), and if we get a request, we first check the mapping: if a record exists for the same (user_id, post_id) pair, we check the time stamp. If the timestamp is not older than 10 minutes, we ignore the request. We don't even update the time stamp! Or perhaps, we could increment a counter, so the value is not just the view_timestmap, but also how many view requests arrived in the timeframe since view_timestmap. This allows to detect spam events: if for example we get 100 or more view requests within 10 minutes, there's a high probability the user is spamming requests to increase the view count. Now, if the the view timestamp is older than 10 minutes, then we overwrite it and reset counter to 1, and we finally increase the view count on the post indicated by post_id.

Having this kind of mechanism in place should also make it very clear how to count reach. That is, views by unique users.

Simply speaking, if we see no entry at all for (user_id, post_id), then this is the first time this user sees this post, and we can increase the reach counter.

We can do the same thing with detail views, but this makes it likely that we will duplicate all that book keeping code.

So to avoid that, we add another parameter to the tuple above: (user_id, post_id, mode) where mode is a constant that indicates whether we are handling the summary views or the detail views. We can also use it for other things, like 'comments', 'reposts', etc. The output from the process are two bits of information: should this count as a new view? and should it count as an increase in reach?

Thinking about this in go, we can define the following structs:

type PostViewMode int
const (
    SummaryView PostViewMode = 0
    DetailView  PostViewMode = 1

    MaxViewMode PostViewMode = 2
)

type PostViewParams struct {
    UserId int
    PostId int
    Mode PostViewMode
}

type PostViewVerifyResult struct {
    IsValid bool
    IsUnique bool
}

func VerifyPostView(params PostViewParams) PostViewVerifyResult

type UserPostViews struct {
    Timestamp timestamp
    Counter int
}

type PostAnalytics struct {
    Totals  [MaxViewMode]int
    Uniques [MaxViewMode]int
}

bucket UserPostViewsFilter(PostViewParams, UserPostViews)
bucket PostAnalyticsBucket(postId int, PostAnalytics)

You may have noticed the PostAnalytics struct uses an array instead of a named fields like `TotalViews, UniqeViews, TotalDetailViews, UniqueDetailViews, etc...`.

This streamlines the processing of verifying and updating data.

Simply put, given a mode, we update the counts this way:

result := VerifyPostView(params)
if result.IsValid {
    analytics.Totals[params.Mode] += 1
    if result.IsUnique {
        analytics.Uniques[params.Mode] += 1
    }
}

This is generally a good way to keep the number of codepaths small, speically when they'd be mostly identical.

Imagine if we had to do this instead, with named fields, and no "mode" parameter to VerifyPostView.

// code path for handling summary view
result := VerifyPostSummaryView(userId, postId)
if result.IsValid {
    summaryAnalytics.Totals += 1
    if result.IsUnique {
        summaryAnalytics.Uniques += 1
    }
}

// different (but identical) code path for handling detail views
result := VerifyPostDetailView(userId, postId)
if result.IsValid {
    detailAnalytics.Totals += 1
    if result.IsUnique {
        detailAnalytics.Uniques += 1
    }
}

// as we add more modes, we duplicate the code more and more

Visualizing

To draw a chart, what do we need? We need data points on a scale that makes sense for the chart's dimensions.

If we're drawing data from the last 7 days and the chart has some width in it, it might make sense to take data grouped by the hour: every dot on the chart represents the value of the views counter at that point in time, one dot for the views on 2024-05-26 14:00, the next dot is the value of the views counter on 2024-05-26 15:00, and so on.

If we are viewing the chart over the range of a year, it doesn't really make sense to use the granularity of an hour. A day would make more sense: what was the views counter set to at 2024-05-26 0:00

If we imagine viewing the chart with the range of 10 years, the granulairty of a single week might be enough.

So let's go with three levels of granularity: hour, day, week.

That should satisfy most conditions reasonably well.

What if you want to see analytics in a finer detail? We could keep 10 minute granularity view counts only for the 72 hours, for example. Otherwise it'll just waste too much data.

The expected usage pattern is the user wants to visualize the exposure for some post he made over time. In the same chart, we want to show the impressions, the reach, and the engagement numbers over a period of time. The granularity will be decided by the time range. I'm not worried too much right now about who will decide the granularity or how.

The query parameters will probably include the post id, the time range, the metric (the view mode from above), and the granularity. The output is a list of pairs, each pair is a timestamp and the view counts for that timestamp.

The timestamp in the output can be chosen either to represent the beginning or the end of the "time window" specified by the granularity. For example, if the granulairty is a day, and the timestamp is "2024-05-26" and the views is 550, does that mean the post had 550 at the start of the day or the end of the day? Does the timestamp represent the start or the end of the time period?

It doesn't matter which convention we pick, as long as we remain consistent and stick to it throughout the entire codebase.

So what should we pick? I think we should pick the choice that simplifies the process of storing and querying.

How do we "store" the data? Everytime we register a new view event, we have to know to which time bucket it belongs to, for each granularity we care about, and update the count for that granularity.

It's easier to define the granularity if we can "truncate" the timestamp to find the appropriate time value.

For example, let's say the time is 2024-05-26 14:12

Hour granularity: 2024-05-26 14:00

Day Granularity: 2024-05-26 00:00

What about the week granularity? I think we can just use the Go standard library time trimming function.

package main

import (
	"fmt"
	"time"
)

func main() {
	t := time.Date(2024, 5, 26, 14, 12, 0, 0, time.UTC)
	fmt.Println(t)
	fmt.Println(t.Truncate(time.Hour))
	fmt.Println(t.Truncate(time.Hour * 24))
	fmt.Println(t.Truncate(time.Hour * 24 * 7))
}

Outputs:

2024-05-26 14:12:00 +0000 UTC
2024-05-26 14:00:00 +0000 UTC
2024-05-26 00:00:00 +0000 UTC
2024-05-20 00:00:00 +0000 UTC

For the week, it seems to truncate it to a Monday. It probably has to do with the underlying representation.

We can try in another language. Let's say, Javascript.

The time truncate works by doing (time - time % duration). It should be pretty easy to replicate in Javascript using just unix timestamps (JS timestamps are to millisecond accuracy).

let t = Date.parse("2024-05-26T14:12") // 1716700320000

let s = 1000
let m = s * 60
let h = m * 60
let d = h * 24
let w = d * 7

// truncate to hour
new Date(t - t % h).toJSON() // 2024-05-26T05:00:00.000Z

// truncate to day
new Date(t - t % d).toJSON() // 2024-05-26T00:00:00.000Z

// truncate to week
new Date(t - t % w).toJSON() // 2024-05-23T00:00:00.000Z

Notice the week start point is now different. Go does not use unix timestamps internally, so that's probably why.

For general consistency it's probably better to stick to unix timestamps.

The answer is clear now: the timestamp represents the start of the time window for this bucket in this granularity.

Now, it should be very clear how to update counts:

Everytime we encounter a view, after updating the totals, we also update the totals in the current bucket for each time granularity

Remember the above line where we did:

analytics.Totals[params.Mode] += 1

This means we already know the totals, so all we have to do is store in the specific time bucket for each granularity.

Note however that this process does not guarantee that we will have entries for all the time buckets.

To give an example, let's say there are the following entries in our database. Assume they are all for the same post, for the same granularity (hour), for the same metric (total impressions).

 Timestamp           Value
-------------------|-------------------
 2024-05-26 13:00  | 2454
 2024-05-26 14:00  | 2465
 2024-05-26 17:00  | 2470
 2024-05-26 18:00  | 2493
 2024-05-26 19:00  | 2509
 2024-05-26 20:00  | 2538
 2024-05-26 21:00  | 2552
-------------------|-------------------

Notice there are no entries for hours 15:00 and 16:00. Why does this happen? Because the process we perform is to set the total values in the bucket for the current time window. If no event occurs during the specific time window, there will be no entry.

How do we deal with this?

Well, if the client queries for views from, say 15:00 to 20:00, we have nothing to show for 15:00, so we have to find the last entry before 15:00 and use its value. In this case, the entry is the 14:00 entry, and the view count at the end of that entry is the view count at the end of 15:00 (aka the view count at 16:00). How do you get the last entry directly before the 15:00 entry depends on your database engine. If it's a low level storage engine that can iterate on entries by order with a cursor, you just move the cursor back one step at the beginning.

With that, we can now describe the data model.

Note: all of the above was kind of a thinking process. I'm not necessarily going to use everything as-is. Don't be too surprised if the next section uses slightly different names or even structures than what I described above. I tried to streamline it a little bit more.

Data Model

As always, no actual code has been written yet, so think of this as a starting point to get you started, not as the final destination to arrive to.

type Metric int
const (
    SummaryViews Metric = 0
    DetailViews  Metric = 1
    Engagements  Metric = 2

    MaxMetric    Metric = 3
)

type Granularity int
const (
    AllTime Granularity = 0
    Weekly  Granularity = 1
    Daily   Granularity = 2
    Hourly  Granularity = 3

    MaxGranularity Granularity = 4
)

type UserPostKey struct {
    UserId int
    PostId int
    Metric Metric
}

type UserRepeatTracking struct {
    Timestamp timestamp
    Count int
}

type PostMetricsKey struct {
    PostId int
    Granularity Granularity
    Metric Metric
    Timestamp timestamp
}

bucket UserViews(UserPostKey, UserRepeatTracking)
bucket PostMetricsTotal(PostMetricsKey, int)
bucket PostMetricsUnique(PostMetricsKey, int)

Processes

Handling an analytic event

Some user interacted in some what with a post:

• View

• Expand

• Engagement (like, comment, etc)

The input is: user id, post id, metric type.

In response to this event, we want to record some analytics.

• Check the values in the `UserViews` bucket that corresponds to the given input. If a value already exists, check the time stamp. If the activity is within a certain threshold (e.g. 10 minutes), increase the counter and don't proceed with any anlytics recording. If the timestamp is older than the threshold, or there's no entry, reset the entry with the current timestamp and a counter value of 1. It's important to note whether there was an entry at all or not. If there was no entry, this counts as a new unique event. If not, it's just a regular event but not unique

  • For future consideration: If the value of the counter is way too high (e.g. 1000 events) it might be a suspicious activity. (For now we don't specify how handle it)

• Compute the appropriate timestamps for all granularities:

  • AllTime: The zero time

  • Weekly: Now, truncated by a 7 day duration

  • Daily: Now, truncated by a day duration

  • Hourly: Now, truncated by an hourly duration

• If we are to add the event to the total counter:

  • Fetch the counter from the PostMetricsTotal buckets that corresponds to the AllTime granularity.

  • Increment it by one

  • Store the number in the PostMetricsTotal bucket for all the given granularities (using the timestamps we computed)

  • If we are to consider this event a unique event, do the same thing for the PostMetricsUnique bucket

Displaying metrics for a post

Given a specific post, the user wants to see how many total views it got, and how many unique impressions it's got. He also wants to see the total and unique values for the detail expands and engagements.

The input is the post id.

• Construct a PostMetricsKey query key that has the post id set to the input, while the granularity and timestamps are kept to the zero value.

• Loop over the metrics values (0..<MaxMetric), and for each one, copy the key, set the Metric to the value from the loop, and load the value from both the PostMetricsTotal and PostMetricsUnique.

  • Alternatively, instead of looping explicitly, utilize the appropriate functionality in your database system that loads all entries where for the given post it at granularity 0 and timestamp 0. In other words, allow the database system to naturally load all the metrics without you explicitly looping over the metric values.

• Collect the result into the appropriate structure for a response. Something like this could work:

    type PostMetrics struct {
         Uniques [MaxMetric]int
         Totals  [MaxMetric]int
    }

Displaying post metrics in a chart

To show the various engagement metrics change (grow) over time.

Input: Post ID, time range (start, end), granularity, the list of metrics.

• For each listed matreic, construct the appropriate PostMetricsKey, using the given post id and granularity and start time. Iteratively all the entries for this metric that fall within the given time range.

  • If there's a significant gap between the timestamp in the first entry that matches the query and the requested range's start time, find the one entry immediately before the requested start time.

• Return the data to the client to be displayed using the appropriate UI component (out of scope for this document).

Keep in mind that this kind of document only serves as a starting point, not as a final design.

As you start to implement this in actual code, it's almost certain that you would need to make some changes and adjustments.

If this document helps you get started and get along far enough to a working solution, then it has served its purpose.

Background

We are running a website and we want to provide some products and services that require payment to access.

At first we're not worrying about the actual payment process. Instead, we'll focus first on managing the data that represents user's license to access the products.

I'll introduce interfacing with some payment provider in the later part of this document.

One thing I want to bring to your attention is that we will not be using an "abstraction" for the payment processor service.

Instead we'll just be managing our own data, and at key points interact with the external service to verify conditions and update our data.

Requirements

• Allow multiple separate products

• Allow access by:

  • subscription

  • usage credits

  • a combination thereof

• Allow trial access

• Allow lifetime access

• Allow multiple tiers of access (basic, pro, premium, etc)

  • With different credit point cost for features, depending on the tier

Thinking about the data model

With these requirements in mind, it's tempting to think about "modes" of access, and data to describe product "packages", etc. But I think we can build a simple core model that supports all the requirements.

Instead of thinking about disjoint features, I want to think about their union.

It should be possible for the product to be subscription based but also limited by usage. The subscription gives you base level access to the features, but if you use too much you'll need to make additional payments to recharge your credits.

It took me a few iterations to arrive at this design, so don't be surprised if it appears counter-intuitive.

type UserLicense struct {
    Id int
    UserId int
    ProductId int

    RequiresCredits bool
    Credits int

    HasExpiration bool
    Expiration timestamp

    AutoRenews bool
    RenewPeriod Period
}

type Period struct {
    Years int
    Months int
    Days int
}

type UserProductKey struct {
    UserId int
    ProductId int
}

bucket UserLicenses(Id int, UserLicense)
bucket LicenseLookup(UserProductKey, Id int) // user-product => license-id

This design models the access right of a certain user to a certain product.

It does not describe a "package".

To lookup what access a user has to a product, we use the LicenseLookup bucket to find the license id and then load the license from the UserLicenses bucket.

In principle, it's possible for different users to have different access modes to the same product: one user has life-time access, another user has a subscription based access, another user has credit-based access, etc.

The flags RequiresCredits and HasExpiration determine the access mode.

If RequiresCredit is set to true, then credit charges may apply. If it's false, then credit charges will never apply.

If HasExpiration is set, then the user cannot access the product past the expiration date.

If AutoRenew is set, then we expect the expiration date to be extended by RenewalPeriod after the expiration date has passed. This means when checking for expiration, we should allow a grace period for the payment to go through and for our system to update its records. (I'm not sure if RenewalPeriod belongs here; this is a point to revisit when I go to actually implement this).

If RequiresCredit and HasExpiration are both false, then the user has unlimited lifetime access to the product. Notice how we don't have an explicit flag for it. It just emerges naturally.

Similarly, trial access can be implemented by granting the user access to the product with a short expiration period, and we can apply some restrictions such as making sure that they have never had access to the product in the past.

If they did have access to the product in the past but it expired, the entry would be still be in the UserLicenses bucket, and it would be still be found using the LicenseLookup bucket.

Product ID

The product id has no inherent meaning. It's just a numeric value that we use across the code and the database to denote a specific product or service.

The only thing that matters is that this never changes, ever, and that two separate products never ever have the same id.

Access Tiers

The data model itself has no concept of tiers, but tiers can simply be encoded as different product ids.

const TierA = 10
const TierB = 11
const TierC = 12

Verifying access right to a service can be done by verifying all relevant tiers, and using the first one that matches.

Credit Points

Credit points are currency-like values for use within our system. They are integers only and have no fractions.

Using a Product

Verifying and Granting Access

type ProductAccessCost struct {
    ProductId int
    CreditCost int
}

type ProductAccessResult struct {
    Allowed bool
    BalanceBefore int
    BalanceAfter int
}

func VerifyProductAccess(tx Tx, UserId int, cost []ProductAccessCost) ProductAccessResult
func ChargeProductAccess(tx Tx, UserId int, cost []ProductAccessCost) ProductAccessResult

Notice that VerifyProductAccess and ChargeProductAccess take a list of ProductAccessCost. This is to support the concept of "Access Tiers" as explained above.

For example, if you are on a basic tier, this feature costs 20 points, but if you are on a pro tier, it costs 5 points.

The result would have the Allowed flag set to true if the user's access has not expired, and if he can fulfill the credit costs (that is, that his credit points >= the credit cost, but this restriction only applies if his access has the RequiresCredits flag).

VerifyProductAccess would just check the user can access one of the product ids listed, and it would report how his credits would change if he were to use this 'set'.

ChargeProductAccess would actually charge the credit points against the user. It can potentially do other things we haven't covered here, such as logging the user's access to the product.

Interfacing with an external payment service

We can use an external payment provider like Paypal, Stripe, LemonSqueezy, Paddle, GumRoad, or some other service.

We need two essential features from such a service:

• A way to verify some proof of payment

• A way to verify subscription status

Both boil down of course to verifying payment at different points in time.

LemonSqueezy

To simplify the interface with LemonSqueezy, I'll base the design on the assumption that they issue a LicenseKey per product.

LemonSqueezy License Key Validation API

LemonSqueezy has its own version of productId, and it also has a variantId to handle different tiers or access modes.

Mapping between our internal product ids and LemonSqueezy's product-and-variant-id can be done in a variety of ways, but in my opinion we can just use a switch statement or a series of if-else blocks.

The variantId basically determines the payment method (free, onetime, recurring) and the assumed access mode. For example, if you offer a trial, a basic membership, and a premium membership, they would each be considered a variant of the same product.

type LicenseSource struct {
    Id int // same as UserLicense.Id
    Source string // e.g. "LemonSqueezy" or "Stripe"
}

type LemonSqueezyOrder struct {
    Id int // same as UserLicense.Id
    OrderItemId int
    LicenseKey string // if applicable
    SubscriptionId int // if applicable
}

bucket LicenseSources(Id int, LicenseSource)
bucket LemonSqueezyOrders(Id int, LemonSqueezyOrder)
index ActiveSubscriptions(Expiration date, LicenseId)

License Key Verification

We provide a UI for the user to enter his License Key that he acquired via LemonSqueezy.

The user sends us his key, and we verify it using the validate API (linked above).

POST https://api.lemonsqueezy.com/v1/licenses/validate license_key=1234-1234-abcd-abcd

If the license is valid, the response will have have valid: true and it will also contain some meta data. We care about the following:

• store_id

• product_id

• variant_id

• order_item_id

• customer_email

If the store_id does not match our store, then the key is invalid from our point of view.

If the customer_email does not match the logged in user, we reject this key and send an appropriate error message to the user to let them that they need to login with the same email address.

If store_id and customer_email are valid, we create (or update) the user license information to grant them access to the product based on the product_id and variant_id. For the purposes of this document, I'm assuming this will be done manually via code. For example:

switch variantId {
    case Lemon_ProductA_TierB:
        userAccess.ProductId = ProductA_TierB
        userAccess.HasExpiration = true
        userAccess.ExpirationDate = time.Now().AddDate(0, 1, 0)
        userAccess.AutoRenew = true
        userAccess.RenewPeriod.Months = 1
    case .....:
        .... etc
}

We use the order_item_id to find the subscription object

GET https://api.lemonsqueezy.com/v1/subscriptions?filter[order_item_id]=123456

Now, we update the LicenseSources and LemonSqueezyOrders buckets to contain the relevant information.

Renew Subscription

This requires a daily process. The simplest way is have a goroutine (or thread, or equivalent) scheduled to run at a set time everyday, e.g. 1am. Have it query all licenses that have expired but expected to auto renew.

The query uses the ActiveSubscriptions index. Since keys are sorted from lowest, the first key is the oldest expired subscription. We start iterating from the first key and stop as soon as we find a key that is in the future.

In a SQL database, this could roughly look something like:

select license_id form active_subscriptions
where expiration_date < today + 1
order by expiration_date

For each item, verify the status of the subscription with the merchant service.

It would be nice if LemonSqueezy provided us an API that takes a list of subscription ids and reports the status for all of them at once, but for now I'm afraid we'll have to do it one at a time.

https://docs.lemonsqueezy.com/api/subscriptions

Check the status field. If it's "active", then we can assume the renewal payment was successful, and we can update the expiration date on our database (including updating the corresponding entry on the ActiveSubscriptions index).

If it's "past_due", we check the grace period. If we're past the grace period, we cancel the subscription from our end and send a request to LemonSqueezy to cancel it as well:

https://docs.lemonsqueezy.com/api/subscriptions#cancel-a-subscription

Similarly, if it's any of "unpaid", "cancelled", or "expired", then we consider it cancelled, and update our data accordingly.

Note: when we cancel a subscription on our end, we also remove its entry from the ActiveSubscriptions index.

All behaviors we associate with computer applications emerge from how data is processed over time.

The only thing computers can do is data processing. They can't do anything else.

It makes no sense to talk about "business logic" as if it's a separate thing from the data processing.

The purpose of reading code is to deduce how data is being processed. When code is written in such a way to purposely obfuscate what the data is and how it's processed, we say the code is hard to read.

We advocate straight forward programming: writing code in the most simple and direct way to perform its duty:

• Process the data correctly

  This is the essence of programming. If you can't do this you're not a programmer.

• Lend itself to be easy to understand

  If you can't deduce the data model and the operations being performed on it, then the code is not easy to understand. (skill issues non-withstanding)

The purpose of software architecture is to model data processing such that:

• Building blocks allow a wide range of desired behaviors

  You get the desired behavior by composing building blocks together in different ways. You can experiment until you find what works well

• Code paths can be simplified and collapsed

  The data model is rich enough to describe a wide range of behaviors that can be implemented by very few code paths

We oppose architectures that:

• Hide the data and obfuscates its processing
  Making systems opaque and difficult to reason about is not a virtue

• Overfit to the current iteration of the requirements

  A good architecture is flexible, allowing exploration of the design space

• Emphasize abstract processes that are not defined in terms of data
  Good abstractions are defined in terms of data.
  Bad abstractions are defined in terms of abstract objects.

We Encourage proven techniques to untangle code complexities

• Separate data processing into stages
  Each stage prepares the data to feed to the next stage.

• Semantic Compression
  Extract common code patterns into reusable functions that operate on data

• Collapsing code paths
  When the same code path can process many cases without modification

We de-emphasize irrelevant concerns

• How to organize code in files and folders?

• How big should files be?

• How long should functions be?

• How to format code?

It doesn’t matter. Use your taste. Find what works for you.

There will not be much in the way of actual code, but there should be enough details for most programmers to be able to implement the feature on their own as an exercise, even if they are not experienced enough to come up with the design on their own.

If you are experienced, everything here probably seems trivial. You might even wonder what's the point.

There's a phenomenon where experienced people don't realize the knowledge and experience they take for granted is actually not available to most people. I've noticed it when working with other people: things that seem obvious to me are puzzling to others. Likewise, some things are puzzling to me, but appear obvious to those proficient at them.

Authentication is not a particularly difficult problem. If anything, it's among the easiest problems you'll ever encounter. But it feels like a good starting point for the series, because almost any problem you want you solve will require you to have users and accounts first.

Background:

What does it mean for a user to register and login? It means they reserve a handle (e.g. a username, email address) to identify themselves, and they can use it to interact with the service as themselves.

This document will be limited to handling the following operations:

• Registration

• Sessions

  • Login

  • Logout

• Email Verification

• Password Reset

Almost all operations require the use of "tokens".

Outline of Operations:

Registration

The user reserves an identifier and specifies a password. Potentially providing other information, such as an email address and a phone number.

Email Verification

The user tells us back a secret token or short code that we sent to his email (or phone number).

Login

The user provides his username and password to prove that he is the one sitting on the machine, and as a result, a session is created.

Session

The website remembers that the user has logged in, without requiring him to re-enter his password all the time to verify himself.

The server issues a secret token to associate with the user id, the client keeps this token and sends it along with every request.

Password reset

If the user forgets his password, a fallback mechanism is provided via email.

Usually done by producing a password-reset token and sending an email with a special URL that, when clicked, opens a page that allows resetting the form.

Logout

The client forgets the session token, and optionally requests the server to expire the token

Important Considerations

The Session

The session token counts as a form of authentication, but it's weaker than the password. It's possible that some time after the user has logged in, someone else is now sitting at the machine.

• Some operations should require the user to re-enter his password.

  • Viewing sensitive information

  • Attempting to change the password

  • Making a payment, specially if a large sum

• Tokens need to expire after a while to minimize risks.

Building Blocks

Passwords

Passwords are to never be stored in plain text as-is. They must be encrypted using a "one way" encryption scheme, such that:

• It's not practically possible to deduce the password given the encrypted form

• It's very easy to validate a specific password against a specific hash

The consensus seems to be on the "bcrypt" algorithm.

Here's the Go implementation: https://pkg.go.dev/golang.org/x/crypto/bcrypt

Username and User Id:

The system should be designed to allow the user to change their username without breaking anything else in the system.

Now, whether you want to allow the user to change his identifier is a separate question.

The implication is that the actual user identification used throughout the system is a numerical id, and the user-facing handler is merely a way to get the actual user id.

Generating Tokens

Tokens are easy to generate: fill a 16 byte buffer with random data from the OS provided high entropy random data source (/dev/random on linux).

Then encode the buffer to base64 (or hex, or any other text encoding)

Verifying and using tokens

Some operations require sending the user a URL with a token.

When the page is first loaded, the server verifies the token: that it exists, has not expired, and is useable for the given purpose (password reset, etc).

When the form on the page is submitted, the token is submitted with it as well, and the server again verifies the token.

If the token is one-time use, the server expires the token after performing the requested operation.

Token expiration and deletion

Tokens are only useful for specific operations. There's no point in keeping old tokens around forever. Once a token is expired, it can either be deleted immediately, or left alone for some other process to delete old tokens.

For example, you can run a daily process to delete any token that has expired more than 10 days ago.

Sessions

A session is merely a token that identifies a user.

The client and the server agree on a special identifier name to use in request headers (or cookies) to identify the session token.

Data Design

To support all the process needed for authentication, we will define types, buckets, and indices.

Refer to the Data Design Spec for a brief explanation of the notation used.

type Account struct {
	Id int // auto generated
	Username string
	Email string
	EmailVerified bool
	PasswordHash string
	Created time.Time
}

type AccountToken struct {
	Token string // unique (id)
	Type string // session, email_verification, etc
	AccountId int
	Created timestamp
	Expires timestamp // or can be defined as a duration
}

bucket Accounts(Id int, Account)
bucket Usernames(Username string, AccountId int)
bucket Emails(Email string, AccountId int)
bucket Tokens(Token string, AccountToken) // string to object
index  AccountTokens(AccountId int, Token string) // iterates tokens associated with account

// Valid session token types
const TokenTypeSession = "session"
const TokenTypePasswordReset = "password_reset"
const TokenTypeEmailVerification = "email_verification"

Operations

Her's a basic description of operations in plain language.

New Account Registration

• User provides desired username, email, and password

• Verify that no existing account uses the username

  • By checking the Usernames index does not have an entry for the given username (and the same for the email, if you want)

• Hash the password using bcrypt

• Generate a new Id on the AccountsBucket

  • Each bucket has an auto incrementing sequence

• Generate a verification token for the provided email, and send an email (as a background task) containing a verification link

• Store the Account in the Accounts bucket.

• Create a session token and send it to the client.

Verify Email:

• A verification URL with a token is already sent to the email address

• When the user opens the URL, the token will be sent to the server

• The server looks up the token, makes sure it has not expired, and that it is

in fact an email verification token

• The email associated with the token is now considered verified. Update the EmailVerified flag on the account.

• This token is one time use, so expire it and schedule it for deletion

Change Email:

When the user changes their email address, the following steps must be taken:

• Set the EmailVerified flag on the account to false

• Use the AccountTokens index to find any "pending" email verification token associated with the account and expire it (or delete it). This is crucial because the account is only associated with the account id rather than the email.

• Create a new verification token and send it (as a URL) to the provided email address. (same as what we did during registration)

Login:

• User enters his username and password

• The password is sent to the server in plain text

• Use the Usernames bucket to find the account id

• Load the account and compare the password against the PasswordHash on the account, using the bcrypt algorithm.

  • Note: an empty password hash does not match any password!

• If account exists and password matches, create a session token and return it to the client along with basic user information

• If the client is SPA based, store the session token in localStorage and add it to the headers in every request using a special header name that both the client and the server agree on.

Password Reset:

If the user has forgotten their password, they are allowed to request a password reset.

• The user enters their username or their email, which ever they happen to remember

• The server finds the account id associated with the account, creates a password reset token, and sends it to the email associated with the account.

• When the user clicks the URL, he is taken to a form for entering a new password

• The new password and the token are sent to the server

• The server verifies the token: that it has not expired, and that it is of the appropriate type

• The password is hashed, and a session token is generated and returned to the client.

Password Change

If the user wants to change his password while logged in, we require him to enter his original password, because as mentioned above, the session is a weaker form of authentication than the password.

• User enters existing password and new password

• Existing password is validated against hashed password stored on the account

• New password is hashed, and its hash replaces the old password hash on the account

Session Invalidation

This is an admin function: if there's suspicion that a user account has been hacked, it should be possible for an admin to invalidate all current sessions associated with the user account.

• Use the AccountTokens index to find all tokens associated with the account

• If the token is not expired, expire it immediately

  • No need to check for token type. We actually should expire all tokens of all types, because each one can be used to gain illegitimate access to the account.

Session Recognition

You want a certain class of request handlers on the server side to require a valid session before proceeding with handling the request.

The exact requirements vary from one handler to another, but at a minimum, we want to return a "401 unauthorized" if the request does not have a session.

• Grab the session token from the specified location: either a special http header, a special token name, or both

• Find the token object, make sure its type is "session", and that is is not expired.

• Find the account associated with it and make sure it exists.

• If any of the above is not fulfilled, return a 401 response and stop end the handler

• If all conditions are satisfied, keep around the session and account information so the rest of the request handler has access to them.

Conclusion

I have mixed feelings about writing this document. On one hand, I think this kind of data modelling and informal explanation of basic operations is useful in general. On the other hand, I'm not sure of:

• How useful is it for this particular "feature":

  • To me it seems very basic. A few months ago I would have assumed that every programmer just has enough skills to come up with this on their own.

• How detailed should I go:

  • I tried to keep it high level and abstract. I made almost no assumptions about programming language, framework, or database.

  • I kept it focused on the server side data model

• How actionable this information is:

  • Would a team of beginners, having no prior experience implementing this feature, be able to use this document to actually create a fully working implementation?

• Noise to signal ratio:

  • How much of the document is actually useful? How much of it is just blabber that makes it difficult to access the useful information hidden in it? (assuming there's any at all)

Data as structs

The core convention is describing data in terms of structs and slices, in the syntax of the Go programming language.

This will be in contrast to the more common convention of describing data in terms of relational tables.

This choice is deliberate. You can of course use relational tables to store the data. You can also use "json" documents to store the data.

The basic data types are going to be: int, float, string, boolean.

In addition, I will often refer to 'date' and 'timestamp' as if they were built-in types.

A 'date' is usually a string in the format 'yyyy-mm-dd'. The idea is that it has no time component, and no timezone. It's a little bit abstract in that sense.

A timestamp is probably stored as a 64bit millisecond with respect to some epoch (usually the unix epoch), along with a timezone (either as an identifier, or as a difference from utc). The details might vary from system to system, but the core assumption will be that it can be converted to some specific local time for the purpose of input/output on the user's machine.

Example:

type StoreItem struct {
	Id int
	Slug string // url slug; one per store item
	Title string
	Subtitle string
	Description string
	BrandId int // reference to some brand
	Images []string // urls relative to the user images endpoint
	UnitPrice int
	UnitCurrency string
}

type Brand struct {
	Id int
	Name string
	Logo string // image url
	Header string // image url
	Homepage string // company url
	CatchPhrase string
	Description string
}

Storage Bucket

Defining a type does not mean that it will be stored as-is in the database. Things are only stored when they are put in a bucket. A bucket uses a lookup key to retrieve a type of object

Examples:

bucket StoreItems(Id int, StoreItem)
bucket Brands(Id int, Brand)
bucket StoreItemSlugs(Slug string, Id int)

In a relational database, you may choose to implement a bucket in terms of a table, and the lookup key may be understood as the primary key for the table.

Lookup Index

A lookup index is also a mapping, but instead of being one to one, it's basically many to many, mapping a lookup term to a list of targets.

Examples:

index BrandItems(BrandId int, StoreItemId int)
index CurrencyItems(Currency string, StoreItemId int)

The BrandItems index allows us to iterate over all the store items that are from a certain brand.

The CurrencyItems index allows us to iterate over all the store items that define their price in a certain currency.

In a relational database, the index may be assumed to be some kind of optimization, since SQL allows you to use arbitrary queries with arbitrary lookup conditions.

However, when modelling data, it's important to be explicit about these lookup indices.

When using the lookup index, you provide a key and iterate on matches.

When setting the lookup index, you provide a match and a list of keys. It's the opposite.

Think of the index on the back of a book. If you edit a page and want to update the index, the operation requires that you provide the page number and a list of important terms that appear on the page. The system will then find each term on the index, and update it by adding the given page number. It will also need to find keys on the index that used to point to the given page number but no longer need to, and remove the page number from their matches.

The amount of code will be minimal. The idea is that any programmer, even if they are not very experienced, can use this design document to implement the feature on their own.

The first item I’m thinking of is basic authentication: account creation, email verification, password encryption, password reset, login, validating sessions, etc.

I’ve noticed a lot of people don’t know how to implement it and instead rely on third party services, which frankly I think is pathetic. It’s not that hard, and if you can’t bother to do it properly, I have to question whether you can get anything at all done.

The next item I’m think of is basic integration with LemonSqueezy: managing product ids, validating & activating license keys, using the license key as an alternative form of email verification, supporting multiple modes: subscription, lifetime access, credit based access, and trial period access.

If you’re interested in these kinds of topics, stay tuned.

    input_field("label", &object.field)

We can take inspiration from this idea to define input components in (p)react without using callbacks.

We want to write this:

    <Input ref={ref(object, 'field')} />

instead of this:

    <input value={object.field}
        onChange={e => object.field = e.target.value}
    />

Here's why:

• Zero callbacks

• Code volume smaller

• No duplicating object.field

Contemporary react based apps overuse closures. We should reduce the amount of closures as much as we can. Assume by default that closures come with performance penalties and find other default ways of achieving tasks.

For implementing this idea, we still need a callback, but it will be a single regular function, not a dynamic closure at every usage site.

We need a way to pass the ref to the input element without closures. The easy way to do this is to encode the reference as a string into the DOM attributes of the input element.

Object ID

We can easily come up with numeric ids for object references, but we need them to be stable; they should not change between render cycles.

Javascript does not give us the "address" of object references, but internally they are addresses, and the builtin Map object uses those addresses.

Here's a pair of functions to get an id for an object and also get an object back from the id.

let idmap = new Map<unknown, number>()
let objmap = new Map<number, unknown>()

export function objectId(obj: unknown): number {
    let v = idmap.get(obj)
    if (v === undefined) {
        v = nextObjectId()
        idmap.set(obj, v)
        objmap.set(v, obj)
    }
    return v
}

export function objectById<T = any>(id: number): T | null {
    return objmap.get(id) as T ?? null;
}

Leaking memory

Whenever we call objectId(...) on something, we hold a strong reference to it in the idmap and objmap. This means we should never call it on "temporary" local objects.

It also means we need a function to clear the idmap and objmap that we can call at the appropriate time. If the application we're developing is an SPA (single page application) with client side routing, we can call this function when the route changes.

export function clear() {
    idmap.clear()
    objmap.clear()
}

One possible alternative is to use WeakMap instead of map, but then it can't work with strings, which we want to be able to work with.

Field Ref

For the field ref, we can define a type with two keys: obj and key.

This is the closest thing to an arbitrary C pointer in Javascript (in terms of what we can do with it): we can use it to read and write arbitrary data without having to know anything about where that data is coming from.

export function ref<T, K extends keyof T>(obj: T, key: K): Ref<T[K]> {
    return { obj, key } as Ref<T[K]>
}

export function get<T>(r: Ref<T>): T {
    return r.obj[r.key]
}

export function set<T>(r: Ref<T>, value: T) {
    r.obj[r.key] = value
}

The definition of Ref<T> is constructed such that T is the type stored in obj.key.

Making this type in Typescript requires an ugly hack:

declare const _ref_type: unique symbol;
export type Ref<T = any> = RawRef & {
    [_ref_type]: T
}

export type RawRef<T = any> = {
    obj: T;
    key: keyof T;
}

When we call ref(object, "field"), the Typescript checker will:

• Ensure that object.field is valid

• Deduce the type of the returned ref

Example code:

type Person = {
    name: string;
    age: number;
}

let p: Person = { name: "Hello", age: 20 }

let x = ref(p, "name")
let y = ref(p, "age")

The Typescript checker automatically deduces that x has type Ref<string> and that y has type Ref<number>.

If you type a bad field name like this, it will report an error:

let z = ref(p, "inv")

Ref String Encoding

When we call ref(....) the object we get back will be a temporary local object. It will not have a stable id, so we can't just use objectId(...) on it; we'd get something different every time, and we'd leak memory, as discussed above.

Instead, we can get the id of the object and the field name. String literals have stable references, so we can call objectId on them.

function encodeRefAttr(ref: refs.Ref): string {
    let objectId = refs.objectId(ref.obj);
    let fieldId = refs.objectId(ref.key);
    return `${objectId}:${fieldId}`;
}

function decodeRefAttr(sref: string): refs.Ref {
    let [objectIdStr, fieldIdStr] = sref.split(":");
    return refs.ref(
        refs.objectById(parseInt(objectIdStr)),
        refs.objectById(parseInt(fieldIdStr)),
    );
}

Note the absence of any "validation" code in the decoder. We just assume the references are valid. If someone puts garbage in, they just get garbage out, similar to trying to pass garbage to a C function that expects a pointer.

Now, here's another helper to read the attribute of a specific DOM element.

function getAttr(el: HTMLElement | EventTarget | null, attr: string): string {
    if (el instanceof HTMLElement) {
        return el.getAttribute(attr) ?? "";
    } else {
        return "";
    }
}

It does not seem like much but it removes edge cases we don't care about, and we can also pass event.target to it without Typescript complaining.

The input handler

Now we have all the building blocks to create our Input component and define the input handler callback:

const INPUT_REF_ATTR = "input-ref";

function onInput(event: Event) {
    let target = event.target as HTMLInputElement;
    let ref = decodeRefAttr(getAttr(target, INPUT_REF_ATTR));
    let refValue = refs.get(ref);
    let value: any = target.value;
    refs.set(ref, value);
    core.scheduleRedraw();
}

export function inputAttrs(ref?: refs.Ref): any {
    if (!ref) {
        return {};
    }
    return {
        [INPUT_REF_ATTR]: encodeRefAttr(ref),
        value: refs.get(ref),
        onInput: onInput,
    };
}

export function Input(props: { ref: refs.Ref<string> }) {
    return <input {...inputAttrs(ref)} />
}

This is a "bare-bones" implementation, but I think it illustrates the idea.