Posts on Prskavec.Net

A Knowledge Format Is Not a Knowledge Base

ladislav@prskavec.net (Prskavec.Net) — Thu, 16 Jul 2026 09:00:00 +0200

Google’s Open Knowledge Format sounded relevant to me immediately. OKF v0.1 is a directory of Markdown files with YAML frontmatter, stored in Git and readable by people and agents. That is close to how I already use Logseq, Codex, and Claude Code with my own knowledge base.

After reading the specification, I do not think I should adopt it. OKF could be an export format one day, but it leaves out too much of what makes a knowledge base useful.

First, a correction: OKF is not another markup language. It is a small profile on top of Markdown and YAML. The required surface is deliberately thin: every concept is a file, its path is its identity, and its frontmatter must contain a type. Titles, descriptions, tags, resources, timestamps, indexes, logs, and citations are optional or recommended.

That simplicity is attractive. It also standardizes the least difficult part of a knowledge system.

Backlinks are part of the data model

OKF defines a link from concept A to concept B as a directed, untyped edge. The meaning, such as “depends on”, “references”, or “joins with”, stays in the surrounding prose. A consumer may scan the bundle and compute the reverse edge, but the format does not require it.

This distinction matters. In Logseq, a backlink is not decoration in a graph view. When I open B, I can see every page that mentions B, including connections I did not remember to encode on B itself. The forward link records what the author of A knew. The backlink shows how the rest of the corpus has accumulated around B.

Adding an explicit link from B back to A is not the same thing. It duplicates the edge and may misstate the relationship: “A references B” does not imply “B references A”. It also creates a synchronization problem when pages move or relationships change.

An OKF consumer can reconstruct backlinks. Google’s reference visualizer does this and presents them as “Cited by”. But another consumer can ignore reverse links and still conform to the specification. Both tools support the same format while exposing materially different knowledge bases.

Conformance does not imply shared meaning

The rest of the interoperability contract is similarly thin. type is the only required field, but type names are not registered. Consumers must tolerate unknown values. Links are untyped. Citations are recommended rather than required. Broken links are explicitly permitted.

These are sensible choices for a forgiving file exchange format. They also mean that two conformant bundles can agree on syntax while disagreeing on nearly all of the semantics. One producer’s Playbook may not resemble another’s. A consumer can parse both files without knowing how to compare, validate, or use them.

That is portability of text. It is not yet portability of knowledge.

We already have document formats

If the problem is how to store rich, readable technical documents in plain text, we already have mature options. reStructuredText has field lists, citations, hyperlink targets, directives, and extensible interpreted-text roles. AsciiDoc has document attributes, document-to-document cross-references, and includes.

Neither format gives every tool Logseq-style backlinks automatically. That is the point: backlinks are a corpus-level index and navigation contract, not an inline-markup feature. Putting YAML in front of Markdown does not solve that layer either.

Markdown plus YAML is a reasonable transport choice with modest parsing requirements. But choosing a transport is different from defining a knowledge architecture. If a team prefers AsciiDoc or reStructuredText, it should not need to convert its source documents just to participate in a knowledge exchange. A useful standard could define concepts, identifiers, relationships, provenance, and lifecycle independently, then provide mappings for several text formats.

Markdown is the least interesting part of my knowledge base

My setup works because it is selective. Journals are a raw inbox. External sources are retained locally when needed. Durable pages are synthesized rather than copied. Claims carry source notes with searchable anchors. A curation log records what was promoted, skipped, or left for later. Every page edit is reviewed against its sources instead of trusted because the prose sounds coherent.

None of that follows from the file extension.

A directory can be perfectly conformant with OKF and still be a bad knowledge base: duplicated notes, stale claims, missing provenance, disconnected pages, and generated summaries nobody has checked. An agent can parse the corpus, but parsing is not the same as trusting or navigating it.

For me, a stronger interchange model would need at least:

stable concept identifiers that survive file moves
standardized backlink or reverse-edge behavior
relationship types that consumers can understand without reading prose
claim-level provenance, not only an optional citations section
a way to distinguish raw capture, durable synthesis, and deprecated material
mappings for existing systems and formats instead of one mandatory source syntax

This does not all need to live inside every document. A manifest or generated graph index may be better than duplicating metadata across files. The important part is that the behavior belongs to the contract, so each consumer does not invent it again.

Where OKF could still fit

OKF may become useful as a lowest-common-denominator export. If an agent or catalog I want to use accepts OKF, a Logseq plugin could map page properties to YAML, convert wiki links to Markdown links, and generate a backlink index. My knowledge base would remain the source of truth.

I would not build that adapter before there is a consumer worth using. OKF is still a v0.1 draft, and changing a working knowledge base to satisfy a thin interchange profile would add maintenance without improving the knowledge.

The problem is real: agents need portable context, and knowledge should not be locked inside one vendor’s catalog. But a collection of readable files is only the substrate. A great knowledge base also needs relationships, provenance, selection, and maintenance. Another file profile does not provide those by itself.

Sources

Google Cloud, Introducing the Open Knowledge Format (2026-06-12)
Open Knowledge Format v0.1 draft specification
Open Knowledge Format reference implementation and visualizer
Docutils, reStructuredText Markup Specification
Docutils, reStructuredText Interpreted Text Roles
Asciidoctor, Document Attributes
Asciidoctor, Document to Document Cross References
Asciidoctor, Includes

How I Use Logseq as a Personal Knowledge Base

ladislav@prskavec.net (Prskavec.Net) — Sat, 27 Jun 2026 09:00:00 +0200

This setup was inspired by Andrej Karpathy’s LLM Wiki, but my version is deliberately local and boring: Logseq journals as the inbox, source files as the archive, and compact pages as the durable layer.

Google is making a similar argument with Open Knowledge Format: Markdown plus YAML front matter as a shareable knowledge format. I think the format is the easy part. Markdown, reStructuredText, AsciiDoc, or any plain text format with metadata can work. For company-wide use, the harder part is the documentation pipeline: how you ingest sources from different places, keep them fresh, and assign curators, human or LLM, to maintain the useful layer.

My Logseq graph is not where every thought becomes permanent. It is where raw material waits until I decide whether it deserves a second life.

The split in this repository is blunt:

journals/ is the raw inbox.
sources/ is where bulky external material lives.
pages/ is the durable knowledge layer.
log.md is the audit trail of curation work.

Day to day, that is not formal. I capture links, screenshots, meeting fragments, commands, papers, talks, and half-formed notes into Logseq journals. Later, I run a curation pass and promote only the material I expect to search for again.

The flow

Journals are the inbox

Daily journals are intentionally messy. They contain:

links I may or may not read later
meeting scraps
debugging notes
command fragments
screenshots
TODOs
short observations
notes for work projects

I do not try to make journals beautiful. If I had to polish every note at capture time, I would capture less.

The rule is that journals are source material, not the knowledge base. Most journal bullets should never be promoted. TODO-only items, meeting logistics, unexplained fragments, and one-off status updates usually stay where they are.

What gets promoted is the material with reuse value:

architecture decisions
operational patterns
incident/debugging lessons
recurring themes
commands with explanation
glossary terms
papers, talks, docs, and articles worth finding again

External sources need a local copy

Some source material is too large, too valuable, or too likely to disappear behind a login wall. The graph has a sources/ directory for that.

For conference talks, I fetch subtitles with yt-dlp and store .vtt files under sources/transcripts/<conference>-<year>/. Each talk gets a small metadata file with the URL, title, speakers, runtime, transcript path, and the page that summarizes it.

The summary goes into pages/. The transcript stays in sources/ so future search can still find an exact phrase, command, or speaker quote that did not make the summary.

X.com needs a different path because direct fetching is unreliable from this workspace. Long posts and threads are exported locally as Markdown into sources/x-com-posts/. The stable join key is the numeric status ID. When a journal has an X link, I grep the exports for that ID and cite the local file.

Attachments such as screenshots, PDFs, CSVs, and images live in assets/, using Logseq’s normal local asset links.

The result is boring in a useful way: a broken website or login wall does not erase the source trail.

Pages are the knowledge layer

The durable pages are compact Markdown notes. They are not dumps from journals. They answer the questions I expect a future reader, including a future AI agent, to ask:

What is this?
Why does it matter?
How does it work?
What are the gotchas?
Where did this come from?

The common shape is:

## Summary
## Key Ideas
## Operational Notes
## Gotchas
## Links
## Source Notes

Not every page needs every heading. The format is only there to force the useful work: make the page searchable, connect it to related pages, and show where the material came from.

A recent addition to that shape is claim-level footnotes. ## Links is a fine roster of further reading, and ## Source Notes records which journal a page grew from, but neither tells a reader which specific source backs a specific sentence. When a page states a number, a benchmark result, or a direct attribution, I now tie it to its source with a footnote at the point of the claim, so the reader lands on the primary source instead of guessing which of ten links in the list is the right one. Logseq renders standard Markdown footnotes: [^1] next to the claim, and [^1]: Title, date, URL collected at the bottom of the page. The curate-journals skill treats this as a rule now: any paper, post, or video a page relies on has to be reachable from the page itself, because the journal it came from usually holds only a bare link.

Logseq links are the connective tissue. A page about local models can point to [[Claude Code]], [[AI/Harness engineering]], or [[Observability/LLM observability]] without needing a separate taxonomy project. If a page name contains a slash, the file uses Logseq’s triple-lowbar encoding, so AI/Local models and RAG lives as pages/AI___Local models and RAG.md.

The curation pass

The curation workflow is deliberately mechanical:

The repo has a Claude skill for this: .claude/skills/curate-journals/SKILL.md. AGENTS.md says what the graph is and what good curation looks like; the skill turns that into the repeatable pass.

Find unprocessed journal changes from the last Curation hash: in log.md, plus any untracked journal files.
Read each journal in full and note the [[tag]] blocks.
Classify each bullet as durable, raw, or already promoted.
Map durable items to existing pages before creating new pages.
Investigate external links, transcripts, and local X.com exports.
Write synthesis into pages/, not a dump of journal bullets.
Cross-link related pages.
Append a new batch entry to log.md.
Update pages/contents.md if a new durable page was created.

The last step saves future work. log.md records which journals were processed, which pages changed, which external links were investigated, which videos or transcripts were fetched, what was skipped, and what backlog remains.

That makes curation resumable. A later pass can see why a link was ignored or whether a conference playlist was already fetched.

The git hash is the part that makes this work for real daily notes. A date scan only finds journal files newer than the last processed date. That misses the normal Logseq habit: I open an old day, add one new block, and save it. The curate-journals skill reads the previous Curation hash: and diffs journals/ against it, so a re-edited journals/2026_06_24.md shows up even if journals/2026_06_25.md was already processed.

The pass starts with roughly this:

git diff --name-only <stored-hash> HEAD -- journals/
git status --short journals/

The union is the work list: journal files changed since the last curation pass, plus new untracked journals. The curation log then records the next hash so the following pass has a new high-water mark.

The skill also keeps the pass conservative. It explicitly says to leave TODOs, meeting logistics, PR breadcrumbs, isolated IDs, and ungrounded fragments in the journal. That matters because the easiest way to ruin a personal knowledge base is to promote everything.

Agents make the graph more useful

This graph is written for humans, but I also expect coding agents to read it.

The repository has explicit instructions for Codex and Claude Code. They say, in effect: this is a private Logseq knowledge base, not an app repo; journals are the raw inbox; pages are the durable layer; do not copy raw fragments into pages; synthesize, cite sources, and keep sensitive details private.

For journal work, the agent loads curate-journals and follows the same checklist every time. That gives the agent a smaller job than “organize my notes”: find the new journal material, decide what has durable value, update the right page, and write down what happened.

That changes how I use AI tools. Instead of asking an agent to remember context from a chat, I ask it to inspect the repo:

search journals for a topic
find the existing page
read source notes
summarize a talk transcript
connect related pages
draft a post from the durable layer

The agent does not need everything pasted into the prompt. It can search the repository, read the source material, and leave behind better Markdown for the next run.

What I do not promote

The part that keeps the system usable is omission.

I do not move every note into pages/. Most raw material stays raw. That keeps the durable layer useful.

I leave behind:

tasks without reusable context
meeting logistics
temporary debugging noise
private operational details that do not need to be repeated
links that looked interesting but did not turn into a durable idea
source material that I cannot verify

The durable layer works because it is selective.

The result

I am not trying to build a perfect wiki. I want a working memory system with:

fast capture in journals
source retention for long-form material
compact durable pages
explicit source notes
a curation log
enough structure for Logseq, Codex, Claude Code, and future me to navigate it

The design choice is to treat personal knowledge as ingestion, not filing. Capture stays cheap because journals can stay messy. Curation happens later, when there is enough signal to write a page worth keeping.

That delay is the trick. It keeps the inbox easy and the knowledge layer clean enough to be useful months later.

Agent Harnesses: Why You Shouldn't Bet Your Company on Claude Code or Codex

ladislav@prskavec.net (Prskavec.Net) — Tue, 02 Jun 2026 09:00:00 +0100

In less than two years, coding agents went from novelty to daily driver. Claude Code and OpenAI Codex read your repo, run commands, edit files, and finish real work. I use them every day. You probably do too.

Using a tool is one thing. Standardizing your whole company on a single vendor’s closed agent is another. That decision is easy to make and very expensive to walk back, so it is worth thinking through before you make it.

What is an agent harness?

When people say “Claude Code” or “Codex,” they usually mean two things glued together:

The model, the LLM that does the reasoning (Claude, GPT, and so on).
The harness, everything around the model: the tool-calling loop, file editing, command execution, context management, permissions, prompts, and the UI.

The harness is what turns a chat model into an agent. It decides which tools exist, how context is assembled, when to ask for permission, and how errors get recovered. The model is a commodity you rent by the token. The harness is where your actual workflow lives.

And the harness is the valuable part. The vendor owns it.

The case against vendor lock-in

1. You couple your workflow to someone else’s roadmap

Adopt Claude Code or Codex wholesale and your engineers’ muscle memory, your CI integrations, your custom commands, and your prompts all get shaped by that one harness. When the vendor changes pricing, drops a feature, throttles your usage, or pivots, you pay for it. You don’t control the roadmap and you can’t fork it.

2. The model is the easy part to swap, if the harness lets you

A new best model shows up every few months. The model that fits your codebase best in Q1 may be second-best by Q3. A good setup lets you switch when the price/performance curve moves. A closed vendor harness ties you to their model family, and that costs you the biggest lever you have: picking the best model for the task at the best price.

3. Data and compliance boundaries are not negotiable for many companies

Where does your code go? What gets logged, retained, and used for training? With a closed harness you take the vendor’s answers as given. For regulated industries, air-gapped environments, or anyone with serious IP concerns, that is not enough. An open harness lets you choose providers, route to self-hosted or enterprise endpoints, and keep an audit trail you actually own.

4. Lock-in is cheap to enter and expensive to leave

The start is great. Then you have hundreds of engineers with vendor-specific habits, internal tooling built against a proprietary CLI, and no abstraction layer. Migrating later means retraining people and rebuilding integrations. That switching cost is exactly what gives the vendor pricing power over you.

5. Your competitive edge should not be a SaaS subscription

If everyone in your industry runs the same off-the-shelf agent the same way, where is your advantage? The leverage is in how you wire agents into your codebase, your domain knowledge, your tests, and your review process. That belongs in something you control.

The alternative: own the harness, rent the model

This is not an argument against AI coding agents. It is an argument for putting the boundary in the right place. Here is the setup I recommend.

Own the harness with pi

pi is a minimal, extensible terminal coding harness. The idea is to adapt the tool to your workflow instead of the other way around, and you do that without forking the agent’s internals. You extend it with TypeScript extensions, skills, prompt templates, and themes, then package and share them across your team over npm or git.

That gives you what you want from a company harness:

Your custom tools, prompts, permissions, and skills become version-controlled artifacts in your repo, not settings trapped in a vendor’s cloud.
It runs in interactive, print/JSON, RPC, and SDK modes, so the same harness powers your engineers’ terminals and your CI pipelines and internal apps.
It ships with sensible defaults but stays small, so you add only the complexity you need.

You get the ergonomics of Claude Code or Codex, but the workflow logic is yours and travels with your codebase.

Rent the model through a gateway

Keep the model pluggable. The simplest path is OpenRouter, a model gateway that gives you access to basically every model worth using behind one interface. Swap Anthropic, OpenAI, Google, and open-weight models on price/performance without rewriting anything.

If your data or compliance requirements are stricter, and for many companies they are, go through a cloud provider gateway instead. Amazon Bedrock and Azure AI Foundry give you the same model menu inside your existing cloud account, with your org’s networking, IAM, logging, and data-residency guarantees already in place. For regulated or IP-sensitive work, that is often the right answer.

Either way, standardize on the OpenAI-compatible API shape as the lowest common denominator, so swapping providers is a config change and not a migration.

Measure and route

Once the model is a swappable component, you can benchmark cost and quality per task and route on purpose: a cheap fast model for boilerplate, a strong one for tricky refactors. With pi owning the loop and a gateway in front, that routing is yours to tune.

“But Claude Code / Codex is just better right now”

Maybe today. The gap between the leading harness and the open ones is closing fast, and the open ones are built out in the open. Optimizing for “best this quarter” while you lock yourself in for years is a bad trade, especially for something as foundational as how your company writes software.

Use the good tools. Just don’t let one of them become the thing you can’t leave.

Takeaways

The harness, not the model, is where your workflow lives, and closed vendors own it.
Treat the model as a swappable commodity. Lock-in takes that lever away.
Data, compliance, and cost control all get harder when the harness is a black box.
Use an extensible harness like pi and keep your prompts, tools, and skills version-controlled in your own repo.
Rent models through a gateway: OpenRouter for breadth, or Bedrock / Azure AI Foundry when compliance and data residency matter.

Adopt agents fast, but keep the boundary, the harness and the choice of model, in your own hands.

Schema-First Telemetry in Go: A New Approach to Observability

ladislav@prskavec.net (Prskavec.Net) — Fri, 18 Apr 2025 08:05:13 +0200

Modern Go services generate telemetry (metrics, logs, traces) with instrumentation embedded directly in code. Usually this is a code-first approach: developers call APIs (counters, loggers, tracers) and attach key-value attributes on the fly. It works, but at scale it leads to fragmented data and missing context. Schema-first telemetry turns that around: you define a schema (like an IDL file) for your telemetry data up front, including its semantic metadata, and emit telemetry from generated code. This post introduces schema-first telemetry through Meta’s research, compares it to how OpenTelemetry handles schemas today, looks at what others in the community are doing, and shows Go examples of how you might implement it on top of OpenTelemetry.

The Limitations of Code-First Telemetry

Most of us instrument applications in a code-first fashion. For example, using OpenTelemetry in Go, you might record a metric like this:

// Code-first instrumentation example (OpenTelemetry)
counter := metric.Must(meter).NewInt64Counter("request_count")
counter.Add(ctx, 1,
 attribute.String("service.id", "foo"),
 attribute.String("endpoint", "/api/v1/resource"),
 attribute.Int("status_code", 200),
)

In this snippet, the keys "service.id", "endpoint", and "status_code" are just strings. You choose them in code, perhaps following suggested naming conventions. This is straightforward, but all the knowledge about these fields is implicit. The code is the only place where the “schema” exists. There’s no formal description of what "service.id" means or what values "status_code" can take, beyond maybe documentation or tribal knowledge. Common problems with the code-first approach:

Discoverability: Consumers of telemetry (engineers writing queries or dashboards) have to guess or search what attributes exist and what they mean. With many services, each might use slightly different keys for the same concept (e.g., "userId" vs "user.id").
Consistency: Nothing enforces consistency. Developers may mis-name a field or use it differently than others. OpenTelemetry offers semantic conventions, standard names for common attributes, but these are encouraged, not enforced.
Lack of semantic info: The backend treats most data as opaque strings. It might not know that "status_code" is an integer HTTP status, or that "endpoint" is a URL path. Units and types are often lost; everything becomes a string label.
Change management: Evolving telemetry is risky. Rename an attribute in code and any dashboard or alert that depends on the old name breaks. Without a schema there’s no easy way to notify or automatically migrate consumers.

In a code-first world, adding a new attribute (say a shard ID) to a metric is just another key-value in the code. Easy for the developer, but that new field’s existence and meaning still has to reach everyone downstream, usually through documentation or code review.

What is Schema-First Telemetry?

A schema-first approach means you define the structure and metadata of telemetry before writing instrumentation code, much like designing a schema for a database or an API. Yuri Shkuro (Meta) describes using Thrift IDL files to define telemetry data types (metrics, log events, and so on) with rich annotations for semantics. The schema becomes the single source of truth for what telemetry your app can emit.

Key aspects of schema-first telemetry:

IDL-based Schema Definition: You write an IDL (Interface Definition Language) file describing your telemetry data. For example, a metric “RequestCounter” might be defined as a struct with fields for service_id, endpoint, status_code, etc., each with a type and description.
Rich Metadata Annotations: The schema can include annotations for things like human-readable descriptions, units, acceptable values, privacy levels, etc. For instance, you might annotate that status_code is an HTTP status and an integer, or tag fields with @DisplayName and @Description in the IDL. This metadata travels with the schema.
Code Generation: A tool generates language-specific code (for Go, it could generate Go structs or helper functions) from the schema. Developers use this generated code instead of manually writing key strings. This ensures they only emit data that conforms to the schema.
Compile-time checks: Because telemetry fields are now struct fields or typed constants, many errors (typos in attribute names, wrong types) become compile-time errors. Try to record a string into a field defined as an int and the code won’t compile, which is a real win for robustness.
Schema Registry & Versioning: The schemas are versioned and stored in a registry (or metadata store). Both producers and consumers (backends, query tools) can retrieve the schema to know how to parse and interpret telemetry data. When updating a schema, you might bump a version or use a compatibility check in CI.

Example: Defining and Using a Telemetry Schema

Let’s illustrate with a simplified example inspired by Meta’s approach. Suppose we want a metric for counting requests. In schema-first style, we define it in an IDL (here using pseudo-Go syntax for clarity):

// Schema (could be in Thrift, Proto, etc.)
type ServiceID string // alias type for service identifier
type StatusCode int32 // alias type for HTTP status code
struct RequestCounter {
 1: ServiceID service_id // e.g., "orderservice"
 2: string endpoint // e.g., "/api/v1/submit"
 3: StatusCode status_code // e.g., 200
 // (We can add more fields later, like shard_id)
}

This schema says: a RequestCounter metric event has three fields with specified types. We could add metadata annotations too, such as descriptions or units (not shown here for brevity).

From this schema, a code generator would produce a Go struct and helpers. In Go, the generated code might look like:

// Generated code (simplified)
type RequestCounter struct {
 ServiceID string
 Endpoint string
 StatusCode int32
}
// ... plus maybe helper methods or validators

Now, to emit telemetry in our application, we use this struct with the OpenTelemetry API:

// Using schema-first generated struct in instrumentation
rc := RequestCounter{
 ServiceID: "orderservice",
 Endpoint: "/api/v1/submit",
 StatusCode: 200,
}
// The telemetry SDK could provide a helper to record struct data:
telemetry.Record(ctx, rc)

Under the hood, telemetry.Record could iterate over the struct’s fields and convert them to OpenTelemetry attributes, using a mapping of field names to attribute keys. For example, ServiceID -> "service.id", Endpoint -> "endpoint", StatusCode -> "http.status_code", based on the schema definition. The result is similar telemetry data, but produced in a schema-driven way rather than ad-hoc in code.

Adding a new field in schema-first starts with the schema. Say we want to add shard_id to the RequestCounter. We add it to the IDL:

 4: string ShardID // add shard identifier for sharded services

This change is made in the schema (with perhaps a description annotation for clarity). The codegen then updates the Go struct:

type RequestCounter struct {
 ServiceID string
 Endpoint string
 StatusCode int32
 ShardID string // new field
}

Now the application code can start populating ShardID when recording the metric:

rc := RequestCounter{
 ServiceID: "orderservice",
 Endpoint: "/api/v1/submit",
 StatusCode: 200,
 ShardID: "shard-42", // new dimension
}
telemetry.Record(ctx, rc) // now includes shard_id

From the developer’s side, adding the field was a small change: one extra line in the IDL, plus using the new field in code. From the system’s side, a lot more happens behind the scenes to keep it seamless and safe. That is the telemetry lifecycle in a schema-first model.

Telemetry Lifecycle in a Schema-First Model

In a schema-first system like the one at Meta, there is an end-to-end flow that ensures schema changes and data emission stay in sync:

Schema Definition/Change: A developer edits the schema file (IDL) to add or modify telemetry fields (for example, add shard_id to RequestCounter). This is done alongside the code that will emit the data. Both changes can go in a single pull request.
CI Validation: A continuous integration step checks the schema change for backward compatibility (no forbidden changes like renaming a field outright). It may also generate the new code and run tests. If the schema change would break consumers (e.g., deleting a field or changing its type in a non-backwards-compatible way), the CI rejects it. This ensures stability.
Schema Registry Update: When the change is approved and merged, an actualization service updates the central metadata store or schema registry with the new schema version. Now the organization’s observability platform “knows” about the new shard_id field for RequestCounter.
Telemetry Emission: The application runs with the updated code. The OpenTelemetry SDK (or a custom telemetry SDK) serializes the telemetry. In Meta’s case, they even serialize it as a binary Thrift payload on the wire for efficiency. Importantly, each telemetry data point can carry a reference to the schema version used.
Backend Ingestion: The telemetry backend receives the data. Seeing the schema reference (e.g., a schema URL or version), it can parse the data using the correct schema. Because the backend was informed of the schema update, it already understands the new field by the time data arrives. This avoids the usual problem of “surprise” fields nobody knows about.
Query & Analysis: When you go to query this data (maybe in a monitoring UI or via a query language), the tools can pull the schema metadata from the registry. This means UIs can present human-friendly names or descriptions for fields, enforce correct units or types in queries, and even suggest joins across telemetry streams if they share common schema elements. The rich annotations we discussed (like descriptions, units, etc.) are available to make exploration easier.

OpenTelemetry Today: Schemas and Semantic Conventions

How does this compare with OpenTelemetry (OTel), the open-source observability framework many Go developers use? OTel wasn’t originally schema-first, but it recognized some of these issues and introduced a schema concept in version 1.8 of the spec.

OTel provides Semantic Conventions, a catalog of standard attribute names for common scenarios (HTTP, database, messaging, and so on). They’re defined in YAML files and published as part of the spec. In Go they’re available as constants (for example, semconv.HTTPMethodKey for the "http.method" attribute). Semantic conventions add some structure and consistency: if everyone follows them, all services use "http.method" to mean the same thing. But as Shkuro et al. point out, these are “only conventions”. Nothing prevents a developer from deviating or misusing them, and there is no automatic validation that your values make sense (e.g., that an "http.status_code" attribute is a number and not a string saying “OK”).

To help conventions evolve, OpenTelemetry also introduced Telemetry Schemas identified by a Schema URL. An instrumentation library can embed a schema URL like https://opentelemetry.io/schemas/1.9.0 in the data it emits, signaling “I’m using version 1.9.0 of the conventions.” If a backend sees data from an older schema version, it can in theory automatically translate attributes to the newer schema (for example, if a key was renamed between OTel 1.8.0 and 1.9.0). So OTel schemas enable version negotiation and transformation, a bit like a migration script for telemetry data, which is useful for handling gradual changes in conventions without breaking dashboards.

But OTel’s Schema URL mechanism is still about evolving conventions. It doesn’t give you a per-application custom schema with rich types, and it doesn’t formally describe the data’s shape beyond listing name changes between versions. So it helps with compatibility, but it doesn’t reach the full schema-first vision of strongly typed telemetry with all semantics captured.

Comparing Approaches:

Schema Definition: Meta’s approach defines exact fields and types for each telemetry signal (e.g., the fields in RequestCounter). OTel defines common fields in a looser way (a list of recommended keys for things like HTTP, etc., which you may or may not use).
Enforcement: In schema-first, your code won’t compile if you try to emit something that’s not in the schema or of the wrong type. In OTel, you can always add any attribute key on a span or metric, with no compile-time checks. Errors or inconsistencies might only surface in the backend, or not until someone queries and finds nonsense values.
Evolution: Schema-first supports safe evolution through versioned schemas and automated compatibility checks. OTel’s schema URLs let backends adapt to renamed fields, but they don’t solve deeper changes (splitting a field in two, changing data types) and don’t prevent problems if producers and consumers aren’t in lockstep.
Metadata Richness: In schema-first, metadata like unit (e.g. milliseconds vs seconds for a latency metric) can be captured in the schema. In OTel’s code-first model, if you use a semantic convention the unit might be suggested by convention (e.g., "ms" for milliseconds), but it’s not enforced. It’s up to the developer to remember and the backend to interpret. Many existing telemetry systems treat all labels as opaque strings with no unit or type info.

OpenTelemetry is moving toward more schema awareness (schema URLs, generated semantic-convention libraries), but it is still largely a code-first ecosystem. The schema-first approach, as championed by Meta, goes further: it treats telemetry schemas as first-class artifacts in your development workflow.

Evolving the Ecosystem: Schema-First Telemetry Elsewhere

Meta isn’t alone in seeing the need for better telemetry metadata. Other organizations and open-source projects are pushing in the same direction:

Elastic Common Schema (ECS): Elastic created ECS as a unified schema for logs and metrics (primarily for Elasticsearch ingestion). It defines a common set of field names and types across many domains (cloud, host, network, http, and so on) so disparate sources can be correlated. For example, ECS says use source.ip for an IP address rather than every app inventing its own field. In 2023, Elastic announced it is working with OpenTelemetry to merge ECS with OTel’s semantic conventions, aiming for one common open schema. This is still about standardizing field naming, not a compiled per-app schema, but it shows the industry’s appetite for schema convergence. If ECS and OTel unify, a Go developer might one day rely on a single set of well-known field names for all telemetry.
CloudEvents schema references: The CloudEvents standard (for describing event payloads in a common way) includes a dataschema attribute, a URI pointing to the schema of the event’s data. This is conceptually similar to OTel’s schema URL: it tells consumers where to find the schema to interpret the event. It doesn’t enforce the schema on the producer side, but it encourages formal schemas for event data.
OpenTelemetry Weaver (Project OTel Weaver): This is a proof-of-concept CLI tool from the OpenTelemetry community exploring schema-first ideas (GitHub - f5/otel-weaver: OTel Weaver a schema-first approach to OpenTelemetry (status: POC)) (GitHub - f5/otel-weaver: OTel Weaver a schema-first approach to OpenTelemetry (status: POC)). OTel Weaver can generate a client SDK from a telemetry schema and allows searching and resolving of schema definitions. It’s inspired by the vision laid out in an OpenTelemetry discussion (OTEP 0243) about Application Telemetry Schema and Component Telemetry Schema. The goal is to let developers define custom schemas (for their specific application’s telemetry) on top of the global conventions, and have tools to generate code and validate telemetry against those schemas. As of now, this is in early stages (not production-ready), but it shows that the open-source community is actively working on enabling schema-first workflows in the future.
Datadog Reference Tables: Some observability vendors tackle the metadata problem by letting users enrich telemetry after the fact. Datadog, for instance, has Reference Tables that let you join telemetry data with external data (CMDB info, business metadata). It isn’t schema-first, but it comes from the same pain point: needing more context and meaning in telemetry. Reference Tables enrich a posteriori (add metadata after data is collected), whereas schema-first bakes it in a priori. At Meta they had experience with an a posteriori metadata store and found it can drift out of sync if not tightly coupled, which is why they moved to schema-first for critical systems.
Chronosphere’s approach: Chronosphere (an observability platform) emphasizes standardizing telemetry metadata and naming across an organization. In a blog post on OTel attribute naming, they note that “metadata plays a key role in making observability data meaningful” and advocate for clear conventions and company-wide standards. This isn’t a full schema-first system, but the recommended practice of defining a naming-standard document and running workshops to align teams is essentially a manual schema-governance process. The goals are the same: consistency, discoverability, and interoperability of telemetry.

There’s broad recognition that telemetry needs stronger contracts. Whether through formal schemas, unified conventions, or metadata enrichment, the industry is converging on telemetry that is more self-describing and can evolve without breaking consumers. Schema-first is one of the leading techniques to get there.

Go Example: Schema-First Telemetry in Practice

Back to the Go developer’s perspective. How might you apply schema-first ideas in Go today with OpenTelemetry? The full tooling isn’t here yet, but you can adopt some patterns:

Define Structs for Telemetry Data: Instead of sprinkling string keys throughout your code, define Go structs to represent the telemetry events or metrics you intend to emit. This gives you a single place to see all fields and their types.
```
// Define a struct as a pseudo-schema for a log or metric event
type OrderSubmittedEvent struct {
 OrderID string // business identifier
 Value float64 // value of the order
 CustomerID string // customer identifier
 Region string // region of the request
}
```
Add comments or struct tags to document the fields. This struct is a mini-schema. It doesn’t enforce anything on its own, but it’s a start.

Use Constructor Functions to Enforce Required Fields or Types: You can provide functions to create these events, which can apply validation.

func NewOrderSubmittedEvent(orderID string, value float64, customerID, region string) OrderSubmittedEvent {
 // Here you might enforce some constraints or default values
 return OrderSubmittedEvent{orderID, value, customerID, region}
}

Conversion to OTel Attributes: Implement a method to convert your struct to OTel attributes. For example:
```
import "go.opentelemetry.io/otel/attribute"

func (e OrderSubmittedEvent) Attributes() []attribute.KeyValue {
 return []attribute.KeyValue{
 attribute.String("order.id", e.OrderID),
 attribute.Float64("order.value", e.Value),
 attribute.String("customer.id", e.CustomerID),
 attribute.String("region", e.Region),
 }
}
```
Here we decide on the actual keys (like "order.id", "customer.id"). Ideally, follow existing semantic conventions or your organization’s schema guidelines for naming (e.g., if “region” should be cloud.region per OTel conventions, use that). This method is essentially encoding the schema knowledge.
Emit Telemetry Using the Struct: Now use the struct in your instrumentation code:
```
meter := global.Meter("myapp/orders")
ordersCounter := metric.Must(meter).NewInt64Counter("orders_submitted_total")

// When an order is submitted:
evt := NewOrderSubmittedEvent(orderID, value, customerID, region)
ordersCounter.Add(ctx, 1, evt.Attributes()...)
```
This ensures every time you record orders_submitted_total, you attach the full set of attributes as defined in the struct. If you forget a field, the compiler will complain if it’s not set (since the struct requires you to provide all fields in NewOrderSubmittedEvent). If you ever rename a field in the struct, you’ll update it in one place (the Attributes() method), rather than hunting down every occurrence of the string literal.
Optional, schema registration: If you want to go one step further, you could maintain a simple registry or metadata map in your code that describes these events. For example:
```
var TelemetrySchema = map[string]map[string]string{
 "OrderSubmittedEvent": {
 "order.id": "ID of the order",
 "order.value": "Value of the order in USD",
 "customer.id":"ID of the customer who placed the order",
 "region": "Geographical region of the request",
 },
}
```
This could expose the schema programmatically (say, an HTTP endpoint that returns the schema JSON, or a log line at startup). It’s a rudimentary schema registry. Consumers (or your team’s documentation) could pull it to know what telemetry to expect. It’s not as robust as a real schema registry, but it builds the habit of treating telemetry fields as a contract.

In a true schema-first setup, much of this would be automated: an IDL, codegen, and so on. But even without full tool support, these Go patterns get you closer to the benefits, with type safety, clarity, and a self-documenting approach.

Benefits: Why Schema-First Telemetry Matters

For developers, schema-first means a better experience when instrumenting code. You get auto-complete for telemetry fields (they’re struct fields or constants) and compile-time feedback when you misuse them. It cuts the cognitive load of remembering “what keys should I use here?” because the schema lives in one place. As Meta’s engineers found, a type-safe API is “more ergonomic and robust” than generic key-value pairs. It’s like the difference between raw SQL strings and an ORM with defined models: the latter catches errors earlier and is easier to navigate in an IDE.

For telemetry consumers (SREs, analysts, and so on), the data comes with a map. It’s more discoverable: you can list all metrics and their dimensions from the schema registry without reading the source. It’s more interpretable: units and descriptions are right there, so you don’t have to guess whether a duration is in seconds or milliseconds, or what “xyz_flag=true” really means. Streams correlate more reliably because shared dimensions are clearly identified. If logs and metrics both carry a customer.id field per schema, you can join on it with confidence; in a code-first world where one used user and the other customerId, you might not even realize they refer to the same thing.

For platform engineers and architects, schema-first enables better governance and tooling. You can enforce policies centrally, like “every metric must have a unit annotation” or “PII must be marked in the schema and handled accordingly.” It also opens the door to optimizations: if a field is an enum with a limited set of values, you can encode it more efficiently than a free-form string. Meta reported performance gains from serializing telemetry as a struct (no repeated keys per record) and using binary encoding. The schema also allows automated compatibility layers, similar to OTel’s schema transforms but potentially more powerful, because you know the data types, not just the names.

Schema-first isn’t free. It adds an upfront step, writing or updating the schema, that code-first avoids. The Meta paper acknowledges this friction but calls it “a worthy trade-off” given the benefits. Tooling (schema wizards, templates) can soften the overhead, and once a schema exists, using it is straightforward, especially when it’s wired into your build (Meta integrates schema codegen into their Buck build).

Conclusion

Schema-first application telemetry brings strong typing and rich metadata to observability. For Go developers it means moving from ad-hoc strings in code to defined structs and schemas that describe your metrics, logs, and traces. Meta’s implementation shows the payoff: compile-time validation, easier multi-signal correlation, and automatic enforcement of consistency. OpenTelemetry today gives you semantic conventions and schema versioning, which solve part of the problem, and a fuller schema-first model is coming with community efforts like OTel Weaver and the unification of industry schemas.

Even an informal schema-first mindset can improve the telemetry your Go services emit. Start by defining and documenting your telemetry fields in one place, whether an IDL, a Go struct, or a markdown file in your repo. Use code generation or helper libraries so everyone shares the same definitions. Over time this can grow into a more automated schema-first toolchain as the ecosystem matures.

Telemetry with a schema is like code with types: safer, clearer, and easier to change. As observability keeps growing in importance for cloud-native systems, schema-first techniques may well become a standard part of the Go developer’s toolkit for building reliable, introspectable services.

Sources:

Build nice terminal UI with Bubble Tea

ladislav@prskavec.net (Prskavec.Net) — Thu, 04 Jul 2024 12:54:08 +0200

If you want to build text-based user interfaces in Go, Bubble Tea is a good option. It makes building TUI (Text User Interface) applications easier, and the results can look good and stay interactive.

What is Bubble Tea?

Bubble Tea is a Go library from Charm_, based on The Elm Architecture, which keeps state and UI updates predictable. It works well for dashboards, command-line tools, and games.

Key Features

Elm-inspired architecture: manage application state in a structured, predictable way.
Rendering: build interactive interfaces with the Bubble Tea rendering engine.
Concurrency: use Go’s concurrency model for responsive applications.
Cross-platform: works across operating systems.

Getting Started

I will use simple application that I created for demo purpose.

It’s a simple RSS reader that fetches the latest items from a feed and shows them in the terminal. It uses Bubble Tea for the TUI, so you can navigate the news items and read the full content of each article.

Main app run model and give TUI data. Project have all components related to TUI in tui package.

package main

import (
 "fmt"
 "log"
 "os"

 "github.com/abtris/rss-bubbletea/tui"
 tea "github.com/charmbracelet/bubbletea"
 "github.com/mmcdole/gofeed"
)

func main() {
 file, _ := os.Open("data/podcast.xml")
 defer file.Close()
 fp := gofeed.NewParser()
 feed, err := fp.Parse(file)
 if err != nil {
 log.Fatal("parse feed failed", err)
 }

 model, err := tui.NewModel(feed)
 if err != nil {
 log.Fatal("create model failed", err)
 }

 if len(os.Getenv("DEBUG")) > 0 {
 f, err := tea.LogToFile("debug.log", "debug")
 if err != nil {
 fmt.Println("fatal:", err)
 os.Exit(1)
 }
 defer f.Close()
 }

 p := tea.NewProgram(model, tea.WithAltScreen(), tea.WithMouseCellMotion())
 if err := p.Start(); err != nil {
 log.Fatal("start failed: ", err)
 }
}

model tui/model.go using list component from Bubbles component library.

package tui

import (
 "log"

 md "github.com/JohannesKaufmann/html-to-markdown"
 "github.com/charmbracelet/bubbles/list"
 "github.com/mmcdole/gofeed"
)

type model struct {
 list list.Model
 choice string
 content string
 detail bool
 quitting bool
}

const width = 80
const height = 40
const title = "RSS Reader"

func NewModel(data *gofeed.Feed) (*model, error) {
 var items []list.Item
 converter := md.NewConverter("", true, nil)
 for _, rssItem := range data.Items {
 markdown, err := converter.ConvertString(rssItem.Description)
 if err != nil {
 log.Println("Convert to markdown", err)
 }
 i := item{
 title: rssItem.Title,
 desc: "Published at " + rssItem.Published + "\n\n" + markdown,
 }
 items = append(items, i)
 }
 l := list.New(items, list.NewDefaultDelegate(), width, height)
 l.Title = title

 return &model{
 list: l,
 choice: "",
 content: "",
 detail: false,
 }, nil
}

interaction with interface is covered by tui/update.go

package tui

import (
 tea "github.com/charmbracelet/bubbletea"
)

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 switch msg := msg.(type) {
 case tea.WindowSizeMsg:
 m.list.SetWidth(msg.Width)
 return m, nil

 case tea.KeyMsg:
 switch keypress := msg.String(); keypress {
 case "q", "ctrl+c":
 m.quitting = true
 return m, tea.Quit

 case "enter":
 i, ok := m.list.SelectedItem().(item)
 if ok {
 m.detail = true
 m.choice = string(i.title)
 m.content = string(i.desc)
 }
 return m, nil
 case "b":
 if m.detail {
 m.choice = ""
 m.content = ""
 }
 case "p":
 if m.detail {
 changeIndex := m.list.Index() + 1
 if changeIndex <= 0 {
 changeIndex = 0
 }
 m.list.Select(changeIndex)
 i, ok := m.list.SelectedItem().(item)
 if ok {
 m.choice = string(i.title)
 m.content = string(i.desc)
 }
 }
 case "n":
 if m.detail {
 changeIndex := m.list.Index() - 1
 maxLength := len(m.list.Items())
 if changeIndex > (maxLength - 1) {
 changeIndex = maxLength - 1
 }
 m.list.Select(changeIndex)
 i, ok := m.list.SelectedItem().(item)
 if ok {
 m.choice = string(i.title)
 m.content = string(i.desc)
 }
 }
 }
 }

 var cmd tea.Cmd
 m.list, cmd = m.list.Update(msg)
 return m, cmd
}

and view tui/view.go take care about rendering. I’m using glamour for rendering markdown content.

package tui

import (
 "log"

 "github.com/charmbracelet/glamour"
)

func (m model) View() string {
 var s string
 if len(m.choice) > 0 {
 s += "# " + title
 s += "\n## " + m.choice
 s += "\n\n"
 s += m.content
 renderer, err := glamour.NewTermRenderer(
 glamour.WithAutoStyle(),
 glamour.WithWordWrap(width),
 )
 if err != nil {
 return ""
 }
 out, err := renderer.Render(s)
 if err != nil {
 log.Println(err)
 }
 return out
 }
 return m.list.View()
}

Whole demo is recorded using VHS.

Conclusion

Bubble Tea is a solid library for building terminal UIs in Go. Whether you’re writing a small command-line tool or a more involved interactive app, it gives you what you need without much ceremony.

I had talk about Bubble Tea on Go meetup #15.

Exploring Terramate - A Solution to Terraform Orchestration Challenges

ladislav@prskavec.net (Prskavec.Net) — Sun, 10 Mar 2024 12:22:10 +0100

Terraform by HashiCorp has become a leading tool for defining and provisioning infrastructure as code. Its declarative configuration language lets developers and operations teams manage infrastructure precisely. But as projects grow in size and complexity, you run into orchestration, state management, and configuration spread across many environments. Terramate is built to help with exactly those problems.

What Terramate does

Terramate complements Terraform with extra structure and tooling for managing multiple stacks, environments, and configurations. It targets the pain points of large-scale Terraform deployments: dependency management, stack management, and environment differentiation.

Key features

Stack management: Terramate organizes Terraform configurations into stacks, which makes complex infrastructure easier to manage. The hierarchy lets teams handle parts of their infrastructure in a more modular way.
Orchestration: Terramate can define and manage dependencies between stacks, so components deploy in the right order, a common problem in multi-stack projects. You can define and reuse data across stacks with variables and metadata, and integrate Terramate into CI/CD pipelines to automate testing and deployment. It supports Terraform, OpenTofu, Terragrunt, Kubernetes, Pulumi, AWS CloudFormation, AWS Cloud Development Kit (CDK), Azure Resource Manager (ARM), Bicep, and others.
Code generation: Generate code from templates and modules to cut duplication and keep configuration consistent across your infrastructure.
Git integration: Detects and manages stacks with changes in a branch, commit, or pull request.

Is it the right tool for you?

Whether Terramate fits depends on your situation: how complex your infrastructure is, the specific problems you have, and how your team likes to work. It’s worth weighing it against other tools in the Terraform space, like Terragrunt, before deciding.

My use case is focus make our infrastructure in Azure Cloud automated. Our infrastructure team give us recommendation to make modules smallest as possible to work with their pipelines. It’s nice idea but it’s very hard to do it with complex infrastructure that have tens of modules. So I decided to use Terramate for my project. Split modules into stacks and use features for ordering to make graph how I want run them to be sure that order is right.

I really like generating code. I make templates for backend and providers, that can works for everyone. But I can setup templates specific for project. We generate common variables, injected data from others team etc. That simplify many parts that was duplicated on many places.

Conclusion

Terramate helps if you’re orchestrating and managing large Terraform projects. Stack management, environment handling, and dependency management take some of the pain out of infrastructure as code. If you’re running into these problems, it’s worth a look.

I will about more details how we using Terramate after our project will be released to public.

OpenTelemetry and Go Gin Framework

ladislav@prskavec.net (Prskavec.Net) — Sat, 21 Oct 2023 09:06:54 +0100

On OpenTelemetry website, you can read the definition what is OpenTelemetry:

OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

It is not vendor free way to instrument your code. In Go, we don’t have in Oct 2023 full support. Traces, metrics, logs, are not in the stable version. You can still start using it, at least for traces and metrics.

I’m working on a project in which I need to make traces from Python to my Go API, and we want all stored in logs and traces. I started with Gin framework, which is my default simple framework for APIs. There are many good alternatives, but I like to use this one.

It’s great if you Google otel gin you get otelgin library for instrument your code.

You find good example where you see how to use it.

r.Use(otelgin.Middleware("my-server"))

You add a new line to your code with otelgin middleware, and you are done! Or not?

Context Propagation

If you need to connect traces across services, you need setup propagation, and you find in every example of your tracer.

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

I will focus on propagation.TraceContext{}. TraceContext is W3C spec on how to pass information about TraceId, SpanId across services in Header traceparent. All work for you if you call another system. But it doesn’t work for me.

I tested with this command:

curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" http://localhost:8080

I can’t find my correct traceId in my logs.

You need to configure otelgin middleware with the propagator too. Not just tracer.

otelginOption := otelgin.WithPropagators(propagation.TraceContext{})
r.Use(otelgin.Middleware("my-server", otelginOption))

After that, all start working.

How to debug it?

I use two ways to debug.

First, I’m using otel-desktorp-viewer to debug traces locally without running an observability server.

Second, I’m using log/slog library for logging (available in Go 1.21+). I make simple logging middleware for Gin that records traceId and spanId.

func GinSlogMiddleware(logger *slog.Logger) gin.HandlerFunc {
 return func(c *gin.Context) {
 start := time.Now().UTC()
 path := c.Request.URL.Path
 query := c.Request.URL.RawQuery
 c.Next()
 ctx := c.Request.Context()
 end := time.Now().UTC()
 latency := end.Sub(start)
 fields := slog.Group("http",
 slog.Int("status", c.Writer.Status()),
 slog.String("method", c.Request.Method),
 slog.String("path", path),
 slog.String("query", query),
 slog.String("ip", c.ClientIP()),
 slog.String("user-agent", c.Request.UserAgent()),
 slog.Duration("latency", latency),
 )
 if len(c.Errors) > 0 {
 for _, e := range c.Errors.Errors() {
 logger.ErrorContext(ctx, e, fields)
 }
 } else {
 logger.InfoContext(ctx, path, fields)
 }
 }
}

You can see that the key is logging with Context, which contains information about traces. Context handler will log information about traces.

type ContextHandler struct {
 slog.Handler
}

func (h ContextHandler) Handle(ctx context.Context, r slog.Record) error {
 r.AddAttrs(h.addTraceFromContext(ctx)...)
 return h.Handler.Handle(ctx, r)
}

func (h ContextHandler) addTraceFromContext(ctx context.Context) (as []slog.Attr) {
 if ctx == nil {
 return
 }
 span := trace.SpanContextFromContext(ctx)
 traceID := span.TraceID().String()
 spanID := span.SpanID().String()
 traceGroup := slog.Group("trace", slog.String("id", traceID))
 spanGroup := slog.Group("span", slog.String("id", spanID))
 as = append(as, traceGroup)
 as = append(as, spanGroup)
 return
}

And my logger init looks as this:

func InitLog() *slog.Logger {
 jsonHandler := slog.NewJSONHandler(os.Stdout)
 ctxHandler := ContextHandler{jsonHandler}
 return slog.New(ctxHandler)
}

You can make a custom JSONHandler if you need to override the logging format, for example, for support Elastic ECS if your destination is Elastic stack.

Resources and libraries supporting log/slog aren’t always ready, but I expected that will resolved in the future.

Watch out for basic HTTP client settings in Go

ladislav@prskavec.net (Prskavec.Net) — Wed, 25 Jan 2023 13:55:01 +0100

Many articles have been written about this problem, but I keep seeing this problem coming back over and over again. Most programming languages don’t have basic settings for HTTP made to run in production. We’ll demonstrate this with the example of Go, but other languages are often similar, sometimes better, sometimes worse.

If you take a standard library, and you want to make a request, it will come out something like this.

func main() {
 url := "http://localhost:3000"
 var httpClient = &http.Client{}
 response, _ := httpClient.Get(url)
 fmt.Println(response)
}

All the HTTP client settings are given by default transport.

var DefaultTransport RoundTripper = &Transport{
 Proxy: ProxyFromEnvironment,
 DialContext: defaultTransportDialContext(&net.Dialer{
 Timeout: 30 * time.Second,
 KeepAlive: 30 * time.Second,
 }),
 ForceAttemptHTTP2: true,
 MaxIdleConns: 100,
 IdleConnTimeout: 90 * time.Second,
 TLSHandshakeTimeout: 10 * time.Second,
 ExpectContinueTimeout: 1 * time.Second,
}

Your HTTP client will behave according to that basic setup. All the timeouts are defined, and you can’t simply hit that client later.

If you want to tweak it, if you need to touch the base timeout, you can set it like this:

func main() {
 url := "http://localhost:3000"
 var httpClient = &http.Client{
 Timeout: time.Second * 10,
 }
 response, _ := httpClient.Get(url)
 fmt.Println(response)
}

This will override the 30s timeout to 10s which is about the maximum value I would accept for a synchronous POST request, although it’s well beyond what I’m willing to expect as a user.

The way to keep things under control is to have a custom client where we overload the transport and use custom settings.

type HTTPClientSettings struct {
 Connect time.Duration
 ConnKeepAlive time.Duration
 ExpectContinue time.Duration
 IdleConn time.Duration
 MaxAllIdleConns int
 MaxHostIdleConns int
 ResponseHeader time.Duration
 TLSHandshake time.Duration
}

func NewHTTPClientWithSettings(httpSettings HTTPClientSettings) (*http.Client, error) {
 var client http.Client
 tr := &http.Transport{
 ResponseHeaderTimeout: httpSettings.ResponseHeader,
 Proxy: http.ProxyFromEnvironment,
 DialContext: (&net.Dialer{
 KeepAlive: httpSettings.ConnKeepAlive,
 DualStack: true,
 Timeout: httpSettings.Connect,
 }).DialContext,
 MaxIdleConns: httpSettings.MaxAllIdleConns,
 IdleConnTimeout: httpSettings.IdleConn,
 TLSHandshakeTimeout: httpSettings.TLSHandshake,
 MaxIdleConnsPerHost: httpSettings.MaxHostIdleConns,
 ExpectContinueTimeout: httpSettings.ExpectContinue,
 }

 // So client makes HTTP/2 requests
 err := http2.ConfigureTransport(tr)
 if err != nil {
 return &client, err
 }

If we have custom settings, then the usage is similar, just directly specifying the individual parameters.

func main() {
 url := "http://localhost:3000"

 httpClient, err := NewHTTPClientWithSettings(HTTPClientSettings{
 Connect: 5 * time.Second,
 ExpectContinue: 1 * time.Second,
 IdleConn: 90 * time.Second,
 ConnKeepAlive: 30 * time.Second,
 MaxAllIdleConns: 100,
 MaxHostIdleConns: 10,
 ResponseHeader: 5 * time.Second,
 TLSHandshake: 5 * time.Second,
 })
 if err != nil {
 fmt.Println("Got an error creating custom HTTP client:")
 fmt.Println(err)
 return
 }

 response, _ := httpClient.Get(url)
 fmt.Println(response)
}

This is a good solution, but it still needs to solve everything. Sometimes we would like a different timeout or deadline for a specific case (PUT, POST), and we want to keep the HTTP client the same. This can be achieved with context.WithTimeout or context.WithCancel context. The usage is then seen in the following example.

func main() {
 url := "http://localhost:3000"

 req, err := http.NewRequest(http.MethodGet, url, nil)
 if err != nil {
 log.Fatal(err)
 }

 ctx, cancel := context.WithCancel(context.Background())
 _ = time.AfterFunc(1*time.Second, func() { cancel() })

 httpClient, err := NewHTTPClientWithSettings(HTTPClientSettings{
 Connect: 5 * time.Second,
 ExpectContinue: 1 * time.Second,
 IdleConn: 90 * time.Second,
 ConnKeepAlive: 30 * time.Second,
 MaxAllIdleConns: 100,
 MaxHostIdleConns: 10,
 ResponseHeader: 5 * time.Second,
 TLSHandshake: 5 * time.Second,
 })
 if err != nil {
 fmt.Println("Got an error creating custom HTTP client:")
 fmt.Println(err)
 log.Fatal(err)
 }

 response, _ := httpClient.Do(req.WithContext(ctx))
 fmt.Println(response)
}

Still, it can happen that you are not on the client but on the server side, and you can’t solve it entirely well. That’s why in Go 1.20, they added ResponseController where you get access to these methods:

Flush()
FlushError() error // alternative Flush returning an error
Hijack() (net.Conn, *bufio.ReadWriter, error)
SetReadDeadline(deadline time.Time) error
SetWriteDeadline(deadline time.Time) error

Here you can, for example, extend the request deadline if you need it. Find the complete discussion of timeout handling in Handler on GitHub.

So remember to set up HTTP clients for production, test performance, measure latency and adjust your settings accordingly. It’s good to know your limits and adjust accordingly. Please keep it to the basic settings in production. It can cause annoying errors that are hard to detect because they often look like random errors.

If you’re interested in this, come and talk about Go and release 1.20 at the next Prague Go Meetup on February 21.

Pipelines as Code

ladislav@prskavec.net (Prskavec.Net) — Mon, 28 Nov 2022 14:08:54 +0100

“Pipelines as code” is not exactly a new concept, I heard about it a few years ago with the launch of Tecton. I wrote a design document at that time to create a new CI that would accept various rules (Jenkins, CircleCI, TravisCI and Github Actions) and you wouldn’t have to learn new syntax. Unfortunately, the project never materialized, so I didn’t pursue this much longer.

Solomon Hykes left Docker a few years ago and started Dagger.io, where they started adopting Cue lang and working on an interesting new project. When I got access to early access I wasn’t too excited about it, because writing pipelines in Cue is not something I would recommend to developers. After experiencing how hard it was to push Grafana dashboards in Jsonnet, I’m cautious.

A few weeks ago, they launched a technical preview of Dagger Go SDK and when I tried out the basic demo based on the video, I thought it was great. I decided to test on my project ga-badge that generates badge for Github Actions. Here’s the relevant PR with the changes, showing how simple the change is. Then here’s a log of when it’s dropped. I use Mage for dropping as recommended by the Dagger folks for the multi repository. I’m sure there are other ways to do it, but it seemed fine to me at first.

I’ll be talking about the Dagger Go SDK at Go Meetup #9 on November 30, 2022, and if you’re interested, stop by the discussion.

How to make the perfect Grafana dashboard?

ladislav@prskavec.net (Prskavec.Net) — Fri, 30 Oct 2020 12:09:14 +0100

First, there is no such thing as a perfect dashboard. You may think that your dashboard is perfect but the next incident can completely break this perception. This is normal, everything is imperfect in this world. We have to work with that. Improving dashboards is an iterative process. You have to make it a part of the culture in the same way as post mortems or other best practices for running services.

How to start

I always recommend starting with simple things. First, I’d like to recommend reading Grafana best practices articles¹. There are lists of things that will help you formulate the overall high-level strategy. Don’t fill all the things at once - start slow. You can always add more metrics as you go.

It’s good to start with the first dashboard and basic things, such as absent metrics (to detect if your metrics are even working) and up/down metrics (are all instances of database healthy?).

Generally, it’s good not to forget about observability rules. I prefer The Four Golden Signals². You can also use the USE method³ or RED method⁴.

Finally, add important business metrics that are relevant to your service (number of new registrations, outgoing emails, created teams or projects, etc.).

What to do next

Maintaining dashboards is not an easy task. Creating hundreds of dashboards is simple but maintaining them is a nightmare. How to prevent this? I recommend thinking about structure and using the drill-down strategy. Use tools, such as Jsonnet, to generate consistent dashboard over teams, environments and services. You will be able to reuse common parts, such as metrics for load balancers or databases used across the company.

What is a drill-down strategy? It means that you have a central dashboard with overview as a single point of contact for your oncall engineers and then there are more detailed dashboards linked from the central dashboard.

In this dashboard, they see the most important things. You need work to minimize the noise there and focus only on important signals.

You have to keep in mind that nobody is constantly looking at the dashboard (except the guys working in NOC). You have to think about it and make the visualization and error signaling significant to the person at 2am in the morning, after the pager woke them up. If a signal is visible, the operator can click on a graph and to get to another dashboard that gives more insight into the problem. This can be repeated for more levels of depth. Based on my experience, it is only useful to stick to a maximum of 2 to 4 levels of depth.

Dashboards should be using variables and templates. You can let the user switch between different variable values to display the relevant data for a data region, availability domain or a specific application.

Make templates for reusable components to make your setup DRY and easy to maintain, such as a common header for all your dashboards.

Links to detailed dashboards should also use variables and templates, to minimize the amount of code. For instance, you can reuse the same dashboard type for all your applications.

Make the process work!

Create a process to review the dashboards and include it in your regular operations review. You should iterate on improvements of metrics and dashboards, ideally on a weekly basis. The best way to do this review is to make it a part of your incident post-mortem process. You can use your incidents to inspire you to improve your alarms and margins for your dashboards.

Alarms have always a higher priority than your dashboards and you should focus on good alarms with actionable runbooks for solving issues.

Dashboards have to add more context and overview for your service. This helps with finding correlation with other events⁵. If you can, integrate the logs and traces into a single tool or add links between them to find what you need.

Take regular feedback from the team and operators to make it easy to make changes in alarms and maintain the dashboards.

https://grafana.com/docs/grafana/latest/best-practices/ ↩︎
traffic, errors, latency, saturation ↩︎
utilization, saturation, errors ↩︎
rate, errors, duration ↩︎
for example deployments are displayed as annotations in Grafana, logs - Loki and traces - Opentracing/Tempo ↩︎

The Future of Continuous Integration Engines

ladislav@prskavec.net (Prskavec.Net) — Fri, 23 Oct 2020 20:02:34 +0200

Thanks to Daria Grudzien for correction and comments.

Continuous Integration Engines (CIEs) have a long history. The term Continuous Integration (CI) has been used for the first time by Grady Booch in 1991 ¹². First known CIE is Tinderbox ³ (1997). Personally I started using CruiseControl (launched in 2001) around 2003. After 2005 Java programmers switched from CruseControl to Hudson. Few years later phpUnderControl (2007) became a popular tool, and later on Hudson gave way to Jenkins (released in 2011). These days Jenkins is being replaced by JenkinsX (2017). Many other CIEs have been developed in the meantime: Teamcity (2006), Bamboo (2008), as well as other commercial versions and cloud first CIEs like Travis (2011) , CircleCI (2016) and Github Actions (2018).

Nowadays we have a wide range of CIEs available on every platform, which solve a variety of problems. To help select the best one for the job, in 2019 I wrote a post “What you need from your CI” which is still valid.

Today I’d like to introduce a new question— what is the future of CIE and what we can expect as developers and customers of CIE software.

Modularity

These days modularity is key. Architectural changes introduced by tools like JenkinsX (2017) and Tekton pipelines (2018) help in designing new CIEs with that feature in mind. Such an approach allows developers to easily change the language describing workflows by replacing modules which translate workflow description to Tekton CRDs (Custom Resource Definition). I like that approach and believe that the teams behind Tekton and JenkinsX are doing great work on increasing flexibility of workflow development.

Flexibility

Flexibility is the second consideration. I don’t think that all CIEs have to be running on Kubernetes. Good alternatives are HashiCorp Nomad,the simplest platform like AWS Lambda and LambdaCI or simply a local computer. Portable software which can be easily run anywhere can be very useful. This still gives space to different CIEs and can be a major differentiator among them.

Language

This approach isn’t available among the existing tooling, but it would be nice to have a de facto standard for describing workflows for CIE. Thanks to the modular every CI can support such a standard workflow description, making switching CIEs for projects would become easy. Currently Yaml format is the king - it’s used by Travis, CircleCI, Github Actions among others. But I hope that we will find a better approach like HCL (HashiCorp Configuration Language) which is more of a language than a serialization format (noyaml.com).

Observability

It’s very hard to make any good decisions without sufficient data. Something I see quite often is a lack of good solutions for handling builds, failing tests, flaky tests etc. For applications we have open metrics, Prometheus is commonly supported by many libraries and applications. An open format like this is missing in CIEs and it would be great to fill those gaps.

Scalability

Speed, reliability and cost are all relevant from the perspective of scalability. In the current price model you often pay for VMs which is cons ineffective for teams collocated in one timezone. If you have a “follow the sun” model and similar resource usage across workdays — what about weekends? Many large companies have their own CIE, but often it’s run on dedicated hardware which doesn’t scale up with the number of developers or tasks in the queue.

It’s possible to keep the same spending level while improving developer experience by scaling the CI up and down depending on current usage. If you discover that your usage pick lasts for 4-5 hours a day—you can scale down the number of instances running CI outside of those hours. But don’t forget that running builds in parallel needs to be fast by default.

Bringing It All Together

There are multiple CIEs available in the market today addressing some of the considerations I listed above. Do you know of a Continuous Integration Engine which addresses all those, or do still we have gaps in the market? Is there space for someone to make a significant impact on the market?

https://en.wikipedia.org/wiki/Continuous_integration ↩︎
In 1994, Grady Booch used the phrase continuous integration in Object-Oriented Analysis and Design with Applications (2nd edition) ↩︎
TinderBox by Mozilla - https://oduinn.com/2014/06/04/farewell-to-tinderbox/ ↩︎

Grafana dashboards and Jsonnet

ladislav@prskavec.net (Prskavec.Net) — Sat, 17 Oct 2020 19:57:41 +0200

I know Grafana for a long time, but in my previous career, we used different tools as Datadog or other older tools. We switched from Datadog to Grafana last year to unite with other teams in one central place.

I was looking at how others maintain their dashboards and some teams just edit and store JSON outputs from Dashboards. I heard on Monitorama conference about Jsonnet a few years ago. I thought this would be a better approach than editing JSON files directly.

Grafana has a library called grafonnet-lib that helps you with creating widgets in grafana via jsonnet. You can find prepared components for a dashboard, annotation, graph, singlestat, template, and more. What I find out later was that many options and parts were missing. You need to edit directly or send PRs into the library. I sent a few PRs to fix it. We found more and more missing parameters and I gave up after seeing that many PR were not merged. Today I found that the Grafana team is working on a better solution. In Grafana 7.0 they are generating jsonnet from spec. I hope this will be a good solution to have all parameters available and without the need to fix it by ourselves.

How we structure our dashboards

This is list of files in our repository:

├── Makefile
├── README.md
├── build
│ ├── dev
│ ├── prod
│ └── staging
├── dashboards
│ ├── actions.jsonnet
│ ├── backend.jsonnet
│ ├── mongo.jsonnet
│ ├── oncall.jsonnet
│ ├── redis.jsonnet
│ └── traffic.jsonnet
├── environments
│ ├── dev.libsonnet
│ ├── prod.libsonnet
│ └── staging.libsonnet
├── grafonnet (copy of library)
└── templates
├── config.libsonnet
├── header.libsonnet
├── mql.libsonnet

The makefile has these commands to prepare dashboards and command for deploy.

$ make
Usage:
make <target>
Targets:
help Display this help
clean Clean built dashboards
build Build dashboards
deploy Deploy dashboards
auth Get fresh token from grafana

We use the same dashboards for every environment and there are configuration changes that are stored in folder environments. We use one grafana instance for all environments but all settings related include name and title of dashboards are specific for the environment. Dashboards have similar names as Team - Environment - Service.

In template folder we have config.libsonnet that is the main config file reused in all dashboards. In the folder we have common parts, such as a header with the message: # Don\'t edit here. Edit in repository, and make PR there!. We extend code as tutorial with libsonnet files.

The command to compile jsonnet into build/*.json looks like:

jsonnet --ext-code-file "env=environments/$(patsubst %/,%, $(dir $(patsubst build/%, %, $@))).libsonnet" dashboards/$(basename $(notdir $@)).jsonnet > $@

From the files in the dashboards folder, we create json output that is split to folders by environments. We are following guidance from Grafana Labs at KubeCon: Foolproof Kubernetes Dashboards for Sleep-Deprived On Calls that recommends Hierarchical Dashboards.

Our dashboards are organized in this order just on two levels. We have the main one called oncall for overview with summary for traffic, databases, etc. and if you click on that in the main dashboard that opens for you a more detailed dashboard with specific information and you can select region, availability zone/domain or application in a template.

We are using a jsonnet standard library to generate the list for templates that are big and easy to maintain.

.addTemplate(
template.new(
'id',
'$datasource',
'map($app;' + std.manifestJsonEx(dataMap, '') + ')',
hide = 2,
refresh = 1,
)
)

and if we get issues with newlines in a map we can easily fix using standard library too and remove newlines:

.addTemplate(
template.new(
'id',
'$datasource',
'map($app;' + std.strReplace(std.manifestJsonEx(dataMap, ''),'\n','') + ')',
hide = 2,
refresh = 1,
)
)

Making these changes manually in many places would be exhausting.

Summary

The advantage of using jsonnet is more organized code for dashboards. We try to make it dry and if we have some graph type in the dashboard more times, we create a function to generate it with parameters that are different, such as application name. All code is directly in the dashboard jsonnet file. You can move that to separate files if you need it in more places.

The disadvantage of this solution is the learning curve, people don’t know jsonnet, and there isn’t much documentation on how to learn it. But from the year of experience, I see more benefits.

Updating README in Github personal repository

ladislav@prskavec.net (Prskavec.Net) — Sat, 11 Jul 2020 08:45:17 +0200

Github introduce new feature for repository that have special name as your nickname on Github. If you add README.md there they display content in profile header.

I see first on twitter where my friend mention this. I like it and make similar version. Later I see this tweet about automate content of readme.

I’m not big friend with python and I need something else there that I make my own version in Golang using Github Actions.

First I created template for readme and use go to proceed template. I read RSS/Atom from my blog to update readme and I used just Golang standard library. Output is just printed to stdout.

package main

import (
 "encoding/xml"
 "fmt"
 "io/ioutil"
 "log"
 "net/http"
 "os"
 "text/template"
)

// Item type for RSS
type Item struct {
 Title string `xml:"title"`
 Link string `xml:"link"`
 Desc string `xml:"description"`
 GUID string `xml:"guid"`
 PubDate string `xml:"pubDate"`
}

// Channel type for RSS
type Channel struct {
 Title string `xml:"title"`
 Link string `xml:"link"`
 Desc string `xml:"description"`
 Items []Item `xml:"item"`
}

// Rss type for RSS as root
type Rss struct {
 Channel Channel `xml:"channel"`
}

func readFeed(url string) []string {
 resp, err := http.Get(url)
 if err != nil {
 log.Fatal(err)
 }
 defer resp.Body.Close()

 rss := Rss{}

 decoder := xml.NewDecoder(resp.Body)
 err = decoder.Decode(&rss)
 if err != nil {
 log.Fatal(err)
 }

 output := []string{}
 for _, item := range rss.Channel.Items {
 output = append(output, fmt.Sprintf("[%s](%s)\n", item.Title, item.Link))
 }
 return output
}

func main() {
 content, err := ioutil.ReadFile("README.md.template")
 if err != nil {
 log.Fatal(err)
 }

 text := string(content)
 blogURL := os.Getenv("BLOG_URL")
 rssItems := readFeed(blogURL)
 data := struct {
 Title string
 Items []string
 }{
 Title: "Last blog posts",
 Items: rssItems,
 }

 t, err := template.New("readme").Parse(text)
 if err != nil {
 log.Fatal(err)
 }
 err = t.Execute(os.Stdout, data)
 if err != nil {
 log.Fatal(err)
 }
}

For automating this I created workflow file .github/workflows/build.yml.

name: Build README

on:
 push:
 workflow_dispatch:
 schedule:
 - cron: "0 4 * * *"

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - name: Check out repo
 uses: actions/checkout@v2
 - name: Set up Go
 uses: actions/setup-go@v2
 with:
 go-version: "^1.14.3"
 - run: go version

 - name: Update README
 env:
 BLOG_URL: ${{ secrets.BLOG_URL }}
 run: |-
 go run main.go > README.md
 cat README.md
 - name: Commit and push if README changed
 run: |-
 git diff
 git config --global user.email "readme-bot@github.com"
 git config --global user.name "README-bot"
 git diff --quiet || (git add README.md && git commit -m "Updated README")
 git push

I run cron daily, you can change as you need it. I run my program go run main.go and stdout is forward into file and commit into repository if diff exists. Github Actions manage access into repository and you don’t need add ssh key or something else. The blog URL is defined in repository <repo>/settings/secrets with name BLOG_URL, but can be directly set in this yaml file too as normal environment variable.

Running Terratest in CircleCI

ladislav@prskavec.net (Prskavec.Net) — Wed, 06 Mar 2019 00:00:00 +0000

Running golang test on CircleCI is easy but if you need DIND (docker in docker) that is made by setup_remote_docker is all easy and you have something like this:

version: 2
jobs:
 build:
 docker:
 - image: circleci/golang:1.13
 environment:
 TEST_RESULTS: /tmp/test-results
 steps:
 - checkout
 - setup_remote_docker
 - run: mkdir -p $TEST_RESULTS
 - restore_cache:
 keys:
 - v1-pkg-cache
 - run: go get gotest.tools/gotestsum
 - run: docker pull appropriate/curl
 - run: cd test && go mod download
 - run:
 name: Run unit tests
 command: |
 make test-unit
 - run: cp tmp/unit-report.xml /tmp/test-results/unit-report.xml
 - save_cache:
 key: v1-pkg-cache
 paths:
 - "/go/pkg"
 - store_artifacts:
 path: /tmp/test-results
 destination: raw-test-output
 - store_test_results:
 path: /tmp/test-results

But the issue is the network that is limit for security reasons and this helper will not work in DIND mode.

import http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
// It can take a few seconds for the Docker container boot up, so retry a few times
maxRetries := 5
timeBetweenRetries := 2 * time.Second
url := fmt.Sprintf("http://localhost:%d", serverPort)

// Setup a TLS configuration to submit with the helper, a blank struct is acceptable
tlsConfig := tls.Config{}

// website::tag::5::Verify that we get back a 200 OK with the expected text
http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries)

you need to replace http_helper with docker container with curl:

opts := &docker.RunOptions{
 Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"},
 OtherOptions: []string{"--network", "testdockercomposelocal_teststack-network"},
 }
tag = "appropriate/curl"
output := docker.Run(t, tag, opts)
assert.Equal(t, expectedServerText, output)

The important thing is to have the same network with docker-compose that I’m using in the application and this container with curl with correct network parameter. Curl have advantage for support an exponential backoff algorithm.

New start in 2020

ladislav@prskavec.net (Prskavec.Net) — Wed, 20 Apr 2016 00:00:00 +0000

I clean up old posts and restart this with a new look. I can focus on some best practices that I collected over a year that can help multiple people.

This blog is powered by JAMStack. I’m using static generator Hugo and theme Academic.

My workflow using Github and deploy and publish is using Netlify. Netlify makes a preview site for every PR and that helps me review every change in site.