When Your Markdown Notes Need to Act Like a Database: Diving into SilverBullet
I've been using Markdown for notes and documentation for years, but it always felt like I was leaving so much on the table. How do you query across files? How do you create dynamic dashboards? SilverBullet finally tackles this, turning simple Markdown into a surprisingly powerful, queryable knowledg
by Sunil Band
Your Notes Are Data, But Your Tools Don't Treat Them That Way
We all take notes. Whether it's daily logs, project documentation, or just jotting down thoughts, Markdown has become the de facto standard for its simplicity and portability. But here's the rub: once you accumulate a few hundred Markdown files, they start feeling less like an organized knowledge base and more like a pile of text. Finding connections, aggregating information, or building dynamic views across your notes becomes a manual, often frustrating, process.
Traditional note-taking apps often force you into proprietary formats or rigid database schemas. Plain text files give you freedom but lack structure. What if you could have the best of both worlds? What if your Markdown files weren't just static documents, but living records that you could query, link, and transform, much like data in a database? That's the promise of SilverBullet.
SilverBullet isn't just another Markdown editor. It's a self-hosted, programmable knowledge base that treats your Markdown files as its primary data store. It brings database-like capabilities, scripting, and a powerful query language directly into your plain text workflow. This fundamentally changes how you interact with your notes, elevating them from mere text to actionable data.
The Problem with Plain Markdown
Think about a typical Markdown setup: you have files organized in folders, maybe some tags using frontmatter. When you want to see all tasks due next week, or all notes related to a specific client, you're usually resorting to grep or your editor's global search. This works, but it's not intelligent. It can't understand relationships, filter by specific properties, or aggregate data.
For example, imagine you have meeting notes, project plans, and daily logs, all in Markdown. You might want to:
- List all
[[Tasks]]across all projects that arestatus: pending. - Generate a report of
[[Client]]meetings from the last month, showing who attended. - Create a dashboard of
[[Project]]progress based onprogress: 75%frontmatter fields.
Doing this with vanilla Markdown is a nightmare. You'd need custom scripts, external tools, or just a lot of manual copy-pasting. SilverBullet offers a far more elegant solution.
Querying Your Knowledge Graph
SilverBullet's core innovation lies in its data-oriented approach to Markdown. It parses your Markdown files, extracts metadata, links, and even embedded data structures, and then makes all of this queryable. It uses a powerful query language (inspired by Datalog and Datomic) that lets you ask sophisticated questions about your notes.
Let's start with a simple example. Imagine you have a bunch of notes, some with frontmatter like this:
---
tags: [meeting, project-alpha]
dproject: Project Alpha
attendees: [Sunil, Alice, Bob]
date: 2023-10-26
---
## Project Alpha Standup
- Discussed sprint progress.
- Alice to follow up on API integration.And another note:
---
tags: [task, project-alpha]
dproject: Project Alpha
status: pending
due: 2023-11-03
assignee: Alice
---
## API Integration Task
- Need to integrate with the new auth endpoint.
- See [[Project Alpha Standup]] for context.With SilverBullet, you can embed queries directly into your Markdown pages. Here's how you might list all pending tasks for 'Project Alpha':
```query
page
| where tags = "task"
| where dproject = "Project Alpha"
| where status = "pending"
| select name, due, assignee
When you render this page in SilverBullet, that query block executes and displays a dynamic table:
| name | due | assignee |
| :------------------ | :--------- | :------- |
| API Integration Task | 2023-11-03 | Alice |
This is incredibly powerful. Your dashboard page can now dynamically pull in relevant tasks from anywhere in your vault, without you manually updating it. The `d` prefix on `dproject` or `dname` refers to **data properties**, which SilverBullet extracts from frontmatter or specific inline syntaxes. `name` is a special property referring to the page title.
### Beyond Simple Queries: Linking and Transclusion
SilverBullet goes further. It understands **page links** (`[[Page Name]]`) and **block links** (`[[Page Name#Block Name]]`). You can use these to build a true knowledge graph. More importantly, you can **transclude** content.
Want to pull a specific section from another note into your daily log? Just use a special syntax:
```markdown
## Daily Log 2023-10-27
### Important Updates
[[Project Alpha Standup#Discussed sprint progress.]]
### Today's Tasks
```query
page
| where tags = "task"
| where assignee = "Sunil"
| where status = "pending"
| select name, due
This means you're not just linking *to* information, you're embedding *live* information. If the source content changes, your transcluded view updates automatically. This is a game-changer for maintaining consistency across documentation and dashboards.
## Programmability with Lua
Queries are great for retrieving and displaying data, but what if you need more complex logic? This is where SilverBullet's **Lua scripting** comes in. You can embed Lua code blocks directly into your Markdown, allowing you to manipulate data, perform calculations, or even interact with SilverBullet's API.
Let's say you want to calculate the number of days until a task is due, or dynamically generate a list of pages based on a more complex condition than the query language can handle. Here's a conceptual example of a Lua block:
```markdown
```lua
local sb = require "silverbullet"
local tasks = sb.query.query("page | where tags = \"task\" and status = \"pending\" | select name, due")
print("## Pending Tasks (Lua Generated)\n")
for _, task in ipairs(tasks) do
local due_date = os.time{year=tonumber(string.sub(task.due, 1, 4)), month=tonumber(string.sub(task.due, 6, 7)), day=tonumber(string.sub(task.due, 9, 10))}
local now = os.time()
local days_left = math.ceil((due_date - now) / (24 * 60 * 60))
print(string.format("- [[%s]] (due in %d days)", task.name, days_left))
end
This Lua script would fetch all pending tasks, calculate the days remaining until their due date, and then print a Markdown-formatted list. This opens up a world of possibilities for custom reporting, data processing, and even simple automation right within your notes.
The integration with Lua isn't just for displaying output; you can also define **commands** and **templates** using Lua, making your knowledge base highly extensible. Need a custom command to archive a task? Write a Lua function. Need a template for new project pages that pre-fills certain frontmatter? Lua can do it.
## The Self-Hosted Advantage and Trade-offs
SilverBullet is designed to be **self-hosted**. You run it locally or on your own server. This is a huge win for privacy and control. Your data stays yours, in plain text Markdown files. You're not locked into a proprietary cloud service, and you're not dependent on an internet connection to access your core knowledge.
However, this also presents a **trade-off**. Setting it up initially requires a bit more effort than signing up for a SaaS app. While the core server is lightweight and easy to run (often just a single command with Docker), managing updates, backups, and ensuring uptime is on you. If you're looking for a zero-config solution, SilverBullet might feel a bit heavy. For developers and power users who value ownership and extensibility, this trade-off is often well worth it.
Another point is the learning curve for the query language and Lua scripting. While powerful, they are not as immediately intuitive as a simple text search. You'll need to invest a little time to grasp the syntax and concepts, but the payoff in terms of dynamic knowledge management is substantial.
## Wrapping up
SilverBullet is a fascinating project that challenges the traditional boundaries of what a note-taking app can be. By treating Markdown as a queryable data source and integrating a powerful scripting language, it transforms your collection of plain text files into a dynamic, programmable knowledge graph. This isn't just about better organization; it's about unlocking new ways to connect, analyze, and act upon the information you've gathered.
If you're tired of your Markdown notes feeling like a static archive and want to experience what it's like to query your personal knowledge base, **clone the SilverBullet repository and try running it locally**. Explore its query language, experiment with embedded Lua, and see how quickly your Markdown files start behaving like a database. It's a fundamental shift in how you can interact with your own thoughts and data.
When Your Database Needs to Think: Adding AI Search with pgvector
Semantic search using embeddings is a game-changer, but integrating it often feels like a separate service problem. What if your existing Postgres database could handle it natively?
When Your Data Visualization Needs to Be More Than Just a Static Chart: Diving into Plotly.js
Tired of static charts that only tell half the story? Plotly.js offers a powerful way to bring your data to life with rich interactivity, letting users explore and understand complex datasets directly in their browser. It's not just about pretty graphs; it's about making data explorable.
When Your E2E Tests Need to Be More Than Just Clicking Buttons: Diving into Playwright's API Capabilities
Most developers use Playwright for UI automation, clicking through pages and asserting on elements. But its underlying API client is a hidden gem, allowing you to combine UI interactions with direct API calls for faster, more robust end-to-end tests.
