Commit Graph

29 Commits

Author SHA1 Message Date
William Casarin
3c5a83392e nostrdb: filter: use relative data offsets for easy cloning
Instead of storing exact pointers inside of our filter elements, just
store offsets. This will allow us to clone filters very easily without
having to mess around with fixing up the pointers afterwards.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
1c63c3b9bb nostrdb: filter: add ndb_filter_end
This is a pretty scary looking function that realloc our large variable
filter buffer into a compact one. This saves up a bunch of memory when
we are done building the filter.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
0bd4717e01 nostrdb: query: include note size in query results
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
62625c6ff3 nostrdb: ndb: add ndb_poll_for_notes
The polling variant of ndb_wait_for_notes. This makes more sense for
realtime apps like notedeck

Changelog-Added: Add ndb_poll_for_notes
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b8bef86ea1 nostrdb: port kernelkind's to the new bech32 parser
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
934ea80f85 nostrdb: blocks: add word count interface
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
588cebd18d nostrdb: header: add ptr helpers for swift
swift is kind of dumb when it comes to opaque pointers

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Charlie Fish
c1befa5221 nostrdb/tce: add AUTH to-client-event
This was committed to damus, but this should be in nostrdb or else we
will lose it when we update.

Damus: 84cfeb1604 ("nip42: add initial relay auth support")
Link: https://groups.google.com/a/damus.io/g/patches/c/Zx3dk01e0yg/m/t59TsVkXAQAJ
Signed-off-by: Charlie Fish <contact@charlie.fish>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00
William Casarin
8b3c86c5de nostrdb/query: add tag index and tag queries
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00
William Casarin
a45f4d3087 nostrdb/Query Plans
Instead of running queries off filters directly, we do some simple
heuristics and determine a reasonable query plan for the given filter.

To test this, also add a kind index query plan and add a test for it.

We still need tag, author, and created_at index scans. This is up next!

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00
William Casarin
77601e77ee nostrdb/filter: rename FILTER_GENERIC to FILTER_TAG
it's a bit more intuitive

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00
William Casarin
9f1b9ab945 nostrdb/Initial nostrdb queries
Still a lot more work to do, but this is at least a proof of concept for
querying nostrdb using filters.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00
William Casarin
8b03ed6175 nostrdb/filters: remove ndb_filter_group from public API
We can just use a list of filters instead when subscribing

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00
William Casarin
4d14ca8d0a nostrdb/filters: add ndb_filter_group_init function
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
81d65cd5bf nostrdb/subs: subs and monitor cleanup
We need to free these resources when we're done with them.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
0df18ae1a4 nostrdb/test: switch reaction test to use subscriptions
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
8c5ec32eaa nostrdb/Initial nostrdb relay subscriptions
This adds some initial code for the nostrdb relay subscription monitor.

When new notes are written to the database, they are checked against
active subscriptions. If any of the subscriptions are matched, the note
primary key is written to the inbox queue for that subscription.

We also add an ndb_wait_for_notes() method that simply waits for notes
to be written by the subscription monitor.

Changelog-Added: Added filter subscriptions
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
c2383060aa nostrdb/blocks: add ndb_blocks_flags function
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
c677233dcb nostrdb/blocks: expose block iterator internals
so we don't need heap allocation. we will be calling this a lot in tight
render loops, we don't want to be allocating on each frame.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
d063362bd7 nostrdb/blocks: write note blocks on ingest
When ingesting notes, parse text/longform contents and store them in nostrdb.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
f2795aa71c nostrdb/blocks: add ndb_blocks_free
In some situations we will need to have owned note blocks. For
example, when we try to fetch note blocks from the database and it's not
there yet. We will need to parse the content on the spot and return an
owned copy, since it will not be immediately available in the database.

Add a new flag field to note blocks that lets us know if it's owned by
malloc or nostrdb.

We the add a free function that checks this flag and frees the object if
its set. If it is not set then it doesn nothing because it likely came
from the database.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
c831976078 nostrdb/blocks: add total_size
Fix this mistake that we have with ndb_notes where we don't know the
total size of the object

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
c2c73c3af6 nostrdb/header: move bech32 around
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
dfa145dd4a nostrdb/parser: fix bech32 block decoding
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
034f2cc02f nostrdb/blocks: add note block iterator
This adds an api that walks along and pulls compact note block data out of
nostrdb.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
d73422db38 nostrdb/content_parser: add initial db decoders
We need to pull the data out as well! Let's add some initial decoders.
We still need tests to make sure it's working.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
e0a2dcf3db nostrdb/Inital embedded content parser
This adds some initial code for nostrdb content parsing.

We still need to write tests for encoding and decoding, so this is
likely not working yet.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
76862776b8 nostrdb/varint: switch to 64 bit varints
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00
William Casarin
1ffbd80c67 nostrdb: move everything to src
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:43 -07:00