Commit Graph

82 Commits

Author SHA1 Message Date
William Casarin
155ac27bb5 nostrdb: introduce ndb_id_u64_ts
This will be the key used by our note_profile_kind indee

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
be1d149f4b nostrdb: misc: move some functions around
because this will make the changes nicer

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
9e0dc47e98 nostrdb: rename: ndb_u64_tsid to ndb_u64_ts
technically more accurate. we are about to introduce a new type called:

	ndb_ts_u64_id

which would be confusing if we didn't do this

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
0916b14b32 nostrdb: make the subscription monitor threadsafe
This was the only thing that wasn't threadsafe. Add a simple mutex
instead of a queue so that polling is quick.

This also means we can't really return the internal subscriptions
anymore, so we remove that for now until we have a safer
interface.

Fixes: https://github.com/damus-io/nostrdb/issues/55
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
6818d001f2 nostrdb: mem: reduce default queue size
This was overkill and was using lots of memory

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
4bf9160502 nostrdb: fix heap corruption on windows
windows thinks this is heap corruption... so I
guess we have to trust it.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
02df1e209b nostrdb: windows: fix threading bugs
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3186b0e1d3 nostrdb: fix windows build
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
de0935582c nostrdb: ndb_filter_{eq,is_subset_of}: make interfaces const
this makes rust happier

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
573de6b881 nostrdb: ndb_filter_is_subset_of
subset testing for filters. Can be used to see if one subset is
redundant in the presence of a another in the local relay model

Changelog-Added: Add ndb_filter_is_subset_of
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
44ab702792 nostrdb: add ndb_filter_eq
filter equality testing. this works because field elements are sorted

Changelog-Added: Add ndb_filter_eq for filter equality testing
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
1fdf234c46 nostrdb: rename get_elems to find_elements
This is more accurate

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3018200e95 nostrdb: add ndb_subscription_filters
Expose a way to get the set of filters for a subscription. On the rust
side, we should likely ndb_filter_clone each filter asap, because the
result of this function will only be valid up until the subscription
ends.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
47b79fc02e nostrdb: ingest: support kind 6 reposts
This also enables processing raw json via ndb import

Fixes: https://github.com/damus-io/nostrdb/issues/46
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
ddd30054e8 nostrdb: nostrdb: fix ndb_builder_find_str.
This will find strings which match the beginning of other strings,
which seems wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
8c446f804c nostrdb: filter: retain const variant of get_int_elemnet
otherwise rust gets bitchy at as

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
e92018aee5 nostrdb: filter: allow mutable int elements
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b4b84e6895 nostrdb: resync with repo
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
6001063754 nostrdb: nostrdb: fix ndb_builder_find_str.
This will find strings which match the beginning of other strings,
which seems wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
827731b9cb nostrdb: filter: retain const variant of get_int_elemnet
otherwise rust gets bitchy at as

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
56d44d0004 nostrdb: filter: allow mutable int elements
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
bbed448ccb nostrdb: ndb_filter_from_json
Changelog-Added: Add method for parsing filter json
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
fc30b68c40 nostrdb: Makefile: build using ccan/ versions of files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b326f007f2 nostrdb: expose filter introspection methods
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
a86d8416fc nostrdb: expose ndb_filter_get_elements
This can be used to iterate though filter elements

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b5c57dc935 nostrdb: make more things const
rust is happier this way

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
7d6814a481 nostrdb: add ndb_filter_json method
Changelog-Added: Add ndb_filter_json method for creating json filters
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
8dd048681b nostrdb: Fix issue where id tag filters are pushed as strings
When creating filters, sometimes IDs are pushed as strings, so if there
is ever a 0 byte, the id prematurely ends, causing the filter to not
match

Fixes: https://github.com/rust-nostr/nostr/issues/454
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3171959d85 nostrdb: debug: improve tag index display
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
57db252783 nostrdb: ndb_note_json: return length
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
92e1e4b08f nostrdb: api: add ndb_note_json
add a way to write an ndb note as json to a buffer

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
ffc50bb2c1 nostrdb: fix realloc corruption
can't figure out why this is happening, but let's disable it for now
while we test. we shouldn't hit this code path anyways once we switch
over to local notes in damus ios

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
a562be009d nostrdb: add ability to register a subscription callback
Since Damus iOS is not an immediate-mode UI like android, we would
rather not poll for results. Instead we need a way to register a
callback function that is called when we get new subscription results.

This is also useful on the android side, allowing us to request a new
frame to draw when we have new results, instead of drawing every second.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
30c9bc7db7 nostrdb: add ndb_unsubscribe
We didn't have a way to unsubscribe from subscriptions. Now we do!

Apps like notecrumbs may open up many local subscriptions based on
incoming requests. We may need to make the MAX_SUBSCRIPTIONS size much
larger, but this should be okish for now.

Changelog-Added: Add ndb_unsubscribe to unsubscribe from subscriptions
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
0ac03df841 nostrdb: build: fix compile warning
A small size_t/uint64 conversion issue

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
cc9585b6e3 nostrdb: plan: use a less efficient plan for author query plans
This is less efficient for now but we don't have a small-author-list
query plan yet.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
bd17dcfac6 nostrdb: plan: add created_at query plan
This introduces the basic created_at query plan. We scan the created_at
+ id index in descending order looking for items that match a filter.
This is a very general query plan, but might not be very efficient for
anything other than local timelines.

Changelog-Added: Add general created_at query plan for timelines
Closes: https://github.com/damus-io/nostrdb/issues/26
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
560e9e53cd nostrdb: fix a few note size compile issues
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
2d5f86b142 nostrdb: filter: make sure clone copies metadata
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
89686d758a nostrdb: filter: make sure to return clone errors
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
6c26add1da nostrdb: filter: add ndb_filter_clone
Clone filters when moving them into subscriptions. This will allow us to
fix the double free issue on the rust side.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3c5a83392e nostrdb: filter: use relative data offsets for easy cloning
Instead of storing exact pointers inside of our filter elements, just
store offsets. This will allow us to clone filters very easily without
having to mess around with fixing up the pointers afterwards.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
1c63c3b9bb nostrdb: filter: add ndb_filter_end
This is a pretty scary looking function that realloc our large variable
filter buffer into a compact one. This saves up a bunch of memory when
we are done building the filter.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
0bd4717e01 nostrdb: query: include note size in query results
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
5788c077c4 nostrdb: silence annoying debug
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
1b77b4f0e0 nostrdb: filters: copy filter metadata into subscription
This fixes a few ownership issues

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
62625c6ff3 nostrdb: ndb: add ndb_poll_for_notes
The polling variant of ndb_wait_for_notes. This makes more sense for
realtime apps like notedeck

Changelog-Added: Add ndb_poll_for_notes
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b128330b2a nostrdb: tce: fix build for previous TCE change
Fixes: 34093cd1 ("tce: add AUTH to-client-event")
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Charlie Fish
c1befa5221 nostrdb/tce: add AUTH to-client-event
This was committed to damus, but this should be in nostrdb or else we
will lose it when we update.

Damus: 84cfeb1604 ("nip42: add initial relay auth support")
Link: https://groups.google.com/a/damus.io/g/patches/c/Zx3dk01e0yg/m/t59TsVkXAQAJ
Signed-off-by: Charlie Fish <contact@charlie.fish>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00
William Casarin
8b3c86c5de nostrdb/query: add tag index and tag queries
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:39:59 -07:00