Commit Graph

149 Commits

Author SHA1 Message Date
William Casarin
af5961ce26 nostrdb: query: add missing since check to kind query
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
58de0025aa nostrdb: monitor: lock monitor when we're freeing subscriptions
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
c931108741 nostrdb: subs: fix memory leak in ndb_subscribe
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
20255198fd nostrdb: bug: add missing break statement
probably harmless but it writes the note twice...

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
289a8e262a nostrdb: migrations: make migrations asyncronous
This also seems to fix some issues with older migrations.

Fixes: https://github.com/damus-io/nostrdb/issues/58
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
05baba9c03 nostrdb: flags: make some indexes optional
Make fulltext indices and note blocks optional. This will be useful for
quickly building databases when testing, since more stuff in the write
queue when writing can slow things down.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
e0461d3458 nostrdb: writer: rename any_note to needs_commit
This is a bit more clear as to what this variable actually means

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
62aa72c215 nostrdb: leak: fix memory leak when failing to write like stats
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
287b35a8fb nostrdb: migration: dont fail v3 -> v4 on 0 migrations
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
478d7b4060 nostrdb: add authors query plan
This fixes author queries

Fixes: https://github.com/damus-io/nostrdb/issues/52
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
2c4728508b nostrdb: earlier since check in ndb_query_plan_execute_created_at
this avoids a lookup if we dont need it

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
d24a3f0ce5 nostrdb: simplify ndb_query_plan_execute_ids
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
efba599779 nostrdb: ids: fix typo in ndb_query_plan_execute_ids
We should be specifying that we've matched the id here, not authors. Not
that this would have any effect.. but still.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
19243d49e1 nostrdb: always show migration text
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
6845d0df47 nostrdb: migrate notes to have pubkey indices
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
8e79ad582a nostrdb: add note pubkey and pubkey_kind indices
We need these for profile queries

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
282c02eed4 nostrdb: add ndb_db_is_index
This function can be used to check if a db is an index or not. We
will use this in future functions that rebuild indices.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
155ac27bb5 nostrdb: introduce ndb_id_u64_ts
This will be the key used by our note_profile_kind indee

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
be1d149f4b nostrdb: misc: move some functions around
because this will make the changes nicer

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
9e0dc47e98 nostrdb: rename: ndb_u64_tsid to ndb_u64_ts
technically more accurate. we are about to introduce a new type called:

	ndb_ts_u64_id

which would be confusing if we didn't do this

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
0916b14b32 nostrdb: make the subscription monitor threadsafe
This was the only thing that wasn't threadsafe. Add a simple mutex
instead of a queue so that polling is quick.

This also means we can't really return the internal subscriptions
anymore, so we remove that for now until we have a safer
interface.

Fixes: https://github.com/damus-io/nostrdb/issues/55
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
6818d001f2 nostrdb: mem: reduce default queue size
This was overkill and was using lots of memory

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
4bf9160502 nostrdb: fix heap corruption on windows
windows thinks this is heap corruption... so I
guess we have to trust it.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
02df1e209b nostrdb: windows: fix threading bugs
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3186b0e1d3 nostrdb: fix windows build
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
de0935582c nostrdb: ndb_filter_{eq,is_subset_of}: make interfaces const
this makes rust happier

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
573de6b881 nostrdb: ndb_filter_is_subset_of
subset testing for filters. Can be used to see if one subset is
redundant in the presence of a another in the local relay model

Changelog-Added: Add ndb_filter_is_subset_of
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
44ab702792 nostrdb: add ndb_filter_eq
filter equality testing. this works because field elements are sorted

Changelog-Added: Add ndb_filter_eq for filter equality testing
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
1fdf234c46 nostrdb: rename get_elems to find_elements
This is more accurate

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3018200e95 nostrdb: add ndb_subscription_filters
Expose a way to get the set of filters for a subscription. On the rust
side, we should likely ndb_filter_clone each filter asap, because the
result of this function will only be valid up until the subscription
ends.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
47b79fc02e nostrdb: ingest: support kind 6 reposts
This also enables processing raw json via ndb import

Fixes: https://github.com/damus-io/nostrdb/issues/46
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
ddd30054e8 nostrdb: nostrdb: fix ndb_builder_find_str.
This will find strings which match the beginning of other strings,
which seems wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
8c446f804c nostrdb: filter: retain const variant of get_int_elemnet
otherwise rust gets bitchy at as

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
e92018aee5 nostrdb: filter: allow mutable int elements
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b4b84e6895 nostrdb: resync with repo
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
6001063754 nostrdb: nostrdb: fix ndb_builder_find_str.
This will find strings which match the beginning of other strings,
which seems wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
827731b9cb nostrdb: filter: retain const variant of get_int_elemnet
otherwise rust gets bitchy at as

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
56d44d0004 nostrdb: filter: allow mutable int elements
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
bbed448ccb nostrdb: ndb_filter_from_json
Changelog-Added: Add method for parsing filter json
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
fc30b68c40 nostrdb: Makefile: build using ccan/ versions of files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b326f007f2 nostrdb: expose filter introspection methods
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
a86d8416fc nostrdb: expose ndb_filter_get_elements
This can be used to iterate though filter elements

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b5c57dc935 nostrdb: make more things const
rust is happier this way

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
7d6814a481 nostrdb: add ndb_filter_json method
Changelog-Added: Add ndb_filter_json method for creating json filters
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
8dd048681b nostrdb: Fix issue where id tag filters are pushed as strings
When creating filters, sometimes IDs are pushed as strings, so if there
is ever a 0 byte, the id prematurely ends, causing the filter to not
match

Fixes: https://github.com/rust-nostr/nostr/issues/454
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3171959d85 nostrdb: debug: improve tag index display
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
57db252783 nostrdb: ndb_note_json: return length
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
92e1e4b08f nostrdb: api: add ndb_note_json
add a way to write an ndb note as json to a buffer

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
ffc50bb2c1 nostrdb: fix realloc corruption
can't figure out why this is happening, but let's disable it for now
while we test. we shouldn't hit this code path anyways once we switch
over to local notes in damus ios

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
a562be009d nostrdb: add ability to register a subscription callback
Since Damus iOS is not an immediate-mode UI like android, we would
rather not poll for results. Instead we need a way to register a
callback function that is called when we get new subscription results.

This is also useful on the android side, allowing us to request a new
frame to draw when we have new results, instead of drawing every second.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00