Wallet pagination for auctions
List of auctions, bids and reveals suffers from the same issues as tx history did (#888). This is document describes the potential API and reindexing necessary to get the similar features as the tx history. The indexes are categorized two ways: time index and count index. Count index refers to the ordered list of items, where adding new entries from new blocks does not change the ordering of the previous ones. Wallet TX pagination uses Block height + tx index in the block - for this, and time index is references into this.
Names have four separate endpoints and entries in the db: namestate, auction, bid and reveal. The indexing and pagination should be consistent across these three. Different from txs, names are only indexed for the confirmed txs. All these have two separate indexes for the owned and non-owned. Names and auctions currently are the same, but should be separate.
Time index note: Because all entries are only indexed by block, their time will be same as the block median time. So instead of having separate time indexes, we can introduce new index for median time -> block height, which can be used for querying by time. (Note: Confirmed tx index for time could also be removed and use time -> height index)
TX Pagination uses tx hashes as the cursor, unfortunately same can't be done with the auctions. Same name can appear several, after names expire - which are essentially different auctions. We can instead use block height + tx index (returned via API/Seek) in order to iterate over auctions/bids/reveals. NOTE: TXs use block height + index internally.
Sorted index for the auctions can be fully expressed by combination of height + tx index (as names themselves are not unique and have multiple auctions, we can't use them as cursor). We will go into details per item below.
NOTE: layout keys not chosen for this document, it must be done on implementation step. Listed layouts will use descriptive names instead of one symbol prefixes.
General block time -> height index
layout.heightByTime[time] -> height - Uses monotonic time.
General considerations
- Wherever
height + tx-index occurs, it can be aliased to txhash and use TX pagination entries to look up height and index (TXCount). It makes no sense for some: E.g. auctions. But this could be useful for bids/reveals.
Names (Latest namestate)
Names are the latest state that wallet knows from the chain. This is not an auction, there can be multiple auctions per name. NOTE: This endpoint wont have account to it, as namestate is more general state.
Indexes:
layout.namestateUniqueIndex -> number.layout.namestateByIndex[uniqueNameIndex++] -> namestate
- uniqueIndex is there to make sure, next/prev page does not get shifted when new name is indexed.
This is used to iterate over all names in consistent manner.
layout.namestateOwnedByIndex[uniqueNameIndex] -> dummy
- This allows us to iteretate over owned names separately given unique name index and resolve the name
from the namestateByIndex entries.
layout.namestateIndexByName[nameHash] -> uniqueNameIndex
- Given a name cursor, we can internally resolve the index, this is used by owned/non-owned. Also ensures we don't have name with two indexes.
HTTP API:
-
GET /wallet/:id/name - get latest namestates, but with new parameters.
params
reverse - boolean - should we reverse the list.limit - number - number of namestates to list at once.after - string - list of namestates after the name.owned - boolean - list of namestates that are owned.
- Uses
layout.namestateByIndex for iteration.
- Uses
layout.namestateOwnedByIndex if owned is set.
- Uses
layout.namestateIndexByName[nameHash] to get from cursor to internal iteration.
This endpoint does not have time param, as this refers to the latest state instead of auction.
-
GET /wallet/:id/name/:name - same as it was.
- Uses
layout.namestateIndexByName[nameHash] -> layout.namestateByIndex[index]
Auctions
Auctions are almost same as names, but they can happen several times if the name expires. They can be uniquely identified by the block height + name, but tx-index is there to ensure the ordering follows blockchains ordering in a block.
Indexes:
layout.auctionsByCount[auction-height][tx-index][nameHash] -> namestate
- Can have multiple names under same height.
- This can be used to list all auctions by chronological order.
layout.auctionsByName[nameHash][auction-height][tx-index] -> dummy (uses auctionsByCount to resolve)
- or
layout.auctionsByName[nameHash][auction-height] -> tx-index (tx-index does not matter for sorting)
- Can have multiple heights under same name.
- tx-index is only used for recovering auctionsByCount entry.
Account indexes:
layout.auctionsByCountAccount[account][height][tx-index][nameHash] -> dummylayout.auctionsByNameAccount[account][nameHash][height][tx-index] -> dummy
- or
layout.auctionsByNameAccount[account][nameHash][height] -> tx-index (tx-index does not matter for sorting)
HTTP API:
-
GET /wallet/:id/auction - get all wallet auctions. (Must no longer include bids/reveals)
params:
account - optional string - instead of whole wallet, do specific account.reverse - boolean - should we reverse the list.limit - number - number of auctions to list at once.time - number - list auctions after timestamp(or before reverse)
- Uses
layout.heightByTime[time]
after - height:txindex - cursor is little bit tricky here. Cursor alternatives:
height <<< ceil(log2(MAX_TX_IN_BLOCK)) | tx-index - gives a single number.name:height - this has advantage that the namestate json contains these two.- (Or
afterName, afterHeight)
- Uses
layout.auctionsByCount[height][index] for iteration (with or w/o after).
- Uses
layout.auctionsByCountAccount[height][index] if account is set.
-
GET /wallet/:id/auction/:name - Get all wallet auctions for the name. (Must no longer include bids/reveals)
params
account - optional string - instead of whole wallet, do specific account lookup.reverse - boolean - should we reverse the list.limit - number - number of auctions to list at once.time - number - list auctions after timestamp(or before reverse)
- Uses
layout.heightByTime[time]
after - height.
- Uses
layout.auctionsByName[name][height] to iterate over name auctions (with or w/o after).
- Uses
layout.auctionsByNameAccount[account][nameHash][height] if account is set.
-
GET /wallet/:id/auction/:name/:height - Get specific auction.
params
account - optional string - make sure we check specific account only.
- Uses
layout.auctionsByName[nameHash][height] to grab specific auction.
- Uses
layout.auctionsByNameAccount[account][nameHash][height] if account is set.
This also enables Names endpoint:
GET /wallet/:id/name/:name/:height - to get namestate for specific auction, not latest. Similar to historical proofs, but namestates.
- Uses
layout.auctionsByName[nameHash][height] to get indexes.
Bids
Bids are different from auctions. Block and TX can have multiple bids for the wallet, so if we want sorted set we need to include all this. Another important note: Not all bids for the same name are part of the same auction, so they need to be separated as well.
Indexes:
layout.bidByOutput[height][tx-index][out-index] -> BlindBid
- This does not need auction height, it can be extracted from the BlindBid, sorting is good enough with height.
layout.bidByOutputOwned[height][tx-index][out-index] -> dummylayout.bidByName[nameHash][height][tx-index][out-index] -> dummylayout.bidByNameOwned[nameHash][height][tx-index][out-index] -> dummy
Account indexes:
layout.bidByOutputAccount[account][height][tx-index][out-index] -> BlindBid
- This does not need auction height, it can be extracted from the BlindBid, sorting is good enough with height.
layout.bidByOutputOwnedAccount[account][height][tx-index][out-index] -> dummylayout.bidByNameAccount[account][nameHash][height][tx-index][out-index] -> dummylayout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummy
HTTP API:
This also enables Auctions endpoint:
GET /wallet/:id/auction/:name/:height/bid - get all bids specific to single auction.
params
account filter by account.reverse - boolean - should we reverse the list.limit - number - number of auctions to list at once.time - number - list bids after timestamp
- Uses
layout.heightByTime[time]
after - This same as well: height + tx-index + out-indexowned - Filter the bids that are ours only.
- Uses
layout.auctionsByName[name][height] to get the start height, end height can be grabbed with treeInterval + biddingPeriod
- Uses
layout.bidByName[nameHash][height][tx-index][out-index]
- Uses
layout.bidByNameAccount[account][nameHash][height][tx-index][out-index] if account is set.
- Uses
layout.bidByNameOwned[nameHash][height][tx-index][out-index] if owned is set.
- Uses
layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index] if owned and account is set.
Reveals
Reveals are similar to bids. There can be several reveals in a single block and in a single tx.
Indexes:
layout.revealByOutput[height][tx-index][out-index] -> BidReveal
- This does not need auction height, it can be extracted from the BidReveal, sorting is good enough with height.
layout.revealByOutputOwned[height][tx-index][out-index] -> dummylayout.revealByName[nameHash][height][tx-index][out-index] -> dummylayout.revealByNameOwned[nameHash][height][tx-index][out-index] -> dummy
Account indexes:
layout.revealByOutputAccount[account][height][tx-index][out-index] -> BidReveal
- This does not need auction height, it can be extracted from the BidReveal, sorting is good enough with height.
layout.revealByOutputOwnedAccount[account][height][tx-index][out-index] -> dummylayout.revealByNameAccount[account][nameHash][height][tx-index][out-index] -> dummylayout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummy
HTTP API:
This also enables Auctions endpoint:
GET /wallet/:id/auction/:name/:height/reveal - get all reveals specific to single auction.
params
account filter by account.reverse - boolean - should we reverse the list.limit - number - number of auctions to list at once.time - number - list reveals after timestamp
- Uses
layout.heightByTime[time]
after - This same as well: height + tx-index + out-indexowned - Filter the reveals that are ours only.
- Uses
layout.auctionsByName[name][height] to get the start height, end height can be grabbed with treeInterval + revealdingPeriod
- Uses
layout.revealByName[nameHash][height][tx-index][out-index]
- Uses
layout.revealByNameAccount[account][nameHash][height][tx-index][out-index] if account is set.
- Uses
layout.revealByNameOwned[nameHash][height][tx-index][out-index] if owned is set.
- Uses
layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index] if owned and account is set.
Wallet pagination for auctions
List of auctions, bids and reveals suffers from the same issues as tx history did (#888). This is document describes the potential API and reindexing necessary to get the similar features as the tx history. The indexes are categorized two ways:
timeindex andcountindex. Count index refers to the ordered list of items, where adding new entries from new blocks does not change the ordering of the previous ones. Wallet TX pagination uses Block height + tx index in the block - for this, and time index is references into this.Names have four separate endpoints and entries in the db:
namestate,auction,bidandreveal. The indexing and pagination should be consistent across these three. Different from txs, names are only indexed for the confirmed txs. All these have two separate indexes for the owned and non-owned. Names and auctions currently are the same, but should be separate.Time index note: Because all entries are only indexed by block, their time will be same as the block median time. So instead of having separate time indexes, we can introduce new index for median time -> block height, which can be used for querying by time. (Note: Confirmed tx index for time could also be removed and use time -> height index)
TX Pagination uses tx hashes as the cursor, unfortunately same can't be done with the auctions. Same name can appear several, after names expire - which are essentially different auctions. We can instead use block height + tx index (returned via API/Seek) in order to iterate over auctions/bids/reveals. NOTE: TXs use block height + index internally.
Sorted index for the auctions can be fully expressed by combination of height + tx index (as names themselves are not unique and have multiple auctions, we can't use them as cursor). We will go into details per item below.
NOTE: layout keys not chosen for this document, it must be done on implementation step. Listed layouts will use descriptive names instead of one symbol prefixes.
General block time -> height index
layout.heightByTime[time] -> height- Uses monotonic time.General considerations
height + tx-indexoccurs, it can be aliased totxhashand useTX paginationentries to look upheightandindex(TXCount). It makes no sense for some: E.g. auctions. But this could be useful for bids/reveals.Names (Latest namestate)
Names are the latest state that wallet knows from the chain. This is not an auction, there can be multiple auctions per name. NOTE: This endpoint wont have account to it, as namestate is more general state.
Indexes:
layout.namestateUniqueIndex -> number.layout.namestateByIndex[uniqueNameIndex++] -> namestateThis is used to iterate over all names in consistent manner.
layout.namestateOwnedByIndex[uniqueNameIndex] -> dummyfrom the namestateByIndex entries.
layout.namestateIndexByName[nameHash] -> uniqueNameIndexHTTP API:
GET /wallet/:id/name- get latest namestates, but with new parameters.paramsreverse- boolean - should we reverse the list.limit- number - number of namestates to list at once.after- string - list of namestates after thename.owned- boolean - list of namestates that areowned.layout.namestateByIndexfor iteration.layout.namestateOwnedByIndexifownedis set.layout.namestateIndexByName[nameHash]to get from cursor to internal iteration.This endpoint does not have
timeparam, as this refers to the latest state instead of auction.GET /wallet/:id/name/:name- same as it was.layout.namestateIndexByName[nameHash]->layout.namestateByIndex[index]Auctions
Auctions are almost same as names, but they can happen several times if the name expires. They can be uniquely identified by the block height + name, but tx-index is there to ensure the ordering follows blockchains ordering in a block.
Indexes:
layout.auctionsByCount[auction-height][tx-index][nameHash] -> namestatelayout.auctionsByName[nameHash][auction-height][tx-index] -> dummy(uses auctionsByCount to resolve)layout.auctionsByName[nameHash][auction-height] -> tx-index(tx-index does not matter for sorting)Account indexes:
layout.auctionsByCountAccount[account][height][tx-index][nameHash] -> dummylayout.auctionsByNameAccount[account][nameHash][height][tx-index] -> dummylayout.auctionsByNameAccount[account][nameHash][height] -> tx-index(tx-index does not matter for sorting)HTTP API:
GET /wallet/:id/auction- get all wallet auctions. (Must no longer include bids/reveals)params:account- optional string - instead of whole wallet, do specific account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list auctions after timestamp(or before reverse)layout.heightByTime[time]after-height:txindex- cursor is little bit tricky here. Cursor alternatives:height <<< ceil(log2(MAX_TX_IN_BLOCK)) | tx-index- gives a single number.name:height- this has advantage that the namestate json contains these two.afterName,afterHeight)layout.auctionsByCount[height][index]for iteration (with or w/oafter).layout.auctionsByCountAccount[height][index]ifaccountis set.GET /wallet/:id/auction/:name- Get all wallet auctions for the name. (Must no longer include bids/reveals)paramsaccount- optional string - instead of whole wallet, do specific account lookup.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list auctions after timestamp(or before reverse)layout.heightByTime[time]after-height.layout.auctionsByName[name][height]to iterate over name auctions (with or w/oafter).layout.auctionsByNameAccount[account][nameHash][height]ifaccountis set.GET /wallet/:id/auction/:name/:height- Get specific auction.paramsaccount- optional string - make sure we check specific account only.layout.auctionsByName[nameHash][height]to grab specific auction.layout.auctionsByNameAccount[account][nameHash][height]ifaccountis set.This also enables Names endpoint:
GET /wallet/:id/name/:name/:height- to get namestate for specific auction, not latest. Similar to historical proofs, but namestates.layout.auctionsByName[nameHash][height]to get indexes.Bids
Bids are different from auctions. Block and TX can have multiple bids for the wallet, so if we want sorted set we need to include all this. Another important note: Not all bids for the same name are part of the same auction, so they need to be separated as well.
Indexes:
layout.bidByOutput[height][tx-index][out-index] -> BlindBidlayout.bidByOutputOwned[height][tx-index][out-index] -> dummylayout.bidByName[nameHash][height][tx-index][out-index] -> dummylayout.bidByNameOwned[nameHash][height][tx-index][out-index] -> dummyAccount indexes:
layout.bidByOutputAccount[account][height][tx-index][out-index] -> BlindBidlayout.bidByOutputOwnedAccount[account][height][tx-index][out-index] -> dummylayout.bidByNameAccount[account][nameHash][height][tx-index][out-index] -> dummylayout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummyHTTP API:
GET /wallet/:id/bid- get all wallet bids.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This could be built usingheight+tx-index+out-index.?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.owned- Filter the bids that are ours only.layout.bidByOutput[height][tx-index][out-index]index for iteration.layout.bidByOutputAccount[account][height][tx-index][out-index]ifaccountis set.layout.bidByOutputOwned[height][tx-index][out-index]ifownedis set.layout.bidByOutputOwnedAccount[account][height][tx-index][out-index]ifaccountandownedis set.GET /wallet/:id/bid/:name- get all bids specific to name. (Include all auctions)paramsaccountfilter by account.Account Indexesinstead of normal indexes for all operations.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This is same as all bids:height+tx-index+out-indexowned- Filter the bids that are ours only.layout.bidByName[nameHash][height][tx-index][out-index]index.layout.bidByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.bidByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.This also enables Auctions endpoint:
GET /wallet/:id/auction/:name/:height/bid- get all bids specific to single auction.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This same as well:height+tx-index+out-indexowned- Filter the bids that are ours only.layout.auctionsByName[name][height]to get the start height, end height can be grabbed with treeInterval + biddingPeriodlayout.bidByName[nameHash][height][tx-index][out-index]layout.bidByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.bidByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.Reveals
Reveals are similar to bids. There can be several reveals in a single block and in a single tx.
Indexes:
layout.revealByOutput[height][tx-index][out-index] -> BidReveallayout.revealByOutputOwned[height][tx-index][out-index] -> dummylayout.revealByName[nameHash][height][tx-index][out-index] -> dummylayout.revealByNameOwned[nameHash][height][tx-index][out-index] -> dummyAccount indexes:
layout.revealByOutputAccount[account][height][tx-index][out-index] -> BidReveallayout.revealByOutputOwnedAccount[account][height][tx-index][out-index] -> dummylayout.revealByNameAccount[account][nameHash][height][tx-index][out-index] -> dummylayout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummyHTTP API:
GET /wallet/:id/reveal- get all wallet reveals.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This could be built usingheight+tx-index+out-index.?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.owned- Filter the reveals that are ours only.layout.revealByOutput[height][tx-index][out-index]index for iteration.layout.revealByOutputAccount[account][height][tx-index][out-index]ifaccountis set.layout.revealByOutputOwned[height][tx-index][out-index]ifownedis set.layout.revealByOutputOwnedAccount[account][height][tx-index][out-index]ifaccountandownedis set.GET /wallet/:id/reveal/:name- get all reveals specific to name. (Include all auctions)paramsaccountfilter by account.Account Indexesinstead of normal indexes for all operations.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list reveals after timestamplayout.heightByTime[time]after- This is same as all reveals:height+tx-index+out-indexowned- Filter the reveals that are ours only.layout.revealByName[nameHash][height][tx-index][out-index]index.layout.revealByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.revealByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.This also enables Auctions endpoint:
GET /wallet/:id/auction/:name/:height/reveal- get all reveals specific to single auction.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list reveals after timestamplayout.heightByTime[time]after- This same as well:height+tx-index+out-indexowned- Filter the reveals that are ours only.layout.auctionsByName[name][height]to get the start height, end height can be grabbed with treeInterval + revealdingPeriodlayout.revealByName[nameHash][height][tx-index][out-index]layout.revealByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.revealByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.