Developers

Indexer

The indexer is an open-source Node.js library that rebuilds the AXIS orderbook from contract events and serves it to your code and over HTTP.

What it is#

• The npm package @axis-markets/indexer, MIT licensed, with its source at https://github.com/axis-markets/indexer.
• A library you embed in your own Node.js process. It consumes AXIS contract events from a data source you provide, keeps the live orderbook and the backing of every maker in memory, writes orders, trades, swaps and failed transactions to a history storage you provide, and can serve read-only HTTP routes.
• The base of the AXIS API. The hosted service embeds it with its own streaming Stellar RPC data source and adds quotes, depth, candles, the 24h ticker and the WebSocket API, which are not part of the package.

What it is not:

• Not a router. It finds no routes and computes no quotes. It exposes the book so you can build a router on it.
• Not a wallet. It never signs or submits transactions.
• Not a standalone service. There is no binary, CLI or configuration file, and running the package directly does nothing.
• Not a complete stack. It ships no DataSource implementation and only an in-memory HistoryStorage.

How it builds the book#

Contract events#

Each AXIS event changes the book as follows:

A new event does not report the allowance granted in the same call, so with a polling data source the indexer reloads the maker's backing of the sold token after every new. Every touched order is written to the history storage, the contract state is saved after refresh, freeze and config, and every change is emitted as an event. The event layouts are in Events.

Expiration#

The contract emits nothing when an order expires. An order whose expires time has passed simply stops filling. Every expirationInterval (10 seconds by default) the indexer:

1. Takes the expired orders out of the live book.
2. Marks them EXPIRED, writes them to the archive and emits order with the action expire.
3. Stops tracking the backing those orders needed.

The contract keeps the entry of an expired order, so the indexer keeps the order in memory too. Its owner can still revive it with update (a mod with an amount) or remove it (a mod with zero), and a new order can take its ID. A revived order moves back to the active orders as the same record. Expired orders are never served as live: /order/:id returns 404 for them. See Order lifecycle and rent.

Effective depth and backing#

Every order is backed by its maker's balance and allowance in the token it sells, shared by all the maker's orders selling that token (see Backing). The indexer keeps one backing record per maker and token for every live order: the token it sells, which funds it, and the token it buys, which the maker must be able to receive, with room under a trustline limit for the payment (headroom).

• The budget is min(balance, allowance). It is zero while the record is not loaded, when the maker is not authorized for the token, and once the indexed ledger has passed the allowance's liveUntil.
• The budget is split across the maker's live orders selling the token, oldest first. Each order's share is its backed amount in the API.
• A maker's orders in other tokens have their own budgets.

See Effective depth for how routers use these numbers. How the records stay current depends on the data source:

At most concurrency loads run at once. A failed load is logged. A record that never loaded is retried by the next sweep, one that loaded before only once it is older than staleAfter. With a streaming data source there is no sweep, so a failed subscribeBacking stays failed while the pair is tracked.

Restart and resume#

init() restores the indexer from its history storage:

1. It reads the last cursor (getCursor) and the contract state (loadContractState).
2. It loads the active orders and the EXPIRED archive, keeps the newest record per order ID and replays them in creation order. Orders that expired while the indexer was down are archived as EXPIRED without an order event.
3. It starts loading, or subscribing to, the backing of every live order. Backing is not persisted.
4. It starts its timers and calls dataSource.init(network, contractAddress, cursor), which resumes after the stored cursor.

Until the backing of every loaded order has arrived, the missing budgets count as zero and the built-in GET / reports loading.

Install and embed#

pnpm add @axis-markets/indexer

The package is CommonJS. It depends on @stellar/stellar-sdk 17 and, for its HTTP server, on Express 5.

const {Indexer, InMemoryHistoryStorage} = require('@axis-markets/indexer')
const {MyDataSource} = require('./my-data-source') //your DataSource implementation

const indexer = new Indexer({
    dataSource: new MyDataSource(),
    historyStorage: new InMemoryHistoryStorage(), //development only
    network: 'testnet',
    contractAddress: 'CA6P26K4QNNIMTYP22ILTSCXQQDEKNWJIZJPC34YEZYOLBNT7YUPX7XS',
    apiPort: 8070 //optional: serve the built-in HTTP routes
})

indexer.on('order', ({action, order}) => console.log(action, order.id.toString(), order.amount.toString()))
indexer.on('trade', trade => console.log('fill', trade.toJSON()))

indexer.init()
    .then(() => console.log('indexer running'))
    .catch(e => {
        console.error(e)
        process.exit(1)
    })

process.on('SIGTERM', () => {
    indexer.dispose() //does not close the HTTP server
    process.exit(0)
})

Attach your listeners before init(), since the data source can deliver events as soon as it starts. With apiPort set, the routes answer at once, for example curl "http://localhost:8070/order?limit=5&pretty_print". /markets caches its list for 30 minutes, so a request made while the book is still loading keeps showing a partial list.

Options#

The second load exists because a data source may serve ledger state that lags behind its event stream. With a streaming data source the sweep and the second load are off, so refreshInterval, staleAfter and recheckDelay have no effect, and concurrency only bounds the reloads after skip events.

Methods, properties and events#

Methods#

watchBacking is how a service shows balances and allowances of a connected wallet before it places any order. The AXIS API uses it for accounts subscribed on the WebSocket account channel.

Properties#

The dispatcher also exposes the reads behind the HTTP routes: getOrder(id), getOrders(filter), getMarkets(filter), getAccount(owner) and getBacking(filter). serialize(order) returns an order as the API shows it, with backed and backing, and allocateBacking(owner, asset) returns a Map from order ID to backed amount.

Events#

Indexer is an EventEmitter. Payloads are live objects that keep changing, so serialize them before you queue or send them: order.toJSON(), trade.toJSON(), or indexer.dispatcher.serialize(order) to include the backing.

fill comes with the fill and filled actions and describes the fill from the maker's side: {sold, bought, taker, trade, ts}, where sold is the amount of the order's selling token delivered, bought the amount of its buying token received, trade the trade ID and ts the time in UNIX seconds. The fill of a crossfill taker order also carries crossfill: true. Amounts and IDs are bigint.

Reading the book#

dispatcher.graph.sellingGraph.get(x)?.get(y) returns the live orders that a taker selling x can fill to receive y, best price first. Their price is what the taker pays per unit received: the maker's buying per selling, times 10^18. Combine it with the backing to build your own router or depth view:

const {graph} = indexer.dispatcher
const XLM = 'CDLZFC3SYJYDZT7K67VZ75HPJVIEUVNIXF47ZG2FB2RMQQVU2HHGCYSC'
const USDC = 'CBIELTK6YBZJU5UP2WWQEUCYKLPU6AUNZ2BQ4WWFEIE3USCIHMXQDAMA'

//orders selling USDC for XLM, cheapest first
for (const order of graph.sellingGraph.get(XLM)?.get(USDC) ?? []) {
    const backed = indexer.dispatcher.allocateBacking(order.owner, order.selling).get(order.id) ?? 0n
    console.log(order.id, order.price, order.amount, backed)
}

The market vectors hold live orders only. graph.allOrders also holds the expired orders that can still be revived, and graph.isLive(id) tells them apart. graph.lastLedger is the last ledger the indexer has seen.

The DataSource interface#

A data source connects the indexer to the chain. It delivers the AXIS contract events in chain order and reads makers' balances and allowances. The package ships only the interface: implement it on top of Stellar RPC getEvents or any event feed. The data source of the AXIS API, which streams every transaction from Stellar RPC, is part of the hosted service and is not published. Extend the exported DataSource class, which provides defaults for the optional members.

Handlers#

The indexer assigns these handlers in init(), before it calls dataSource.init(). Call them one event at a time, in chain order. The indexer applies every event as it arrives and does not deduplicate.

Data source methods#

A BackingRecord has these fields:

Streaming data sources#

A data source that follows ledger state itself can push backing changes instead of being polled:

A streaming source still implements loadBacking, which the indexer calls after each skip event, once for each token of the skipped order.

Event objects#

Every event object carries these fields:

The other fields depend on the event:

IDs, prices and amounts are bigint, and addresses are strings (G..., C...).

A FailureEvent describes a failed AXIS call rather than an event, so it has position, ledger and ts but no cursor. Its position is the transaction position plus the index of the failed call within it, and it becomes the Failure ID, which is the deduplication key and the /failures cursor. The call may come from the transaction or from another contract, and caught: true marks a failure that a calling contract caught, in a transaction that succeeded. Its other fields are txHash, fn, caller, orders, takerOrder (for crossfill), result, reason (contract, transfer, resources, auth or unknown), and, when the diagnostic events show them, error ({contract, code, name?}) and transfer ({token, fn, from, to, amount}). The indexer stores it as a Failure, served by GET /failures.

Skeleton implementation#

An outline of a polling data source on Stellar RPC, without retries, rate limiting or gap handling. Decoding follows Decoding with Stellar RPC. An RPC event ID has the form <toid>-<event index>, so shifting the TOID left by 16 bits and adding the index gives a monotonic position. This scheme yields the same trade and swap IDs as the AXIS API.

const {DataSource} = require('@axis-markets/indexer')
const {rpc, scValToNative} = require('@stellar/stellar-sdk')

class RpcEventSource extends DataSource {
    constructor({rpcUrl, startLedger}) {
        super()
        this.server = new rpc.Server(rpcUrl)
        this.startLedger = startLedger //first ledger to scan on a fresh index
    }

    async init(network, contractAddress, cursor) {
        this.contractAddress = contractAddress
        this.pageCursor = cursor //undefined on a fresh index
        this.timer = setInterval(() => this.poll().catch(e => this.onError(e)), 5000)
    }

    async poll() {
        if (this.polling)
            return
        this.polling = true
        try {
            const filters = [{type: 'contract', contractIds: [this.contractAddress]}]
            const request = this.pageCursor ?
                {filters, cursor: this.pageCursor, limit: 200} :
                {filters, startLedger: this.startLedger, limit: 200}
            const {events, cursor} = await this.server.getEvents(request)
            for (const event of events) {
                this.dispatch(event)
            }
            this.pageCursor = cursor
        } finally {
            this.polling = false
        }
    }

    dispatch(event) {
        const [name, ...topics] = event.topic.map(topic => scValToNative(topic))
        const data = scValToNative(event.value)
        const [toid, index] = event.id.split('-')
        const base = {
            cursor: event.id,
            position: BigInt(toid) << 16n | BigInt(index),
            ledger: event.ledger,
            ts: Math.floor(Date.parse(event.ledgerClosedAt) / 1000)
        }
        switch (name) {
            case 'new': {
                const [id, owner, price, amount, expires] = data
                const [selling, buying] = topics
                return this.onOrderEvent({...base, action: 'new', id, owner, selling, buying, price, amount, expires: Number(expires)})
            }
            case 'mod': {
                const [id, price, amount, expires] = data
                return this.onOrderEvent({...base, action: 'mod', id, price, amount, expires: Number(expires)})
            }
            case 'trade': {
                const [order, taker, maker, sold, bought, left] = data
                const [soldAsset, boughtAsset] = topics
                return this.onTradeEvent({...base, id: base.position, order, taker, maker, soldAsset, boughtAsset, sold, bought, left})
            }
            case 'swap': {
                const [trader, sold, bought] = data
                const [soldAsset, boughtAsset] = topics
                return this.onSwapEvent({...base, id: base.position, trader, soldAsset, boughtAsset, sold, bought})
            }
            case 'skip':
                return this.onSkipEvent({...base, order: data})
            case 'refresh':
                return this.onMarketEvent({...base, base: topics[0], quote: topics[1]})
            case 'freeze':
                return this.onFreezeEvent({...base, frozen: data})
            case 'config':
                return this.onConfigEvent({
                    ...base,
                    safetyAdmin: data.safety_admin,
                    oracle: data.oracle,
                    listingMinDays: data.listing_min_days,
                    marketListingFee: data.market_listing_fee,
                    minTradeSize: data.min_trade_size,
                    ledgerTime: data.ledger_time
                })
        }
    }

    async loadBacking(asset, owner, spender) {
        //read the spendable balance and the trustline authorization of `owner` in `asset`,
        //and allowance(owner, spender) from the token contract, for example with simulated calls
        throw new Error('Not implemented')
    }

    async dispose() {
        clearInterval(this.timer)
    }
}

module.exports = {RpcEventSource}

getEvents returns events after the cursor it receives, and its response cursor moves past the scanned ledgers even when they hold no AXIS event. The cursor stored with each event is the event ID, which RPC also accepts as a cursor, so a restart resumes right after the last stored cursor. A skip or a failure that arrived after the last stored write is replayed, which is harmless: a skip only reloads backing and failures are deduplicated by ID.

The HistoryStorage interface#

The history storage persists what the indexer cannot rebuild from memory: active and archived orders, the trade log, the contract state and the cursor. Extend the exported HistoryStorage class.

Write methods#

Implement them as async methods, or return a Promise. The indexer calls them without awaiting them, calls .catch() on the result and only logs a failure. The same applies to dispose() of the data source and of the storage.

cursor is the data source cursor of the last processed event. Order IDs can be reused once an order is gone, so the archive can hold several records with the same ID, told apart by position.

Read methods#

pair is the canonical key returned by the exported toPair(x, y), base/quote. Cursors, IDs and positions are bigint, and every cursor is exclusive. At startup the indexer pages through loadActiveOrders and through loadArchivedOrders with status: 3, so these queries must be complete and stable.

When you implement a storage on a database:

• Write each record and its cursor in one transaction. After a crash the indexer resumes after the stored cursor, so a record saved without its cursor is replayed, and a cursor saved without its record loses that record.
• Apply the writes in call order and stop the process when a write fails, rather than let memory and storage diverge.
• Make storeTrade idempotent by id if your data source can deliver an event twice.
• Index orders by position, owner, pair and status, and the trade log by id, pair, taker and maker.
• Without storeContractState and loadContractState, the contract state starts empty after a restart and stays incomplete until new config, freeze and refresh events arrive.

Records#

The load methods must return instances of the exported entry classes, not plain objects: the indexer calls isExpired() and applyExpiration() on replayed orders, and the history routes call toJSON() on every row.

Trade.fromEvent, Swap.fromEvent and Failure.fromEvent rebuild a stored record from its fields. Order has no deserializer: assign the stored fields to new Order(). IDs, amounts and positions are bigint.

InMemoryHistoryStorage#

InMemoryHistoryStorage ships with the package. It keeps everything in process memory, scans linearly and loses all data on restart, after which the next init starts from an undefined cursor. Use it for tests and development, and implement HistoryStorage on a database for production. It serves orders newest first by position, trades, swaps and failures newest first by id, and stores each trade and failure once per id.

Built-in HTTP routes#

With apiPort, the indexer starts an Express server with open CORS, ?pretty_print and the JSON error shape of the REST API. It serves these read-only routes:

Parameters, limits and response shapes match the AXIS API. To mount the routes in your own Express app instead, call orderbookRoutes(app, indexer) (markets, orders, accounts, backing and contract) and historyRoutes(app, indexer) (order history, trades and failures). apiPort itself calls the exported initApiServer(indexer, port), which listens on all interfaces. GET / is not part of the exported route sets: build your own status route from indexer.dispatcher.ready, indexer.dispatcher.graph.lastLedger and indexer.contractState.frozen.

The package also exports OrderBookDispatcher, BackingTracker, ContractState, the entry classes, toPair, canonicalPair, compareAssets and PRECISION.

Operational notes#

• Memory. The indexer holds every live order, every expired order that can still be revived and the backing records in memory. An expired order stays until its owner removes or revives it, or a new order takes its ID. History lives in the storage.
• Restarts. With a persistent storage, a restart replays the stored orders, reloads the backing and resumes the data source from the stored cursor, so only the downtime is caught up.
• Event retention. Stellar RPC servers keep events for a limited window. A stored cursor older than that window cannot be resumed. A streaming data source can expose cursor, which the indexer saves after every ledger, so the resume point stays recent while the contract is quiet. A polling data source moves the stored cursor only with AXIS events.
• Fresh index. The book is built from events only. Start a new index at or before the contract's first event, or from an event feed with full history. Orders created before the starting point are missing, and their later fills cannot be applied.
• Switching data sources. Cursors are opaque to the indexer and specific to each data source, and positions (trade IDs and order cursors) can be computed differently. Start a fresh index with an empty storage when you change the data source, the network or the contract.
• Clock. Expirations use the host clock, so keep it synchronized.
• Shutdown. dispose() does not stop the server started by apiPort. Mount the routes in your own server if you need a clean shutdown.

For the place of the indexer in the system, see Architecture.