Skip to content
Browse documentation
Working with data

Data sources

Load inline data, local or remote files, and database tables.

Choose a data source above, then use its example and parameter table below. CSV and JSON each support inline content and files.

Connect a source

Create an AVA instance, then call source({ type, options }):

typescript
import { AVA } from '@antv/ava';
 
const ava = new AVA({
  llm: { model: 'YOUR_MODEL', apiKey: process.env.OPENAI_API_KEY },
});
 
await ava.source({
  type: 'json',
  options: { data: [{ city: 'Hangzhou', revenue: 120 }] },
});
 
const schema = await ava.schema();
ParameterTypeRequiredMeaning
typeSourceTypeYescsv, csv-file, json, json-file, parquet, excel, text, sqlite, mysql, postgresql, mongodb, supabase, or clickhouse.
optionsSource-specific objectYesUse the matching table below. Field names are relative to options.

source() returns no value. Calling it again replaces the current source and clears its cached schema and profile. Use schema() to inspect the actual table and field names, and call await ava.dispose() when finished.

The examples use Node.js. Supabase and ClickHouse include their required constructor settings alongside the source call. For frontend applications, see browser usage.

CSV

Inline CSV

Pass the CSV content as a string with type: 'csv'.

typescript
await ava.source({
  type: 'csv',
  options: { csv: 'city,revenue\nHangzhou,120\nShanghai,200' },
});
FieldTypeRequiredMeaning / default
csvstringYesRaw CSV content string
optionsCSVReadOptionsNoCSV reader options.

CSV files

Use type: 'csv-file' with a local path or HTTP(S) URL. For a protected URL, supply request headers in options.headers.

typescript
await ava.source({
  type: 'csv-file',
  options: {
    path: 'https://example.com/sales.csv',
    headers: { Authorization: 'Bearer YOUR_TOKEN' },
    options: { header: true, delim: ';' },
  },
});
FieldTypeRequiredMeaning / default
pathstringYesLocal file path or http(s) URL
headersRecord<string, string>NoHTTP headers for remote sources (e.g. Authorization)
optionsCSVReadOptionsNoCSV reader options.

CSV reader options

Use these fields in options.options for csv and csv-file sources with DuckDB. Every field is optional. AVA forwards supplied values to read_csv; omitted values use DuckDB reader defaults or detection. Defaults below describe individual reader settings; automatic detection can determine the effective header, delimiter, and types.

FieldTypeRequiredMeaning / default
headerbooleanNoFirst line of each file contains the column names. Default false
namesstring[]NoColumn names, as a list
columnsRecord<string, string>NoColumn names → types struct, e.g. { col1: 'INTEGER' }. Disables schema auto-detection
typesstring[] | Record<string, string>NoColumn types, by position (list) or by name (struct)
all_varcharbooleanNoSkip type detection and read all columns as VARCHAR. Default false
normalize_namesbooleanNoRemove non-alphanumeric chars from column names; prefix reserved keywords with _
force_not_nullstring[]NoColumns that must not be matched against the NULL string
delimstringNoField delimiter (up to 4 bytes). Default ','
quotestringNoQuote character. Default '"'
escapestringNoEscape character for the quote within quoted values. Default '"'
commentstringNoCharacter used to initiate comments
new_linestringNoNew line character(s): '\r' | '\n' | '\r\n'
auto_detectbooleanNoAuto detect CSV parameters. Default true
auto_type_candidatesstring[]NoTypes the sniffer considers when detecting column types, e.g. ['BIGINT', 'DATE']
dateformatstringNoDate format used when parsing dates
timestampformatstringNoTimestamp format used when parsing timestamps
decimal_separatorstringNoDecimal separator for numbers. Default '.'
thousandsstringNoThousands separator (single char, different from decimal_separator)
nullstrstring | string[]NoString(s) that represent a NULL value
allow_quoted_nullsbooleanNoAllow the conversion of quoted values to NULL. Default true
sample_sizenumberNoNumber of sample lines for auto detection. Default 20480
buffer_sizenumberNoSize of the read buffers in bytes. Default 16 * max_line_size
files_to_sniffnumberNoFiles used by the sniffer for multi-file schema detection; -1 = all. Default 10
max_line_sizenumberNoMaximum line size in bytes. Default 2000000
parallelbooleanNoUse the parallel CSV reader. Default true
skipnumberNoNumber of lines to skip at the start of each file. Default 0
ignore_errorsbooleanNoIgnore any parsing errors encountered. Default false
null_paddingbooleanNoPad short rows with NULL values on the right. Default false
strict_modebooleanNoThrow an error upon encountering any issue. Default true
store_rejectsbooleanNoSkip faulty lines and store them in the rejects table. Default false
rejects_tablestringNoTemp table name for faulty lines. Default 'reject_errors'
rejects_scanstringNoTemp table name for faulty scans. Default 'reject_scans'
rejects_limitnumberNoUpper limit on faulty lines recorded per file; 0 = no limit. Default 0
encodingstringNoFile encoding: 'utf-8' | 'utf-16' | 'latin-1'. Default 'utf-8'
compressionstringNoCompression method: 'none' | 'gzip' | 'zstd'. Default auto (from file extension)
union_by_namebooleanNoAlign columns from different files by name instead of position. Default false
hive_partitioningbooleanNoInterpret the path as a Hive partitioned path. Default auto-detected
filenamebooleanNoAdd the source file path to each row as a filename column. Default false

JSON

Inline JSON

Pass an array of objects with type: 'json'.

typescript
await ava.source({
  type: 'json',
  options: { data: [{ city: 'Hangzhou', revenue: 120 }] },
});
FieldTypeRequiredMeaning / default
dataany[]YesArray of rows to load.

JSON files

Use type: 'json-file' with a local path or HTTP(S) URL.

typescript
await ava.source({
  type: 'json-file',
  options: { path: './sales.json' },
});
FieldTypeRequiredMeaning / default
pathstringYesLocal file path or http(s) URL
headersRecord<string, string>NoHTTP headers for remote sources (e.g. Authorization)

Parquet

Read a Parquet file from a local path or HTTP(S) URL.

typescript
await ava.source({
  type: 'parquet',
  options: { path: './sales.parquet' },
});
FieldTypeRequiredMeaning / default
pathstringYesLocal file path or http(s) URL
headersRecord<string, string>NoHTTP headers for remote sources (e.g. Authorization)

Excel

Read a workbook from a local path or HTTP(S) URL. Each sheet becomes a separate table; call schema() to discover their names.

typescript
await ava.source({
  type: 'excel',
  options: { path: './sales.xlsx' },
});
FieldTypeRequiredMeaning / default
pathstringYesLocal file path or http(s) URL
headersRecord<string, string>NoHTTP headers for remote sources (e.g. Authorization)

Text

Extract structured rows from unstructured text. This source calls your configured model.

typescript
await ava.source({
  type: 'text',
  options: { text: 'Hangzhou revenue was 120. Shanghai revenue was 200.' },
});
FieldTypeRequiredMeaning / default
textstringYesUnstructured text to extract using the configured model.

SQLite

Open a local SQLite database in read-only mode.

typescript
await ava.source({
  type: 'sqlite',
  options: { path: './analytics.sqlite' },
});
FieldTypeRequiredMeaning / default
pathstringYesLocal SQLite database file path.

MySQL

Connect to a database and discover its tables.

typescript
await ava.source({
  type: 'mysql',
  options: {
    host: 'localhost',
    database: 'analytics',
    user: 'reader',
    password: process.env.MYSQL_PASSWORD,
  },
});
FieldTypeRequiredMeaning / default
hoststringYesDatabase hostname.
portnumberNoDefault: 3306.
databasestringYesDatabase to expose.
userstringNoDatabase user; omitted if not supplied.
passwordstringNoDatabase password; omitted if not supplied.
sshSSHOptionsNoOptional SSH tunnel.

PostgreSQL

Connect to a database and discover its tables.

typescript
await ava.source({
  type: 'postgresql',
  options: {
    host: 'localhost',
    database: 'analytics',
    user: 'reader',
    password: process.env.POSTGRES_PASSWORD,
  },
});
FieldTypeRequiredMeaning / default
hoststringYesDatabase hostname.
portnumberNoDefault: 5432.
databasestringYesDatabase to expose.
userstringNoDatabase user; omitted if not supplied.
passwordstringNoDatabase password; omitted if not supplied.
schemastringNoSchema the tables live in (defaults to public)
sshSSHOptionsNoOptional SSH tunnel.

MongoDB

Load collections from the specified database. Use structured connection fields or supply connection, which takes precedence over them.

typescript
await ava.source({
  type: 'mongodb',
  options: {
    connection: process.env.MONGODB_URI!,
    database: 'analytics',
  },
});
FieldTypeRequiredMeaning / default
connectionstringNoAdvanced connection string, e.g. host=localhost port=27017 or mongodb://user:pass@host:27017/db. Takes precedence over all structured fields when provided.
hoststringNoMongoDB hostname or IP address (default localhost).
portnumberNoMongoDB port number (default 27017).
databasestringYesDatabase containing the collections to load.
userstringNoMongoDB username.
passwordstringNoMongoDB password.
authSourcestringNoAuthentication database (MongoDB authSource).
srvbooleanNoUse SRV connection format (for MongoDB Atlas).
tlsbooleanNoEnable TLS/SSL encryption. Alias: ssl.
sslbooleanNoAlias for tls.
tlsCAFilestringNoPath to CA certificate file for TLS.
tlsAllowInvalidCertificatesbooleanNoAllow invalid certificates (testing only, default false).

Supabase

Provide your project reference and a Supabase Management API access token. Your application obtains and refreshes the token.

typescript
const ava = new AVA({
  llm: { model: 'YOUR_MODEL', apiKey: process.env.OPENAI_API_KEY },
  engine: { type: 'supabase' },
});
await ava.source({
  type: 'supabase',
  options: {
    projectRef: 'YOUR_PROJECT_REF',
    accessToken: process.env.SUPABASE_ACCESS_TOKEN!,
  },
});
FieldTypeRequiredMeaning / default
accessTokenstringYesOAuth access token from the Supabase Management API OAuth flow
projectRefstringYesProject reference (as returned by GET https://api.supabase.com/v1/projects)

ClickHouse

Provide a server endpoint and database. Use url or host; url takes precedence when both are present.

typescript
const ava = new AVA({
  llm: { model: 'YOUR_MODEL', apiKey: process.env.OPENAI_API_KEY },
  engine: { type: 'clickhouse' },
});
await ava.source({
  type: 'clickhouse',
  options: {
    url: 'http://localhost:8123',
    database: 'analytics',
    username: 'reader',
    password: process.env.CLICKHOUSE_PASSWORD,
  },
});

Connection fields

FieldTypeRequiredMeaning / default
hoststringhost or urlHost or host:port; AVA adds http:// if the scheme is absent.
urlstringhost or urlFull server URL; takes precedence over host.
portnumberNoOverrides the endpoint's port.
databasestringYesDatabase to discover and query.
usernamestringNoClient username; takes precedence over user. Client default: default.
userstringNoAlias for username.
passwordstringNoClient password; empty by default.

AVA requires an endpoint even though the underlying client has a localhost default.

Forwarded client fields

ClickHouseSourceOptions also accepts the following fields from @clickhouse/client. This table follows the installed client type declarations (1.23.1); these fields are owned by that dependency. AVA forwards them when creating the client.

FieldTypeDefault / meaning
pathnamestringEmpty; extra proxy path.
request_timeoutnumber30000 milliseconds.
max_open_connectionsnumber10 sockets per host.
compressionObjectRequest and response compression; nested fields below.
access_tokenstringUnset; ClickHouse Cloud JWT. Do not combine with username/password.
applicationstringEmpty; application name.
clickhouse_settingsClickHouseSettingsEmpty; server settings map. AVA enforces readonly: '1' on every query, overriding a conflicting setting.
logObjectClient logging options below.
session_idstringEmpty; no explicit server session.
rolestring | string[]Unset; server roles.
http_headersRecord<string, string>Empty; additional request headers.
additional_headersRecord<string, string>Deprecated client alias; use http_headers.
keep_aliveObjectSocket reuse options below.
jsonPartial<JSONHandling>Native JSON.parse / JSON.stringify; callbacks below.
tracerClickHouseTracerUnset; client tracing implementation.
use_multipart_paramsbooleanfalse; always send query parameters as multipart data.
use_multipart_params_autobooleanfalse; use multipart when encoded parameters exceed 4096 characters.
tlsTLS options objectUnset; certificate fields below.
http_agenthttp.Agent | https.AgentUnset; a custom agent supersedes max_open_connections, tls, and keep_alive. Experimental client API.
set_basic_auth_headerbooleantrue; send a Basic Authorization header. Experimental.
capture_enhanced_stack_tracebooleanfalse; capture extended client stacks. Experimental.
max_response_headers_sizenumberUnset; Node.js response header byte limit applies.
connectionConnection<Stream.Readable>Unset; experimental client backend connection. Overrides the default HTTP transport and its options.

All forwarded fields are optional. Server settings are an extensible, server-version-dependent map rather than an AVA-defined finite list. The dependency's ClickHouseSettings type and your server determine accepted keys. Tracer, agent, logger, and connection objects implement the corresponding client contracts; they are not plain AVA configuration objects.

Nested client options

FieldTypeRequired / default
compression.responseboolean | { codec: 'gzip' | 'zstd' | 'br' }Optional; false. true selects gzip.
compression.requestboolean | RequestCompressionOptional; false. true selects gzip.
compression.request.codec'gzip' | 'zstd' | 'br'Required in the object form.
compression.request.levelnumberOptional for gzip/zstd; codec default.
compression.request.qualitynumberOptional for Brotli; default 4.
log.LoggerClassnew () => LoggerOptional; client's default logger.
log.levelClickHouseLogLevelOptional; WARN. Values: TRACE, DEBUG, INFO, WARN, ERROR, OFF.
keep_alive.enabledbooleanOptional; true.
keep_alive.idle_socket_ttlnumberOptional; 2500 ms; 0 disables idle socket management.
keep_alive.eagerly_destroy_stale_socketsbooleanOptional; false.
tls.ca_certBufferRequired when tls is supplied.
tls.certBufferRequired for mutual TLS.
tls.keyBufferRequired for mutual TLS.
json.parse<T>(input: string) => TOptional; parses a JSON string.
json.stringify<T>(input: T) => stringOptional; serializes a value.

Zstd compression requires Node.js 22.15 or newer in this client. Response compression may conflict with the engine's enforced read-only server settings; leave it disabled unless the server permits it.

SSH options

Nested under options.ssh for MySQL and PostgreSQL.

FieldTypeRequiredMeaning / default
hoststringYesSSH server hostname.
portnumberNoDefault: 22.
userstringYesSSH login user.
passwordstringNoSSH login password; no password supplied when omitted.