Skip to main content

Fauna

Availability
Available Airbyte Cloud Available Airbyte OSS
Support Level
Community
Latest Version
0.1.1
Definition Id
3825db3e-c94b-42ac-bd53-b5a9507ace2b

This page guides you through setting up a Fauna source.

Overview

Availability
Available Airbyte Cloud Available Airbyte OSS
Support Level
Community
Latest Version
0.1.1
Definition Id
3825db3e-c94b-42ac-bd53-b5a9507ace2b

The Fauna source supports the following sync modes:

  • Full Sync - exports all the data from a Fauna collection.
  • Incremental Sync - exports data incrementally from a Fauna collection.

You need to create a separate source per collection that you want to export.

Preliminary setup​

Enter the domain of the collection's database that you are exporting. The URL can be found in the docs.

Full sync​

Follow these steps if you want this connection to perform a full sync.

  1. Create a role that can read the collection that you are exporting. You can create the role in the Dashboard or the fauna shell with the following query:
CreateRole({
  name: "airbyte-readonly",
  privileges: [
    {
      resource: Collections(),
      actions: { read: true }
    },
    {
      resource: Indexes(),
      actions: { read: true }
    },
    {
      resource: Collection("COLLECTION_NAME"),
      actions: { read: true }
    }
  ],
})

Replace COLLECTION_NAME with the name of the collection configured for this connector. If you'd like to sync multiple collections, add an entry for each additional collection you'd like to sync. For example, to sync users and products, run this query instead:

CreateRole({
  name: "airbyte-readonly",
  privileges: [
    {
      resource: Collections(),
      actions: { read: true }
    },
    {
      resource: Indexes(),
      actions: { read: true }
    },
    {
      resource: Collection("users"),
      actions: { read: true }
    },
    {
      resource: Collection("products"),
      actions: { read: true }
    }
  ],
})
  1. Create a key with that role. You can create a key using this query:
CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})
  1. Copy the secret output by the CreateKey command and enter that as the "Fauna Secret" on the left. Important: The secret is only ever displayed once. If you lose it, you would have to create a new key.

Incremental sync​

Follow these steps if you want this connection to perform incremental syncs.

  1. Create the "Incremental Sync Index". This allows the connector to perform incremental syncs. You can create the index with the fauna shell or in the Dashboard with the following query:
CreateIndex({
  name: "INDEX_NAME",
  source: Collection("COLLECTION_NAME"),
  terms: [],
  values: [
    { "field": "ts" },
    { "field": "ref" }
  ]
})

Replace COLLECTION_NAME with the name of the collection configured for this connector. Replace INDEX_NAME with the name that you configured for the Incremental Sync Index.

Repeat this step for every collection you'd like to sync.

  1. Create a role that can read the collection, the index, and the metadata of all indexes. It needs access to index metadata in order to validate the index settings. You can create the role with this query:
CreateRole({
  name: "airbyte-readonly",
  privileges: [
    {
      resource: Collections(),
      actions: { read: true }
    },
    {
      resource: Indexes(),
      actions: { read: true }
    },
    {
      resource: Collection("COLLECTION_NAME"),
      actions: { read: true }
    },
    {
      resource: Index("INDEX_NAME"),
      actions: { read: true }
    }
  ],
})

Replace COLLECTION_NAME with the name of the collection configured for this connector. Replace INDEX_NAME with the name that you configured for the Incremental Sync Index.

If you'd like to sync multiple collections, add an entry for every collection and index you'd like to sync. For example, to sync users and products with Incremental Sync, run the following query:

CreateRole({
  name: "airbyte-readonly",
  privileges: [
    {
      resource: Collections(),
      actions: { read: true }
    },
    {
      resource: Indexes(),
      actions: { read: true }
    },
    {
      resource: Collection("users"),
      actions: { read: true }
    },
    {
      resource: Index("users-ts"),
      actions: { read: true }
    },
    {
      resource: Collection("products"),
      actions: { read: true }
    },
    {
      resource: Index("products-ts"),
      actions: { read: true }
    }
  ],
})
  1. Create a key with that role. You can create a key using this query:
CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})
  1. Copy the secret output by the CreateKey command and enter that as the "Fauna Secret" on the left. Important: The secret is only ever displayed once. If you lose it, you would have to create a new key.

Export formats​

This section captures export formats for all special case data stored in Fauna. This list is exhaustive.

Note that the ref column in the exported database contains only the document ID from each document's reference (or "ref"). Since only one collection is involved in each connector configuration, it is inferred that the document ID refers to a document within the synced collection.

Fauna TypeFormatNote
Document Ref{ id: "id", "collection": "collection-name", "type": "document" }
Other Ref{ id: "id", "type": "ref-type" }This includes all other refs, listed below.
Byte Arraybase64 url formatting
Timestampdate-time, or an iso-format timestamp
Query, SetRefa string containing the wire protocol of this valueThe wire protocol is not documented.

Ref types​

Every ref is serialized as a JSON object with 2 or 3 fields, as listed above. The type field must be one of these strings:

Reference Typetype string
Document"document"
Collection"collection"
Database"database"
Index"index"
Function"function"
Role"role"
AccessProvider"access_provider"
Key"key"
Token"token"
Credential"credential"

For all other refs (for example if you stored the result of Collections()), the type must be "unknown". There is a difference between a specific collection ref (retrieved with Collection("name")), and all the reference to all collections (retrieved with Collections()). This is why the type is "unknown" for Collections(), but not for Collection("name")

To select the document ID from a ref, add "id" to the "Path" of the additional column. For example, if "Path" is ["data", "parent"], change the "Path" to ["data", "parent", "id"].

To select the collection name, add "collection", "id" to the "Path" of the additional column. For example, if "Path" is ["data", "parent"], change the "Path" to ["data", "parent", "collection", "id"]. Internally, the FQL Select is used.

Changelog​

VersionDatePull RequestSubject
0.1.12022-12-1220275Fix index lookup with no values
0.1.02022-11-1715274Add Fauna Source
Was this page helpful?