Skip to main content

Airbyte Local Deployment

System Requirements​

To experiment with the ELT pipeline via local Airbyte instance, let's first ensure your computer is configured correctly.

Windows​

For Windows users, the Windows Subsystem for Linux 2 (WSL2) must be installed. You must be running Windows 10 version 2004 or higher (build 19041 or higher), or Windows 11 to use the commands in this tutorial.

  1. Open PowerShell or the Windows Command Prompt in administrator mode, then enter the following command to install WSL2:

    wsl --install

    Note: In case the installation command returns "This application requires the Windows Subsystem for Linux Optional Component", you should run:

    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

  2. Restart your machine after installation.

  3. To ensure that WSL2 has been installed successfully, run the following command:

    wsl -l -v

    Expected output:

      NAME                   STATE           VERSION
    * Ubuntu                 Running         2

  4. After you've set up WSL2 successfully, install Docker desktop on Windows by following the steps here.

Mac​

  1. Make sure you have Homebrew installed. If not you can follow the instructions listed here.

  2. Make sure you have Docker Desktop installed. If not you can install it with Homebrew in your terminal:

    brew install --cask docker

Linux​

  1. In order to install Docker and docker-compose on Linux, please follow the instructions listed here and choose your Linux distribution. Remember to also run the post-installation steps listed here.

Local Deployment​

In this step, we'll deploy Airbyte locally on your machine. Make sure you have installed Docker and Docker Compose on your system as shown in the Overview section.

To deploy Airbyte locally, check out the KYVE Airbyte fork

git clone https://github.com/KYVENetwork/airbyte
cd airbyte
git checkout feat/mainnet-support

Then start Airbyte locally by running

./run-ab-platform.sh

After the deployment is complete, you'll be able to access the Airbyte UI at http://localhost:8000/. The default credentials are:

BASIC_AUTH_USERNAME=airbyte
BASIC_AUTH_PASSWORD=password

When you first access the Website, you will be directed to the onboarding screen. Enter an email to proceed.

Airbyte Preferences

Load KYVE source​

You can browse on the Sources page and search for the integrated KYVE source.

Airbyte Source Page
note

For using the latest KYVE source with custom modification you should build and load the source-kyve manually. See the following section for the instructions.

Custom-build KYVE source​

If you wish to test a modified version of the KYVE source connector, follow these steps to build the connector and load it into Airbyte:

  1. In a new terminal window change to the source-kyve directory:

    cd airbyte-integrations/connectors/source-kyve

  2. Build the Docker image for the KYVE source:

    docker build . -t airbyte/source-kyve:dev

  3. In the Airbyte UI, navigate to the settings page and add a new source connector.

    Airbyte Settings

  4. Fill out the fields as follows, then click Add:

    • Connector display name: KYVE (or any other name)
    • Docker repository name: airbyte/source-kyve
    • Docker image tag: dev
    • Connector documentation URL: https://docs.kyve.network/

    Airbyte Add New Connector