WeaviateIngestOperator¶

Use the WeaviateIngestOperator to interact with Weaviate APIs to create embeddings for given text and ingest into the database. Alternatively, you could also provide custom vectors for your text that can be ingested into the database.

Using the Operator¶

The WeaviateIngestOperator requires the input_data as an input to the operator. Use the conn_id parameter to specify the Weaviate connection to use to connect to your account.

An example using the operator to ingest data with custom vectors retrieved from XCOM:

tests/system/weaviate/example_weaviate_operator.py[source]

    batch_data_with_vectors_xcom_data = WeaviateIngestOperator(
        task_id="batch_data_with_vectors_xcom_data",
        conn_id="weaviate_default",
        collection_name=COLLECTION_NAME,
        input_data=store_data_with_vectors_in_xcom(),
        trigger_rule="all_done",
    )

An example using the operator to ingest data with custom vectors retrieved from a python callable:

tests/system/weaviate/example_weaviate_operator.py[source]

    batch_data_with_vectors_callable_data = WeaviateIngestOperator(
        task_id="batch_data_with_vectors_callable_data",
        conn_id="weaviate_default",
        collection_name=COLLECTION_NAME,
        input_data=get_data_with_vectors(),
        trigger_rule="all_done",
    )

An example using the operator to ingest data without vectors retrieved from XCOM for which the operator would generate embedding vectors:

tests/system/weaviate/example_weaviate_operator.py[source]

    batch_data_without_vectors_xcom_data = WeaviateIngestOperator(
        task_id="batch_data_without_vectors_xcom_data",
        conn_id="weaviate_default",
        collection_name="QuestionWithOpenAIVectorizerUsingOperator",
        input_data=xcom_data_without_vectors["return_value"],
        trigger_rule="all_done",
    )

An example using the operator to ingest data without vectors retrieved from a python callable for which the operator would generate embedding vectors:

tests/system/weaviate/example_weaviate_operator.py[source]

    batch_data_without_vectors_callable_data = WeaviateIngestOperator(
        task_id="batch_data_without_vectors_callable_data",
        conn_id="weaviate_default",
        collection_name="QuestionWithOpenAIVectorizerUsingOperator",
        input_data=get_data_without_vectors(),
        trigger_rule="all_done",
    )