Sync Pipeline Data Source

POST 
/api/v1/pipelines/:pipeline_id/data-sources/:data_source_id/sync

Run ingestion for the pipeline data source by incrementally updating the data-sink with upstream changes from data-source.

Request

Path Parameters

data_source_id uuidrequired
pipeline_id uuidrequired

Cookie Parameters

session any

Responses

200

Successful Response

application/json

Schema

Schema

id uuidrequired

Unique identifier

created_at

object

Creation datetime

anyOf

MOD1

string

updated_at

object

Update datetime

anyOf

MOD1

string

name Name (string)required
project_id uuidrequired

embedding_model_config_id

object

The ID of the EmbeddingModelConfig this pipeline is using.

anyOf

MOD1

string

pipeline_type PipelineType (string)
Enum for representing the type of a pipeline
Possible values: [PLAYGROUND, MANAGED]
Default value: MANAGED

managed_pipeline_id

object

The ID of the ManagedPipeline this playground pipeline is linked to.

anyOf

MOD1

string

embedding_config

object

required

oneOf

AzureOpenAIEmbeddingConfig

type Type (string)

Type of the embedding model.

Default value: AZURE_EMBEDDING

component

object

Configuration for the Azure OpenAI embedding model.

model_name Model Name (string)

The name of the OpenAI embedding model.

Default value: text-embedding-ada-002

embed_batch_size Embed Batch Size (integer)

The batch size for embedding calls.

Possible values: > 0 and <= 2048

Default value: 10

num_workers

object

The number of workers to use for async embedding calls.

anyOf

MOD1

integer

additional_kwargs object

Additional kwargs for the OpenAI API.

api_key

object

The OpenAI API key.

anyOf

MOD1

string

api_base Api Base (string)

The base URL for Azure deployment.

Default value:

api_version Api Version (string)

The version for Azure OpenAI API.

Default value:

max_retries Max Retries (integer)

Maximum number of retries.

Default value: 10

timeout Timeout (number)

Timeout for each request.

Default value: 60

default_headers

object

The default headers for API requests.

anyOf

MOD1

property name* string

reuse_client Reuse Client (boolean)

Reuse the OpenAI client between requests. When doing anything with large volumes of async API calls, setting this to false can improve stability.

Default value: true

dimensions

object

The number of dimensions on the output embedding vectors. Works only with v3 embedding models.

anyOf

MOD1

integer

azure_endpoint

object

The Azure endpoint to use.

anyOf

MOD1

string

azure_deployment

object

The Azure deployment to use.

anyOf

MOD1

string

class_name Class Name (string)

Default value: AzureOpenAIEmbedding

CohereEmbeddingConfig

type Type (string)

Type of the embedding model.

Default value: COHERE_EMBEDDING

component

object

Configuration for the Cohere embedding model.

model_name Model Name (string)

The modelId of the Cohere model to use.

Default value: embed-english-v3.0

embed_batch_size Embed Batch Size (integer)

The batch size for embedding calls.

Possible values: > 0 and <= 2048

Default value: 10

num_workers

object

The number of workers to use for async embedding calls.

anyOf

MOD1

integer

api_key

object

required

The Cohere API key.

anyOf

MOD1

string

truncate Truncate (string)

Truncation type - START/ END/ NONE

Default value: END

input_type

object

Model Input type. If not provided, search_document and search_query are used when needed.

anyOf

MOD1

string

embedding_type Embedding Type (string)

Embedding type. If not provided float embedding_type is used when needed.

Default value: float

class_name Class Name (string)

Default value: CohereEmbedding

GeminiEmbeddingConfig

type Type (string)

Type of the embedding model.

Default value: GEMINI_EMBEDDING

component

object

Configuration for the Gemini embedding model.

model_name Model Name (string)

The modelId of the Gemini model to use.

Default value: models/embedding-001

embed_batch_size Embed Batch Size (integer)

The batch size for embedding calls.

Possible values: > 0 and <= 2048

Default value: 10

num_workers

object

The number of workers to use for async embedding calls.

anyOf

MOD1

integer

title

object

Title is only applicable for retrieval_document tasks, and is used to represent a document title. For other tasks, title is invalid.

anyOf

MOD1

string

task_type

object

The task for embedding model.

anyOf

MOD1

string

api_key

object

API key to access the model. Defaults to None.

anyOf

MOD1

string

api_base

object

API base to access the model. Defaults to None.

anyOf

MOD1

string

transport

object

Transport to access the model. Defaults to None.

anyOf

MOD1

string

class_name Class Name (string)

Default value: GeminiEmbedding

HuggingFaceInferenceAPIEmbeddingConfig

type Type (string)

Type of the embedding model.

Default value: HUGGINGFACE_API_EMBEDDING

component

object

Configuration for the HuggingFace Inference API embedding model.

model_name

object

Hugging Face model name. If None, the task will be used.

anyOf

MOD1

string

embed_batch_size Embed Batch Size (integer)

The batch size for embedding calls.

Possible values: > 0 and <= 2048

Default value: 10

num_workers

object

The number of workers to use for async embedding calls.

anyOf

MOD1

integer

pooling

object

Pooling strategy. If None, the model's default pooling is used.

anyOf

Pooling

Enum of possible pooling choices with pooling behaviors.

string

Possible values: [cls, mean, last]

query_instruction

object

Instruction to prepend during query embedding.

anyOf

MOD1

string

text_instruction

object

Instruction to prepend during text embedding.

anyOf

MOD1

string

token

object

Hugging Face token. Will default to the locally saved token. Pass token=False if you don’t want to send your token to the server.

anyOf

MOD1

string

MOD2

boolean

timeout

object

The maximum number of seconds to wait for a response from the server. Loading a new model in Inference API can take up to several minutes. Defaults to None, meaning it will loop until the server is available.

anyOf

MOD1

number

headers

object

Additional headers to send to the server. By default only the authorization and user-agent headers are sent. Values in this dictionary will override the default values.

anyOf

MOD1

property name* string

cookies

object

Additional cookies to send to the server.

anyOf

MOD1

property name* string

task

object

Optional task to pick Hugging Face's recommended model, used when model_name is left as default of None.

anyOf

MOD1

string

class_name Class Name (string)

Default value: HuggingFaceInferenceAPIEmbedding

OpenAIEmbeddingConfig

type Type (string)

Type of the embedding model.

Default value: OPENAI_EMBEDDING

component

object

Configuration for the OpenAI embedding model.

model_name Model Name (string)

The name of the OpenAI embedding model.

Default value: text-embedding-ada-002

embed_batch_size Embed Batch Size (integer)

The batch size for embedding calls.

Possible values: > 0 and <= 2048

Default value: 10

num_workers

object

The number of workers to use for async embedding calls.

anyOf

MOD1

integer

additional_kwargs object

Additional kwargs for the OpenAI API.

api_key

object

The OpenAI API key.

anyOf

MOD1

string

api_base

object

The base URL for OpenAI API.

anyOf

MOD1

string

api_version

object

The version for OpenAI API.

anyOf

MOD1

string

max_retries Max Retries (integer)

Maximum number of retries.

Default value: 10

timeout Timeout (number)

Timeout for each request.

Default value: 60

default_headers

object

The default headers for API requests.

anyOf

MOD1

property name* string

reuse_client Reuse Client (boolean)

Reuse the OpenAI client between requests. When doing anything with large volumes of async API calls, setting this to false can improve stability.

Default value: true

dimensions

object

The number of dimensions on the output embedding vectors. Works only with v3 embedding models.

anyOf

MOD1

integer

class_name Class Name (string)

Default value: OpenAIEmbedding

VertexAIEmbeddingConfig

type Type (string)

Type of the embedding model.

Default value: VERTEXAI_EMBEDDING

component

object

Configuration for the VertexAI embedding model.

model_name Model Name (string)

The modelId of the VertexAI model to use.

Default value: textembedding-gecko@003

embed_batch_size Embed Batch Size (integer)

The batch size for embedding calls.

Possible values: > 0 and <= 2048

Default value: 10

num_workers

object

The number of workers to use for async embedding calls.

anyOf

MOD1

integer

location Location (string)required

The default location to use when making API calls.

project Project (string)required

The default GCP project to use when making Vertex API calls.

embed_mode VertexEmbeddingMode (string)

The embedding mode to use.

Possible values: [default, classification, clustering, similarity, retrieval]

Default value: retrieval

additional_kwargs object

Additional kwargs for the Vertex.

client_email

object

required

The client email for the VertexAI credentials.

anyOf

MOD1

string

token_uri

object

required

The token URI for the VertexAI credentials.

anyOf

MOD1

string

private_key_id

object

required

The private key ID for the VertexAI credentials.

anyOf

MOD1

string

private_key

object

required

The private key for the VertexAI credentials.

anyOf

MOD1

string

class_name Class Name (string)

Default value: VertexTextEmbedding

BedrockEmbeddingConfig

type Type (string)

Type of the embedding model.

Default value: BEDROCK_EMBEDDING

component

object

Configuration for the Bedrock embedding model.

model_name Model Name (string)

The modelId of the Bedrock model to use.

Default value: amazon.titan-embed-text-v1

embed_batch_size Embed Batch Size (integer)

The batch size for embedding calls.

Possible values: > 0 and <= 2048

Default value: 10

num_workers

object

The number of workers to use for async embedding calls.

anyOf

MOD1

integer

profile_name

object

The name of aws profile to use. If not given, then the default profile is used.

anyOf

MOD1

string

aws_access_key_id

object

AWS Access Key ID to use

anyOf

MOD1

string

aws_secret_access_key

object

AWS Secret Access Key to use

anyOf

MOD1

string

aws_session_token

object

AWS Session Token to use

anyOf

MOD1

string

region_name

object

AWS region name to use. Uses region configured in AWS CLI if not passed

anyOf

MOD1

string

max_retries Max Retries (integer)

The maximum number of API retries.

Possible values: > 0

Default value: 10

timeout Timeout (number)

The timeout for the Bedrock API request in seconds. It will be used for both connect and read timeouts.

Default value: 60

additional_kwargs object

Additional kwargs for the bedrock client.

class_name Class Name (string)

Default value: BedrockEmbedding

configured_transformations

object[]

Deprecated don't use it, List of configured transformations.

Array [

id uuid

configurable_transformation_type ConfigurableTransformationNames (string)required

Name for the type of transformation this is (e.g. SIMPLE_NODE_PARSER). Can also be an enum instance of llama_index.ingestion.transformations.ConfigurableTransformations. This will be converted to ConfigurableTransformationNames.

Possible values: [CHARACTER_SPLITTER, PAGE_SPLITTER_NODE_PARSER, CODE_NODE_PARSER, SENTENCE_AWARE_NODE_PARSER, TOKEN_AWARE_NODE_PARSER, MARKDOWN_NODE_PARSER, MARKDOWN_ELEMENT_NODE_PARSER]

component

object

required

Component that implements the transformation

anyOf

MOD1

object

CharacterSplitter

A splitter that splits text into characters.

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

chunk_size Chunk Size (integer)

The token chunk size for each chunk.

Possible values: > 0

Default value: 1024

chunk_overlap Chunk Overlap (integer)

The token overlap of each chunk when splitting.

Default value: 200

separator Separator (string)

Default separator for splitting into words

Default value:

paragraph_separator Paragraph Separator (string)

Separator between paragraphs.

Default value:

secondary_chunking_regex

object

Backup regex for splitting into sentences.

anyOf

MOD1

string

class_name Class Name (string)

Default value: SentenceSplitter

PageSplitterNodeParser

Split text into pages.

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

page_separator

object

Separator to split text into pages.

anyOf

MOD1

string

class_name Class Name (string)

Default value: base_component

CodeSplitter

Split code using a AST parser.

Thank you to Kevin Lu / SweepAI for suggesting this elegant code splitting solution. https://docs.sweep.dev/blogs/chunking-2m-files

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

language Language (string)required

The programming language of the code being split.

chunk_lines Chunk Lines (integer)

The number of lines to include in each chunk.

Possible values: > 0

Default value: 40

chunk_lines_overlap Chunk Lines Overlap (integer)

How many lines of code each chunk overlaps with.

Possible values: > 0

Default value: 15

max_chars Max Chars (integer)

Maximum number of characters per chunk.

Possible values: > 0

Default value: 1500

class_name Class Name (string)

Default value: CodeSplitter

SentenceSplitter

Parse text with a preference for complete sentences.

In general, this class tries to keep sentences and paragraphs together. Therefore compared to the original TokenTextSplitter, there are less likely to be hanging sentences or parts of sentences at the end of the node chunk.

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

chunk_size Chunk Size (integer)

The token chunk size for each chunk.

Possible values: > 0

Default value: 1024

chunk_overlap Chunk Overlap (integer)

The token overlap of each chunk when splitting.

Default value: 200

separator Separator (string)

Default separator for splitting into words

Default value:

paragraph_separator Paragraph Separator (string)

Separator between paragraphs.

Default value:

secondary_chunking_regex

object

Backup regex for splitting into sentences.

anyOf

MOD1

string

class_name Class Name (string)

Default value: SentenceSplitter

TokenTextSplitter

Implementation of splitting text that looks at word tokens.

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

chunk_size Chunk Size (integer)

The token chunk size for each chunk.

Possible values: > 0

Default value: 1024

chunk_overlap Chunk Overlap (integer)

The token overlap of each chunk when splitting.

Default value: 20

separator Separator (string)

Default separator for splitting into words

Default value:

backup_separators undefined[]

Additional separators for splitting.

class_name Class Name (string)

Default value: TokenTextSplitter

MarkdownNodeParser

Markdown node parser.

Splits a document into Nodes using Markdown header-based splitting logic. Each node contains its text content and the path of headers leading to it.

Args: include_metadata (bool): whether to include metadata in nodes include_prev_next_rel (bool): whether to include prev/next relationships

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

class_name Class Name (string)

Default value: base_component

MarkdownElementNodeParser

Markdown element node parser.

Splits a markdown document into Text Nodes and Index Nodes corresponding to embedded objects (e.g. tables).

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

llm

object

LLM model to use for summarization.

anyOf

LLM

The LLM class is the main class for interacting with language models.

Attributes: system_prompt (Optional[str]): System prompt for LLM calls. messages_to_prompt (Callable): Function to convert a list of messages to an LLM prompt. completion_to_prompt (Callable): Function to convert a completion to an LLM prompt. output_parser (Optional[BaseOutputParser]): Output parser to parse, validate, and correct errors programmatically. pydantic_program_mode (PydanticProgramMode): Pydantic program mode to use for structured prediction.

callback_manager Callback Manager

system_prompt

object

System prompt for LLM calls.

anyOf

MOD1

string

messages_to_prompt Messages To Prompt (string)

Function to convert a list of messages to an LLM prompt.

completion_to_prompt Completion To Prompt (string)

Function to convert a completion to an LLM prompt.

output_parser

object

Output parser to parse, validate, and correct errors programmatically.

anyOf

pydantic_program_mode PydanticProgramMode (string)

Pydantic program mode.

Possible values: [default, openai, llm, function, guidance, lm-format-enforcer]

Default value: default

query_wrapper_prompt

object

Query wrapper prompt for LLM calls.

anyOf

BasePromptTemplate

metadata objectrequired

template_vars string[]required

kwargs

object

required

property name* string

output_parser

object

required

anyOf

template_var_mappings

object

Template variable mappings (Optional).

anyOf

MOD1

object

function_mappings

object

Function mappings (Optional). This is a mapping from template variable names to functions that take in the current kwargs and return a string.

anyOf

MOD1

property name* string

class_name Class Name (string)

Default value: base_component

summary_query_str Summary Query Str (string)

Query string to use for summarization.

Default value: What is this table about? Give a very concise summary (imagine you are adding a new caption and summary for this table), and output the real/existing table title/caption if context provided.and output the real/existing table id if context provided.and also output whether or not the table should be kept.

num_workers Num Workers (integer)

Num of workers for async jobs.

Default value: 4

show_progress Show Progress (boolean)

Whether to show progress.

Default value: true

nested_node_parser

object

Other types of node parsers to handle some types of nodes.

anyOf

NodeParser

Base interface for node parser.

include_metadata Include Metadata (boolean)

Whether or not to consider metadata when splitting.

Default value: true

include_prev_next_rel Include Prev Next Rel (boolean)

Include prev/next node relationships.

Default value: true

callback_manager Callback Manager

id_func

object

Function to generate node IDs.

anyOf

MOD1

string

class_name Class Name (string)

Default value: base_component

class_name Class Name (string)

Default value: MarkdownElementNodeParser

]

config_hash

object

Hashes for the configuration of the pipeline.

anyOf

PipelineConfigurationHashes

Hashes for the configuration of a pipeline.

embedding_config_hash

object

Hash of the embedding config.

anyOf

MOD1

string

parsing_config_hash

object

Hash of the llama parse parameters.

anyOf

MOD1

string

transform_config_hash

object

Hash of the transform config.

anyOf

MOD1

string

transform_config

object

Configuration for the transformation.

anyOf

AutoTransformConfig

mode Mode (string)

Default value: auto

chunk_size Chunk Size (integer)

Chunk size for the transformation.

Possible values: > 0

Default value: 1024

chunk_overlap Chunk Overlap (integer)

Chunk overlap for the transformation.

Default value: 200

AdvancedModeTransformConfig

mode Mode (string)

Default value: advanced

segmentation_config

object

Configuration for the segmentation.

anyOf

NoneSegmentationConfig

mode Mode (string)

Default value: none

PageSegmentationConfig

mode Mode (string)

Default value: page

page_separator Page Separator (string)

Default value: ---

ElementSegmentationConfig

mode Mode (string)

Default value: element

chunking_config

object

Configuration for the chunking.

anyOf

NoneChunkingConfig

mode Mode (string)

Default value: none

CharacterChunkingConfig

chunk_size Chunk Size (integer)

Possible values: > 0

Default value: 1024

chunk_overlap Chunk Overlap (integer)

Default value: 200

mode Mode (string)

Default value: character

TokenChunkingConfig

chunk_size Chunk Size (integer)

Possible values: > 0

Default value: 1024

chunk_overlap Chunk Overlap (integer)

Default value: 200

mode Mode (string)

Default value: token

separator Separator (string)

Default value:

SentenceChunkingConfig

chunk_size Chunk Size (integer)

Possible values: > 0

Default value: 1024

chunk_overlap Chunk Overlap (integer)

Default value: 200

mode Mode (string)

Default value: sentence

separator Separator (string)

Default value:

paragraph_separator Paragraph Separator (string)

Default value:

SemanticChunkingConfig

mode Mode (string)

Default value: semantic

buffer_size Buffer Size (integer)

Default value: 1

breakpoint_percentile_threshold Breakpoint Percentile Threshold (integer)

Default value: 95

preset_retrieval_parameters

object

Preset retrieval parameters for the pipeline.

dense_similarity_top_k

object

Number of nodes for dense retrieval.

anyOf

MOD1

integer

Possible values: >= 1 and <= 100

dense_similarity_cutoff

object

Minimum similarity score wrt query for retrieval

anyOf

MOD1

number

Possible values: <= 1

sparse_similarity_top_k

object

Number of nodes for sparse retrieval.

anyOf

MOD1

integer

Possible values: >= 1 and <= 100

enable_reranking

object

Enable reranking for retrieval

anyOf

MOD1

boolean

rerank_top_n

object

Number of reranked nodes for returning.

anyOf

MOD1

integer

Possible values: >= 1 and <= 100

alpha

object

Alpha value for hybrid retrieval to determine the weights between dense and sparse retrieval. 0 is sparse retrieval and 1 is dense retrieval.

anyOf

MOD1

number

Possible values: <= 1

search_filters

object

Search filters for retrieval.

anyOf

MetadataFilters

Metadata filters for vector stores.

filters

object[]

required

Array [

anyOf

MetadataFilter

Comprehensive metadata filter for vector stores to support more operators.

Value uses Strict* types, as int, float and str are compatible types and were all converted to string before.

See: https://docs.pydantic.dev/latest/usage/types/#strict-types

key Key (string)required

value

object

required

anyOf

MOD1

integer

MOD2

number

MOD3

string

MOD4

Array [

string

]

MOD5

Array [

number

]

MOD6

Array [

integer

]

operator FilterOperator (string)

Vector store filter operator.

Possible values: [==, >, <, !=, >=, <=, in, nin, any, all, text_match, contains, is_empty]

Default value: ==

]

condition

object

anyOf

FilterCondition

Vector store filter conditions to combine different filters.

string

Possible values: [and, or]

files_top_k

object

Number of files to retrieve (only for retrieval mode files_via_metadata and files_via_content).

anyOf

MOD1

integer

Possible values: >= 1 and <= 5

retrieval_mode RetrievalMode (string)

The retrieval mode for the query.

Possible values: [chunks, files_via_metadata, files_via_content, auto_routed]

Default value: chunks

retrieve_image_nodes Retrieve Image Nodes (boolean)

Whether to retrieve image nodes.

Default value: false

class_name Class Name (string)

Default value: base_component

eval_parameters

object

Eval parameters for the pipeline.

llm_model SupportedLLMModelNames (string)

The LLM model to use within eval execution.

Possible values: [GPT_3_5_TURBO, GPT_4, GPT_4_TURBO, GPT_4O, GPT_4O_MINI, AZURE_OPENAI, CLAUDE_3_5_SONNET]

Default value: GPT_4O

qa_prompt_tmpl Qa Prompt Tmpl (string)

The template to use for the question answering prompt.

Default value: Context information is below. --------------------- {context_str} --------------------- Given the context information and not prior knowledge, answer the query. Query: {query_str} Answer:

llama_parse_parameters

object

Settings that can be configured for how to use LlamaParse to parse files within a LlamaCloud pipeline.

anyOf

LlamaParseParameters

Settings that can be configured for how to use LlamaParse to parse files within a LlamaCloud pipeline.

languages ParserLanguages (string)[]

Possible values: [af, az, bs, cs, cy, da, de, en, es, et, fr, ga, hr, hu, id, is, it, ku, la, lt, lv, mi, ms, mt, nl, no, oc, pi, pl, pt, ro, rs_latin, sk, sl, sq, sv, sw, tl, tr, uz, vi, ar, fa, ug, ur, bn, as, mni, ru, rs_cyrillic, be, bg, uk, mn, abq, ady, kbd, ava, dar, inh, che, lbe, lez, tab, tjk, hi, mr, ne, bh, mai, ang, bho, mah, sck, new, gom, sa, bgc, th, ch_sim, ch_tra, ja, ko, ta, te, kn], >= 1

parsing_instruction Parsing Instruction (string)

Default value:

disable_ocr Disable Ocr (boolean)

Default value: false

annotate_links Annotate Links (boolean)

Default value: false

disable_reconstruction Disable Reconstruction (boolean)

Default value: false

disable_image_extraction Disable Image Extraction (boolean)

Default value: false

invalidate_cache Invalidate Cache (boolean)

Default value: false

output_pdf_of_document Output Pdf Of Document (boolean)

Default value: false

do_not_cache Do Not Cache (boolean)

Default value: false

fast_mode Fast Mode (boolean)

Default value: false

skip_diagonal_text Skip Diagonal Text (boolean)

Default value: false

gpt4o_mode Gpt4O Mode (boolean)

Default value: false

gpt4o_api_key Gpt4O Api Key (string)

Default value:

do_not_unroll_columns Do Not Unroll Columns (boolean)

Default value: false

extract_layout Extract Layout (boolean)

Default value: false

html_make_all_elements_visible Html Make All Elements Visible (boolean)

Default value: false

html_remove_navigation_elements Html Remove Navigation Elements (boolean)

Default value: false

html_remove_fixed_elements Html Remove Fixed Elements (boolean)

Default value: false

guess_xlsx_sheet_name Guess Xlsx Sheet Name (boolean)

Default value: false

page_separator

object

anyOf

MOD1

string

bounding_box Bounding Box (string)

Default value:

bbox_top

object

anyOf

MOD1

number

bbox_right

object

anyOf

MOD1

number

bbox_bottom

object

anyOf

MOD1

number

bbox_left

object

anyOf

MOD1

number

target_pages Target Pages (string)

Default value:

use_vendor_multimodal_model Use Vendor Multimodal Model (boolean)

Default value: false

vendor_multimodal_model_name Vendor Multimodal Model Name (string)

Default value:

vendor_multimodal_api_key Vendor Multimodal Api Key (string)

Default value:

page_prefix Page Prefix (string)

Default value:

page_suffix Page Suffix (string)

Default value:

webhook_url Webhook Url (string)

Default value:

take_screenshot Take Screenshot (boolean)

Default value: false

is_formatting_instruction Is Formatting Instruction (boolean)

Default value: true

premium_mode Premium Mode (boolean)

Default value: false

continuous_mode Continuous Mode (boolean)

Default value: false

s3_input_path S3 Input Path (string)

Default value:

s3_output_path_prefix S3 Output Path Prefix (string)

Default value:

project_id

object

anyOf

MOD1

string

azure_openai_deployment_name

object

anyOf

MOD1

string

azure_openai_endpoint

object

anyOf

MOD1

string

azure_openai_api_version

object

anyOf

MOD1

string

azure_openai_key

object

anyOf

MOD1

string

input_url

object

anyOf

MOD1

string

http_proxy

object

anyOf

MOD1

string

auto_mode Auto Mode (boolean)

Default value: false

auto_mode_trigger_on_regexp_in_page

object

anyOf

MOD1

string

auto_mode_trigger_on_text_in_page

object

anyOf

MOD1

string

auto_mode_trigger_on_table_in_page Auto Mode Trigger On Table In Page (boolean)

Default value: false

auto_mode_trigger_on_image_in_page Auto Mode Trigger On Image In Page (boolean)

Default value: false

structured_output Structured Output (boolean)

Default value: false

structured_output_json_schema

object

anyOf

MOD1

string

structured_output_json_schema_name

object

anyOf

MOD1

string

max_pages

object

anyOf

MOD1

integer

max_pages_enforced

object

anyOf

MOD1

integer

extract_charts Extract Charts (boolean)

Default value: false

data_sink

object

The data sink for the pipeline. If None, the pipeline will use the fully managed data sink.

anyOf

DataSink

Schema for a data sink.

id uuidrequired

Unique identifier

created_at

object

Creation datetime

anyOf

MOD1

string

updated_at

object

Update datetime

anyOf

MOD1

string

name Name (string)required

The name of the data sink.

sink_type ConfigurableDataSinkNames (string)required

Possible values: [PINECONE, POSTGRES, QDRANT, AZUREAI_SEARCH, MONGODB_ATLAS, MILVUS]

component

object

required

Component that implements the data sink

anyOf

MOD1

object

CloudPineconeVectorStore

Cloud Pinecone Vector Store.

This class is used to store the configuration for a Pinecone vector store, so that it can be created and used in LlamaCloud.

Args: api_key (str): API key for authenticating with Pinecone index_name (str): name of the Pinecone index namespace (optional[str]): namespace to use in the Pinecone index insert_kwargs (optional[dict]): additional kwargs to pass during insertion

supports_nested_metadata_filters Supports Nested Metadata Filters (boolean)

Default value: true

index_name Index Name (string)required

namespace

object

anyOf

MOD1

string

insert_kwargs

object

anyOf

MOD1

object

class_name Class Name (string)

Default value: CloudPineconeVectorStore

CloudPostgresVectorStore

supports_nested_metadata_filters Supports Nested Metadata Filters (boolean)

Default value: false

database Database (string)required

host Host (string)required

port Port (integer)required

user User (string)required

table_name Table Name (string)required

schema_name Schema Name (string)required

embed_dim Embed Dim (integer)required

hybrid_search

object

anyOf

MOD1

boolean

perform_setup Perform Setup (boolean)

Default value: true

class_name Class Name (string)

Default value: CloudPostgresVectorStore

CloudQdrantVectorStore

Cloud Qdrant Vector Store.

This class is used to store the configuration for a Qdrant vector store, so that it can be created and used in LlamaCloud.

Args: collection_name (str): name of the Qdrant collection url (str): url of the Qdrant instance api_key (str): API key for authenticating with Qdrant max_retries (int): maximum number of retries in case of a failure. Defaults to 3 client_kwargs (dict): additional kwargs to pass to the Qdrant client

supports_nested_metadata_filters Supports Nested Metadata Filters (boolean)

Default value: true

collection_name Collection Name (string)required

url Url (string)required

max_retries Max Retries (integer)

Default value: 3

client_kwargs object

class_name Class Name (string)

Default value: CloudQdrantVectorStore

CloudAzureAISearchVectorStore

Cloud Azure AI Search Vector Store.

supports_nested_metadata_filters Supports Nested Metadata Filters (boolean)

Default value: true

search_service_endpoint Search Service Endpoint (string)required

search_service_api_version

object

anyOf

MOD1

string

index_name

object

anyOf

MOD1

string

filterable_metadata_field_keys

object

anyOf

MOD1

object

embedding_dimension

object

anyOf

MOD1

integer

client_id

object

anyOf

MOD1

string

client_secret

object

anyOf

MOD1

tenant_id

object

anyOf

MOD1

string

class_name Class Name (string)

Default value: CloudAzureAISearchVectorStore

CloudMongoDBAtlasVectorSearch

Cloud MongoDB Atlas Vector Store.

This class is used to store the configuration for a MongoDB Atlas vector store, so that it can be created and used in LlamaCloud.

Args: mongodb_uri (str): URI for connecting to MongoDB Atlas db_name (str): name of the MongoDB database collection_name (str): name of the MongoDB collection vector_index_name (str): name of the MongoDB Atlas vector index fulltext_index_name (str): name of the MongoDB Atlas full-text index

supports_nested_metadata_filters Supports Nested Metadata Filters (boolean)

Default value: false

db_name Db Name (string)required

collection_name Collection Name (string)required

vector_index_name

object

anyOf

MOD1

string

fulltext_index_name

object

anyOf

MOD1

string

class_name Class Name (string)

Default value: CloudMongoDBAtlasVectorSearch

CloudMilvusVectorStore

Cloud Milvus Vector Store.

supports_nested_metadata_filters Supports Nested Metadata Filters (boolean)

Default value: false

uri Uri (string)required

collection_name

object

anyOf

MOD1

string

token

object

anyOf

MOD1

embedding_dimension

object

anyOf

MOD1

integer

class_name Class Name (string)

Default value: CloudMilvusVectorStore

project_id uuidrequired

Example (from schema)

{
  "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "created_at": "2024-07-29T15:51:28.071Z",
  "updated_at": "2024-07-29T15:51:28.071Z",
  "name": "string",
  "project_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "embedding_model_config_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "pipeline_type": "MANAGED",
  "managed_pipeline_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "embedding_config": {},
  "configured_transformations": [
    {
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "configurable_transformation_type": "CHARACTER_SPLITTER",
      "component": {}
    }
  ],
  "config_hash": {},
  "transform_config": {},
  "preset_retrieval_parameters": {
    "dense_similarity_top_k": 0,
    "dense_similarity_cutoff": 0,
    "sparse_similarity_top_k": 0,
    "enable_reranking": true,
    "rerank_top_n": 0,
    "alpha": 0,
    "search_filters": {},
    "files_top_k": 0,
    "retrieval_mode": "chunks",
    "retrieve_image_nodes": false,
    "class_name": "base_component"
  },
  "eval_parameters": {
    "llm_model": "GPT_4O",
    "qa_prompt_tmpl": "Context information is below.\n---------------------\n{context_str}\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: {query_str}\nAnswer: "
  },
  "llama_parse_parameters": {},
  "data_sink": {}
}

422

Validation Error

application/json

Schema

Schema

detail

object[]

Array [

loc

object[]

required

Array [

anyOf

MOD1

string

MOD2

integer

]

msg Message (string)required

type Error Type (string)required

]

Example (from schema)

{
  "detail": [
    {
      "loc": [
        "string",
        0
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}