List Pipeline Data Sources

GET 
/api/v1/pipelines/:pipeline_id/data-sources

Get data sources for a pipeline.

Request

Path Parameters

pipeline_id uuidrequired

Cookie Parameters

session any

Responses

200

Successful Response

application/json

Schema

Schema

• Array [

id uuidrequired

Unique identifier

created_at

object

Creation datetime

anyOf

MOD1

string

updated_at

object

Update datetime

anyOf

MOD1

string

name Name (string)required

The name of the data source.

source_type ConfigurableDataSourceNames (string)required

Possible values: [S3, AZURE_STORAGE_BLOB, GOOGLE_DRIVE, MICROSOFT_ONEDRIVE, MICROSOFT_SHAREPOINT, SLACK, NOTION_PAGE, CONFLUENCE, JIRA, BOX]

custom_metadata

object

Custom metadata that will be present on all data loaded from the data source

anyOf

MOD1

property name*

object

anyOf

MOD1

object

MOD2

Array [

]

MOD3

string

MOD4

integer

MOD5

number

MOD6

boolean

component

object

required

anyOf

MOD1

object

CloudS3DataSource

bucket Bucket (string)required

The name of the S3 bucket to read from.

prefix

object

The prefix of the S3 objects to read from.

anyOf

MOD1

string

aws_access_id

object

The AWS access ID to use for authentication.

anyOf

MOD1

string

aws_access_secret

object

The AWS access secret to use for authentication.

anyOf

MOD1

s3_endpoint_url

object

The S3 endpoint URL to use for authentication.

anyOf

MOD1

string

class_name Class Name (string)

Default value: CloudS3DataSource

CloudAzStorageBlobDataSource

container_name Container Name (string)required

The name of the Azure Storage Blob container to read from.

account_url Account Url (string)required

The Azure Storage Blob account URL to use for authentication.

blob

object

The blob name to read from.

anyOf

MOD1

string

prefix

object

The prefix of the Azure Storage Blob objects to read from.

anyOf

MOD1

string

account_name

object

The Azure Storage Blob account name to use for authentication.

anyOf

MOD1

string

account_key

object

The Azure Storage Blob account key to use for authentication.

anyOf

MOD1

tenant_id

object

The Azure AD tenant ID to use for authentication.

anyOf

MOD1

string

client_id

object

The Azure AD client ID to use for authentication.

anyOf

MOD1

string

client_secret

object

The Azure AD client secret to use for authentication.

anyOf

MOD1

class_name Class Name (string)

Default value: CloudAzStorageBlobDataSource

CloudGoogleDriveDataSource

folder_id Folder Id (string)required

The ID of the Google Drive folder to read from.

service_account_key objectrequired

The service account key JSON to use for authentication.

class_name Class Name (string)

Default value: CloudGoogleDriveDataSource

CloudOneDriveDataSource

user_principal_name User Principal Name (string)required

The user principal name to use for authentication.

folder_path

object

The path of the OneDrive folder to read from.

anyOf

MOD1

string

folder_id

object

The ID of the OneDrive folder to read from.

anyOf

MOD1

string

client_id Client Id (string)required

The client ID to use for authentication.

tenant_id Tenant Id (string)required

The tenant ID to use for authentication.

required_exts

object

The list of required file extensions.

anyOf

MOD1

Array [

string

]

class_name Class Name (string)

Default value: CloudOneDriveDataSource

CloudSharepointDataSource

site_name

object

The name of the SharePoint site to download from.

anyOf

MOD1

string

site_id

object

The ID of the SharePoint site to download from.

anyOf

MOD1

string

folder_path

object

The path of the Sharepoint folder to read from.

anyOf

MOD1

string

folder_id

object

The ID of the Sharepoint folder to read from.

anyOf

MOD1

string

drive_name

object

The name of the Sharepoint drive to read from.

anyOf

MOD1

string

client_id Client Id (string)required

The client ID to use for authentication.

tenant_id Tenant Id (string)required

The tenant ID to use for authentication.

required_exts

object

The list of required file extensions.

anyOf

MOD1

Array [

string

]

class_name Class Name (string)

Default value: CloudSharepointDataSource

CloudSlackDataSource

channel_ids

object

Slack Channel.

anyOf

MOD1

string

latest_date

object

Latest date.

anyOf

MOD1

string

earliest_date

object

Earliest date.

anyOf

MOD1

string

earliest_date_timestamp

object

Earliest date timestamp.

anyOf

MOD1

number

latest_date_timestamp

object

Latest date timestamp.

anyOf

MOD1

number

channel_patterns

object

Slack Channel name pattern.

anyOf

MOD1

string

class_name Class Name (string)

Default value: CloudSlackDataSource

CloudNotionPageDataSource

database_ids

object

The Notion Database Id to read content from.

anyOf

MOD1

string

page_ids

object

The Page ID's of the Notion to read from.

anyOf

MOD1

string

class_name Class Name (string)

Default value: CloudNotionPageDataSource

CloudConfluenceDataSource

server_url Server Url (string)required

The server URL of the Confluence instance.

authentication_mechanism Authentication Mechanism (string)required

Type of Authentication for connecting to Confluence APIs.

user_name

object

The username to use for authentication.

anyOf

MOD1

string

api_token

object

The API token to use for authentication.

anyOf

MOD1

space_key

object

The space key to read from.

anyOf

MOD1

string

page_ids

object

The page IDs of the Confluence to read from.

anyOf

MOD1

string

cql

object

The CQL query to use for fetching pages.

anyOf

MOD1

string

label

object

The label to use for fetching pages.

anyOf

MOD1

string

class_name Class Name (string)

Default value: CloudConfluenceDataSource

CloudJiraDataSource

Cloud Jira Data Source integrating JiraReader.

email

object

The email address to use for authentication.

anyOf

MOD1

string

api_token

object

The API/ Access Token used for Basic, PAT and OAuth2 authentication.

anyOf

MOD1

server_url

object

The server url for Jira Cloud.

anyOf

MOD1

string

cloud_id

object

The cloud ID, used in case of OAuth2.

anyOf

MOD1

string

authentication_mechanism Authentication Mechanism (string)required

Type of Authentication for connecting to Jira APIs.

query Query (string)required

JQL (Jira Query Language) query to search.

class_name Class Name (string)

Default value: CloudJiraDataSource

CloudBoxDataSource

folder_id

object

The ID of the Box folder to read from.

anyOf

MOD1

string

authentication_mechanism BoxAuthMechanism (string)required

The type of authentication to use (Developer Token or CCG)

Possible values: [developer_token, ccg]

developer_token

object

Developer token for authentication if authentication_mechanism is 'developer_token'.

anyOf

MOD1

client_id

object

Box API key used for identifying the application the user is authenticating with

anyOf

MOD1

string

client_secret

object

Box API secret used for making auth requests.

anyOf

MOD1

user_id

object

Box User ID, if provided authenticates as user.

anyOf

MOD1

string

enterprise_id

object

Box Enterprise ID, if provided authenticates as service.

anyOf

MOD1

string

class_name Class Name (string)

Default value: CloudBoxDataSource
project_id uuidrequired
data_source_id uuidrequired

The ID of the data source.

pipeline_id uuidrequired

The ID of the pipeline.

last_synced_at date-timerequired

The last time the data source was automatically synced.

sync_interval

object

The interval at which the data source should be synced.

anyOf

MOD1

number

sync_schedule_set_by

object

The id of the user who set the sync schedule.

anyOf

MOD1

string

• ]

Example (from schema)

[
  {
    "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "created_at": "2024-07-29T15:51:28.071Z",
    "updated_at": "2024-07-29T15:51:28.071Z",
    "name": "string",
    "source_type": "S3",
    "custom_metadata": {},
    "component": {},
    "project_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "data_source_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "pipeline_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "last_synced_at": "2024-07-29T15:51:28.071Z",
    "sync_interval": 0,
    "sync_schedule_set_by": "string"
  }
]

422

Validation Error

application/json

Schema

Schema

detail

object[]

Array [

loc

object[]

required

Array [

anyOf

MOD1

string

MOD2

integer

]

msg Message (string)required

type Error Type (string)required

]

Example (from schema)

{
  "detail": [
    {
      "loc": [
        "string",
        0
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}

Loading...