Skip to main content

Configuring File Storage

File storage is an integral part of LlamaCloud. Without it, many key features would not be possible. This page walks through how to configure file storage for your deployment -- which buckets you need to create and for non-AWS deployments, how to configure the S3 Proxy to interact with them.

Requirements​

  • A valid blob storage service. We recommend the following:
  • Because LlamaCloud heavily relies on file storage, you will need to create the following buckets:
    • llama-platform-parsed-documents
    • llama-platform-etl
    • llama-platform-external-components
    • llama-platform-file-parsing
    • llama-platform-raw-file
    • llama-cloud-output
    • llama-platform-file-screenshots

Connecting to AWS S3​

Below are two ways to configure a connection to AWS S3:

We recommend that users create a new IAM Role and Policy for LlamaCloud. You can then attach the role ARN as a service account annotation.

// Example IAM Policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*"], // this is not secure
      "Resource": [
        "arn:aws:s3:::llama-platform-parsed-documents",
        "arn:aws:s3:::llama-platform-parsed-documents/*",
        ...
      ]
    }
  ]
}

After creating something similar to the above policy, update the backend, jobsService, jobsWorker, and llamaParse service accounts with the EKS annotation.

# Example for the backend service account. Repeat for each of the services listed above.
backend:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<role-name>

For more information, feel free to refer to the official AWS documentation about this topic.

AWS Credentials​

Create a user with a policy attached for the aforementioned s3 buckets. Afterwards, you can configure the platform to use the aws credentials of that user by setting the following values in your values.yaml file:

global:
  cloudProvider: "aws"
  config:
    accessKeyId: "<your-access-key-id>"
    secretAccessKey: "<your-secret-access-key>"

Overriding Default Bucket Names​

We allow users to override the default bucket names in the values.yaml file.

global:
  config:
    parsedDocumentsCloudBucketName: "<your-bucket-name>"
    parsedEtlCloudBucketName: "<your-bucket-name>"
    parsedExternalComponentsCloudBucketName: "<your-bucket-name>"
    parsedFileParsingCloudBucketName: "<your-bucket-name>"
    parsedRawFileCloudBucketName: "<your-bucket-name>"
    parsedLlamaCloudParseOutputCloudBucketName: "<your-bucket-name>"
    parsedFileScreenshotCloudBucketName: "<your-bucket-name>"

Configuring S3 Proxy​

LlamaCloud was first developed on AWS, which means that we started by natively supporting S3. However, to make a self-hosted solution possible, we need a way for the platform to interact with other providers.

We leverage the open-source project S3Proxy to translate the S3 API requests into requests to other storage providers. A containerized deployment of S3Proxy is supported out of the box in our helm charts.

S3 Proxy can be enabled (is enabled by default) and can be further configured in your values.yaml file. The following is an example for how to connect your LlamaCloud deployment to Azure Blob Storage. For more examples of connecting to different providers, please refer to the project's Examples page.

s3Proxy:
  enabled: true

  config:
    S3PROXY_ENDPOINT: "http://0.0.0.0:80"
    S3PROXY_AUTHORIZATION: "none"
    S3PROXY_IGNORE_UNKNOWN_HEADERS: "true"
    S3PROXY_CORS_ALLOW_ORIGINS: "*"
    JCLOUDS_PROVIDER: "azureblob"
    JCLOUD_REGION: "eastus"
    JCLOUDS_AZUREBLOB_AUTH: "azureKey"
    JCLOUDS_IDENTITY: "fill-out"
    JCLOUDS_CREDENTIAL: "fill-out"
    JCLOUDS_ENDPOINT: "fill-out"