Storing Petabytes of Container Logs in Doris on S3

[emahdij]

Hi everyone,

I'm new to the community, and congrats on the recent release and achievements!

I've been researching different systems to store my container logs, which are massive (over 1 PB per month) and semi-structured (dynamic schema). Since I lack local storage, object storage like S3 is necessary. Reads and queries are infrequent, but caching retrieved data would be ideal. Unlike Elasticsearch, scaling isn't a priority for my use case; storage and occasional querying are the main goals.

I couldn't find answers to these specific questions in the documentation, and any help would be greatly appreciated. If Doris proves suitable, I'd be happy to share my experience in a blog post.

Questions:

1. Table/Database Limits: I need separate tables or databases for each container log. Are there limitations on creating a large number (thousands or even 1 million) in Doris?

2. Iceberg on S3: The documentation mentions Doris reading/writing Iceberg in S3. Is this compatible with variants, and how does it differ from the built-in S3 cold storage system?

3. S3 Encryption: Is there a built-in encryption system for storing data in S3 (both cold storage and Iceberg)?

4. Brokers vs. S3 Solutions: How do Brokers differ from the aforementioned S3 storage solutions (cold and Iceberg)?

5. Caching and Dynamic Buckets: Which solution supports caching and dynamic buckets?

6. Benchmarks: Are there any benchmarks comparing these storage options?

7. selectdb Kubernetes Operator Update: The selectdb Kubernetes operator seems outdated. Is there an expected timeline for an update?

[yagagagaga]

1. In theory, there is no quantity limit for databases and tables. But there is a quota limit for the space occupied by each database, which defaults to 1PB. You can modify this value default_db_data_quota_bytes in the configuration file
2. Doris (Iceberg catalog) supports reading data from Iceberg, regardless of whether Iceberg is built on S3 or HDFS. This is completely different from the built-in S3 cold storage system.
3. No
4. Do you mean Broker? It is not recommended to use now.
5. You can learn more about dynamic buckets via this link. And I don't know what you mean about caching buckets.
6. No
7. If you want to get the latest operators at any time, you can join the k8s user group. 寻找社区微信群/qq群/钉钉群 #16887