Release H: Minio Events
- 1 Introduction
- 1.1 Notifications to Postgres
- 1.1.1 Minio ENV
- 1.2 Notifications to Kafka
- 1.2.1 Minio ENV
- 1.1 Notifications to Postgres
- 2 Links
Introduction
MinIO bucket notifications allows user to send notifications on bucket events to external services
Notifications to Postgres
To send bucket notifications to Postgres you first need to setup the following environment variables:
Minio ENV
- name: MINIO_NOTIFY_POSTGRES_ENABLE_PRIMARY
value: "on"
- name: MINIO_NOTIFY_POSTGRES_CONNECTION_STRING_PRIMARY
value: "postgresql://minio:minio@postgres.default:5432/minio?sslmode=disable"
- name: MINIO_NOTIFY_POSTGRES_TABLE_PRIMARY
value: "minioevents"
- name: MINIO_NOTIFY_POSTGRES_FORMAT_PRIMARY
value: "namespace"
- name: MINIO_NOTIFY_POSTGRES_MAX_OPEN_CONNECTIONS_PRIMARY
value: "2"
- name: MINIO_NOTIFY_POSTGRES_QUEUE_DIR_PRIMARY
value: "/opt/minio/events"
- name: MINIO_NOTIFY_POSTGRES_QUEUE_LIMIT_PRIMARY
value: "100000"
- name: MINIO_NOTIFY_POSTGRES_COMMENT_PRIMARY
value: "PostgreSQL Notification Event Logging for MinIO"In this example I have used an identifier of PRIMARY but you can use whatever you prefer.
Note: I included the "sslmode=disable" parameter in the Postgres connection string as I'm not running Postgres over HTTPS.
When Minio starts you should see the following line in your log: SQS ARNs: arn:minio:sqs::PRIMARY:postgresql.
This indicates event notifications to Postgres are available.
The next thing you need to do is enable eventing on whatever bucket you want to monitor.
/minio$ mc event add myminio/py-bucket arn:minio:sqs::PRIMARY:postgresql
Successfully added arn:minio:sqs::PRIMARY:postgresql
/minio$ mc event list myminio/py-bucket arn:minio:sqs::PRIMARY:postgresql
arn:minio:sqs::PRIMARY:postgresql s3:ObjectCreated:*,s3:ObjectRemoved:*,s3:ObjectAccessed:* Filter:The second command lists the events Minio will notify on.
When you upload a file:
/minio$ mc cp test.txt myminio/py-bucket
.../go/minio/test.txt: 13 B / 13 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 687 B/s 0sYou see a new record in the minioevents table in your Postgres database
Key | Value |
|---|---|
py-bucket/test.txt | {"Records": [{"s3": {"bucket": {"arn": "arn:aws:s3:::py-bucket", "name": "py-bucket", "ownerIdentity": {"princi |
Note: The minioevents table will be created if it does not already exists
Notifications to Kafka
To send bucket notifications to Kafka you first need to setup the following environment variables (if you are using tls authentication):
Minio ENV
- name: MINIO_NOTIFY_KAFKA_ENABLE_PRIMARY
value: "on"
- name: MINIO_NOTIFY_KAFKA_BROKERS_PRIMARY
value: "my-cluster-kafka-external-0.kafka:9098"
- name: MINIO_NOTIFY_KAFKA_TOPIC_PRIMARY
value: "my-topic"
- name: MINIO_NOTIFY_KAFKA_TLS_PRIMARY
value: "on"
- name: MINIO_NOTIFY_KAFKA_TLS_SKIP_VERIFY_PRIMARY
value: "on"
- name: MINIO_NOTIFY_KAFKA_CLIENT_TLS_CERT_PRIMARY
value: "/etc/kafka/ssl/user.crt"
- name: MINIO_NOTIFY_KAFKA_CLIENT_TLS_KEY_PRIMARY
value: "/etc/kafka/ssl/user.key"Note: You will also need to copy the required tls certs into a secret and mount them somewhere in the Minio pod.
In this example I have used an identifier of PRIMARY but you can use whatever you prefer.
When Minio starts you should see the following line in your log: SQS ARNs: arn:minio:sqs::PRIMARY:kafka
This indicates event notifications to Kafka are available.
The next thing you need to do is enable eventing on whatever bucket you want to monitor.
/minio$ mc event add myminio/encrypt arn:minio:sqs::PRIMARY:kafka
Successfully added arn:minio:sqs::PRIMARY:kafka
/minio$ mc event list myminio/encrypt arn:minio:sqs::PRIMARY:kafka
arn:minio:sqs::PRIMARY:kafka s3:ObjectCreated:*,s3:ObjectRemoved:*,s3:ObjectAccessed:* Filter:The second command lists the events Minio will notify on.
When you upload a file:
/minio$ mc cp test.txt myminio/encrypt
.../go/minio/test.txt: 13 B / 13 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 78 B/s 0sYou see a new record in "my-topic"
