Howto forward to S3 Bucket
Configuration
The configuration of the component is very versatile and depends on how you intend to use it.
You could hardcode the s3 bucket and endpoint directly in the config
section, but using environment variables allows you to easily adapt to multi cluster setups.
env:
s3_bucket: <NAME_OF_S3_BUCKET>
s3_endpoint: <URL_TO_S3_STORAGE>
secrets:
shared_key: '?{vaultkv:${cluster:tenant}/${cluster:name}/fluentd-forwarder/${_instance}/sharedkey}'
s3_accesskey: '?{vaultkv:${cluster:tenant}/${cluster:name}/fluentd-forwarder/${_instance}/s3_accesskey}'
s3_secretkey: '?{vaultkv:${cluster:tenant}/${cluster:name}/fluentd-forwarder/${_instance}/s3_secretkey}'
Using config_vars
helps reducing recurring configuration values, for example if you want to filter different namespaces.
In this example each namespace will be stored in a different directory.
config: |
<system>
log_level info
</system>
<source>
@type forward
port 24224
null
<security>
shared_key "#{ENV['SHARED_KEY']}"
self_hostname "#{ENV['HOSTNAME']}"
</security>
</source>
<match kubernetes.**>
@type rewrite_tag_filter
<rule>
key \$['kubernetes']['namespace_name']
pattern ^(.+)$
tag kube.$1
</rule>
</match>
<match kube.my_namespace>
%(match_tmpl)s
path my_namespace/
</match>
<match kube.my_other_namespace>
%(match_tmpl)s
path my_namespace/
</match>
The repetitive part of the <match>
configuration can be exported using config_vars
.
config_vars:
match_tmpl: |
@type s3
<format>
@type json
</format>
aws_key_id "#{ENV['S3_ACCESSKEY']}"
aws_sec_key "#{ENV['S3_SECRETKEY']}"
s3_bucket "#{ENV['S3_BUCKET']}"
s3_endpoint "#{ENV['S3_ENDPOINT']}"
time_slice_format %Y-%m-%d_%H%M
<buffer time>
@type memory
compress text
chunk_limit_size 256m
timekey 5m
timekey_wait 1m
timekey_use_utc true
</buffer>