Howto forward to S3 Bucket

Configuration

The configuration of the component is very versatile and depends on how you intend to use it. You could hardcode the s3 bucket and endpoint directly in the config section, but using environment variables allows you to easily adapt to multi cluster setups.

env:
  s3_bucket: <NAME_OF_S3_BUCKET>
  s3_endpoint: <URL_TO_S3_STORAGE>

secrets:
  shared_key: '?{vaultkv:${cluster:tenant}/${cluster:name}/fluentd-forwarder/${_instance}/sharedkey}'
  s3_accesskey: '?{vaultkv:${cluster:tenant}/${cluster:name}/fluentd-forwarder/${_instance}/s3_accesskey}'
  s3_secretkey: '?{vaultkv:${cluster:tenant}/${cluster:name}/fluentd-forwarder/${_instance}/s3_secretkey}'

Using config_vars helps reducing recurring configuration values, for example if you want to filter different namespaces. In this example each namespace will be stored in a different directory.

config: |
  <system>
    log_level info
  </system>
  <source>
    @type forward
    port 24224
    null
    <security>
      shared_key "#{ENV['SHARED_KEY']}"
      self_hostname "#{ENV['HOSTNAME']}"
    </security>
  </source>

  <match kubernetes.**>
    @type rewrite_tag_filter
    <rule>
      key \$['kubernetes']['namespace_name']
      pattern ^(.+)$
      tag kube.$1
    </rule>
  </match>

  <match kube.my_namespace>
    %(match_tmpl)s
    path my_namespace/
  </match>

  <match kube.my_other_namespace>
    %(match_tmpl)s
    path my_namespace/
  </match>

The repetitive part of the <match> configuration can be exported using config_vars.

config_vars:
  match_tmpl: |
    @type s3
    <format>
      @type json
    </format>
    aws_key_id "#{ENV['S3_ACCESSKEY']}"
    aws_sec_key "#{ENV['S3_SECRETKEY']}"
    s3_bucket "#{ENV['S3_BUCKET']}"
    s3_endpoint "#{ENV['S3_ENDPOINT']}"
    time_slice_format %Y-%m-%d_%H%M
    <buffer time>
      @type memory
      compress text
      chunk_limit_size 256m
      timekey 5m
      timekey_wait 1m
      timekey_use_utc true
    </buffer>