Ctrlk
Portal
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Audit configuration

ReadonlyREST can collect audit events containing information about a request and how the system has handled it and send them to configured outputs. Here is an example of the data points contained in each audit event. We can leverage all this information to build interesting Kibana dashboards, or any other visualization.

{
    "error_message": null,
    "headers": [
      "Accept",
      "Authorization",
      "content-length",
      "Host",
      "User-Agent"
    ],
    "acl_history": "[[::LOGSTASH::->[auth_key->false]], [kibana->[auth_key->false]], [::RO::->[auth_key->false]], [::RW::->[kibana->true, indices->true, auth_key->true]]]",
    "origin": "127.0.0.1",
    "final_state": "ALLOWED",
    "task_id": 1158,
    "type": "SearchRequest",
    "req_method": "GET",
    "path": "/readonlyrest_audit-2017-06-29/_search?pretty",
    "indices": [
      "readonlyrest_audit-2017-06-29"
    ],
    "@timestamp": "2017-06-30T09:41:58Z",
    "content_len_kb": 0,
    "error_type": null,
    "processingMillis": 0,
    "action": "indices:data/read/search",
    "matched_block": "::RW::",
    "id": "933409190-292622897#1158",
    "content_len": 0,
    "logged_user": "simone",
    "presented_identity": "simone"
  }

Configuration

The audit outputs are disabled by default. To enable them, add audit.enabled: true and optionally configure audit.outputs.

Note: Even when audit.enabled is false or not set, the built-in ACL log is a special case — it writes a human-readable decision line to Elasticsearch logs for every request by default. See The default ACL log below.

The following is the explicit equivalent of the default behaviour when no audit section is configured at all:

When audit.enabled: true and no outputs are specified, ROR defaults to storing events in a local Elasticsearch index.

Global audit settings

Setting
Default
Description

enabled

false

Master switch for the audit subsystem

default_acl_log_enabled

true

Controls the built-in ACL log output (see below)

outputs

default index output

List of audit outputs

Each entry in outputs accepts an optional name field. Names are only needed for per-block routing: when you want a specific block to send events to only a subset of outputs, you reference them by name using enabled_audit_sinks or disabled_audit_sinks (see Block-level audit control). If you do not need per-block routing, you can omit name from all outputs.

The default ACL log

When default_acl_log_enabled: true (the default), ROR writes a human-readable ACL decision line to Elasticsearch logs for every request, using the logger named tech.beshu.ror.accesscontrol.logging.AccessControlListLoggingDecorator. This happens regardless of whether any outputs are configured.

The default ACL log is exposed as a named output with the reserved name default_acl_log. You can use this name in block-level enabled_audit_sinks and disabled_audit_sinks to include or exclude it from per-block routing:

To replace the default ACL log with a custom one — for example to send it to a different file — disable it globally and add a log output with the acl serializer:

logger_name is the log4j2 logger name that ROR uses when writing to this output (default: readonlyrest_audit). Setting a custom value lets you route these log lines to a dedicated appender in log4j2.properties — for example to write them to a separate file. See Custom logging settings via log4j2 for an example appender configuration.

The acl serializer produces exactly the same single-line human-readable format as the built-in ACL log: a concise decision summary that includes the request identity, matched block, final state, and key request details. The format is unchanged compared to previous versions.

Block-level audit control

The audit section inside each access_control_rules block lets you tune audit behaviour per block.

Setting
Default
Description

enabled

true

When false, no audit events are emitted when this block is matched, regardless of global settings

log_allowed_events

true

When false, allowed requests matched by this block are not written to audit. Denied requests, errors, and index-not-found responses are always written

enabled_audit_sinks

(all sinks)

Whitelist of sink names. Only the listed sinks receive events from this block. Use sink name values from audit.outputs, plus default_acl_log for the built-in ACL log

disabled_audit_sinks

(none)

Blacklist of sink names. All sinks except the listed ones receive events from this block

enabled_audit_sinks and disabled_audit_sinks are mutually exclusive — you cannot specify both on the same block.

⚠️IMPORTANT: When audit.enabled: false for a specific block, there will be no audit events at all when that block is matched — this suppresses both custom outputs and the default ACL log. This is a change in behaviour from previous versions, where block-level audit: {enabled: false} only suppressed the ES audit sinks while the ACL log continued to write.

Per-sink routing example

Multiple outputs

You can configure multiple audit outputs, including mixing output types:

Each output can be individually toggled with enabled: true/false (default: true).

Backward compatibility

The verbosity: error and verbosity: info block-level settings from earlier versions are still accepted. They are treated as aliases for audit: {log_allowed_events: false} and audit: {log_allowed_events: true} respectively.

All other global audit settings (audit.enabled, audit.outputs and their sub-settings) are unchanged. Existing configurations that do not use the new settings will continue to work without modification. The default_acl_log_enabled setting defaults to true, so the ACL log continues to fire exactly as before for configurations that do not set it explicitly.

The 'index' output specific configurations

Custom audit indices name and time granularity

By default, the ReadonlyREST audit index name template is readonlyrest_audit-YYYY-MM-DD. You can customize the name template using the index_template settings.

Example: tell ROR to write on the monthly index.

⚠️IMPORTANT: Notice the single quotes inside the double-quoted expression. This is the same syntax used for Java's SimpleDateFormat.

Custom audit cluster

It's possible to set up a custom audit cluster responsible for storing audit events. When a custom cluster is specified, items will be sent to defined cluster nodes instead of the local one.

⚠️IMPORTANT: Audit events are sent to audit nodes using a round-robin strategy. All audit nodes must belong to the same Elasticsearch cluster. Otherwise, each audit cluster will contain only a subset of audit events. If you intend to send audit events to multiple clusters, define one output per Elasticsearch cluster.

Setting audit.cluster is optional, it accepts a non-empty list of audit cluster nodes URIs.

The 'data_stream' output specific configurations

Custom audit data stream name

To change the default data stream name readonlyrest_audit, add the following configuration to your readonlyrest.yml config:

Here, custom_audit_data_stream is the Elasticsearch data stream where audit events will be stored.

If the specified data stream does not exist, it will be automatically created by the ReadonlyREST plugin. This creation process includes setting up the following components, each dedicated specifically to the configured data stream:

  • A dedicated Index Lifecycle Policy ({{data-stream-name}}-lifecycle-policy).

  • Necessary index settings and mappings (component templates: {{data-stream-name}}-mappings and {{data-stream-name}}-settings).

  • A customized Index Template ({{data-stream-name}}-template).

Custom audit cluster

It's possible to set a custom audit cluster responsible for audit events storage. When a custom cluster is specified, items will be sent to defined cluster nodes instead of the local one.

⚠️IMPORTANT: Audit events are sent to audit nodes using a round-robin strategy. All audit nodes must belong to the same Elasticsearch cluster. Otherwise, each audit cluster will contain only a subset of audit events. If you intend to send audit events to multiple clusters, define one output per Elasticsearch cluster.

Setting audit.cluster is optional, it accepts a non-empty list of audit cluster nodes URIs.

Data stream settings

Here are the default settings set for the audit data stream created by the ReadonlyREST plugin:

Audit data stream
Audit data stream template
Index lifecycle policy defaults

Managing Elasticsearch data streams, such as the ReadonlyREST audit data stream, should be customized based on your specific use case. Aspects like:

  • data retention policies (how long to keep and when to delete data),

  • migrating old indices into the new data stream,

  • handling transitions between different index lifecycle phases (e.g., hot, warm, cold, delete),

depend on your business requirements, data volume and characteristics, and how the data is analyzed and used.

Therefore, we encourage you to configure these settings yourself to best fit your needs. Elasticsearch provides flexible tools, like Index Lifecycle Management (ILM), that allow automating data management based on user-defined rules. Customizing your configuration helps optimize storage costs and search performance.

You can manage and update settings related to your audit data stream directly from Kibana's Index Management UI.

Steps to Change Data Stream Settings using Kibana

  1. Open Kibana and Navigate to Index Management

    • In Kibana, go to Management > Stack Management > Index Management.

    • Select the Data Streams tab to see the list of available data streams.

  2. Select Your Audit Data Stream

    • Find your audit data stream (e.g., custom_audit_data_stream) in the list.

    • Click on it to view details such as indices backing the data stream, mappings, and lifecycle policies.

  3. Edit Index Lifecycle Policy (ILM)

    • If you want to update rollover criteria, retention period, or other lifecycle actions:

      • Navigate to Index Lifecycle Policies under Stack Management.

      • Select the ILM policy associated with your audit data stream.

      • Modify phases such as hot, warm, or delete to adjust settings like maximum size, max age, or deletion timing.

      • Save your changes — they will be applied automatically to the indices backing the data stream.

  4. Update Index Template

    • To change index settings or mappings for new backing indices:

      • Go to Index Templates in Stack Management.

      • Locate the template associated with your audit data stream (usually matching the data stream name or pattern).

      • Edit the template's settings or mappings as needed.

      • Save the updated template; new indices created for the data stream will use these settings.

  5. Verify Changes

    • After updating policies or templates, monitor your data stream to ensure rollover and retention behave as expected.

    • You can also query audit events via Kibana's Discover tab or using the Elasticsearch API.

Important Notes

  • Changes to lifecycle policies and index templates affect new indices created after the update; existing indices are not modified retroactively.

  • To apply mapping changes to existing indices, you may need to reindex data.

  • Ensure you carefully test ILM and template changes in a staging environment before applying to production audit streams.

Rolling Migration from index to data_stream

To migrate ReadonlyREST audit logging from the index output type to data_stream in a rolling update, follow this safe, zero-downtime approach:

  1. Add data_stream as an Additional Output

Temporarily configure both index and data_stream outputs so that audit events are sent to both destinations:

✅ This ensures no audit logs are lost during the transition.

  1. Verify Data Stream Creation

Ensure the data stream is being created and audit events are flowing in.

  1. Monitor for Consistency

Use Kibana or the _search API to confirm that events are present in both audit indices and custom_audit_data_stream.

  1. (Optional) Backfill Historical Data

If you wish to migrate historical audit data from the old audit index, you can reindex it manually:

⚠️ Ensure both audit outputs have the same serializer for data consistency.

⚠️ Data streams are append-only — use "op_type": "create" to avoid overwrites.

⚠️ If the source index contains documents already present in the destination data stream, "conflicts": "proceed" will skip duplicates.

  1. Remove the index Output

After confirming successful logging to the data stream from all nodes, update your config to remove the index output:

  1. Final Verification

Use Kibana dashboards, metrics, or direct queries to confirm that new audit events are flowing into the configured data stream.

The 'log' output specific configurations

The log output writes audit events to Elasticsearch log at INFO level using a dedicated logger.

Built-in rolling file appender

For a self-contained rolling file output — without editing log4j2.properties — use the file_appender sub-section:

When file_appender is present, ROR creates and manages the rolling appender internally, bypassing the default Elasticsearch log routing for this output. The logger_name setting is still accepted and used as the appender name.

Custom logger name

If you want to route log output through a specific log4j2 logger:

The default logger name is readonlyrest_audit.

Custom logging settings via log4j2

For advanced log configuration — custom patterns, external syslog appenders, etc. — configure the logger in $ES_PATH_CONF/config/log4j2.properties. The logger name must match the logger_name value in the output config (or the default readonlyrest_audit):

ACL serializer

The log output type supports a special acl serializer that reproduces the human-readable format written by the default ACL log. This is useful when you want to disable the built-in ACL log (default_acl_log_enabled: false) and replace it with a custom log sink that you can route per-block:

The acl serializer type is only valid for log outputs. Attempting to use it with index or data_stream outputs will produce a configuration error.

Extending audit events

The audit events are JSON documents describing incoming requests and how the system has handled them. To create such events, we use a serializer, which is responsible for the event's serialization and filtering. The example event is in default format and was produced by the default serializer (tech.beshu.ror.audit.instances.BlockVerbosityAwareAuditLogSerializer).

You can:

Predefined serializers:

  • tech.beshu.ror.audit.instances.BlockVerbosityAwareAuditLogSerializer

    • Serializes all non-Allowed events.

    • Serializes Allowed events only when the matched block has log_allowed_events: true (the default).

    • Recommended for standard audit logging, where full request body capture is not required.

    • Fields included:

  • tech.beshu.ror.audit.instances.QueryAuditLogSerializer

    • Similar to the BlockVerbosityAwareAuditLogSerializer regarding Allowed event handling and included JSON fields.

    • Additionally, captures the full request body (content field)

    • Recommended for standard audit logging, where full request body capture is required.

  • tech.beshu.ror.audit.instances.FullAuditLogSerializer

    • Serializes all events of all types, including all Allowed events, regardless of the block's log_allowed_events setting.

    • Included fields are the same as for BlockVerbosityAwareAuditLogSerializer

    • Use this serializer, when you need complete coverage of all events.

  • tech.beshu.ror.audit.instances.FullAuditLogWithQuerySerializer

    • Serializes all events of all types, including all Allowed events, regardless of the block's log_allowed_events setting.

    • Included fields are the same as for QueryAuditLogSerializer (includes content field - full request body)

    • Use this serializer, when you need complete coverage of all events with full request body.

Using configurable serializer:

Configuration should look like that:

The configuration above corresponds to serialized event looking like that:

You can also define nested structure of fields, and use fixed text, number and boolean values:

The configuration above corresponds to serialized event looking like that:

Available placeholders:

Using ECS serializer:

Configuration should look like that:

The configuration above corresponds to serialized event, compatible with ECS 1.6 schema, looking like that:

The ECS schema is highly permissive and ambiguous. The ROR audit events can be mapped to ECS fields in multiple ways. If the provided ECS implementation does not suit your needs, you can define your own ECS-compliant serializer as configurable serializer.

The provided ECS implementation is equivalent to configurable serializer shown below. You can use and adjust it as needed in the configuration.

Custom audit event serializer

You can write your own custom audit events serializer class, add it to the ROR plugin class path and configure it through the YAML settings.

We provided 2 project examples with custom serializers (in Scala and Java). You can use them as an example to write yours in one of those languages.

Create custom audit event serializer in Scala

  1. Checkout https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin

    git clone git@github.com:sscarduzio/elasticsearch-readonlyrest-plugin.git

  2. Install SBT

    https://www.scala-sbt.org/download.html

  3. Find and go to: elasticsearch-readonlyrest-plugin/custom-audit-examples/ror-custom-scala-serializer/

  4. Create own serializer:

    • from scratch (example can be found in class ScalaCustomAuditLogSerializer)

    • extending default one (example can be found in class ScalaCustomAuditLogSerializer)

  5. Build serializer JAR:

    sbt assembly

  6. Jar can be find in:

    elasticsearch-readonlyrest-plugin/custom-audit-examples/ror-custom-scala-serializer/target/scala-2.13/ror-custom-scala-serializer-1.0.0.jar

Create custom audit event serializer in Java

  1. Checkout https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin

    git clone git@github.com:sscarduzio/elasticsearch-readonlyrest-plugin.git

  2. Install Maven

    https://maven.apache.org/install.html

  3. Find and go to: elasticsearch-readonlyrest-plugin/custom-audit-examples/ror-custom-java-serializer/

  4. Create own serializer:

    • from scratch (example can be found in class JavaCustomAuditLogSerializer)

    • extending default one (example can be found in class JavaCustomAuditLogSerializer)

  5. Build serializer JAR:

    mvn package

  6. Jar can be find in:

    elasticsearch-readonlyrest-plugin/custom-audit-examples/ror-custom-java-serializer/target/ror-custom-java-serializer-1.0.0.jar

Configuration

  1. mv ror-custom-java-serializer-1.0.0.jar plugins/readonlyrest/

  2. Your config/readonlyrest.yml should start like this

  3. Start elasticsearch (with ROR installed) and grep for:

Protecting the audit index

To prevent users from modifying or deleting audit data, add a forbid block to your readonlyrest.yml that blocks write and delete actions on the audit indices.

Placement: ACL blocks are evaluated top-to-bottom and the first matching block wins.

  • If no user or service should be able to write to the audit index via the Elasticsearch API, place this block at the very beginning of the ACL. Audit events are written internally by the ReadonlyREST plugin and bypass the ACL entirely, so this does not affect audit collection.

  • If some identities (e.g. a dedicated audit reader service) require access that would conflict with this rule, place the forbid block after their allow blocks but before all other allow rules.

The wildcard pattern readonlyrest_audit-* matches the default index name template. If you configured a custom index_template prefix, adjust the pattern accordingly.

PreviousFor ElasticsearchNextExternal to local groups mapping

Last updated