Skip to main content

Data Streaming (J1DS)

JupiterOne Data Streaming (J1DS) streams your JupiterOne data to an Amazon S3 bucket you own, giving you access to it outside JupiterOne. Two streams are available, and each can be enabled on its own:

StreamContents
Graph changesEntity and relationship changes from your graph
Audit logAccount activity events — who did what, and when

Both write to the same bucket under separate prefixes, so you configure the destination once.

How it Works​

Graph changes. J1DS captures the graph database transaction logs, gathers the change events, and writes those events to your configured AWS S3 bucket. These events represent the "after" state of any change made to an Entity or Relationship in the graph. The events can be Create, Update, or Delete events.

Audit log. J1DS periodically reads your account's activity events and writes them to the same bucket. These are the same records shown under Settings > Audit Events in the JupiterOne UI, in a form suited to long-term retention and querying.

Requirements​

  • A JupiterOne account with the J1DS entitlement (covers both streams)
  • Administrator access to your JupiterOne account
  • An Amazon S3 bucket in the required AWS region (shown in the Data Streaming settings page)
  • Your 12-digit AWS Account ID
  • The S3 bucket name

Configuring Your S3 Bucket Policy​

Before enabling data streaming, you must add a bucket policy to your S3 bucket that grants JupiterOne write access. Without this policy, the Test Connection step will fail.

The S3 Bucket Policy panel on the Data Streaming settings page provides a ready-to-use policy with the correct AWS account ID and role already filled in for your environment. Copy the policy from that panel and replace <BUCKET_NAME> with the name of your S3 bucket.

info

The role name jupiterone-data-streaming is the same for all deployments — do not change it. This policy grants JupiterOne write-only access (s3:PutObject) to the jupiterone/* prefix in your bucket and requires encrypted transport.

Enabling Data Streaming​

  1. Navigate to Settings > Data Streaming.

  2. Review the Setup Instructions panel on the right — it shows the required AWS region for your S3 bucket.

  3. Copy the bucket policy from the S3 Bucket Policy panel and apply it to your S3 bucket.

  4. Turn on the streams you want:

    • Stream graph changes — entity and relationship changes
    • Stream audit log — account activity events

    You can enable either, or both. Turning on at least one enables the fields below.

  5. Enter your S3 Bucket Name.

  6. Enter your AWS Account ID (12 digits).

  7. Check the region confirmation checkbox to confirm your S3 bucket is in the required region.

  8. Click Test Connection — the test must succeed before you can save.

  9. Click Save.

note

You must successfully run Test Connection before saving for the first time. The Save button is not enabled until the connection test passes.

tip

Enabling only the audit log is a supported configuration. Because graph streaming is what requires change capture on your graph database, leaving it off means your account carries none of that overhead while still receiving audit data.

What Happens When You Enable Data Streaming​

  • Data is written to your S3 bucket every few minutes (typically every 5–15 minutes). If there is nothing new, no file is written at all.
  • Only data from the point of enablement forward is captured — there is no historical backfill, for either stream.
  • With graph changes on, JupiterOne begins capturing changes to entities and relationships in your graph.
  • With audit log on, audit events are exported once they have settled. An event becomes eligible a few minutes after it occurs, so the audit stream trails live activity slightly more than the graph stream does. Nothing is skipped as a result — the delay exists precisely so that events still in flight are not missed.

Data Partitioning​

Each stream writes under its own prefix, with the same partitioning below it:

jupiterone/graph/cdc/accountId=<JUPITERONE_ACCOUNT_ID>/year=<YEAR>/month=<MONTH>/day=<DAY>/time=<TIME_UTC>.jsonl.gz
jupiterone/audit/accountId=<JUPITERONE_ACCOUNT_ID>/year=<YEAR>/month=<MONTH>/day=<DAY>/time=<TIME_UTC>.jsonl.gz

Each file is gzip compressed and contains newline-delimited JSON records covering the period since the last export. This partitioning scheme works well with standard discovery tools (e.g. AWS Glue crawler) and allows customers with multiple JupiterOne accounts to collect data into a single target bucket.

Because the two streams occupy separate prefixes, you can grant access to one and not the other, apply different lifecycle rules to each, or crawl them as separate tables.

info

J1DS also writes to jupiterone/.connection-test, which is used to test connectivity from the JupiterOne platform to your S3 bucket.

Data Format​

Graph changes​

Each record represents the "after" state of a graph object following a change. Example:

{
    "operation": "u",
    "eventType": "entity",
    "properties": {
        "_scope": "eb4f2fac-e9a1-474d-8859-5f0e5ef90b16",
        "_source": "integration-managed",
        "_key": "slack-user:team_T0129XXXXXX:user_U09B1XXXXXX",
        "_accountId": "j1dev",
        "_type": "slack_user",
        "_class": ["User"],
        "_id": "3e6fa6e5-158d-5f03-8839-a964652b57dc",
        "_deleted": false,
        "_version": 1,
        "_createdOn": 1755879780646,
        "_beginOn": 1759510655079,
        "username": "some.user",
        "email": "some.user@example.com",
        "displayName": "Some User",
        "active": true
    },
    "labels": ["Entity", "User", "slack_user"]
}
  • eventType — entity or relationship
  • operation — c (create), u (update), or d (delete)
  • properties — the full set of properties for the object after the change
  • labels — the graph labels (classes and type) for the object

Audit log​

Each record is one activity event. Example:

{
    "id": "c95f969f-9f95-4822-a075-8271156a6f3a",
    "accountId": "j1dev",
    "timestamp": 1787747715680,
    "category": "audit/create",
    "resourceType": "iam:token",
    "resourceId": "9c1019cc-adda-44e8-8923-74ebf3c3e79d",
    "performedByUserId": "someone@example.com",
    "data": {
        "username": "someone@example.com",
        "tokenName": "user-session-566c7868",
        "tokenType": "user-session"
    }
}
  • id — unique identifier for the event. Use it to deduplicate (see Known Limitations)
  • accountId — your JupiterOne account, included in the record as well as the path so records stay self-describing when merged across accounts
  • timestamp — when the action occurred, in epoch milliseconds
  • category — the action, prefixed audit/; for example audit/create, audit/update, audit/delete, audit/authenticate
  • resourceType / resourceId — what was acted on
  • performedByUserId — who performed it, resolved to an email address where possible. May be absent: the field is optional, and events raised by the platform itself rather than by a person carry no user. Treat it as nullable when you define a table schema over these files
  • data — event-specific detail; the shape varies by resourceType
note

Records within a single audit file are ordered newest first. Order across files is not guaranteed, so sort by timestamp if sequence matters to you.

Security​

  • When configuring J1DS you must provide the bucket name and the AWS account that the bucket belongs to. J1DS uses the expected bucket owner mechanism when writing, which mitigates S3 bucket hijacking.
  • The bucket policy grants write-only access — J1DS cannot read data from your bucket.
  • All data transfer requires encrypted transport (aws:SecureTransport).

Disabling Data Streaming​

  1. Navigate to Settings > Data Streaming.
  2. Turn off the stream you no longer want — or both.
  3. Click Save.

That stream stops and no new data is written for it. Existing data in your S3 bucket is not affected or deleted, and the other stream is unaffected if you left it on.

Re-enabling Data Streaming​

warning

Re-enabling a stream starts fresh. Anything that occurred while it was disabled is not retroactively captured. Only new data going forward will be streamed. This applies to both streams.

To re-enable, follow the same steps described in Enabling Data Streaming.

Known Limitations​

  1. Graph change data is written on a best-efforts basis and is designed for "at most once" delivery. The system is tolerant to transient failures but cannot buffer transactions indefinitely.
  2. Audit log data is designed for "at least once" delivery, so an event can appear more than once — for example if an export is retried. Deduplicate on the id field, which is stable for a given event.
  3. S3 managed KMS encryption is supported. The PutObject calls made by J1DS (including multipart uploads) work with S3 managed KMS encryption keys.
  4. Audit records carry the fields listed under Data Format. Resource display names and descriptions are not included as separate fields; where available, equivalent detail appears inside data.

Contents