Skip to main content

Data Streaming (J1DS)

JupiterOne Data Streaming (J1DS) streams entity and relationship changes from your JupiterOne graph to an Amazon S3 bucket, giving you real-time access to your data changes outside JupiterOne.

How it Works​

J1DS captures the graph database transaction logs, gathers the change events, and writes those events to your configured AWS S3 bucket. These events represent the "after" state of any change made to an Entity or Relationship in the graph. The events can be Create, Update, or Delete events.

Requirements​

  • A JupiterOne account with the J1DS entitlement
  • Administrator access to your JupiterOne account
  • An Amazon S3 bucket in the required AWS region (shown in the Data Streaming settings page)
  • Your 12-digit AWS Account ID
  • The S3 bucket name

Configuring Your S3 Bucket Policy​

Before enabling data streaming, you must add a bucket policy to your S3 bucket that grants JupiterOne write access. Without this policy, the Test Connection step will fail.

The S3 Bucket Policy panel on the Data Streaming settings page provides a ready-to-use policy with the correct AWS account ID and role already filled in for your environment. Copy the policy from that panel and replace <BUCKET_NAME> with the name of your S3 bucket.

info

The role name jupiterone-data-streaming is the same for all deployments — do not change it. This policy grants JupiterOne write-only access (s3:PutObject) to the jupiterone/* prefix in your bucket and requires encrypted transport.

Enabling Data Streaming​

  1. Navigate to Settings > Data Streaming.
  2. Review the Setup Instructions panel on the right — it shows the required AWS region for your S3 bucket.
  3. Copy the bucket policy from the S3 Bucket Policy panel and apply it to your S3 bucket.
  4. Check Enable integration.
  5. Enter your S3 Bucket Name.
  6. Enter your AWS Account ID (12 digits).
  7. Check the region confirmation checkbox to confirm your S3 bucket is in the required region.
  8. Click Test Connection — the test must succeed before you can save.
  9. Click Save.
note

You must successfully run Test Connection before saving for the first time. The Save button is not enabled until the connection test passes.

What Happens When You Enable Data Streaming​

  • JupiterOne begins capturing changes to entities and relationships in your graph.
  • Changes are streamed to your S3 bucket every few minutes (typically every 5–15 minutes). If there are no new changes, no data is written.
  • Only changes from the point of enablement forward are captured — there is no historical backfill.

Data Partitioning​

Data in your S3 bucket uses the following path structure:

jupiterone/graph/cdc/accountId=<JUPITERONE_ACCOUNT_ID>/year=<YEAR>/month=<MONTH>/day=<DAY>/time=<TIME_UTC>.jsonl

Each .jsonl file is gzip compressed and contains newline-delimited JSON records of change events since the last export. This partitioning scheme works well with standard discovery tools (e.g. AWS Glue crawler) and allows customers with multiple JupiterOne accounts to collect data into a single target bucket.

info

J1DS also writes to jupiterone/.connection-test, which is used to test connectivity from the JupiterOne platform to your S3 bucket.

Data Format​

Each record represents the "after" state of a graph object following a change. Example:

{
    "operation": "u",
    "eventType": "entity",
    "properties": {
        "_scope": "eb4f2fac-e9a1-474d-8859-5f0e5ef90b16",
        "_source": "integration-managed",
        "_key": "slack-user:team_T0129XXXXXX:user_U09B1XXXXXX",
        "_accountId": "j1dev",
        "_type": "slack_user",
        "_class": ["User"],
        "_id": "3e6fa6e5-158d-5f03-8839-a964652b57dc",
        "_deleted": false,
        "_version": 1,
        "_createdOn": 1755879780646,
        "_beginOn": 1759510655079,
        "username": "some.user",
        "email": "some.user@example.com",
        "displayName": "Some User",
        "active": true
    },
    "labels": ["Entity", "User", "slack_user"]
}
  • eventType — entity or relationship
  • operation — c (create), u (update), or d (delete)
  • properties — the full set of properties for the object after the change
  • labels — the graph labels (classes and type) for the object

Security​

  • When configuring J1DS you must provide the bucket name and the AWS account that the bucket belongs to. J1DS uses the expected bucket owner mechanism when writing, which mitigates S3 bucket hijacking.
  • The bucket policy grants write-only access — J1DS cannot read data from your bucket.
  • All data transfer requires encrypted transport (aws:SecureTransport).

Disabling Data Streaming​

  1. Navigate to Settings > Data Streaming.
  2. Uncheck Enable integration.
  3. Click Save.

Streaming stops and no new data is written to your bucket. Existing data in your S3 bucket is not affected or deleted.

Re-enabling Data Streaming​

warning

Re-enabling data streaming starts fresh. Changes that occurred while streaming was disabled are not retroactively captured. Only new changes going forward will be streamed.

To re-enable, follow the same steps described in Enabling Data Streaming.

Known Limitations​

  1. Data is written on a best-efforts basis and is designed for "at most once" delivery. The system is tolerant to transient failures but cannot buffer transactions indefinitely.
  2. S3 managed KMS encryption is supported. The PutObject calls made by J1DS (including multipart uploads) work with S3 managed KMS encryption keys.

Contents