Skip to main content

Data Pipeline Upgrade Overview and Compatibility

Overview​

JupiterOne is upgrading the data pipeline used to bring data into JupiterOne. The project is intended to significantly improve data ingest performance, reducing the time needed to process incoming data and allow the graph to more quickly reflect the data sent from an integration.

There are very few functional changes that impact customers, but if you do make use of JupiterOne's APIs for sending custom data or manipulating data in the graph please review these documents carefully.

info

These changes will start rolling out to customers from Monday 28th April 2025

API Compatibility and Other Changes​

The compatibility changes have been grouped into MAJOR, MEDIUM, and MINOR categories:

  • MAJOR: Indicates a change in behavior that may significantly change API usage or APIs becoming unavailable
  • MEDIUM: Covers changes where APIs change in behaviour but no particular actions are required from customers, or data in the graph may change in a way that impacts downstream processing
  • MINOR: More minor changes, removal of undocumented or unused functionality, or additional metadata being added

CREATE_OR_UPDATE Sync Jobs MAJOR​

The CREATE_OR_UPDATE sync job mode will no longer be supported. Customers using the CREATE_OR_UPDATE mode should use the PATCH mode. More information on how to migrate to this mode is available here.

No Versioned Raw Data MEDIUM​

The Persister APIs previously allowed fetching of Raw Data by version. This will no longer be supported, only the latest version of Raw Data is retained.

New UUIDv5 _id Metadata MINOR​

Existing objects _id will be updated to a new, stable, deterministic UUIDv5. This UUID is derived from the _scope and _key of the object, i.e. the natural identifiers. Customers should not be relying on the _id property being stable for a given entity prior to this change.

Duplicate _key + _scope Constraints Enforced MINOR​

Prior to this upgrade the system would partially tollerate entities that had duplicated _key and _scope combinations. The behaviour in this case was undefined. The new data pipeline will strictly enforce this constraint. Any duplicated _key and _scope data will be dropped at ingest time.

_scope Required and Defaulted MINOR​

The _scope property was previously optional and the system would derive the scope through a combination of the data source and any _integrationInstanceId that may have been provided. _scope is now a required property, and will be set to a default value for certain routes, e.g. certain APIs will default to _scope = api.

No PUT of Raw Data MINOR​

It will no longer be possible to directly set the Raw Data for an entity using a PUT. The raw data must be ingested as part of a sync job and cannot be updated outside of that flow.

Invalid Relationships are Immediately Purged MINOR​

Relationships that are no longer valid (i.e their source or target node is now deleted) will be immediately deleted from the graph and not retained as a "soft deleted" entity.

info

This change does not effect the data retention period of deleted entities, only relationships.

PATCH Sync Jobs cannot target Relationships​

PATCH jobs will not support patching relationships. Either a full DIFF mode sync job must be used or the updateRelationship mutations via GraphQL.

Contents