Custom File Transfer

Leverage JupiterOne's Custom File Transfer to ingest assets into JupiterOne and designate the asset's relationships prior to uploading. This integration allows you to import a batch of assets and relationships via CSV file and track and manage each upload within the integration's instances.

You will need to have your assets and relationships in CSV format, and there is a size limitation of 100MB per file. For larger uploads, see our API documentation.

CSV File Requirements

To ensure data is correctly ingested, please consider the following:

• The first row in the CSV must be headers. These headers will be used as the property values for the final entity.
• We encourage the use of camelCasing for headers for consistency with other data in your graph.
• We encourage the use of Data Type Tokens to improve query-ability of your data (see below for more details).
• If a blank or whitespace header is provided, it will be replaced by idxN where N is the numerical index of that column.
• The number of values in each row must match the number of headers provided on the first row. A mismatch will result in an error.
• All empty (,,) and blank (,"",) values will be replaced with undefined to improve query-ability of your data.
• Whitespace (" " or , ,) values will be maintained.

Data Type Tokens (optional)

In order for the integration to successfully identify different types of data, the column headers can optionally include a token to identify the columns data type. By providing these data type tokens, JupiterOne will store the data in the correct format allowing for advanced J1QL queries to be written.

Supported Data Type Tokens:

• [datetime]: With the [datetime] token present in the header, all valid ISO dates will be parsed and stored ready for queries as a date. To ensure dates are displayed correctly in the JupiterOne app, column header names should end in On. Parsed dates will allow queries such as FIND approvals WITH approvedOn < Date.now - 30days
• [boolean]: With the [boolean] token present in the header, the following values will be parsed as true: true, 1, on, yes. All other values will be considered false. Parsed booleans will allow for queries such as FIND approvals WITH approved = true
• [number]: With the [number] token present in the header, all valid numbers (int, decimal, negative) will be parsed as a number. If a value is not a number, the value will remain unparsed. Parsed numbers will allow queries such as FIND approvals WITH count > 50
• [stringList]: When the [stringList] token is included in a column header, the corresponding cell value will be split into an array of strings using a comma (,) as the delimiter. If no commas are found, the result will be a single-element array containing the entire string.

  • It is recommended to enclose the full list of comma-separated values in double quotes ("). This ensures the list is interpreted as a single CSV field and parsed correctly.

  • For example, if you have an entity property named webUrls[stringList], you can specify the values in the CSV like this:

    • ✅ Correct: "https://www.j1.com,https://www.j1.io"
    • ❌ Incorrect: "https://www.j1.com","https://www.j1.io"

note

Adding a Data Type Token to the column header name that uniquely identifies each row is not supported. The value entered as the Entity Key Property should exactly match the value found in the CSV.

Example walkthrough

Starting CSV:

id,createDate,description,approvedOn,approved,count,weekdays
1,01/01/2024,Granted access to fix the issue,01/01/2024,true,100,"monday,tuesday"
2,02/11/2014,Test access granted,02/11/2014,testBool,100,"tuesday,wednesday"

In this example, createDate and approvedOn are both dates. id and count are both numbers. approved is a boolean. id will not be considered a number, it will be used as the Entity Key Property which is explained later in this document.

For the integration to identify them as their appropriate data type, add the data type tokens to the applicable header. For example, createDate becomes createDate[datetime].

Tips: Add the data type token after the column header name. For timestamps, renaming the column to end with On is preferable for best display support in the JupiterOne app and more consistent with the JupiterOne data model. So for example, createDate becomes createdOn.

Example with Data Type Tokens:

id,createdOn[datetime],description,approvedOn[datetime],approved[boolean],count[number],weekdays[stringList]
1,02/11/2014,Granted access to fix the issue,02/11/2014 11:30:30,true,100,"monday,tuesday"
2,02/11/2014,Test access granted,02/11/2014,testBool,100,"tuesday,wednesday"

Ensure that the date time values are in a valid ISO 8601 format:

id,createdOn[datetime],description,approvedOn[datetime],approved[boolean],count[number],weekdays[stringList]
1,2014-02-11Z,Granted access to fix the issue,2014-02-11T11:30:30Z,true,100,"monday,,tuesday"
2,2014-02-11Z,Test access granted,2014-02-11Z,testBool,100xx,"tuesday,wednesday,"

Below is an example of the resulting entities post-processing. If a value is not successfully parsed for the given data type, the value will remain in its original format. The data type token will be removed prior to use as a property name.

[
  {
    "id": 1,
    "createdOn": 1392076800000, // unix epoch timestamp, will be displayed correctly as ISO timestamp in JupiterOne.
    "description": "Granted access to fix the issue",
    "approvedOn": 1392118230000,
    "approved": true,
    "count": 100,
    "weekdays": ['monday', 'tuesday']
  },
  {
    "id": 2,
    "createdOn": 1392076800000, // unix epoch timestamp, will be displayed correctly as ISO timestamp in JupiterOne.
    "description": "Test access granted",
    "approvedOn": 1392118230000,
    "approved": false, // testBool was parsed as false
    "count": "100xx", // was not parsed
    "weekdays": ['tuesday', 'wednesday']
  }
]

Configuring the integration

Navigate to the Integrations tab in JupiterOne and select Custom File Transfer. Click New Instance to begin configuring the integration.

For this integration, you will need to have your CSV dataset ready to imported.

Creating a Custom File Transfer instance

Provide the following when creating an instance for your custom file transfer:

• The Account Name used to identify the account in JupiterOne. Ingested entities will have this value stored in tag.AccountName when the AccountName toggle is enabled.

• Description to assist in identifying the integration instance, if desired.

• Enter the Dataset Name. For the first example data set, use: EuropaUsers

• Define the Entity Key Property. This is found in the data file. For CSVs, it is the column header. This value should be unique across the entire dataset. Adding a Data Type Token to this column header is not supported.

• Add the Entity Type. This will identify the specific type of entity to be created.

• Select the Entity Class. This value represents the class of data found in the dataset.

info

For more information about JupiterOne _type and _class values, see our JupiterOne data model.

At this point, the integration is able to upload and ingest the defined dataset into your account. Press Create to create the instance and upload your CSV file.

If you wish to add additional datasets, press the Add Dataset button.

Additionally, you can add Direct and Mapped Relationships by selecting the respective Add relationship button. You can find additional information on adding relationships below.

Adding another data set:

• Enter the Dataset Name, the Entity Key Property, and Entity Type.

• Select the Entity Class.

Defining Relationships

When uploading your file, you can additionally define both direct and mapped relationships for the dataset.

When defining a relationship, FORWARDS represents source → target, and REVERSE represents source ← target (target → source).

This determines the relationship between the entities and which direction the arrow points between them, allowing for illustrating ingress and egress accordingly.

To define direct relationships:

Press Add Relationship under Direct Relationships and provide the following:

• The Source Property. The source property indicates what value in the dataset references the target dataset.
• The Relationship Class
• The Target Dataset Name. This needs to exactly match the inputted value for the target Dataset Name.
• The Target Property. The target property indicates what value in the dataset that should be referenced by the source dataset.

Define Mapped Relationships

Mapped relationships can be defined similarly to direct relationships.

To define the mapped relationships:

• Select the Relationship Class.
• Specify the Direction.
• Enter the Target Entity Type.
  note

  To see what _types are available in your account, consider running this J1QL query in your account: FIND * AS e RETURN count(e), e._type. For this tutorial, use: _githubuser

Add Field Mappings

Enter the Source Property and Target Property as desired.

Finalize the integration instance

Click Create once all values are provided to finalize the integration. You will then be prompted to upload your CSV files.

Custom file transfer example

To understand how to set up this integration, the following dataset will be used for a tutorial.

Imagine you want to add User and User Group data from your HR software, called Europa, into JupiterOne.

Here are the two example datasets:

1. Europa Users (europaUsers.csv)

    Id	Name	Username	Email	GroupId
    1	Jane	jane.c	jane.c@company.com	A
    2	Juan	juan.d	juan.d@company.com	B
    3	Julie	julie.e	julie.e@company.com	C

2. Europa User Groups (europaUserGroups.csv)

    Id	Name
    A	Marketing
    B	Sales
    C	Engineering

We can represent both the Europa Users and Europa Groups as assets in the JupiterOne graph.

Further, we can use the native relationship between the Europa Users GroupId property and the Europa Groups Id property to define a direct relationship between the two datasets in this integration.

Finally, if we want, we could also use the Europa Users Email property to define a mapped relationship to any other data that exists in your JupiterOne account.

Define Datasets

• Enter the Dataset Name as: EuropaUsers
• Enter the Entity Key Property. For this tutorial the first column header of the Europa Users dataset we'll use is: Id
• Enter the Entity Type as: europa_user.
• Set the Entity Class as: User.

At this point the integration is able to upload and ingest the Europa Users dataset into your account. With the goal of adding Users Groups and relationships, let's keep going:

• Press the Add Dataset button
• Enter the Dataset Name. For the second dataset: EuropaUserGroups
• Enter the Entity Key Property. For this tutorial we will use the Id column which represents a unique id for each entity.
• Enter the Entity Type. For the second dataset: europa_user_group
• Select the Entity Class. For the second: UserGroup

Excellent! Both datasets are now defined.

Define the direct relationship

The following steps will define the relationship to be built between Users (the source dataset) and User Groups (the target dataset).

User (source) ←HAS→ User Group (target) EuropaUsers.GroupId == EuropaUserGroups.Id

The following steps will build the above relationship:

• Press Add Relationship under Direct Relationships and
• Enter the Source Property. For this tutorial, useGroupId as the source property. It is found in EuropaUsers and references the Europa User Groups dataset.
• Select the Relationship Class. For this tutorial: Has
• Select the Direction. For this tutorial: REVERSE
• Enter the Target Dataset Name. For this tutorial: EuropaUserGroups
• Enter the Target Property. For this tutorial, Id is the target property, found in EuropaUserGroups. Enter Id for this field.

Define Mapped Relationship

The following steps will define the relationship to be built between EuropaUsers and existing data in your account.

• Select the Relationship Class. For this tutorial, use: Is
• Select the Direction. For this tutorial, use: FORWARD
• Enter the Target Entity Type. For this tutorial, we want to map Europa Users to GitHub Users by matching on their emails. The entity type for GitHub Users is github_user. For this tutorial, use: github_user

Add Field Mappings: Enter the Source Property. For this tutorial, use: Email Enter the Target Property. The GitHub user entity use email as the property name. For this tutorial, use: email

And that's it! Click create to finalize the integration instance and upload the CSV files should you wish.

Using stringList for Mapped Relationships

In cases where your Source Property contains a comma-separated list of values, you can enable multiple mapped relationships by using the [stringList] data type token.

When the Source Property includes the [stringList] token in its header (e.g., owners[stringList]), the integration will automatically create one mapped relationship per item in the list.

Example

If your dataset contains a field like this:

owners[stringList]
id-123,id-456,id-789

And you've mapped it to a Target Property like ownerId, JupiterOne will create three mapped relationships, one for each value (id-123, id-456, id-789).

This will result in relationships like:

• Entity A HAS → ownerId: id-123
• Entity A HAS → ownerId: id-456
• Entity A HAS → ownerId: id-789

This is useful when an entity is related to multiple targets and you want each relationship represented individually in your graph.

⚠️ Important Clarification

When defining Mapped Relationships in the integration UI, ensure that the Source Property field matches exactly the name used in your CSV header, including the data type token.

For example:

• If your CSV header is owners[stringList], you must enter owners[stringList] as the Source Property when creating the mapped relationship in the UI.

• If it doesn't appear in the dropdown list, make sure to select Add custom option and manually enter the source property name with the [stringList] suffix exactly as it appears in the CSV header.

• Omitting the [stringList] suffix in the UI (e.g., using just owners) will prevent the system from correctly identifying the data type, which may cause issues.

This ensures that the integration correctly interprets the data type and processes the relationships as expected.

Next steps

Now that your integration instance has been configured, it will begin running the transfer, populating data within JupiterOne. Continue on to our Instance management guide to learn more about working with and editing integration instances.