Custom File Transfer

Leverage JupiterOne's Custom File Transfer to ingest assets into JupiterOne and designate the asset's relationships prior to uploading. This integration allows you to import a batch of assets and relationships via CSV file and track and manage each upload within the integration's instances.

You will need to have your assets and relationships in CSV format, and there is a size limitation of 100MB per file. For larger uploads, see our API documentation.

CSV File Requirements

To ensure data is correctly ingested, please consider the following:

The first row in the CSV must be headers. These headers will be used as the property values for the final entity.
We encourage the use of camelCasing for headers for consistency with other data in your graph.
We encourage the use of Data Type Tokens to improve query-ability of your data (see below for more details).
If a blank or whitespace header is provided, it will be replaced by idxN where N is the numerical index of that column.
The number of values in each row must match the number of headers provided on the first row. A mismatch will result in an error.
All empty (,,) and blank (,"",) values will be replaced with undefined to improve query-ability of your data.
Whitespace (" " or , ,) values will be maintained.

Data Type Tokens (optional)

In order for the integration to successfully identify different types of data, the column headers can optionally include a token to identify the columns data type. By providing these data type tokens, JupiterOne will store the data in the correct format allowing for advanced J1QL queries to be written.

Supported Data Type Tokens:

[datetime]: With the [datetime] token present in the header, all valid ISO dates will be parsed and stored ready for queries as a date. To ensure dates are displayed correctly in the JupiterOne app, column header names should end in On. Parsed dates will allow queries such as FIND approvals WITH approvedOn < Date.now - 30days
[boolean]: With the [boolean] token present in the header, the following values will be parsed as true: true, 1, on, yes. All other values will be considered false. Parsed booleans will allow for queries such as FIND approvals WITH approved = true
[number]: With the [number] token present in the header, all valid numbers (int, decimal, negative) will be parsed as a number. If a value is not a number, the value will remain unparsed. Parsed numbers will allow queries such as FIND approvals WITH count > 50
[stringList]: With the [stringList] token present in the header, all strings will be split on the , (comma) character into a list of strings. If no commas are present, the result will be an array with a single element set to the value provided.

note

Adding a Data Type Token to the column header name that uniquely identifies each row is not supported. The value entered as the Entity Key Property should exactly match the value found in the CSV.

Example walkthrough

Starting CSV:

id,createDate,description,approvedOn,approved,count,weekdays
1,01/01/2024,Granted access to fix the issue,01/01/2024,true,100,"monday,tuesday"
2,02/11/2014,Test access granted,02/11/2014,testBool,100,"tuesday,wednesday"

In this example, createDate and approvedOn are both dates. id and count are both numbers. approved is a boolean. id will not be considered a number, it will be used as the Entity Key Property which is explained later in this document.

For the integration to identify them as their appropriate data type, add the data type tokens to the applicable header. For example, createDate becomes createDate[datetime].

Tips: Add the data type token after the column header name. For timestamps, renaming the column to end with On is preferable for best display support in the JupiterOne app and more consistent with the JupiterOne data model. So for example, createDate becomes createdOn.

Example with Data Type Tokens:

id,createdOn[datetime],description,approvedOn[datetime],approved[boolean],count[number],weekdays[stringList]
1,02/11/2014,Granted access to fix the issue,02/11/2014 11:30:30,true,100,"monday,tuesday"
2,02/11/2014,Test access granted,02/11/2014,testBool,100,"tuesday,wednesday"

Ensure that the date time values are in a valid ISO 8601 format:

id,createdOn[datetime],description,approvedOn[datetime],approved[boolean],count[number],weekdays[stringList]
1,2014-02-11Z,Granted access to fix the issue,2014-02-11T11:30:30Z,true,100,"monday,,tuesday"
2,2014-02-11Z,Test access granted,2014-02-11Z,testBool,100xx,"tuesday,wednesday,"

Below is an example of the resulting entities post-processing. If a value is not successfully parsed for the given data type, the value will remain in its original format. The data type token will be removed prior to use as a property name.

[
  {
    "id": 1,
    "createdOn": 1392076800000, // unix epoch timestamp, will be displayed correctly as ISO timestamp in JupiterOne.
    "description": "Granted access to fix the issue",
    "approvedOn": 1392118230000,
    "approved": true,
    "count": 100,
    "weekdays": ['monday', 'tuesday']
  },
  {
    "id": 2,
    "createdOn": 1392076800000, // unix epoch timestamp, will be displayed correctly as ISO timestamp in JupiterOne.
    "description": "Test access granted",
    "approvedOn": 1392118230000,
    "approved": false, // testBool was parsed as false
    "count": "100xx", // was not parsed
    "weekdays": ['tuesday', 'wednesday']
  }
]

Configuring the integration

Navigate to the Integrations tab in JupiterOne and select Custom File Transfer. Click New Instance to begin configuring the integration.

For this integration, you will need to have your CSV dataset ready to imported.

Creating a Custom File Transfer instance

Provide the following when creating an instance for your custom file transfer:

The Account Name used to identify the account in JupiterOne. Ingested entities will have this value stored in tag.AccountName when the AccountName toggle is enabled.
Description to assist in identifying the integration instance, if desired.
Enter the Dataset Name. For the first example data set, use: EuropaUsers
Define the Entity Key Property. This is found in the data file. For CSVs, it is the column header. This value should be unique across the entire dataset. Adding a Data Type Token to this column header is not supported.
Add the Entity Type. This will identify the specific type of entity to be created.
Select the Entity Class. This value represents the class of data found in the dataset.

info

For more information about JupiterOne _type and _class values, see our JupiterOne data model.

At this point, the integration is able to upload and ingest the defined dataset into your account. Press Create to create the instance and upload your CSV file.

If you wish to add additional datasets, press the Add Dataset button.

Additionally, you can add Direct and Mapped Relationships by selecting the respective Add relationship button. You can find additional information on adding relationships below.

Adding another data set:

Enter the Dataset Name, the Entity Key Property, and Entity Type.
Select the Entity Class.

Defining Relationships

When uploading your file, you can additionally define both direct and mapped relationships for the dataset.

When defining a relationship, FORWARDS represents source -> target, and REVERSE represents source <- target (target -> source).

This determines the relationship between the entities and which direction the arrow points between them, allowing for illustrating ingress and egress accordingly.

To define direct relationships:

Press Add Relationship under Direct Relationships and provide the following:

The Source Property. The source property indicates what value in the dataset references the target dataset.
The Relationship Class
The Target Dataset Name. This needs to exactly match the inputted value for the target Dataset Name.
The Target Property. The target property indicates what value in the dataset that should be referenced by the source dataset.

Define Mapped Relationships

Mapped relationships can be defined similarly to direct relationships.

To define the mapped relationships:

Select the Relationship Class.
Specify the Direction.
Enter the Target Entity Type. :::note To see what _types are available in your account, consider running this J1QL query in your account: FIND * AS e RETURN count(e), e._type. For this tutorial, use: _githubuser :::

Add Field Mappings

Enter the Source Property and Target Property as desired.

Finalize the integration instance

Click Create once all values are provided to finalize the integration. You will then be prompted to upload your CSV files.

Custom file transfer example

To understand how to set up this integration, the following dataset will be used for a tutorial.

Imagine you want to add User and User Group data from your HR software, called Europa, into JupiterOne.

Here are the two example datasets:

Europa Users (europaUsers.csv)

    Id  Name    Username    Email   GroupId
 Jane    jane.c  jane.c@company.com  A
 Juan    juan.d  juan.d@company.com  B
 Julie   julie.e julie.e@company.com C

Europa User Groups (europaUserGroups.csv)

    Id  Name
    A   Marketing
    B   Sales
    C   Engineering

We can represent both the Europa Users and Europa Groups as assets in the JupiterOne graph.

Further, we can use the native relationship between the Europa Users GroupId property and the Europa Groups Id property to define a direct relationship between the two datasets in this integration.

Finally, if we want, we could also use the Europa Users Email property to define a mapped relationship to any other data that exists in your JupiterOne account.

Define Datasets

Enter the Dataset Name as: EuropaUsers
Enter the Entity Key Property. For this tutorial the first column header of the Europa Users dataset we'll use is: Id
Enter the Entity Type as: europa_user.
Set the Entity Class as: User.

At this point the integration is able to upload and ingest the Europa Users dataset into your account. With the goal of adding Users Groups and relationships, let's keep going:

Press the Add Dataset button
Enter the Dataset Name. For the second dataset: EuropaUserGroups
Enter the Entity Key Property. For this tutorial we will use the Id column which represents a unique id for each entity.
Enter the Entity Type. For the second dataset: europa_user_group
Select the Entity Class. For the second: UserGroup

Excellent! Both datasets are now defined.

Define the direct relationship

The following steps will define the relationship to be built between Users (the source dataset) and User Groups (the target dataset).

    User (source) <--HAS-- User Group (target)
    EuropaUsers.GroupId == EuropaUserGroups.Id

The following steps will build the above relationship:

Press Add Relationship under Direct Relationships and
Enter the Source Property. For this tutorial, useGroupId as the source property. It is found in EuropaUsers and references the Europa User Groups dataset.
Select the Relationship Class. For this tutorial: Has
Select the Direction. For this tutorial: REVERSE
Enter the Target Dataset Name. For this tutorial: EuropaUserGroups
Enter the Target Property. For this tutorial, Id is the target property, found in EuropaUserGroups. Enter Id for this field.

Define Mapped Relationship

The following steps will define the relationship to be built between EuropaUsers and existing data in your account.

Select the Relationship Class. For this tutorial, use: Is
Select the Direction. For this tutorial, use: FORWARD
Enter the Target Entity Type. For this tutorial, we want to map Europa Users to GitHub Users by matching on their emails. The entity type for GitHub Users is github_user. For this tutorial, use: github_user

Add Field Mappings: Enter the Source Property. For this tutorial, use: Email Enter the Target Property. The GitHub user entity use email as the property name. For this tutorial, use: email

And that's it! Click create to finalize the integration instance and upload the CSV files should you wish.

Next steps

Now that your integration instance has been configured, it will begin running the transfer, populating data within JupiterOne. Continue on to our Instance management guide to learn more about working with and editing integration instances.

CSV File Requirements​

Data Type Tokens (optional)​

Supported Data Type Tokens:​

Example walkthrough​

Configuring the integration​

Creating a Custom File Transfer instance​

Adding another data set:​

Defining Relationships​

To define direct relationships:​

Define Mapped Relationships​

To define the mapped relationships:​

Add Field Mappings​

Finalize the integration instance​

Custom file transfer example​

Define Datasets​

Define the direct relationship​

Define Mapped Relationship​

Next steps​

Contents