Normalization
JupiterOne normalizes some entity properties on specific classes to make querying easier and to make query results more consistent. In the event that we normalize properties into a property provided in a raw upload, or from an integration, the original raw value is persisted in raw_<property> to prevent any data loss.
Reading diffs:
{
"property": "this is unchanged",
+ "newProperty": "this is added",
- "removedProperty": "this is removed"
- "changedProperty": "This gets Changed"
+ "changedProperty": "this has been changed"
}
Devices and HostAgents
Collect and normalize serial numbers
A property called serials is added containing normalized values of all matching properties. Source properties are not modified. This is primarily feature for improved reliability of our device consolidation features.
| _class | matching property name (one of) | target property | matching property value (one of) |
|---|---|---|---|
| Device, HostAgent | serial OR serialNumber | serials | /^((([^0-9a-f]*[0-9a-f]){7}[0-9a-z\W]*)$\|(([^0-9a-z]*[0-9a-z]){12}[0-9a-z\W]*))$/i |
Example Normalizations
// Example 1:
{
"serial": "AB-CD-EF-12-34-56"
+ "serials": ["ab-cd-ef-12-34-56"]
}
// Example 2:
{
"serialNumber": ["123456789"]
+ "serials": ["123456789"]
}
// Example 3:
{
"serial": "32CCBFD3-2436-41AA-8492-016D20241767",
"serialNumber": ["A12345", "ABCD12345" ],
+ "serials": [
+ "32ccbfd3-2436-41aa-8492-016d20241767",
+ "a12345",
+ "abcd12345"
+ ]
}
Collect associated IP(v4) addresses
A property called associatedIpAddresses is added containing normalized values of all matching properties. Source properties are not modified. This allows easier searching for entities that may be associated with an IP Address in an unknown way, regardless of the semantic meaning of the original IP Address properties.
| _class | matching property name (one of) | target property | matching property value (one of) |
|---|---|---|---|
| Device, HostAgent | ipAddresses OR ipAddress OR publicIpAddress OR privateIpAddress OR lastIpAddress OR network.ipAddress OR lastExternalIpAddress OR IP_Address | associatedIpAddresses | /^(((25[0-5]\|2[0-4][0-9]\|[01]?[0-9][0-9]?)\.){3}(25[0-5]\|2[0-4][0-9]\|[01]?[0-9][0-9]?))$/ |
Example normalizations
// Example 1:
{
"ipAddress": "024.249.245.253",
+ "associatedIpAddresses": [ "024.249.245.253" ]
}
// Example 2:
{
"ipAddress": "024.249.245.253",
"publicIpAddress": "1.9.243.250",
"privateIpAddress": "250.236.221.208",
+ "associatedIpAddresses": [
+ "024.249.245.253",
+ "1.9.243.250",
+ "250.236.221.208"
+ ]
}
Collect associated email addresses
https://html.spec.whatwg.org/multipage/input.html#valid-e-mail-address
A property called associatedEmailAddresses is added containing normalized values of all matching properties. Source properties are not modified. This allows easier searching on any entities that may be associated with an email address in any unknown way, regardless of relationships or the semantic meaning of the original email address properties.
| _class | matching property name (one of) | target property | matching property value (one of) |
|---|---|---|---|
| Device | email OR primaryEmail OR recoveryEmail OR contactEmails OR userEmail OR userEmails OR serviceAccountEmails OR serviceAccountEmail OR publisherEmail OR maintainerEmails OR homeEmail OR verifiedEmail OR verifiedEmails OR userEmails OR emailAddress OR workEmail OR Email OR general.assignedUserEmail OR techLeadEmail OR managerEmail OR cooEmail OR ceoEmail OR ctoEmail OR techContractEmail OR registrantContactEmail OR adminContactEmail OR abuseContactEmail | associatedEmailAddresses | /^([a-zA-Z0-9.!#$%&'*+/=?^_{\|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)\*$)/i |
Example normalizations
// Example 1:
{
"email": "support@jupiterone.io",
"primaryEmail": "Support@JupiterOne.io",
"recoveryEmail: "security@jupiterone.io",
+ "associatedEmails": [
+ "support@jupiterone.io",
+ "security@jupiterone.io
+ ]
}
Normalize make
Source property make may be modified. If so, the original value will be maintained in raw_make.
| _class | matching property name (one of) | target property | matching property value (one of) |
|---|---|---|---|
| Device, HostAgent | make OR Manufacturer OR hardwareManufacturer OR hardwareVendor | make | /^(.+)$/ |
// Example 1:
{
"make": "Intel" // no change
}
// Example 2:
{
"Manufacturer": "intel",
+ "make": "Intel"
}
Further refine make based on MAC address
Source property make may be modified. If so, the original value will be maintained in raw_make.
Using IEEE OUI lookups based on the registration authority, we can determine device make from its MAC Address in over thirty thousand MAC address blocks. Below are a few examples:
| _class | matching property name (one of) | target property |
|---|---|---|
| Device, HostAgent | macAddress | make |
Example normalizations
// Example 1:
{
"macAddress": "6C:29:95:6E:69:F6",
+ "make": "Intel" // normalized from the macAddress based on IEEE OUI lookup
}
// Example 2 (make already defined):
{
"macAddress": "6C:29:95:6E:69:F6",
- "make": "Dell",
+ "make": "Intel", // normalized from the macAddress based on IEEE OUI lookup
+ "raw_make": "Dell" // preserved original raw value
}
Other Example IEEE OUI lookups:
Normalize OS versions
Source property osVersion may be modified. If so, the original value will be maintained in raw_osVersion.
| _class | matching property name (one of) | target property | matching property value (one of) |
|---|---|---|---|
| Device, HostAgent | osVersion OR os_version OR operatingSystemVersion OR os OR OS OR Operating System OR operatingSystem OR operatingsystem | osVersion | /^(?:.*?)(?:\b\|\()(\d+(\.\d+)*)(?:\b\|\))(?:.*)$/i |
Example normalizations
// Example 1:
{
"operatingSystemVersion": "16.5.0",
+ "osVersion": "16.5"
}
// Example 2:
{
- "osVersion": "Mac OS X 12.3.4"
+ "osVersion": "12.3.4", // normalized value
+ "raw_osVersion: "Mac OS X 12.3.4" // preserved original raw value
}
Other normalized values examples:
Normalize OS types
Source property osType may be modified. If so, the original value will be maintained in raw_osType.
| _class | matching property name (one of) | target into | matching property value (one of) |
|---|---|---|---|
| Device, HostAgent | operatingSystem OR operatingsystem OR os OR OS OR osFamily OR os_family OR osName OR osname OR os_name OR Operating System OR os_version OR @osVersion OR osVersion | osType | /(^.*(ios [\d]\|iphone).*$)/i OR /(^.*ipad.*$)/i OR /(^(Cupcake\|Donut\|Eclair\|Froyo\|Gingerbread\|Honeycomb\|Ice Cream Sandwich\|Jelly Bean\|KitKat\|Lollipop\|Marshmallow\|Nougat\|Oreo\|Pie\|Android).*$)/ OR /(^.*ipad.*$)/i OR /(^.*(OS\s?x\|macos).*$)/i OR /((^.*(windows\|microsoft).*$)\|(^.*\s+(win\|ms)(\s\|\d\|-)+.*$)\|(^(win\|ms)(\s\|\d\|-)+.*$))/i OR /^(Linux\|Debian\|Ubuntu\|Fedora\|Red Hat\|CentOS\|Arch(\s?Linux)?\|openSUSE\|Gentoo\|Slackware\|Mint\|Kali(\s?Linux)?\|Zorin(\s?OS)?\|Manjaro\|Mageia).*$/i |
Example normalizations
// Example 1:
{
"Operating System": "Apple iOS 16.6.1"
+ "osType": "iOS"
}
// Example 2:
{
"osName": "Debian GNU/Linux 10 (buster)"
+ "osType": "*nix (Debian)"
}
// Example 3:
{
- "osType": "Apple MAC OS X",
+ "osType": "MacOS",
+ "raw_osType": "Apple MAC OS X" // preserved original raw value
}
Other example value normalizations:
Normalize and collect MAC addresses
Source property macAddress may be modified. If so, the original value will be maintained in raw_macAddress. All matching fields will be merged into an array in the normalized macAddress field:
| _class | matching property name (one of) | normalizes into | matching property value (one of) |
|---|---|---|---|
| Device, HostAgent, Host, Gateway, Finding, Record, NetworkInterface, Printer | macAddress OR macAddresses OR MacAddress OR MACADDRESS OR MACAddress OR MAC_Address OR mac_address OR mac-address OR altMacAddress OR wifiMacAddress OR wifiMacAddresses OR wirelessDeviceMac OR recentDeviceMac OR wifiMac OR mac OR network.macAddress OR bluetoothMac | macAddress | /^([0-9a-f]{1,2})([.:-])?([0-9a-f]{1,2})([.:-])?([0-9a-f]{1,2})([.:-])?([0-9a-f]{1,2})([.:-])?([0-9a-f]{1,2})([.:-])?([0-9a-f]{1,2})$/i |
Example normalizations
{
- "macAddress": "aabbccddeeff",
"altMacAddress": "11.22.33.44.aa.bb",
+ "macAddress": [ // normalized AND collected into array
+ "AA:BB:CC:DD:EE:FF",
+ "11:22:33:44:AA:BB"
+ ],
+ "raw_macAddress": "aabbccddeeff" // preserved original raw value
}
Normalize encryption status
Source property encryptionStatus may be modified. If so, the original raw value will be maintained in raw_encryptionStatus
| _class | matching property name (one of) | target property | matching property value (one of) |
|---|---|---|---|
| Device, HostAgent | encrypted OR encryptionStatus OR isEncryptionEnabled OR encryptionState | encryptionStatus | [true] OR [false] OR /^(encrypted\|true\|yes)/i OR /^(not encrypted\|false\|no\|not_encrypted\|notencrypted\|unencrypted)$/i OR /^(unknown)?/i |
Example normalizations
// Example 1:
{
"encryped": true,
+ "encryptionStatus": "encrypted"
}
// Example 2:
{
"isEncryptionEnabled": "FALSE",
+ "encryptionStatus": "unencrypted"
}
// Example 3:
{
- "encryptionStatus": "yes",
+ "encryptionStatus": "encrypted",
+ "raw_encryptionStatus: "yes" // preserved original raw value
}
Findings
Normalize severity
| _class | _type (one of) | matching property name (one of) | matching property value (one of) |
|---|---|---|---|
| Finding | aws_accessanalyzer_finding OR aws_guardduty_finding OR aws_inspector_finding OR aws_inspectorv2_finding OR aws_macie_finding OR azure_advisor_recommendation OR bugcrowd_submission OR cbdefense_alert OR checkmarx_finding OR cobalt_finding OR crowdstrike_vulnerability OR cycognito_issue OR detectify_finding OR github_code_scanning_finding OR github_finding OR github_repo_finding OR gitlab_finding OR gitleaks_finding OR hacker_one_report OR hackerone_report OR microsoft_defender_vulnerability OR netskope_compliance_finding OR nowsecure_finding OR nuclei_finding OR orca_finding OR orca_finding_alert OR probely_finding OR sast_scan_finding OR snyk_finding OR sysdig_finding OR tenable_container_finding OR tenable_vulnerability_finding OR trivy_finding OR veracode_finding OR qualys_finding OR qualys_host_finding OR wiz_vunerability_finding | severity OR Severity | /(^(info(rmational)?\|(very low)\|none\|negligible))/i OR /(^low)/i OR /(^(medium\|moderate))/i OR /(^high)/i OR /(^(critical\|(very high)\|hazardous\|urgent))/i OR /(^unknown)/i |
aws_ecr_image_scan_finding entities DO NOT have their severity normalized by default. If you would like them to be normalized, please contact support.
Example normalizations
// Example 1:
{
- "severity": "MEDIUM",
+ "severity": "medium",
+ "raw_severity": "MEDIUM" // preserved original value
}
// Example 2:
{
"Severity": "very low",
+ "severity": "info"
}