Docker based Collector

The Docker-based JupiterOne Collector is designed for deployment on a dedicated Linux VM or host using Docker Engine. This guide covers requirements, installation, and management.

System Requirements

Operating System	Any modern 64-bit systemd linux distribution should work. The JupiterOne Collector has been specifically tested against the following Operating Systems using the official Docker Engine: Ubuntu Server 22.04 LTS Ubuntu Server 23.04 RedHat Enterprise Linux 9.4 RedHat Enterprise Linux 8.9
Container Runtime	Whilst any docker compatible container runtime should work (e.g., podman), only the official Docker engine has been tested and certified to work. See Docker installation instructions here. There are specific notes later in this document if you wish to use podman instead of Docker, see Podman Container Runtime.
Docker Engine Version	Minimum: Docker Engine 24.0. Recommended: 28.x or 29.x. Any release at or above the minimum is supported. JupiterOne's own testing covers Docker Engine 24.0, 26.1, 27.5, 28.5, 29.0, 29.1, and 29.8. Docker Engine 29 and later also require Collector images daemon-v1.5.0 and runner-v1.6.2 or later. If your Collector is older than that, update the Collector before you upgrade Docker Engine, because an older Collector cannot update itself once Docker Engine 29 is installed. See Upgrading Docker Engine.
CPU	4 vCPU The CPU requirement is dependent on the integration workload, but 4 vCPU is a good starting point.
RAM	2 GB minimum, 8 - 12 GB recommended. Most integration jobs operate with <1GB of RAM but the memory requirement will depend on the number of entities being handled. Some very large integration instances, those handling millions of entities, may require 10GB+ of available memory.
Networking	The Collector needs to be able to connect to JupiterOne's services at *.us.jupiterone.io or *.eu.jupiterone.io over HTTPS/443. The JupiterOne Collector host also needs connectivity to the integration target; e.g., if running an Active Directory integration, it will need connectivity to your local LDAP port of your AD servers. The container images are pulled from the GitHub package registry at ghcr.io. The container images are all signed by JupiterOne, this requires connectivity to the cosign signing service also hosted at github.
Storage	50 GB available local storage. JupiterOne integrations running on the Collector do not themselves need persistent storage. Only the JupiterOne Collector itself requires persistent storage to maintain a configuration file. The Collector host should have sufficient local storage to hold logs for the various services.

Container Runtime

We recommend using the official Docker Engine. The installation instructions depend on the host OS, and can be found here: https://docs.docker.com/engine/install/

note

You do not need to install Docker Desktop features. For Ubuntu you would use these instructions: https://docs.docker.com/engine/install/ubuntu/

You can confirm you have a working container runtime environment on your host machine using a test command:

sudo docker run hello-world

Which should produce the following output:

This confirms that the docker engine is running and able to pull and launch containers. At this point you are ready to deploy a collector.

Deploying a Collector

To set up a collector, you will need a JupiterOne account.

1. Navigate to Integrations > Collectors, choose New collector. Provide a name for the new collector and select Create:

2. Take the terminal command and run it in your new collector instance.

note

You may need to prefix the command with sudo depending on how your permissions are configured and your current user.

Example command:

docker run -e JUPITERONE_AUTH_TOKEN='...' -e JUPITERONE_COLLECTOR_ID='...' -e JUPITERONE_ACCOUNT_ID='...' -e JUPITERONE_API_BASE_URL='...' -v /etc/.j1config:/etc/.j1config -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/jupiterone/collector-scripts/installer:latest

• docker run is the command to run the installer container.
• -e are environment variables for authentication and configuration.
• -v are volume mounts for configuration and Docker socket access.

Running this command on your Collector host will kick off the process to setup the collector.

3. Confirm the collector is running using docker ps. You should see two containers running (daemon and runner):

• Daemon: Manages local state, upgrades, and health checks.
• Runner: Manages the job queue and launches integration jobs.

To check which versions these are running later, see Upgrading the Collector.

4. Examine the logs of the runner to ensure it's successfully connecting to JupiterOne:

At this point the collector is set up and running. You will see the collector as "Active" in the collector overview after a few minutes:

note

The collector containers are running with a restart policy of "unless-stopped", so they will restart automatically on failure, and will start automatically when the container runtime (re)starts, i.e. at system reboot time.

Assigning an Integration

Assigning an integration job to a collector first requires that there are collectors registered and available. Once collectors are available, the process for defining an integration job and assigning it to a collector is straightforward.

For integrations that are collector compatible, complete the integration configuration as normal. During configuration, you'll notice there's an additional option to choose where the integration should run.

Select Collector on the integration instance, and choose the corresponding collector for which you'd like the integration to run.

Upgrading the Collector

The Collector keeps itself up to date. The daemon checks for new Collector versions every few minutes and replaces the daemon and runner containers automatically, so no action is normally required.

Checking which version is running

Run:

sudo docker ps

In the IMAGE column, the part after the final : is the version. For example, ghcr.io/jupiterone/collector-scripts/runner:runner-v1.6.6 is running runner-v1.6.6.

The daemon and the runner are versioned separately, so check both. Version numbers increase over time and are compared from left to right: runner-v1.6.6 is newer than runner-v1.6.2, and runner-v1.10.0 would be newer than either.

note

After the Collector has updated itself, the daemon container is named daemon- followed by a unique identifier rather than simply daemon. This is expected.

Reinstalling the Collector

If the Collector is not running, or its version is not updating, reinstall it by re-running the installer command from Integrations > Collectors. Reinstalling brings both the daemon and the runner to the current version.

Before you reinstall, remove the existing containers so that their names are free.

First list the Collector's containers. They are the ones whose image name contains collector-scripts:

sudo docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' | grep collector-scripts

Then remove them by the names shown in the first column, for example:

sudo docker rm -f runner daemon

caution

Remove only the container names that the command above listed. Do not match on the words "runner" or "daemon" on their own, because your own containers may have names that contain them.

Upgrading Docker Engine

caution

Check the Collector version before you upgrade Docker, not after.

To run Docker Engine 29 or later, the Collector must already be running daemon-v1.5.0 and runner-v1.6.2 or later. An older Collector cannot communicate with Docker Engine 29 at all, and because it needs Docker in order to update itself, it cannot recover on its own. You would have to reinstall it by hand.

Check the versions first, as described in Checking which version is running.

Docker Engine and the tools that drive it agree on an API version when they connect. Newer Docker Engine releases stop supporting older API versions, which is why a Collector older than your Docker Engine can lose the ability to talk to it.

Once the Collector is up to date, upgrading Docker Engine in place is supported. The daemon and runner containers restart automatically afterwards.

1. Confirm the Collector is up to date. Check that the daemon and runner are running daemon-v1.5.0 and runner-v1.6.2 or later. If either is older, reinstall the Collector before you continue.

2. Check that DOCKER_API_VERSION is not set. This variable pins the Collector to one specific Docker API version. If it is set, the Collector cannot adapt to the upgraded engine and will fail to start. Check both the running container and the host configuration:

   sudo docker inspect runner --format '{{range .Config.Env}}{{println .}}{{end}}' | grep DOCKER_API_VERSION
   sudo grep -r DOCKER_API_VERSION /etc/environment /etc/profile.d/ /etc/systemd/system/docker.service.d/ 2>/dev/null

   No output from either command means the variable is not set, and you can continue. If either command prints a line, remove DOCKER_API_VERSION from the file it names and from the installer command, then reinstall the Collector.

3. Upgrade Docker Engine using the instructions for your host OS.

4. Confirm the Collector recovered. After the Docker service restarts, check that both containers are running again:

   sudo docker ps

   The Collector returns to "Active" in Integrations > Collectors within a few minutes.

note

Restarting the Docker service stops any integration job that is running at that moment, and that job is reported as failed. The job runs again normally at its next scheduled interval. To avoid a failed job, upgrade Docker Engine between scheduled integration runs.

Podman Container Runtime

If you prefer to use podman instead of Docker, follow these steps:

1. Install the podman-docker and podman-remote packages:

sudo dnf install -y podman-docker podman-remote

2. Enable the podman socket:

sudo systemctl enable --now podman.socket

3. Create the JupiterOne configuration directory:

sudo mkdir -p /etc/.j1config/

4. Use the installer command, replacing docker with podman and adding the --privileged flag:

sudo podman run --privileged -e JUPITERONE_AUTH_TOKEN='...' -e JUPITERONE_COLLECTOR_ID='...' -e JUPITERONE_ACCOUNT_ID='...' -e JUPITERONE_API_BASE_URL='...' -v /etc/.j1config:/etc/.j1config -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/jupiterone/collector-scripts/installer:latest

caution

The --privileged flag is required to allow the collector to manage the container runtime when using podman. Review the security implications of running the collector with these privileges.

Removing a Collector

1. Remove the collector from the JupiterOne console using the "Delete Collector" option.
2. On the collector host machine, stop and remove the collector daemon and runner containers. See Reinstalling the Collector for the commands.

If you try deleting a collector that has integrations configured you will see a message like the following:

Troubleshooting

The Collector stops working after a Docker Engine upgrade

If integration jobs start failing and the Collector no longer shows as Active in Integrations > Collectors after Docker Engine has been upgraded, this usually means Docker was upgraded before the Collector was. Reinstalling the Collector resolves it.

To confirm, check the runner logs:

sudo docker logs runner

If that reports No such container, the container has a different name on this host. List the Collector's containers and use the name shown:

sudo docker ps -a --format 'table {{.Names}}\t{{.Image}}' | grep collector-scripts

An error similar to the following means the Collector is older than the Docker Engine now installed on this host:

Error response from daemon: client version 1.41 is too old.
Minimum supported API version is 1.44, please upgrade your client to a newer version

The Collector cannot update itself in this state, so reinstall it manually.

First list the Collector's containers. They are the ones whose image name contains collector-scripts:

sudo docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' | grep collector-scripts

Then remove them by the names shown in the first column, for example:

sudo docker rm -f runner daemon

caution

Remove only the container names that the command above listed. Do not match on the words "runner" or "daemon" on their own, because your own containers may have names that contain them.

Then re-run the installer command from Integrations > Collectors.

If the error persists, confirm that DOCKER_API_VERSION is not set, as described in Upgrading Docker Engine.

The runner or daemon container does not start

If a stopped daemon or runner container is left on the host, a new container cannot be created with the same name, and you will see an error similar to:

Conflict. The container name "/runner" is already in use

Remove the stopped containers, then re-run the installer command from Integrations > Collectors.

First list the Collector's containers. They are the ones whose image name contains collector-scripts:

sudo docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' | grep collector-scripts

Then remove them by the names shown in the first column, for example:

sudo docker rm -f runner daemon

caution

Remove only the container names that the command above listed. Do not match on the words "runner" or "daemon" on their own, because your own containers may have names that contain them.

Known Limitations

• Unable to migrate integration jobs between collectors.
• Integration jobs may run in parallel if multiple jobs are assigned.
• Limited high availability (single node).