Moving an index from one Splunk instance to another

Moving indexes in Splunk can be required in migrations, upgrades or simply as an exercise for backup and restores.

In this example, we will move an index called SSG from a Splunk 7.02 server called UBUNTU07 running on Ubuntu Linux to another 7.02 Splunk server running on another Ubuntu server called SPLUNKOSAUR. We are assuming that Splunk is installed in the default Linux location (/opt/splunk) and that the indexes are in the default Splunk location (/opt/splunk/var/lib/splunk) – if not, adjust the paths accordingly. For Linux/Windows combinations, the copy commands have to be adjusted accordingly but otherwise the overall process remains the same.

Step 1

On the new server, create an index with the same name. In this case we will create an index called SSG on SPLUNKOSAUR:

Create new index

Step 2

Stop Splunk on the new server (SPLUNKOSAUR). From a terminal  run:

sudo /opt/splunk/bin/splunk stop

Step 3

Remove the folder for the newly created index:

sudo rm -r /opt/splunk/var/lib/splunk/ssg

Remove new index

Step 4

On the old server, remove or disable any inputs that send data to the index that has to be migrated. In our case, we have a file monitor input that ingests a Juniper SSG firewall syslog logs into the SSG index:

Disable input

Step 5

Stop Splunk on the old server (UBUNTU07):

sudo /opt/splunk/bin/splunk stop

Step 6

Transfer the folder containing the index to be migrated to the new server. There are several ways to transfer folders between servers. In our case, we will use the scp utility (secure copy) that ships with any Ubuntu installation. scp is using SSH so the servers have to run the SSH daemon. If SSH is not installed, the installation is rather simple. In a terminal prompt run:

sudo apt-get install openssh-server

To check the status of the SSH server run:

sudo service ssh status

Once SSH is installed on the receiving server, you can use the scp utility. To transfer a full folder use the following command:

sudo scp -r /opt/splunk/var/lib/splunk/ssg user_name@server_name_or_ip:”/opt/splunk/var/lib/splunk/ssg”

Replace user_name and server_name_or_ip with the corresponding values. In our case, SPLUNKOSAUR has an IP address of and we will use a user called jdoe so the syntax would be:

sudo scp -r /opt/splunk/var/lib/splunk/ssg jdoe@”/tmp”

scp command

This will transfer the ssg folder to the /tmp folder on the new server.

The user will be prompted for the password and if entered correctly, the copying will be initiated. The duration of the copying process will depend on the amount of data to be transferred and the type of connectivity between servers. If the volume of data is very large and/or the connectivity between servers is slow or inexistent, then the files can be transferred with a portable media such as an USB drive. In any case, the end results have to be the same: the existing SSG index folder from the old server has to end up in the indexes folder on the new server.

scp completed

Step 7

On the new server, transfer the ssg folder from /tmp to /opt/splunk/var/lib/splunk. Depending on the user that you are logged in as, you may have to become root (i.e. use the “sudo -i” command to facilitate all the rest of the commands that you need to configure the new folder properly):

mv ssg /opt/splunk/var/lib/splunk

After the transfer, the /opt/splunk/var/lib/splunk folder on the new server should look something like this:

new server listing

Step 8

Note that the ssg folder has as owner and group the user that was used to transfer the folder (jdoe). Adjust the ownership of the SSG folder to match the existing indexes:

chown root:root /opt/splunk/var/lib/splunk/ssg

The folder should indicate root as owner and group:

change folder owner

Step 9

Start Splunk on the new server (SPLUNKOSAUR):

/opt/splunk/bin/splunk start

Login into the new server and verify that the SSG index has all the data from the old server:

New index migrated

Perform a search to validate that the data is available:

data present

Recreate the inputs as needed. This depends on the nature of the previous inputs, i.e. if the input was for local files, the source of the data will have to be transferred to the new server or the old server configured as a forwarder.

Step 10

As the last step, restart the old server and remove the old index.