This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Concurrent update the same IPAM.IPNode table, python orion-sdk

Hello everybody.

Well, on the previously weeks, my team and me have been working with the IPAM and its python-sdk (https://github.com/solarwinds/orionsdk-python). We wrote an python script that will retrieve an free ip address from a SUBNET based on a given NETWORK_DISTRIBUTION.

An example for a call the python script is:

python free_ipaddress.py hostname1 --distribution NETWORK_DISTRIBUTION

So, basically our script works fine, our script workflow's is like following:

-> Query the database retrieve the FIRST free IP Address from a given subnet, marked as available based on its status

-> Mark the IP as reserved, changing the status value and registering the hostname of its record

-> And then print on stdout the IP and its record values as a JSON

How our workflow works is that the stdout generated by the IPAM python script is consumed by a tool called Terraform (a tool to provision and maintaining infrastructure as a code), and then the VM is deployed with IP Address generated and reserved on the IPAM.

The problem here is that when terraform (plans/apply) deploys multiple infra works on a way that sends the tasks to executes in background, by example, when I want to deploy and register 4 VMs on a same subnet, it will work mostly like execute a shell script like the following:

#!/bin/bash
python free_ipaddress.py hostname1 --distribution NETWORK_DISTRIBUTION &
python free_ipaddress.py hostname2 --distribution NETWORK_DISTRIBUTION &
python free_ipaddress.py hostname3 --distribution NETWORK_DISTRIBUTION &

However, if you execute just one line or line by line of the script, both methods will work fine, since there will be a little time between each execution, even when you sends to execute to background the tasks, but it's not like that when you query to the DB at the same time. The multiple executions will retrieve the same IP, it happens because on the IPAM when the first line runs the second and the third runs as well at the same time, the multiple executions going to background and query the database at the same time, then the three scripts executes the same workflow, but since there's no be deference between each query, the SELECT statement will return the same row, since it's not yet update by any previously execution, and it's status is available. So, every execution takes the same row ID, and then this row is updated 3 times, one by each hostname.

SELECT TOP 1 I.IpNodeId, I.Status, I.DisplayName FROM IPAM.IPNode I WHERE Status=2 AND I.Subnet.DisplayName = 'SUBNET_NAME'

sshot-543.png

In this case the same IP (180.100.2.36) is taken and updated with the three hostnames three times.

If you instead of executing the previously shell script, executes one like the following, it will work fine because the queries to the table aren't executing parallelly, and there is time to update the status of the record, its going to register three different hostnames on a same subnet:

#!/bin/bash
python free_ipaddress.py hostname1 --distribution NETWORK_DISTRIBUTION &&
python free_ipaddress.py hostname2 --distribution NETWORK_DISTRIBUTION &&
python free_ipaddress.py hostnam03 --distribution NETWORK_DISTRIBUTION &&

sshot-545.png

The previously works since &&: command after && is executed if, and only if, command before && returns an exit status of zero.

A workaround approach will be putting a random little sleep time inside of the python script, before the function call, something like:

time.sleep(random.random() * 10)

But I don’t know, this way doesn't like to me, there could be parallel executions if a user wants to register a bigger number of IPs, very minimal, but could by possible.

My point/question is, how to update the IPAM.IPNode table concurrently?

Any help? I will appreciate it, thanks in advance.

  • This approach - query and then update - cannot be made concurrency-safe without transaction support. Since the Orion API does not have general purpose transactions, a different approach is needed.

    Fortunately, the IPAM includes an API for this specific purpose: the StartIpReservation/FinishIpReservation/CancelIpReservation trio of verb calls on IPAM.SubnetManagement. See https://github.com/solarwinds/OrionSDK/wiki/IPAM-4.9-API#ip-address-reservation  for documentation.

  • Honestly - your reply is better documentation that what is in the link.  I was wondering what these functions do exactly, and I had an inkling they are transactional.   But this is the only place on the internet that confirms it.  The 'documentation' in the link has nothing.  The swgger has nothing.  Also still don't know what the minutes parameter is supposed to do.  I see an example of someone putting in 0.  Would that cancel the transaction in the specified number of minutes if it does not get a 'Finish' message beforehand?

  • The minutes parameter (reservationTimeInMinutes) ensures that the status for a given IP would not by override by IPAM for a given amount of time.
    IPAM might change IP status back to free if it still wouldn't be used after the specified amount of time have passed - this might take more than the specified interval, (depending on when the scan is scheduled, but you have a warranty that for X minutes, IPAM will not change the status of your reservation)