Adding a bunch of data sources to vRealize Network Insight can be tedious, especially when the number of data sources goes into the 100s. This is one of the reasons why PowervRNI exists, and it's been helping organizations to add their entire network infrastructure to Network Insight in 1 go.
There's an example script in the PowervRNI GitHub repository called datasource-bulk-import.ps1, that allows you to add data sources of all types in bulk. To make getting started a little easier, I've recorded a video that focuses on adding 100 Cisco Nexus 5K switches to Network Insight in roughly 20 minutes.
The only prereqs are to have PowerShell version 6+ installed, have the switch information ready, and have Network Insight up & running.
Please let me know by commenting on the video if you have any questions!
Version 1.7 of PowervRNI was released yesterday, and this post covers what's new in this version. If you're not familiar with PowervRNI, it is a PowerShell module that you can use to manage vRealize Network Insight. From adding data sources & applications, to retrieve data from it, such a network flows, PowervRNI covers most of the public API endpoints of Network Insight.
What's New?
A bunch! First off, it has been updated to support the new data sources and API calls that have been added in Network Insight 5.0. Second, I've added a few backlog items. I'll go through the changes below.
Data Sources
New: Update-vRNIDataSource
Let's start with a good one. PowervRNI had to ability to add and remove data sources, but not edit them. Update-vRNIDataSource adds the possibility to update one or multiple data sources by updating their Nickname, Notes, or, most importantly, their Username and Password details. Imaging updating the login credentials of 52 switches, all at once. This example does just that:
Network Insight collects routing information from the NSX-v controllers. To do so, it needs the password to log in to those controllers. I've added a new cmdlet called Update-vRNINSXvControllerPassword to do this:
From NSX-v 6.4.5+ and NSX-T 2.5+ - NSX can stream telemetry on latency between VTEPs, physical and virtual NICs. NSX is configured by Network Insight and the new -NSXEnableLatency $True parameter on New-vRNIDataSource, will do that when adding the NSX Manager as a data source. Example:
Maybe you've noticed the domain name is mentioned twice, both in the Username and Domain values. This was mostly due to the input that the vRNI API requires. It's a small change, but now, PowervRNI now detects the usage of a non-local domain and allow you to omit the Domain parameter.
If you have local vRNI accounts with domains other than local, you can make sure it authenticates locally by still adding the parameters: -Domain LOCAL, *or* -UseLocalAuth.
Get-vRNIKubernetesServices
Network Insight can pull Kubernetes information (services, namespaces, clusters, nodes, etc.) and the networking details behind these containers. But, there's only 1 public API added to retrieve information, and it's to retrieve a list of the Kubernetes services:
Network Insight 5.0 also added a new API endpoint to manage passwords of local users. The new cmdlet Set-vRNIUserPassword allows you to change your own password, or as an administrator; change passwords of all users.
In a previous post, Integrating vRealize Automation with Network Insight, I covered how greenfield applications can be automatically pushed into vRealize Network Insight to provide application context with the analytics of Network Insight. This post is about importing brownfield into Network Insight to get insights into the behavior of your existing applications. Specifically, importing CMDB information from ServiceNow.
CMDB
In the case of existing applications, your Configuration Management Database (CMDB) should be your source of truth. There are a lot of CMDB products out there, but ServiceNow (SNOW) is a popular one that a lot of organisations use. That's why this example is based on that specific CMDB, but this can apply to any and all CMDBs. As long as you have the ability to export data or retrieve data from an API; this will work.
ServiceNow Requirements
Honestly, there are not many requirements on the SNOW side for this to work. The only thing this procedure will depend on is that there are application constructs which have relationships towards the VMs that host the application. There doesn't have to be a specific hierarchy in the relations, they only need to exist. Here's an example of a relationship tree (as seen through the SNOW dependancy viewer):
I have declared a top-level application, attached that to a Web Tier which runs on 2 VMs. Those 2 web VMs make use of the App Tier, which runs on 2 App VMs. Finally, those 2 App VMs make use of a DB Tier, which runs on a single VM.
The script I've published below takes this tree and translates it into Network Insight application constructs. The only thing to take note of is that it uses the naming scheme of the VM to determine the name of the tier. For example; with VM VMworld-3TA01-Web-01, the tier will be named Web (it uses the second to last field when split on the character "-"). If you want to change that syntax, have a look at line 56.
Script
You will see a common thread throughout my automation posts; PowerShell. ;-) This script is no different and uses PowervRNI to simplify talking to Network Insight. That means PowervRNI is a requirement, it'll fail if you don't have it installed (Install-Module PowervRNI).
Before running the script, edit it to reflect your environment. It doesn't take parameters just yet, so configure your SNOW and Network Insight details on the top of the script:
## Start editing here ### ServiceNow configuration; login credentials & URL
$CONFIG_SNOW_USER = "admin"
$CONFIG_SNOW_PASS = "VMware!"
$CONFIG_SNOW_URL = "https://myinstance.service-now.com"# vRealize Network Insight config; login credentials & URL
$CONFIG_vRNI_SERVER = "platform.networkinsight.lab.local"
$CONFIG_vRNI_USER = "admin@local"
$CONFIG_vRNI_PASS = "VMware1!"# We're using a filter to find only applications in ServiceNow with this in its name
$APP_FILTER = "VMworld"## Stop editing here ##
After editing those variables, you're free to run it. It will spew out errors and grind to a halt if any of those variables are incorrect. ;-)
Output
When running the script, it will tell you what it discovers and what it'll add it to Network Insight. Here's an example of how it can look:
Demo
If you would like to see whether this is even going to work for you, have a look at the demo I've put together. You'll be able to get a detailed view of what's in SNOW which gets exported. If there's anything that would block you from exporting it this way (for instance, you have a different configuration inside SNOW), please let me know!
VMware is pretty heavily invested in the cloud. Not only to manage your cloud environments, but also products as Software-as-a-Service offerings, next to the regular on-premises offerings. At the time of this post, there are 19 VMware products that you can purchase via the SaaS model. This is a variation between models like VMware Cloud on AWS where you get an Infrastructure-as-a-Service and Network Insight where you get the product hosted in the cloud.
Network Insight SaaS
So there's a SaaS version of Network Insight. Effectively, this means that the Platform (the user-interface and data repository) is hosted in the cloud and VMware takes care of keeping it available, scalable and the lifecycle management (upgrades). It also saves you from running a pretty heavy VM on-prem as well.
You'll get access to the Network Insight interface via cloud.vmware.com, which is exactly the same product and interface as you would get when you host it yourself. The only difference being the color scheme that matches the rest of the cloud services. This also means you have a Collector VM to collect data from data sources (vCenter, NSX, physical network devices, AWS, etc). You would typically host the Collector VM yourself, as it needs access to your internal infrastructure. But the Collector is the only thing you have to deploy, the rest is taken care of.
One of the cool parts of it being exactly the same product is that the API is also exactly the same. This means that PowervRNI can also be used to talk to the SaaS variant. There's one minor difference, as authentication on the Cloud Services Platform (CSP) is centralized to provide a single-sign-on experience on all services. So no separate authentication, but using tokens which are valid for all services.
Cloud Services Platform API Tokens
Using the CSP API is pretty straightforward: so-called Refresh Tokens can be created via your My Account portal and are valid for 6 months. Whenever you want to use the API of a service, first exchange the Refresh Token for an access token. Then use the access token to access the service you want to talk to (i.e. Network Insight).
To create a Refresh Token, log into https://cloud.vmware.com, go to your console. Then 1) access your menu, 2) go to My Account, 3) Select the API Tokens menu on the My Account page and 4) Select New Token to create a new token.
After creating it, the Refresh Token will appear in a long string format. Copy it and start a PowerShell window with PowervRNI (1.4+) loaded.
Connect-NIServer
Usually, you would use Connect-vRNIServer to connect to an on-prem instance of Network Insight - which hasn't changed. The only difference between using on-prem Network Insight and the SaaS variant is the command to connect to it. If you're connecting to the SaaS variant, use Connect-NIServer -RefreshToken $yourtoken
That's it!
After logging into CSP and getting back the access token, you can use the rest of the PowervRNI commands as you would with on-prem Network Insight.
Version 1.2 of PowervRNI was just released and it contains 2 new cmdlets:
Get-vRNIDataSourceSNMPConfig
Set-vRNIDataSourceSNMPConfig
You can use these cmdlets to see and update SNMP configuration on specific data sources. All Arista, Brocade, Cisco, Dell, and Juniper devices are data sources where SNMP can be enabled for richer data collection. This basically turns on interface counter collections, so you can have pretty graphs like this:
Examples
To give you an idea of how to use these new cmdlets, check out the examples below. Of course, there is also documentation available in the module itself, viewable via the Get-Help cmdlet.
Get-vRNIDataSourceSNMPConfig
It's quite a mouthful, but the naming convention of PowerShell is clear: name it after what it does. ;-) So this cmdlet retrieves the SNMP configuration (if there's any present) from a data source. It uses the values from Get-vRNIDataSource as input. Here's an example that goes through all Cisco switch data sources:
As the result, you can see the SNMP configuration is disabled for this example. If it were enabled, you would see the details it's using to poll the data source.
Set-vRNIDataSourceSNMPConfig
As the counterpart to Get-vRNIDataSourceSNMPConfig, the cmdlet Set-vRNIDataSourceSNMPConfig updates the SNMP configuration of a data source. It also takes the output of Get-vRNIDataSource as input, so you can use it in a piped manner.
As a result, the data source with the nickname Core01 will get its SNMP configuration updated to use SNMP v2c and use the community snmpcommunity, and of course enable SNMP collection.
Feedback
As interest in PowervRNI grows, I'm seeing a growing number of people using it to automate installation and operational tasks. Please continue to give me feedback on the module so we can continuously improve it!
PowerShell is about making things easy and consumable. Same goes for PowervRNI, making repetitive tasks in vRealize Network Insight easier and make it possible to extract information from vRNI.
It's not that easy to load a custom module into PowerShell though, you have to download 1 or 2 files, then load it into your PowerShell session by referencing those local files. The PowerShell Gallery is a pretty awesome way to also make module management very easy. All you have to do is use the Install-Module command and it downloads and installs the module for you.
For the module authors, it's also pretty easy to put their modules onto PowerShell Gallery. As I found out this weekend by putting PowervRNI on there.
Install-Module PowervRNI
So, all you have to do to install PowervRNI, using the Install-Module command and the module will be downloaded from the PowerShell Gallery and installed. Then right away, you can load up the module by using Import-Module and you're good to go.
Introduced in vRealize Network Insight 3.2 in the begin of 2017, Applications are logical containers which can contain the structure of applications (including tiering). You can use these application containers to better visualize what network flows are going between specific applications or even application tiers. Next to insights into your applications, you can also use it to take a per-application approach to micro-segmentation, using the planner feature in Network Insight.
The first thing you should know is that the application container has a few concepts to it. The application itself, then tiers that make up the application (for instance, the easy 3 tier app: Web, App and Database tiers) and inside the tiers there are rules that determine what VMs will be placed in that tier. These rules can be based on the same criteria as you can use in the search engine, so you can get really creative. Below is a visual explanation of the different concepts:
Creating Applications with PowervRNI
So, to automate the creation of the application you need to be able to create applications, then the tier and the rules that will place VMs into the tiers. Here are the cmdlets in PowervRNI which will help:
Get-vRNIApplication
Get-vRNIApplicationTier
New-vRNIApplication
New-vRNIApplicationTier
Remove-vRNIApplication
Remove-vRNIApplicationTier
To create a new application, first use New-vRNIApplication to create the container and then New-vRNIApplicationTier to fill that container.
Using New-vRNIApplication
An application is just an empty container, so creating it is easy and straightforward:
That's it. The return value is the Application object, which contains the newly created entity id and a few other details. After running this, you can verify using the interface:
Using New-vRNIApplicationTier
Once you have an application container, you can define the tiers. Due to some limitations of the current vRNI API, you can currently only use VM names and NSX Security Groups to define the tier membership. Below is an example that uses both VM names and security groups:
First, we find the entity ID of the security group called SG-3Tier-Web and store it in $security_group_id
Then we use New-vRNIApplicationTier to create a new tier called Web-Tier with 2 rules:
Put in all VMs where the name begins with 3TA-Web0
Put in all VMs that are in the security group called SG-3Tier-Web
If you then validate this in the interface, you'll see a newly added Web-Tier with the provides rules in it. You'll also notice that there's a "2 Vms" behind the rule. This is because I have 2 VMs called 3TA-Web01 and 3TA-Web02. Both are also in the security group ST-3Tier-Web.
Adding in Bulk
Using the cmdlets described above is pretty quick, but imagine doing that a couple 100-times in a greenfield environment. Not fun.
With that in mind, I've put up an example on how to import a lot of applications at once in the examples/ directory called: application-bulk-import.ps1
Similar to the data source bulk import script, the application bulk import script uses a CSV file as input. The CSV file has the applications, tiers and security groups and/or VM names in it. Here's an example of how it should be formatted.
* Yes, it’s technically not a CSV as I used ‘;’ for the separations. But this way Excel can read it into columns. ;-)
As you might notice, the same Application can contain multiple lines. This is because each line represents an application tier and you can have multiple tiers in 1 single application. You can use either the Security Group or VM Names column to create the filters, or you can use both columns if the VMs need to have a certain name and be a member of a specific security group.
Usage
Once you have the CSV filled with data, you can run the script as follows:
PS ~/PowervRNI/examples> ./application-bulk-import.ps1 -ApplicationsCSV ./application-bulk-import.csv
[02/12/2018 20:06:21] Processing application App01..
[02/12/2018 20:06:21] Application App01 not found, so creating it..
[02/12/2018 20:06:21] Added Tier 'Web'to application 'App01' with filters: name = 'App01-VM1-Web'or name = 'App01-VM2-Web'
[02/12/2018 20:06:22] Processing application App01..
[02/12/2018 20:06:24] Added Tier 'App'to application 'App01' with filters: security_groups.entity_id = '14307:82:1984513309' name = 'App01-VM*-App'
[02/12/2018 20:06:25] Processing application App01..
[02/12/2018 20:06:27] Wanted to use Security Group SG-App01-DB, but it doesn't exist - so skipping this rule!
[02/12/2018 20:06:27] Skipping Tier 'DB' in application 'App01' because filters are empty!
[02/12/2018 20:06:27] Processing application App02..
[02/12/2018 20:06:27] Application App02 not found, so creating it..
[02/12/2018 20:06:29] Wanted to use Security Group SG-App02, but it doesn't exist - so skipping this rule!
[02/12/2018 20:06:29] Skipping Tier 'NoTier'in application 'App02' because filters are empty!
[02/12/2018 20:06:29] Processing application App03..
[02/12/2018 20:06:29] Application App03 not found, so creating it..
[02/12/2018 20:06:29] Added Tier 'Web'to application 'App03' with filters: name = 'App03-Web*'
[02/12/2018 20:06:30] Processing application App03..
[02/12/2018 20:06:30] Added Tier 'DB'to application 'App03' with filters: name = 'App03-DB*'
That's it! Let me know if you have any questions. PowervRNI can be found on GitHub and on twitter for the latest news.
This is the first post in a series on how to use PowervRNI to manage your vRealize Network Insight environment. Starting with Data Sources.
Data Sources
Brief background; data sources in vRNI are endpoints from which vRNI retrieves information from. Examples are: vCenter, NSX, Switches, Routers, Firewalls, an AWS account or converged infrastructure systems like Cisco UCS or HP OneView.
Adding data sources is usually only performed when vRNI is installed into the environment and every time a new endpoint is added to your environment (for example a new vCenter environment, or when a new switch is plugged in).
Why would you want to automate adding data sources? First off, adding all your data sources upon installation can be tedious and repetitive work. And if you are one of those companies who have automated the deployment of a new top-of-rack switch (like some of my customers), you would also want that switch to automatically appear in vRNI.
Using New-vRNIDataSource
One of the beauties of PowerShell, is that the cmdlet naming convention is structured in a way that it tells you exactly what it's going to do. Same goes for New-vRNIDataSource. :-)
PS C:\> Get-Help New-vRNIDataSource
NAME
New-vRNIDataSource
SYNOPSIS
Create new datasources within vRealize Network Insight to retrieve data from.
There are a few parameters with the New-vRNIDataSource cmdlet available. Mostly for providing details on the data source. Let's go through them:
-DataSourceType
A data source type can be one of the following values:
vcenter
nsxv *
ciscoswitch *
aristaswitch
dellswitch *
brocadeswitch
juniperswitch
ciscoucs
hponeview
hpvcmanager
checkpointfirewall
panfirewall
If you provide either nsxv, ciscoswitch or dellswitch as the type, you need to supply specific options for those types.
-Username -Password
Credentials to use to log into the data source.
[-IP ] [-FDQN ]
Supply either the IP address or the FDQN (hostname) of the data source, don't supply both.
-CollectorVMId
This needs to be the entity ID of the vRNI collector appliance that will collect data from this source. This entity ID can be retrieved using the cmdlet Get-vRNINodes (example below).
-Nickname
Within vRNI, this data source will be identified with its nickname. It's usually a good idea to configure this with the hostname of the data source.
[-Enabled ]
By default, the data source will be enabled for collecting. You can disable it by providing $False on this parameter.
[-Notes ]
Optional internal notes about the data source. This won’t show up anywhere but at the data source management within vRNI.
[-Connection ]
With every PowervRNI cmdlet, you can supply an optional parameter identifying the vRNI connection you want to work with. This is useful when working with multiple vRNI instances at once. If you just have a single instance, you don't need to use this parameter.
NSXv, Cisco & Dell Switches Specific Options
With NSX, Cisco and Dell switches there are specific data source options you need to supply.
NSX
-NSXEnableCentralCLI
Do you want to enable collecting NSX Edge information via the NSX Central CLI, or do you want vRNI to directly connect to the NSX Edges: $True to use the Central CLI and $False to directly connect.
-NSXEnableIPFIX
With $True provided, this will enable IPFIX traffic monitoring data coming from NSX to vRNI. Note: this is different than the Netflow data vCenter can send via the VDS.
-NSXvCenterID
The entity ID of the vCenter that is connected to this NSX Manager. This entity ID can be retrieved using the cmdlet Get-vRNINodes (example below).
Cisco Switches
-CiscoSwitchType
Because some Cisco switches have different features, CLIs and different commands to retrieve information, you need to specify what type of Cisco switch you are adding. Here’s a list of options:
CATALYST_3000
CATALYST_4500
CATALYST_6500
NEXUS_5K
NEXUS_7K
NEXUS_9K
Dell Switches
-DellSwitchType
Dell switches also have differences in features, CLIs and commands to retrieve information, so you also need to specify what type of Dell switch you are adding. Here's a list of options:
FORCE_10_MXL_10
POWERCONNECT_8024
S4048
Z9100
S6000
Example
Below is a screenshot of a call to New-vRNIDataSource to add a new vCenter server.
This first looks up the collector entity ID of the collector with the IP address "10.8.x.x" and then uses that entity ID to add a vCenter data source type with.
Adding in Bulk
If adding one by one isn’t quick enough (or if you have a couple dozen devices to go through), you can also use an example script that I put in the examples/ directory of PowervRNI: datasource-bulk-import.ps1
This script takes a CSV file as input and runs through that CSV to add multiple data sources. It contains all available parameters as a column and is formatted as follows:
* Yes, it's technically not a CSV as I used ';' for the separations. But this way Excel can read it into columns. ;-)
The column names represent the cmdlet parameters. This also means that for some data source types, not all columns are used. Hence the "onlyusedforNSX", "onlyusedforCisco" and "onlyusedforDell" values in the line for a vCenter data source.
Usage
Once you have the CSV in order, you can run the script as follows:
[01/15/201817:28:04] Adding a vcenter Data Source with IP 172.16.0.11..
[01/15/201817:28:04] Adding a nsxv Data Source with IP 172.16.0.12..
[01/15/201817:28:05] Adding a ciscoswitch Data Source with IP 172.16.0.13..
That's it! Let me know if you have any questions. PowervRNI can be found on GitHub and on twitter for the latest news.
I am excited to introduce PowervRNI to the world! In the 3.6 release of vRealize Network Insight (vRNI) in November, there was a public API added. Through this API, you can offload certain tedious tasks like creating applications and managing data sources. I was using the private API for a couple of things already, like importing data sources. With the release of the public API, I could move those things to use the public (and supported) API. Ever since then, I've been working on PowervRNI.
PowervRNI
The first release of PowervRNI is on GitHub here. With this PowerShell module which talks to the public API of vRNI, you can grab information about pretty much any entity from vRNI (VM, Host, Network, Problems, Firewall rules, etc, etc) and you can do application and data source management.
Not all API calls to vRNI are supported just yet; I decided to focus on the obvious tasks and go from there. Please let me know if there's anything you'd like to see in there!
Getting Started
Right now, PowervRNI is a simple two-file module. To install it, download it to a PowerShell enabled machine and load it. PowervRNI is supported for PowerShell Desktop & Core, so Windows, MacOS and Linux. Here is an example on how to load it:
I will be posting examples on how to use PowervRNI in the near future to expand the documentation and making it easier to get started. First example will be a bulk import of data sources.
Let me know what you think, poke me on twitter or GitHub.
