XEM-002 topology setup in 3U with 28 Blades. (minima is 4 blade).
Blade install with redhat 6.7
Include 1 Iperf. 2 mausezahn. 3. Screen. 4. Tcpdump 5. Python 2.7 (build-in) 6. Scapy package….
For future plugin developing, will need sshpass and screen.
Topology file is JSON format which use by plugin.
In US side, I have plugin script to install blade from PXE server, and install package, and auto-gen the topology file under session/conf. You will need to create it manually.
This is basic server package…
We have very simple 4 blade in XEM-002 testbed so far. It is ok for ACL, VLAN, MAC-learning, IGMP… plugin.
I already commit the latest code to Automation git server . It include the XEM-002 plugin that Bingming is working on it now. That plugin could perform 100+ ACL sanity test so far…
We will let TW team join developing VLAN, MAC-learning, IGMP (around 2 months) when the member come here.
Please install the same package on NFS server.
Node2 is for MLAG test, please install the same package and configure the LACP port-channel connect to both peer-link.
From the error message. Look like you miss some dependence library.
Please make sure you have below packages.
yum install -y xz
yum install -y libcli
yum install -y libnet
Please verify mausezahn manually before you try automation, make sure you can send the packet. This should be the main reason that automation cannot work normally.
Your clone source already include the PXE installation plugin (plugin_MBM_blade_server_test_set.py). But it is designed for US PXE server environment. It won’t work on Taiwan side. Again, it is not necessary to use PXE to setup your server. I actually recommend to install on HD.
Xem-002 automation is in progress. Bingming had ported ACL, VLAN, MAC-learn, IGMP, Strom-control on separate plugin so far. Since these module just need 4 blade server, They are on the first stage.
The stage two will integrate to single source, in order to perform the same sanity list set. This stage will wait for xem-002’s trunk port ready. I will check with TW RD for the progress. And will start to post xem-002 automation report on stage 2.
Xem-002 don’t have enough uplink could fit on gem-004 topology. This is challenge as the design is single source to handle both gem-004 and xem-002, and work on same topology. I am still designing it.
MLAG is high demand feature on xem-002, it will be handle on stage 2, the mlag single source developing for all product is in progress and you could see the new structure in latest source. I am testing the new structure on daily overnight test and the report for gem-004 already include new MLAG plugin.
I setup separate enclosure with 10g blade for xem-002 and it need separate MLAG server. You can prepare the same environment first.
Since automation is new concept in company. I developing it concurrently with several tasks to make it become well-known by cross-function group.
1.Show the value for product’s quality improvement. We perform each overnight test for MBM product and almost have daily report for the result
2.Show the power to help debug/investigate the root cause. I keep on improve the error log capture function and provide the detail log when detect error.
3.Show the value as the great tool for developing work.
4.Show the power to support customer case to reproduce the issue and collect the data for issue analysis.
We make multi-engineer teams know the value to help developing and debugging now.
It is time to announce automation system to developing teams and start to let team use it.
To make this tool easy starting, We will make each test case as independent testbed and build multiple testbed for user to submit the test request.
We plan to have three testbeds online in a week.
1.MBM CMM power cycle switch stress testbed
2.MBM blade server IPMI ac-cycle stress testbed
3.MBM Switch standalone power cycle stress testbed
Other testbed will be online on following few weeks.
1.MBM PDU power cycle enclosure (CMM, switch , blade server) testbed.
2.MBM switch self-reboot testbed.
3.MBM Switch and blade server boot/traffic/L2-L3 functional stress testbed.
4.SSH-C48Q PDU power-cycle stress testbed.
5.SSH-C48Q self-reboot stress testbed.
Automation version 2.0 design specification and source release on 12/21/2015.
Release note:
Re-origination file structure.
Design multiple test session cluster to enhance multiple test sessions capability. This is the design for our product’s validation which need more independent test session to perform longer period stress testing.
Add Server proxy feature to provide central interface for multiple test sessions. Proxy server provide central entry and load balance test session in cluster.
Simplify plugin logical, easy for plugin development. No installation requirement.
Move topology definition, plugin selection control to Client side. It could make server test session is 100% control by Client. Proxy server could load balance and assign any test request to any session.
Beside original test items, CLOS test is new set of testing items to focus on modern data center’s requirement.
Basically, we build small Clos Fabric networking topology in LAB to simulate modem data center topology.
The traffic performance test is extended to entire CLOS Fabric network.
The traditional Clos Fabric is known as spine-leaf style connection.
We plan to have 4 spine and 4 leaf to build two Fabric plan.
SSE-G3648 will play leaf role and we plan to deploy 4 switches in this topology.
This validation will verify our white-box hardware’s capability with Cumulus OS under modem data center topology.
By the way, our white-box with cumulus is only switch meet the basic requirement on this topology.
I setup ‘git’ as Automation plugin developing source control system in this server (172.30.151.182).
I create two repositories for plugin developing
1. Plugin and Client repository
2. Session repository
You can clone the source as below.
git clone /home/frankwang/RD_LAB/automation_prj/session/
git clone /home/frankwang/RD_LAB/automation_prj/client/