The Power of MoSMB – high performance, high availability SMB 3.0 server on Linux

The MoSMB server stack on linux(Ubuntu, FreeBSD, Solaris) allows admins to achieve high availability, reduce operational costs and migrate virtual machines faster.

It can be widely used to provide file services to SMB clients. In addition, MoSMB also has enterprise grade features that help reduce cost for small and midsize businesses by providing storage for hyper-v server virtualization. MoSMB server is not just a file sharing server but with SMB protocol 3.0 features it has SMB Direct, SMB Multichannel (multiple connections per SMB session), SMB Transparent Failover, SMB Scale-Out, SMB Directory Leasing and many more.

To name a few advantages of MoSMB, with improved performance of SMB Direct, improved SMB bandwidth management, and enabling Hyper-V live migration of VMs and VM storage without failover clustering can be achieved.

Achieving high availability of Hyper-V VMs: One benefit that boosts the return on investments is the use of Mo-SMB file shares as a shared storage for Hyper-V hosts. This is sometimes referred to as “Hyper-V over SMB.” In earlier versions of Hyper-V you had to store Hyper-V VMs on a block-based storage to achieve high availability of the VMs.

With MoSMB, you can configure Mo-SMB file shares to host Hyper-V VM files such as VM configuration, virtual hard disks and snapshot files and expect the same reliability, availability and high performance that you achieve using block-based storage.

When implemented as a shared storage, MoSMB helps deploy a Hyper-V server virtualization infrastructure without spending precious IT dollars on expensive SAN devices. There are several other Hyper-V scenarios where MoSMB can be a useful. You can create a file server cluster running on linux machines, create SMB shares, configure the available properties for the SMB shares, and then have the SMB shares available for Hyper-V hosts for hosting the VM files.

If you deploy VM files over a Mo-SMB share created on unix-like servers (Ubuntu, FreeBSD, Solaris) the SMB Transparent Failover feature can help you provide continuous availability of VMs in case of any maintenance activity for one of the clustered nodes in a file server cluster. This is achievable by implementing MoSMB Scale-Out file server cluster. A MoSMB scale-out file server hosts Mo-SMB shares that are simultaneously online on all the nodes in a file server cluster.

SMB shares accessed over smb clients on Scale-out server

Reducing operational expenditure: You can manage file shares instead of requiring someone with expertise to manage storage fabric and LUNs which, in turn, helps you reduce the operating expenditures associated with managing a SAN environment. A virtual administrator who knows how to manage a file share can easily manage MoSMB shares rather than requiring another administrator to manage the complex SAN environment.

MoSMB as a shared storage for VHDX file sharing: The VHDX file sharing feature of Hyper-V helps you implement guest clustering without exposing SAN storage to Hyper-V guests. The VHDX file that will be shared among the multiple VMs must be kept on a shared storage.

Some Hyper-V features use SMB: It is worth mentioning that SMB has a role to play for some of the notable Hyper-V features such as Storage Live Migration and Shared Nothing Live Migration introduced in Windows Server 2012. Neither of those features require linux failover clustering to be implemented to utilize the SMB 3.0 capabilities to move VM and its storage while the VM is running.

Faster Live Migration of VMs: For Hyper-v live migration of VMs, administrators can select SMB Direct or SMB over RDMA in MoSMB for live migration protocol for transferring VMs to other Hyper-v hosts. RDMA network acceleration will give faster live migration.

A protocol for future virtualization: In SMB 3.1.1 protocol, features like SQOS have been added for cloud deployments. This is being supported in MoSMB server stack and will be available to customers by Q2 2017.

Reference: http://searchservervirtualization.techtarget.com/tip/The-power-of-the-SMB-protocol-in-Hyper-V

Some tips & pointers when load testing a non-windows file server using FSCT

Performance testing is the process of determining a system’s behaviour in terms of responsiveness and stability when the system is subjected to a workload. Whenever, a new server is put into service it is subjected to load testing, a form of performance testing, to identify the behaviour of the system under specific workload and also to identify the bottlenecks in the system.

In the case of a fileserver, the responsiveness can be measured in terms of throughput and latency when multiple users are performing concurrent I/O. Also, it is important to find out the number of users the system can successfully support without hitting a bottleneck.

Fileserver capacity tool (FSCT) is an excellent tool to determine how many users a file server can support. It is one among many tools that Microsoft developed for its internal usage and later made it available for the technical community at large It is a command line based tool and it works by running a set of workloads on the file server. The default workload is “HomeFolders” which runs some basic operations like creating files, directories, renaming and editing the files etc. It tries to determine the server’s behaviour assuming the server’s primary function is to store the user’s home directory. The scenarios are stored in .xml files and can be changed to run with a different configuration than the pre-configured one. The operations are performed from multiple clients simultaneously and the performance in terms of throughput, latency is calculated. As we can configure the test to run with n number of users, we can also calculate the maximum number of users that the fileserver supports. The server can be a Windows or non-windows server.

The FSCTStepbyStepGuide.doc available with the build has very detailed steps about configuring the machines to start the testing. In addition to the steps mentioned in the document the following steps can be followed to run the tool against a non-microsoft server.

Following Requirements are needed to run the tool against a non-windows server:

  • Domain Controller: Windows Server Active Directory Infrastructure
  • File Controller: Windows Server 2k8r2 or higher with at least 1GB free space
  • Client: Windows7 or higher with at least 1GB free space
  • Server: The non-windows server which we will be testing.

On all Windows machines following changes need to be done:

1) Disable UAC

2) Disable UAC remote restrictions on all Windows machines.

3) Ensure following services are running on the Domain controller and File controller:

  • DNS Client
  • Function Discovery Resource Publication
  • SSDP Discovery
  • UPnP Device Host

4) On all Windows machines, Turn On Network discovery and File and Printer sharing.

FSCT configures the server by using the hostname of server and sharename as fsroot followed by a number, if the sharename is not fsroot the preparation phase of server will fail.

Although, you can always rename your sharename to fsroot1, it might not be a good solution. An excellent way to address this limitation is

  • Create a softlink of the share with name fsroot1
  • Add the entry in /etc/samba/smb.conf
  • Restart smbd and nmbd service

Once the machines have been configured you can:

  • Run FSCT to prepare the domain controller, file controller, client and server to run the test.

Test Automation Framework using Python

Ryussi created a test automation framework to simplify and speed up the release process of MoSMB (SMB 3.0 server) and find regression issues during every release.

Main features of the framework

  • A highly modularized test framework with ability to test configuration using both individual and global parameters.
  • Reliability, Robustness and Regression  Test Automation
  • Perform simulated testing of MS Office Apps & Media Apps using Synthetic IO Tool
  • Ability to run tests for long duration in days, weeks and even longer.
  • Automation for Data Integrity
  • Filesystem metadata/data related Data integrity testing using MD5 checksum & other in-build methods provided by synthetic tools
  • Performance Automation which provides ability to perform, generate & extract performance parameters such as IOPS/throughput & latency & generate Excel graphs directly from these values

testAutomation_python

 

Benefits

  • Ability to select test type and sub-set of tests to execute for a given test cycle.
  • Fully automated controller node which propagates to worker nodes.
  • Light weight controller (Docker container) to consolidate all test results at a single location.
  • Easily scalable framework which allows addition of test cases with plug-n-play architecture.

Ryussi Advantage:

At Ryussi, we’ve addressed most of the testing automation challenges by adopting the appropriate derivatives of the solutions discussed above & have successfully delivered quality testing as per the needs of our customers.

  1. Leverage the existing knowledge within the organization to support testing
  2. Creation of new test automation frameworks to suit to user requirements that perform functional regression, stress, error-Injection, limit, and data path testing.
  3. Use of Tables, Graphs & Charts to create performance charting for publishing and also knowledge of other 3rd party vendor neutral testing techniques for generating performance numbers under typical workloads.
  4. Ability to identify weakly developed feature/function and focus on specific areas to find maximum issues early during the test cycle rather than later.
  5. In order to adapt to the changing requirements of the customers & quickly come-up with test strategy to address the change.

To assist you in creating your own NAS automation framework, please contact us as sales@ryussi.com

NAS QA Challenges

Storage QA, especially Network Attached Storage, (NAS) QA, poses many challenges for the OEM/ODM. It is not difficult by itself, but it needs a wide array of expertise like File System knowledge, Protocol functionality, understanding differences between various versions of the Protocol and interoperability between different protocol types & the underlying File System.

Moreover, it needs relevant knowledge of various tools & techniques that can effectively cover all the possible use cases.

All these need to be done before the product is released either to its Beta Users or for GA release of the software/firmware/hardware.

NASQAChallenge

 

Types of Testing:

  1. Functionality Testing of the features offered.
  2. Protocol validation, a.k.a conformance (to RFC) testing.
  3. Interoperability testing across matrix of major OS for verification of functionality & conformance.
  4. Compatibility testing against legacy version of the protocol versions.
  5. Scalability testing.
  6. Data Path testing by doing data ingestion of different types, size, and combination.
  7. Control Path testing, a.k.a Configuration Testing
  8. Out-of-Box (OOB) Testing for getting end-user perspective.
  9. User Interface Testing of CLI, GUI, and API if applicable.
  10. Stress/Load testing for identifying stability across various load condition.
  11. Error injection testing for identifying recoverability aspect of the system.
  12. Limit or Boundary Analysis Testing to identify limits of the overall system.
  13. File System interop with the network filesystem protocol; ext3/4 (POSIX), NTFS. ZFS, Reiser, HPFS or any other proprietary File System against SMB 2.x/3.x and NFS 3/4.x
  14. File System and File System Protocol locking semantics testing.
  15. 3rd Party Authentication/Authorization testing such as NIS+/AD/LDAP/Kerberos, etc;
  16. Solution Testing a.k.a. Use Case Testing, such as use in simple file serving, host virtualization, Backup & DR, High Performance Computing (HPC), etc.
  17. Performance testing for publishing to Solution White Papers, Technical Journals, or online vendor neutral charting of performance numbers under typical workloads.

 

Ryussi Advantage:

At Ryussi, we’ve addressed most of the NAS testing challenges by adopting the appropriate derivatives of the solutions discussed above & have successfully delivered quality testing as per the needs of our customers.

 

  1. Leverage the existing knowledge within the organization to support testing
  2. Use of existing Test Automation frameworks to perform functional regression, Stress, Error-Injection, Limit, and data path testing.
  3. Use of readymade open source tools to carry out the above testing using automation framework & also perform other complex testing such as file-locking, interop, scalability, and solution testing.
  4. Use of Tables, Graphs & Charts to create performance charting for publishing and also knowledge of other 3rd party vendor neutral testing techniques for generating performance numbers under typical workloads.
  5. Ability to identify weakly developed feature/function and focus on specific areas to find maximum issues early during the test cycle rather than later.
  6. In order to adapt to the changing requirements of the customers & quickly come-up with test strategy to address the change.

 

Monitoring System for OS

Monitoring the performance of operating systems and processes is essential to debug processes and systems, effectively manage system resources, making system decisions, and evaluating and examining systems. These tools are primarily divided into two main categories: real time and log-based.

Real time monitoring tools measure the current systems’ state and provide up to date information about the system performance.

Log-based monitoring tools record system performance information for post-processing, analysis, and determine trends in the system performance.

Here we present a survey of the most commonly used tools for monitoring operating system and process performance in Windows- and Unix-based systems that describes the unique challenges of real time and log-based performance monitoring.

Platforms focused

  • Linux all supported versions
  • Windows 8 onwards, windows 2008 onwards.
  • Mac OS 10.6 onwards

Monitoring means logging all events.

Features for monitoring

  • Following are the interested features, drawn along with the platform which states how it can be implemented
Features List Understanding Linux Mac Windows
Process creation and termination Monitor process creation and termination parameters like process creation time, termination time, process control block etc. Using

·  /Proc/

·  ps command

·  auditctl -a task,always

Using

·  top

·  sysctl

·  launchctl

·  launchd

·  lldb

Using

·  wmic process list

Process and memory execution (interactions) Monitoring interactions between executing process and corresponding memory Using

·  /proc/

·  ps command

·  auditctl -a task,always

Using

·  top

·  sysctl

·  launchctl

·  launchd

·  lldb

Using

·  wmic process list

Process injection Here lets take example, Suppose there are porcess which is designed to execute for longer period of time, lets take daemon. However program need to be updated in simple way without stopping execution.here we can inject code. Monitoring such injection. Using

·  ptrace

Using

·  top

File IO activity (manipulations) Mmonitoring file operations like open, read and write. Using

·  Inotify

Using

·  opensnoop

·  lsof

Using

·  FileSystemWatcher Class

File properties Monitoring file metadata Using

·  Inotify

Using

·  opensnoop

·  lsof

Using

·  FileSystemWatcher Class

Executed binary Binary executed clears all memory used. monitor

·  ptrace

Using

·  opensnoop

·  lsof

Using

·  event tracing for windows (ETW)

Registry activity (modifications) / system configuration We need to monitor configuration of os Monitor /etc/*.config files, Most of the settings which defines execution of the process stored in config files. monitor changes in /Library/Preferences/SystemConfiguration/ Monitoring changes in registry
Module (such as DLL) load loaded libraries into memory Using

·  ptrace

·  lld

Using

·  otool

using
EnumProcessModules
Network activity monitor open ports , sockets, protocols used, bandwidth utilization Using

·  netstat,

·  ss(socket statistics)

Using

·  nettop

 

Using

·  getmac

·  hostname

·  ipconfig

·  nbtstat

·  net

·  netsh

·  netstat

·  nslookup

·  pathping

·  ping

·  route

·  tracert

·  tracert

Attached devices Monitor attached external devices such as usb, external hard drives, and other input devices Using

·  ls udev

Using

·  diskutil list

similar to devcon
Network connection monitor open ports and data Using

·  netstat

·  ss

nettop Using

·  net

·  netstat

Application behavior Heuristic scanning looks for code and/or behavioral patterns Set of path maps to follow by application.
System resources Monitor memory pools, pages tables, Using

·  auditctl

·  top

·  ps

·  vmstat

Using

·  top

·  lsof

Tuning set of commands Using wmic
ASEP Registrations Unable to get what is ASEP? Is it Auto Start Entry Points? automount?? monitor Run and RunOnce form registry key
Normal Object Access Which objects come under term “Normal Objects”
Boot Arguments Monitor boot time parameters of the Linux kernel like init, nfsaddrs, no387, etc. monitoring

·  /etc/lilo.conf, /etc/grub.

Monitoring

·  nvram

Monitoring

·  Boot.ini

If you would like to share your experience with each of the features listing here or would like to build on these tools, please do share to help knowledge and expertise reach excellence..

Linux Incremental volume backup

Incremental volume backup for Linux like operating systems can be implemented by tracking block-level writes. The block-level writes tracking can be achieved by adding a block level filter driver through ‘Device mapper’.

The Device Mapper is a kernel component which could transform a block I/O (“BIO”) request based on different policies. By transforming a BIO request, a Device Mapper device could remap the request to a different address, to a different block device(psudo device created based on device mapper on top of original device), or simply perform some bookkeeping task and then pass the request to the underlying original device.

Read More

Heat Orchestration to automate Swift Installation on a Block Storage Device – Part 2

The manual Swift deployment process is very onerous – too many things have to be handled at the same time and too many configuration parameters. Over the course of two notes we are trying to talk through how to install Swift using OpenStack Heat orchestration for a Flash Storage Array (backend store) supporting iSCSI. In the previous part of this note, part 1,  we had defined some of the basic terms and the context they will be used in, some features the script should possess and a look at the Installer. In this part we will look at the details of the Implementation.

Read More

Heat Orchestration to automate Swift Installation on a Block Storage Device – Part 1

The OpenStack Orchestration program is designed to create a human & machine accessible service for managing the infrastructure and applications lifecycle within the OpenStack clouds. Heat is the main project within the program. The manual Swift deployment process is very onerous – too many things have to be handled at the same time and too many configuration parameters. In this case Heat becomes the conductor of the orchestra.

In the following two parts of this note we will talk through how to install Swift using OpenStack Heat orchestration for a Flash Storage Array (backend store) supporting iSCSI.

Read More

Hyper – V Virtual Hard Disk Format – An Overview

Enterprise workloads have been growing for a while now. As the demands for size and performance in virtual environments also grew Virtual Hard Disk drive formats had to evolve to accommodate them. It was with this in view that a new format of the VHD called the VHDX was introduced a couple of years ago by Hyper – V in Windows Server 2012. This had a much greater Storage capacity, data corruption protection during power loss and optimized structural integrity across dynamic and differencing disks. This was designed to provide better performance across the newer disk with larger sectors. In this presentation we look provide a comprehensive view of the format as well as look at some specific usability scenarios. What has been your experience of this file format? Any specific points of information that you may like to share that could help others in the same boat as you?

Read More

MAC Backup Options – An in-depth view of available options and a wish list for a new one (Part 2) !

In the last post we had looked at Time Machine and Time Capsule – the inbuilt Mac OS Back-up and Disaster Recovery options and at the advantages as well as several challenges with those solutions. There are quite a few options available for users to be used in combination with Time Machine or other Mac backup applications to address specific issues. Some of these are :

  1. CCC: Carbon Copy Cloner

 

Approximate Cost – $40

Read More

MAC Backup Options – An in-depth view of available options and a wish list for a new one (Part 1) !

MAC devices form a significant percentage of home and business use devices and it follows that a large amount of data being created today originates from such devices. The rising volume of data brings into focus the MAC OS backup and recovery options available. In the next couple of posts we will touch upon Time Machine and Time Capsule the inbuilt Back-up options available with Mac and then at some options available out there to be used in combination with Time Machine and our own wish list of what a great new Mac OS backup solution.

Read More

OpenStack Swift Backup Strategy – Part 2

In the last post we identified the need for a solution that addressed how to handle disaster recovery for object data storage systems and also comply with data storage and retention laws. We defined a base-line scenario and a likely approach that could work for that scenario along with the pros and cons of that approach. In this post we look at a couple of more possible approaches that could work for the same scenario.

Approach 2: This approach is much more stable and also simpler. We could call it a modified off-host backup.

Read More

OpenStack Swift Backup Strategy

“Do as you please, I’ll back you up” – The Dave Matthews Band

Use case:

Data backup is required not just for disaster recovery, but also for legal compliance. OpenStack Swift in itself has architecture to deal with disasters by way of data replication to Zones that are distributed across geographies.

This however does not really constitute a backup strategy as it does not provide point-in-time state-of-data under protection and neither does it comply with the legal requirements for data retention. This is especially true if the cloud storage is being used for email archival.

Read More