CephFS as SDS

To run the Ceph File System, you must have a running Ceph Storage Cluster with at least one Ceph Metadata Server (MDS) running. For details on installing the Ceph Storage Cluster, see the Installation Guide for Red Hat Enterprise Linux or Installation Guide for Ubuntu. See Chapter 2, Installing and Configuring Ceph Metadata Servers (MDS) for details on installing the Ceph Metadata Server.

REF: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ceph_file_system_guide_technology_preview/what_is_the_ceph_file_system_cephfs

Software-defined storage with LizardFS

Figure 1: The LizardFS master is the center of all operations (source: Skytechnology).
REF: http://www.admin-magazine.com/Articles/Software-defined-storage-with-LizardFS

fiber deployment notes

• 100 Gbps = 24c
• 40 Gbps = 12c
• 10 Gbps = 2c
• LC connector
• SDN switch on each rack
• fiber / network run upper space
• power cord run floor space

SDS configuration and performance

Quote: The three candidates in the test follow different approaches. Ceph is a distributed object store that can also be used as a filesystem in the form of CephFS [4]. GlusterFS and LizardFS, on the other hand, are designed as filesystems; however, although just two nodes are enough to operate a Gluster setup, LizardFS needs an additional control node, for which it has a web interface (Figure 1) that informs you about the state of the cluster.

REF: http://www.admin-magazine.com/Archive/2017/37/SDS-configuration-and-performance

Red Hat Infra

This figure of Red Hat Infra is cool. You can use this to check the completeness of your own infra, as well as the documented controls in ISO27001:2013.

Ceph Day 2016

Tuning Ceph parameters is the key for good performance. Please check the CRUSH  Map Parameters for details.

ZFS vs GlusterFS

REF: https://www.jamescoyle.net/how-to/471-zfs-and-glusterfs-network-storage

ZFS offers superb data integrity as well as compression, raid-like redundancy and de-duplication. As a file system it is brilliant, created in the modern era to meet our current demands of huge redundant data volumes.

The problem with ZFS is that it is not distributed. Distributed file systems can span multiple disks and multiple physical servers to produce one (or many) storage volume. This gives your file storage added redundancy and load balancing and is where GlusterFS comes in.

openstack spec

Compute 與 Storage nodes 應該是建造像 OpenStack 這種 IaaS 最常用的components. 因此提早規劃適當的伺服器規格給他們，不算浪費時間吧 : )

• Compute應考慮刀鋒架構。2U4node的刀鋒伺服器做一個節點挺合適。
• Storage的話，當然是用SDS這種IP Storage，而每個硬碟不要太大，維護。Ceph的話，2U機殼含12Bay硬碟，兩個做SSD cahe，還剩10個用2TB HDD. replication三座，這樣一個單位是20TB.

Storage network trend

一些趨勢觀察報告如下。

• IDC projection: 3x growtg in FC+NAS×iSCSI storage.
• IP storage vs Fibre Channel ('95~) storage.
• Gen5 16Gbps vs Gen6 32Gbps
• SSD/Flash requires faster networking:IOPS vs throughput
• Flash drive speed may be 10,000 times faster than 7200rpm!
• Monitoring and Alerting Policy Suite (MAPS) for network packets. ~ intuitive reporting ex. CRC errors.
• Loop is still the common issue for IP Network! => Brocade VDX could auto detect & correct config.
• Pure Storage: warranty peroid + volume capacity, as simple pricing.
• Introducing forever flash: controller (firmware) upgrade every 3 yrs. All Flash, everywhere.
• ex. Netflix has 1.5 factor gzip compression, and higher IOPS of PB scale data.

vendor locking

昨天聽了NetApp的產品介紹，很酷。但跟EMC一樣啦，很貴，而且vendor locking。代理商回答的妙：你可以不被A代理綁，轉跟B代理買啊。我不是問這個啦，而是NetApp撐不住怎麼辦？EMC都賣給Dell了，老大哥storage營收逐年下滑，這才是憂心的。列舉不錯的特性如下。

• Auto support。偵測到硬碟(或其他模組？)問題，代理商於四小時內寄備品過來。
• force shutdown. 該換的硬碟沒換，短時間內強制關掉storage等候救援。
• redundant. 重要的元件如controller，至少雙備援。
• 4U chassis 可以塞 48個3.5"硬碟。 

hybrid backup

有效的資料備份之前，要落實資料分級。這樣，才能回答多重要的資料，備份頻率要多高，要用多少成本的方法備份。備份頻率與數量定下後，由各單位組長督導備份作業的落實。資料備份是有效對抗檔案綁架或損毀的方法。

• SAN, SDS array, Object Storage (Online)：相對昂貴，集中且架構強健的方案。
• RAID array (Nearline)：balanced, affordable. NAS型態的備份，承載適中，擴充適中。
• Single HDD：大家都會的單碟備份，技術與設備成本最低，但人力與行政成本最高。

混合式備份，可讓TCO降到最低，CP最佳化。

VM / Compute instances backup / restore

雖說VM可以自在遷移，但根本上還是依賴於實體CPU與儲存空間，本體instance還是得生存於storage上啊。除了以regular cycling的方式勤奮migration或backup來避免single point failure之外，還有更有效率的辦法嗎？縮短downtime時間是最簡單的第一步吧。

• Compute-only. 無論KVM全虛擬或是LXC容器，只要沒存太多資料，instance不大，搬移backup都很快。
• Storage dependent. 目前想到的就是分離storage componet，獨立出Compute，就可以單獨把Compute SOP了。接下來就是用哪種SDS來保障Storage availability.

HDD on rack

無論決定用GlusterFS還是Ceph這些SDS軟體架構，都得先規劃出硬體架構為何。

• 硬碟櫃。就像一個很能裝硬碟的特質機殼，像QCT他家有做一個4U裝七十幾顆的，超酷，1U可裝16個之類的。但這種櫃子就是價格高，單次花費大。之前買過一個外殼幾萬塊，正面裝16個硬碟，4U。
• 外接陣列。用USB3之類的外接硬碟盒，一盒裝五個左右，這樣便宜很多，但機架管理比較麻煩。若用小型工業電腦掛載，3U放兩台電腦，4U放四座硬碟盒，最高5x4x4T=80TB。7U放20個硬碟，比起上述4U放16個，有點佔空間。

Software Defined Data Center

整個資料中心用Software Defined來設計，就變成Software Defined Data Center了。其實軟體定義的概念挺實用，他強迫你拆開所有元件，從新pooling，自然就達到最大效率與彈性。除了SDN, SDS之外，若Business Application也都遵循這樣的設計，很酷。

• Controller. IncludingUser (Login) Management，Database (Most important definitions, policies resided)，Orchestration (automation)。
• Storage.  Clustering, redundant data storage based on  SDS.
• Head End. Controlled by Controller and fed data by Storage. Ex. File gateway, video streaming.
• Of course, SDN with strong backbone is required for all Software Defined designs. 

Software Defined, Cloud compatible File service

檔案服務的歷史由來已久。但要升級到Software Defined, Cloud compatible，思路要換一下。不是非得用當紅的SDN, SDS才能做，而可以在基礎架構上改動，使其更有彈性。

• gateway service可用市售appliance或是iptables, pf。單獨跑在一台至多台VM上。SDN from Internet由此進。其實也就是老牌CDN概念。
• file service可用SFTP, FTP, CIFS。單獨跑在一台至多台VM上。SDN in intranet由此進。
• storage service可用NFS, iSCSI，單獨跑在一台至多台VM上，外掛給file service。
• 儲存空間由storage VM來管理。根據需要進行多份replication，如rsync, GlusterFS等等。其實就當做自己在調自己想要的RAID演算法，自己做SDS。

Software Defined Storage, SDS

Red Hat拿下Ceph之後，或說從GlusterFS開始，就大力的鼓吹Software Defined Storage (SDS)。GlusterFS已經是distributed FS的架構，而Ceph更進一步成為大家用OpenStack的Cinder服務首選，能調的地方要比GlusterFS多很多。基本unit的目前主流，是1 primary + 2 rep，Hadoop似乎也是這樣存三份於commidity hardware？不過很能調的SDS，應該就得配合彈性很大的SDN (Network)吧。

舉凡你要多少比例的IOPS與TB容量的取捨, MB/s的tuning，混用SSD, HDD, SAS甚至NVMe的儲存個體，10Gbps甚至40Gbps的混搭傳輸，sequential vs random R/W的調整等等。在充斥著DB, analytics, 以及近年竄紅的video streaming市場，都可對應到object, block, file based的整個cluster network。這個35 billion dollar market，似乎有10%蠢蠢欲動著擁抱新技術呢。