TurnKey App deployment

It's pretty convenient to deploy TurnKey App on IaaS or PaaS platforms. In addition to pre-configured packages, web shell as well as webmin for the specific app is also set! This saves lots of time for sysadmin who needs to manage A LOT OF software at the same time.


REF: https://en.wikipedia.org/wiki/TurnKey_Linux_Virtual_Appliance_Library

Zabbix: compile agent

Zabbix agent is useful for pulling or pushing checks on many platforms, which is a general all-in-one solution for monitoring. Although Nagios can also achieve similar features, it requires further understanding of its modules (plugins and addons). Modular design may be useful in heterogeneous environments.

$ ./configure --enable-agent

You may use the --enable-static flag to statically link libraries. If you plan to distribute compiled binaries among different servers, you must use this flag to make these binaries work without required libraries. Note that --enable-static does not work under Solaris.

REF: https://www.zabbix.com/documentation/2.0/manual/installation/install 

Check_MK: NSCA push checks

In some cases we need to push checks from client to server, for example dynamic ip, behind firewall, etc. NSCA service of Nagios Addon is useful for such case, which can be turned on via 'omd config'. However, if you have many such push hosts, using Zabbix with its native passive poller via agent may be a better choice.

freshness can also be checked via WATO->Host & Service Parameters->Active Checks-> Classical...

REF: http://users.telenet.be/mydotcom/howto/nagios/nscaclient.html

Test and tweak

On a linux system, you can test your send_nsca by running something like this (where nagiosserver is the FQDN of your nagios server) :

 echo -e "localhost \ttestservice \t0 \tTEST " | send_nsca nagiosserver  

On the Nagios server, look in syslog or /var/log/nagios/nagios.log : you should see a mention that nagios received your message, and a complaint that it can't process the service check result because the service doesn't exist. That's because you did not define service "testservice". But it confirms that your nsca setup works.

REF: http://lists.mathias-kettner.de/pipermail/checkmk-en/2013-March/008780.html

CHECKMK MULTISITE CONFIGURATION
1) In WATO go to "Host & Service Parameters" >> "Active Checks" >>
"Classical active and passive Nagios checks"

2) Create a new rule in which the "Service description" is the same name
that is going to receive the NSCA information.  (This name would also be
configured on your NSCA agent on the client being monitored.)

3) Make sure the new rule only applies to the hosts that will be receiving
the NSCA information.  This can be done via tags or by explicitly
specifying host names.  I prefer tags as that way I can just set up a new
host with the correct tags and it will automatically get the new NSCA
Service added to it's checks.

4) In the "Command line" checkbox of the new service, I put something like
this
'echo "ERROR - you did an active check on this service - please do not do
that on this service" && exit 1'.
(This is similar to what MK is doing with CheckMK passive checks in the
CheckMK template file.)

5) Go back to the Main Menu for "Host & Service Parameters" (The Rule
Editor)

6) Go to "Monitoring Configuration"

7) Create a new rule under the section "Service Checks" in the category
"Enable/disable active checks for services"

8) Make sure this rule specifies the Service name that you created back in
step 2 and set it to "Disable active checks".

9) Apply the changes in WATO

It is important to note that at this point, you have now created a Service
for a host (or hosts) that is not actively being checked and is just
sitting as a passive check.  This is good because it will allow NSCA to
pass status information for the Service back to Nagios.

ffmpeg on Mac

Simple howto for regular users in Chinese. ffmpeg parameters are for watermark overlay then scale down to 720p mp4.

1.   下載Mac版的ffmpeg.
http://evermeet.cx/ffmpeg/ffmpeg-3.2.2.dmg

2. 打開此dmg，裡面有一個ffmpeg，將他拉到你的家目錄的「下載項目」資料夾中。
將你要轉檔的檔案，改成無空格英數檔名後，也放到此資料夾中。hdlogo.png也要放
進此資料夾。

3. 打開：應用程式->工具程式->終端機

4. 按順序複製/貼上以下兩行指令。每行打完按enter。ffmpeg那行是一行不能斷。把
輸入與輸出檔案換成你要的。

cd Downloads/

./ffmpeg -y -i 輸入檔案 -vf "movie=hdlogo.png [watermark]; [in][watermark]
overlay=10:10 [wlogo]; [wlogo] scale=1280:720 [out]" -movflags faststart -crf 27
-c:v libx264 -r 30 -c:a aac -b:a 192k -threads 0 輸出檔案.mp4

ffmpeg volume detect

We can get audio level statistics easily using ffmpeg audio filter.

Find out the gain to apply

First you need to analyze the audio stream for the maximum volume to see if normalizing would even pay off:

ffmpeg -i video.avi -af "volumedetect" -f null /dev/null

Replace /dev/null with NUL on Windows. This will output something like the following:

[Parsed_volumedetect_0 @ 0x7f8ba1c121a0] mean_volume: -16.0 dB
[Parsed_volumedetect_0 @ 0x7f8ba1c121a0] max_volume: -5.0 dB
[Parsed_volumedetect_0 @ 0x7f8ba1c121a0] histogram_0db: 87861

As you can see, our maximum volume is -5.0 dB, so we can apply 5 dB gain. If you get a value of 0 dB, then you don't need to normalize the audio.

Apply the volume filter:

Now we apply the volume filter to an audio file. Note that applying the filter means we will have to re-encode the audio stream. What codec you want for audio depends on the original format, of course. Here are some examples:

• Plain audio file: Just encode the file with whatever encoder you need:

  ffmpeg -i input.wav -af "volume=5dB" output.mp3
  

  Your options are very broad, of course.
• AVI format: Usually there's MP3 audio with video that comes in an AVI container:

  ffmpeg -i video.avi -af "volume=5dB" -c:v copy -c:a libmp3lame -q:a 2 output.avi
  

  Here we chose quality level 2. Values range from 0–9 and lower means better. Check the MP3 VBR guide for more info on setting the quality. You can also set a fixed bitrate with -b:a 192k, for example.
• MP4 format: With an MP4 container, you will typically find AAC audio. We can use ffmpeg's build-in AAC encoder.

  ffmpeg -i video.mp4 -af "volume=5dB" -c:v copy -c:a aac -strict experimental -b:a 192k output.mp4
  

  Here you can also use other AAC encoders. Some of them support VBR, too. See this answer and the AAC encoding guide for some tips.

In the above examples, the video stream will be copied over using -c:v copy. If there are subtitles in your input file, or multiple video streams, use the option -map 0 before the output filename.

REF: http://superuser.com/questions/323119/how-can-i-normalize-audio-using-ffmpeg

secure data transfer

FTP may be the most popular protocol for transferring data over Internet. However, it is unencrypted by default. Here're some remedies for this issue.

• FTP over SSL. it may be affected by SSL mismatch of versions between clients and server due to SSL upgrade.
• SFTP from OpenSSH. shell access or chroot may be required.
• VPN. using VPN as secure tunnel would be easier between sites.

Check_MK: Multisite distributed monitoring

Set remote sites as slave for distributed WATO (v1.2.8+).

• Configuration Replication (Distributed WATO)
• Replication method "Slave: push configuration to this site"
• Peer replication priority 0
• Multisite-URL of remote site "http:// / check_mk /"
• SSL - SSL certificate errors Ignore activated
• - The slave host has properly configured xinetd for services and hosts appear in the master slave

Enable TCP connections to livestatus on slaves.

• omd config yoursite
• via xinetd

REF: https://mathias-kettner.de/checkmk_livestatus.html

TriCaster 8000 switcher

REF: http://www.newtek.com/tricaster/8000

Virtually Unlimited Scalability

Accommodate every show imaginable. Expand your inputs through external video routers, break out control of live operation to additional operators or a crew of only one, and output to multiple destinations—for broadcast, live streaming, display, or across the network to another TriCaster.

Connect 10 external sources, and expand with router support

Redundant power supplies

Durable 4RU rackmount chassis

4 removable drive bays

 

 

 

Complete Customization

Personalize your workflow and flex your workspace to fit your show. Mix and match inputs, outputs, and monitor configurations. Automate any function or sequence. And customize PTZ camera angles, virtual sets, and more to deliver a unique experience for every show with command at your fingertips.

Monumental Effects and Graphics

Make an impression on your audience with a full palette of composition tools including 8 M/Es with re-entry, real-time motion tracking, 360° photorealistic virtual sets, and custom full-color, warping video transitions with audio that elevate your production into visual artistry.

Social Sharing

Deliver second and third screen experiences and social shares that trend worldwide—while live. And, with the ability to begin uploading recordings while production is in progress, on-demand viewers can start catching up immediately, and you capture more audiences—sooner.

Ready for IP — and Beyond

Take your productions to unprecedented levels with the optional TriCaster Advanced Edition software upgrade, adding more than 100 production-enabling features without spending a penny on more hardware. You’ll unleash an exponential workflow that knows no bounds with NDI™, our own IP video protocol—access any source, anywhere on your LAN over a standard ethernet connection.

Syslog: push data / rsync

Hosts behind NAT or without fixed ip may be the reason we want them to 'push data' for our monitoring. However, letting clients 'phone back' isn't a very good idea for security or stability. Here are some workarounds.

• syslog. using remote logging but no authentication or encryption is offered. use 
• rsync. push data with script running cron job.

REF: http://lists.mathias-kettner.de/pipermail/checkmk-en/2012-June/006232.html

Dell virtualization with VMware vSphere

REF: http://www.dell.com/learn/us/en/555/virtualization/dell-and-vmware-alliance

• Deliver simple, expandable, and affordable storage for your virtualized workloads with Dell™ EqualLogic™, Compellent, and Dell PowerVault™ storage solutions. Over the years Dell and VMware have worked together to develop and deploy deep integration with VMware storage-related projects, such as SRM, VAAI, and VASA. These storage integrations help to drive efficiency, agility and continuity for your virtualization projects.
• Provide highly efficient and flexible networking with Dell- VMware innovative networking solutions. This includes integration with NSX for simplified management and enhanced workload mobility. And Dell’s distributed-core network switches provide centralized, programmatic control of network service provisioning to help you realize the full potential of the software defined data center.

Value-added software - Dell provides a range of software that complements your VMware virtualization initiative.

• Dell OpenManage provides integration between Dell hardware management and VMware vCenter. Dell Active System Manager provides end-to-end management and automation of infrastructure and workloads in a heterogeneous physical environment.
• Dell Foglight for Virtualization Enterprise helps you visualize, analyze and optimize the performance and capacity of your entire heterogeneous virtualized infrastructure.
• Dell has a complete suite of data protection offerings that can protect your entire virtual environment.

Syslog: with logwatch

There isn't many options in conf for further syslog processing. So remote logging to another server, then using Cacti syslog plugin or Check_MK/Nagios logwatch for filtering or alert is the practical way.

# Remote logging
#*.* @log.server.ip:514

REF: http://www.netadmin.com.tw/article_content.aspx?sn=1609300018

Check_MK: YUM Update package

REF: https://mathias-kettner.de/checkmk_packaging.html#H1:Installation,%20Update%20and%20Removal

The installation and update of a package is done with the command install. Simply add the name of the package file as an argument:

root@linux# check_mk -P install foo-1.0.mkp

http://mathias-kettner.com/check_mk_exchange_file.php?HTML=yes&file=yum-2.0.3.mkp

YUM Update Check

Package	yum
Version	2.0.3
Author	Henri Wahl
Uploaded	2015-10-01 15:00:03
Description	Checks for updates on RPM-based distributions via yum.
Website	https://github.com/HenriWahl/checkmk-agent-plugin-yum
Minimal Version	1.2.4
Packet with	1.2.6p10
Filesize	3.06k
Content	agents/plugins/yum checkman/yum checks/yum web/plugins/wato/yum_check_parameters.py
Checksum (MD5)	405fe7bb6ff1dd897b8723c8d30bf220
Downloads	2185

Download

Draytek routers

Draytek routers may be good choices for small business routing. It supports VPN, VLAN, QoS, monitoring, etc, as well as syslog, web and cli consoles.

REF: https://www.draytek.com/

Check_MK: IDS alerts

Sometimes we need to pinpoint for details of suspicious DNS query alerts from IDS such as snort. tcpdump and tshark are necessary tools for packaet analysis.

# tcpdump -vvv -s 0 -l -n port 53 | grep suspicious.dns.url
# tcpdump -f 'dst host suspicious.ip'
# grep suspicious.dhcp.client /var/log/messages

=> then you can get the computer name as well as physical address of the source.
REF: https://jontai.me/blog/2011/11/monitoring-dns-queries-with-tcpdump/

Check_MK: Security Onion

SOC components could refer to the structure of Security Onion: 

1. HIDS: ossec
2. NIDS: snort
3. Asset data: Bro
4. Packet capture: netsniff-ng
5. Host: syslog, ossec
6. Session/transaction: Bro

Item 4 and 6 may use tcpdump or netflow as alternative, while item 1-3 mabe be integrated via Nagios/Check_MK.

REF: https://securityonion.net

Perl: Windows compiler

REF: http://stackoverflow.com/questions/2948405/how-to-compile-a-perl-script-to-a-windows-executable-with-strawberry-perl

 :: short answer :
  :: perl -MCPAN -e "install PAR::Packer" 
  pp -o <<DesiredExeName>>.exe <<MyFancyPerlScript>> 

Check_MK: Puppet Policies

If you have "pets" much more than "cattle", in cloud terms, you could consider using Check_MK as a simple substitution of automated Puppet policies. Process detection then reaction can perform similar effects as Puppet agent.

To monitor one of the processes:

REF: http://parttimemis.blogspot.tw/2016/05/checkmk-raw-edition-part-7.html

For action:

REF: http://parttimemis.blogspot.tw/2016/11/checkmk-actions.html

IGMP multicast IPv6 bridge issue

# issue: "eth1: received packet with own address as source address"

# first disable ipv6. messages decreased.

sysctl net.ipv6.conf.all.disable_ipv6=1

sysctl net.ipv6.conf.default.disable_ipv6=1

sysctl net.ipv6.conf.lo.disable_ipv6=1

# looking for exact problematic packets

tcpdump -i eth1 -w aalmac.pcap
tshark -r aalmac.pcap -i eth1 -nn -e eth.src -e eth.dst -e ip.src -Tfields

# found. STP related address of Cisco switch. 

Root Cause

• Sending general queries was implemented as an optimisation to speed
  up convergence on start-up. In order to prevent interference with
  multicast routers a zero source address has to be used.
• Unfortunately these packets appear to cause some multicast-aware
  switches to misbehave, e.g., by disrupting multicast packets to us.
• There is a flood of packets back to the system causing the message "kernel: bond0.xxxx: received packet with own address as source address" to appear endlessly.

#  Work around on bridge interface. Newer kernel should resolve this issue already.

brctl setmaxage br1 0

brctl setageing br1 0

REF: https://access.redhat.com/solutions/260053

Perl: while read LINE

# similar to BASH while read LINE loop. Not typing more!

open FH, "mytext.txt";
while (FH) {
 chomp;
 print $_ ;
}

AWS re:Invent Recap 2016

• Big Data 
• IoT
• Serverless

Check_MK: time sync check

Checking fluctuation of system time difference between servers can be done by Check_MK's local checks. Start NTP service on primary server, then `ntpdate -q` the primary server to get the offset interval. Set WARNING and CRITICAL values in your local check script, either shell or perl script, then all is done. Monitoring interval better set one hour or more.

REF: https://mathias-kettner.de/checkmk_localchecks.html

Bitnami: security

Although Bitnami Application Stack is convenient for IaaS deployment, security issue is still needed to be carefully handled for the whole stack. As reference.

REF: https://docs.bitnami.com/virtual-machine/security/
https://community.bitnami.com/t/when-will-bitnami-update-app-stacks-to-fix-openssl-heartbleed/23532

Check_MK: ignored services

REF: http://mathias-kettner.de/checkmk_inventory.html
=> ignored services can be set on GUI.

Excluding items from the inventory

Sometimes the inventory finds things that you do not want to check. Removing that items from the files in autochecks is not a perfect idea: At the next inventory those items will reappear again.

It is better to explicitely exclude them. Check_mk provides three configuration variables for doing that:

Config variable	Meaning
ignored_checktypes	Simple list of checktypes to exclude from inventory
ignored_services	Host specific configuration list of service names to exclude
ignored_checks	Host specific configuration list of checktypes to exclude NEW in 1.1.9i1

Chromebook with Google Play

Chromebook with Google Play looks cool. Works as both tablet and netbook :)


REF: http://www.ithome.com.tw/news/110981

s3 to glacier

aws官方文字介面工具，sync同步檔案至s3。而s3啟用lifecycle，可將資料移動到glacier去。 根據以下兩篇官方文章，是否可以推論：即便檔案移到glacier去，但index還是可以在s3取得，所以sync不會因為檔案的storage class變成glacier了，導致一樣的檔案重傳一次？

https://aws.amazon.com/tw/blogs/aws/archive-s3-to-glacier/
http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

還是會像下述WD產品一樣，會因為lifecycle to glacier而不能正確運作？

https://community.wd.com/t/amazon-glacier-backups/99386

Check_MK: fileinfo

REF: https://mathias-kettner.de/checkmk_check_fileinfo.html

Currently Windows, Linux, FREEBSD, MACOSX and Solaris agents supports this type of check. 

For windows, the configuration is done in check_mk.ini in the section [fileinfo]. Put one or more entries of the format path = C:\Some Dir\*.log into this section. The agent will send all files that match the pattern. Wildcards are only allowed in the filename part, not in the directory path. 

For Linux, FREEBSD, MACOSX and Solaris the configuration is done via fileinfo.cfg located in the $MK_CONFDIR (usually /etc/check_mk/fileinfo.cfg). To add files to the check, simply add one path pattern /tmp/*.log per line. Wildcards or patterns can be used as usual on the console. 

Shrinking qcow2 disk files

REF: https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files

Use the following command to zero out your disk:

dd if=/dev/zero of=/mytempfile
# that could take a some time
rm -f /mytempfile

• rename original qcow2 file.

mv image.qcow2 image.qcow2_backup

• Shrink your disk with compression (smaller disk size, takes longer to shrink, performance impact on slower systems):

qemu-img convert -O qcow2 -c image.qcow2_backup image.qcow2

Ivy as ubuntu sidebar

Ivy (Sidebar, Widgets, RSS) can be installed on Android to simulate the cool sidebar of Ubuntu Touch. Interesting : )
REF: https://play.google.com/store/apps/details?id=com.stinger.ivy

Check_MK: Architecture

REF: https://mathias-kettner.com/check_mk.html

ffmpeg dvvideo transcoding

for NTSC, with PCM audio exported by NLE software (pcm_s16le -ac 2 -ar 48000):

# ffmpeg -y -i $video -i $pcm16_audio -f mov -vcodec dvvideo -r 29.97 -s 720x480 -aspect 16:9 -pix_fmt yuv411p -acodec copy -map 0:v:0 -map 1:a:0 -shortest $output

REF: http://ffmpeg-users.933282.n4.nabble.com/DV25-quicktime-code-td943121.html

completely disable MySQL replication

REF: http://dba.stackexchange.com/questions/21119/how-do-i-completely-disable-mysql-replication

* also 'flush slave' or 'slave stop' or 'reset master'

On the slave server(s):

1. Run "stop slave" to stop replication.
2. Run "reset slave" to tell the slave server to forget it's position in the binary log retrieved from the master server.
3. Add "skip-slave-start" to my.cnf to prevent replication from starting when you restart MySQL.

Linux change the speed and duplex settings

REF: https://www.cyberciti.biz/faq/linux-change-the-speed-and-duplex-settings-of-an-ethernet-card/

Task: Get speed and other information for eth0

Type following command as root user:
# ethtool eth0Output:

Settings for eth0:
     Supported ports: [ TP MII ]
     Supported link modes:   10baseT/Half 10baseT/Full
                             100baseT/Half 100baseT/Full
     Supports auto-negotiation: Yes
     Advertised link modes:  10baseT/Half 10baseT/Full
                             100baseT/Half 100baseT/Full
     Advertised auto-negotiation: Yes
     Speed: 100Mb/s
     Duplex: Full
     Port: MII
     PHYAD: 32
     Transceiver: internal
     Auto-negotiation: on
     Supports Wake-on: pumbg
     Wake-on: d
     Current message level: 0x00000007 (7)
     Link detected: yes

Or use mii-tool command as follows:# mii-tool eth0Output:

eth0: negotiated 100baseTx-FD flow-control, link ok

AWS EC2 Snapshot backup

REF: http://stackoverflow.com/questions/13834324/amazon-ec2-ebs-backup-ami-vs-snapshot

EBS Snapshots are an excellent way to create backups.

You can perform frequent Snapshots of your EBS Volumes via scripts. Weekly, Daily, Hourly, or as frequently as your Credit Card will allow. The only limit is around how many simultaneous snapshots you can be doing - when you hit that, the EBS API will start giving back errors until a few of the in-flight operations complete.

Snapshots can also be copied from Region to Region in order to provide backup against a catastrophic event.

When you snapshot an EBS volume, that snapshot is of the entire volume. Even if it was created from an AMI, your snapshot contains everything you need to create a new instance of the volume. You can pretty easily try this yourself.