mingbo's tech tips

There are many great articles and videos on gray code, why/when we need gray code, how to compute gray code etc..  Here's a comprehensive one: https://en.wikipedia.org/wiki/Gray_code  Among all those articles and videos, this formula is used (^ is logical xor, >> is right shift):  num ^ (num >> 1)  however, I can't find any good explanation about why. I'll try to explain it here in an easy to understand way (not mathematical way). First, we need to understand the easy way of generating sequence of gray code.  You can read the wikipedia page of any other articles out there if you need a very detailed version. Here's my simple version. Let's start with 1 bit, that's easy Decimal Binary Gray 0 0 0 1 1 1 in order to expand to 2 bit, all we need to do is, flip (or mirror) all existing gray code, and prefix 0 to the top half, prefix 1 to the bottom half. so first flip/mirror: Decimal Binary Gray 0 00 0 1 01 1 2 10 1 3 11 0 then p...

What happens when you have many levels of sub-directory of CPU cgroup? what's the expect behavior for each process inside or outside those sub-directory?  I hope this post helps. TL;DR When Linux scheduler processes, each cgroup with running processes will be considered as a single entity the same other processes at the same level.  Scheduler will assign CPU shares to each entity based on its weight (priority). Each group will then distribute CPU share to all processes and cgroups under it, follow the same rule Assumptions:  1. We have N running (running, not all) processes/threads all need lots of CPU (if there's on competition, then each process will get the CPU they need) 2. All processes have default priority/nice value 3. We only have 1 CPU (to make calculation simple) Case 1:   No sub-directory/sub-group (this is same as without CPU cgroup)  each process will get 1/N CPU time, so if N is 10, then each one will get 10% CPU Cas...

Graphite will align timestamp based on the retention defined in storage-schemas.conf,  whenever you send a metric to Graphite, Graphite will align timestamp like this: timestamp  - (timestamp % interval) so for example, let's say you have 1 minute interval (time per point), or  60  seconds for one of your metrics, then you send: my.useful.metric 2022 1414516239 Graphite will save (timestamp aligned): my.useful.metric 2022 1414516200 because 1414516239 - 1414516239 % 60 = 1414516200 and if you send multiple values with same timestamp, or you send multiple values then all of them  are within defined interval ----- that is: (timestamp  - (timestamp % interval)) has same value. the last one wins . Graphite will only save last value ( with exceptions, see detail in last part of this post ). So back to previous example, if you send in below order (doesn't matter when you actually send them, with  exceptions ): my.useful...

It's difficult to find out who's using the security group if you looked at wrong place. Here's how to find out dependent objects: 1. Search group name or group ID at "Network Interfaces" under EC2 Management Console. This will give you all direct usage from instances including EC2, RDS, RedShift .... 2. find out dependencies between/among security groups by using this small program:   https://github.com/mingbowan/sgdeps you will get something like: python sgdeps.py --region us-east-1 mingbotest-A sg-b4566ad1 (mingbotest-A) |-- sg-9b566afe (mingbotest-C2) |-- sg-9f566afa (mingbotest-C1) |  |-- sg-69576b0c (mingbotest-D2) |  |  `-- sg-b4566ad1 (mingbotest-A) ** loop |  `-- sg-64576b01 (mingbotest-D1) |     `-- sg-9b566afe (mingbotest-C2) |-- sg-8b566aee (mingbotest-B2) `-- sg-86566ae3 (mingbotest-B1) then delete dependent rules/groups.

the solution I got runs more than 1 second, which is terrible. Please help if you have better algorithm """ Project Euler Problem #14 ========================== The following iterative sequence is defined for the set of positive integers: n->n/2 (n is even) n->3n + 1 (n is odd) Using the rule above and starting with 13, we generate the following sequence:                   13->40->20->10->5->16->8->4->2->1 It can be seen that this sequence (starting at 13 and finishing at 1) contains 10 terms. Although it has not been proved yet (Collatz Problem), it is thought that all starting numbers finish at 1. Which starting number, under one million, produces the longest chain? NOTE: Once the chain starts the terms are allowed to go above one million. """ limit=1000000 uplimit  = limit * 3 answer = 1 maxval = 0 data = [0] * uplimit data[1] = 1 for in...

""" Project Euler Problem #12 ========================== The sequence of triangle numbers is generated by adding the natural numbers. So the 7^th triangle number would be 1 + 2 + 3 + 4 + 5 + 6 + 7 = 28. The first ten terms would be:                  1, 3, 6, 10, 15, 21, 28, 36, 45, 55, ... Let us list the factors of the first seven triangle numbers:    1: 1    3: 1,3    6: 1,2,3,6   10: 1,2,5,10   15: 1,3,5,15   21: 1,3,7,21   28: 1,2,4,7,14,28 We can see that 28 is the first triangle number to have over five divisors. What is the value of the first triangle number to have over five hundred divisors? """ limit = 100000 data=[0] * limit for i in xrange(1,int(limit/2+1)):     data[i::i] = [ x+1 for x in data[i::i]] def find():     s = 1     while True:         if s % 2 == 0:           ...

basically, when any process need to do I/O and underlying library calls io_schedule() will put CPU into iowait status and then if the CPU is idle (no other active process running), iowait time increases. below is related code snip from 3.x kernel: // in kernel/sched/cputime.c                                                                                                                                                                   /*  * Account for idle time.  * @cputime: the cpu time spent in idle wait  */ void account_idle_time(cputime_t cputime) {   ...

I searched online for answers, but none of them are good enough for me --- most of them says it's equal to running queue, but some observations on an I/O busy NFS server disagree with that. So I checked source code. TLDR;  load average can be considered as a sum of total running processes (the run queue size) and interruptible process over specific time period. This applies to at least version 2.6.x and 3.x (I only checked source code of those two versions) BTW, the uptime command is only read and output the data in /proc/loadavg for load average. (A little bit) Long but official explanation : copied from Linux kernel source code: of v3.14-rc6 /* * Global load-average calculations * * We take a distributed and async approach to calculating the global load-avg * in order to minimize overhead. * * The global load average is an exponentially decaying average of nr_running + * nr_uninterruptible. * * Once every LOAD_FREQ: * * nr_active = 0; * f...

I recently applied a NFS tag from my bank and attached it to my Android phone (Nexus 4). Then I found every time I power on (not boot up) the phone, there will be weird sound. I tried to disabled all the possible setting under sound section in settings, but the sound still there. Then I realized it may caused by the NFS tag, so I went to settings -> Wireless & networks -> more , and unchecked the "NFC" option, and the weird/annoying sound gone.

you may get URL like: for iPhone/iPad: com.rsa.securid.iphone://ctf?ctfData=nnnnnnnn for Andriod: http://127.0.0.1/securid/ctf?ctfData=nnnnnnnn you can convert between them, simply keep the ctf?ctfData=nnnnnnnn part and replace prefix, then import.

Putty is one of the best and must-have freeware for people working on Linux/Unix but use Windows as client like me.  We need to manage many servers and sometimes we are hoping we can run/execute same command on multiple host at same time, or just input same thing to multiple host. I searched online for a tool can do this. And it looks like PuTTYCS (PuTTY Command Sender) is the only one existing. But I’m a little bit disappointing after tried the software, it’s good but not good enough. It can only send command to each window one by one, and you have to wait until last window got input. So I think I should do something, and puttyCluster was born ( https://github.com/mingbowan/puttyCluster ) interface is simple: When you input Windows title pattern in the text box, you will be prompt for how many windows matching the pattern, like this: and you click the edit box under “cluster input”, what ever key you pressed will pass to all those windows simultaneously, even “Ctrl-C”, “Esc” ...

Problem Graphite doesn’t support special characters like “ “ (empty space), “/” slash etc. Because it expect everything to be just ASCII to split/processing them, and then make directories based on metric name. For example:   Metric:     datacenter1.server1.app1.metric1.abc Will create datacenter1/server1/app1/metric1/abc.wsp But Metric: datacentter1.this is a test/with/path.app.test will fail when create directory So any special name not allow to appear in directory/file name is not supported by Graphite.   What we can do?   We can urlEncode the metric name which has special characters. So like “/var/opt” (not valid file name) will become “%2Fvar%2Fopt”(now valid), using urlEncode instead of others (like BASE64) is because this will keep most of data readable.   So what to change? 1. urlEncode metric name before send to Graphite (if you always sending metrics using text/line mode instead of pickle/batch mode, then you may consider modify ...

What is EndPointSnitch :    Cassandra is a distribute database, and data can be on any of the node or nodes (depends on replication factor).  But one single request from one client can hit only one Cassandra node at a time, so Cassandra needs to locate where is the data (on local node or on remote), then proxy the request to the node, wait for result and then return result to client. EndPointSnitch is used here to determine the sorted list of node Cassandra internally should proxy request to (best node is the first one in list). Coming with Cassandra, there’re 5 EndPointSnitch implementations.   EndPointSnitch also needs to provide “datacenter” and “rack” info for ReplicationStrategy to determine where to put replicas SimpleSnitch:   First get host list from ReplicationStrategy (defined for each keyspace, see below), exclude dead node and then return the node list. This snitch will always return "datacenter1" as datacenter and "rack1" as rac...

here's what will happen when node repair is request (nodetool repair) definition:    neighbors, the node(s) has replica of the data other node(s) has.  For example, you have a 7 node cluster, and token is evenly distributed (that is, each node holds 1/7 range). Assume you set replication factor to 3. That means any data you write should have 3 replicas, under default strategy (SimpleStrategy)  , the next two replicas will be put on the next node along the ring. So if the data you write is on node 4, then node 5 and 6 each will also have one replica. And node 4 should also hold one replica for node 3 and one replica for node 2.  Then the neighbors for node 4 will be node 2,3,4,5 and 6.  Using same logic, node 6’s neighbors will be 4,5,6,7 and 1 (remember it’s a ring, when you reaches end, the next will be the first node) you can do the same calculation for other node or for other replication factor. Steps: for each keyspace in Cassandra DB do below: ...

Concept: hector connection pool contains other connection pools. The "other connection pool" here is the connection pool to each Cassandra host. Let's call the master hector pool as hector pool and other pools as host pool. So, for example, your cluster has 3 hosts, then the hector pool will contain 3 host connection pools, each host pool has several connections to one of the 3 hosts. Details: come with hector client library, there are 3 load balancing policies (will call them LBP). 1. RoundRobinBalancingPolicy         this LBP always return next host pool in hector pool. So if you have 3 hosts as A, B and C. Then when you request new connection pool, this LBP will give you A, B, C, A, B, C, A …. 2. LeastActiveBalancingPolicy         This LBP keeps track of how many active connections within each host pool. And when you request a new one, it will return then host pool with least active connections . If multiple host...

My users decided to move from CVS to SVN, and they want to use centralized LDAP server for user authentication, I need to setup a test server for prove of concept test. I found one no longer used server, and installed SVN packages as well as mod_dav_svn. The Apache HTTP server is already there. After I configured Apache HTTP follow the document at http://svnbook.red-bean.com/nightly/en/svn.serverconfig.httpd.html#svn.serverconfig.httpd.prereqs , I tried to use subversion client to connect to it but failed. I checked http server log and found the request was sent to the wrong place: # more error_log ...... [Wed May xx 16:07:36 2012] [error] [client 192.168.128.10] File does not exist: /var/www/html/repos [Wed May xx 16:07:36 2012] [error] [client 192.168.128.10] File does not exist: /var/www/html/favicon.ico [Wed May xx 16:07:49 2012] [error] [client 192.168.128.10] File does not exist: /var/www/html/repos [Wed May xx 16:07:49 2012] [error] [client 192.168.128.10] File d...

gimp 2.8 is out, I downloaded Windows version and installed. But when I try to run it, below error pops-up (the procedure entry point gzdirect could not be located in the dynamic link library zlib1.dll):   if I click on OK, the program can continue, but I want to fix it since I don’t want to miss anything, so I fired up Process explore and when the error pops-up, I located the zlib1.dll using Process Explorer’s DDL view   Here’s the output double click on zlib1.dll, and got below info   I did a file search under GIMP’s directory, there are 2 of them (one for 64-bit and one for 32-bit), and the version is different with the one in Windows directory:   seems the dll file within system directory was loaded instead of the one come with Gimp. I located the plugin’s directory by double click on the name (script-fu.exe) in Process Explorer And then copied zlib1.dll from GIMP’s bin directory (in my case, its C:\Program Files\GIMP 2\bin ) into the plug-ins di...

Symptom: When trying to do “edit table” or “query data” (as below) in pl/sql developer or try to run any SQL statement, the program will hang. Although schema browse works and connection to database server can be made withouth issue. Trouble shooting: I updated Oracle client to 11.2.0.3, upgrade pl/sql developer to the lasted version 9.0.6, no help.  I fired up TCPView (from www.sysinternals.com ) and found there were 2 connections to server: While on server side, we only got one: I did a few more test, and turned out, sometimes, the first connection you made to the database may disappear on server side, when that happens, plsqldeveloper will hang.   So I traced server side Oracle process using “strace” as below (the number after –p is process ID as in above picture 32641/oracle….) strace –p 32641 and in the trace file, I got: write(20, "\2\274\0\0\6\0\0\0\0\0ssed\25\1\0\1\7 EHCC Query H"..., 700) = -1 ECONNRESET (Connection reset by pee...

I installed and configured 389 directory server (aka redhat directory server), created user/group and configured follow the instruction at http://directory.fedoraproject.org/wiki/Howto:PAM on my Redhat 5.x Linux server I tried to reset user password, and then login with ssh, but no matter if the password is reset or not, there's no prompt for changing the expired password. I tried using command "login testUser", it will prompt me for new password. I checked /etc/pam.d/sshd, found out it missed one entry: auth       include      system-auth and system-auth will include ldap related settings: auth        sufficient    pam_ldap.so use_first_pass then the password change prompt pop out after login with ssh: WARNING: Your password has expired. You must change your password now and login again! Changing password for user testUser. Enter login(LDAP) password: So now I can for...

After install openldap on redhat, I can startup the process and access using ldapsearch but cannot access via ssl, I hit errors like below # ldapsearch -d 1 -x -D "cn=Manager,dc=example,dc=com" -w secret -H ldaps://192.168.16.88 cn ldap_create ldap_url_parse_ext(ldaps://192.168.16.88) ldap_bind ldap_simple_bind ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP 192.168.16.88:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying 192.168.16.88:636 ldap_connect_timeout: fd: 3 tm: -1 async: 0 TLS trace: SSL_connect:before/connect initialization TLS trace: SSL_connect:SSLv2/v3 write client hello A TLS: can't connect. ldap_perror ldap_bind: Can't contact LDAP server (-1) did a test using openssl to verify if ssl works:  # openssl s_client -connect 192.168.16.88:636 CONNECTED(00000003) 10484:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:...