Character sets and character encodings tutorial

Change we can believe in

I've finally found the words to articulate how I feel right now.

I feel like that great air-breathing fish, Admiral Ackbar; as he leans back and sighs with relief and awe, the other fish-men in the rebel corsair cheer as the disabled super star destroyer crashes into the death star.

Why PHP won

I've run out of patience with PHP, but I've always recognized the coexistence of its strengths and its weaknesses that narrower minds refused to acknowledge. Some CTO has a long post essentially arguing this.

I first learned PHP in January 2000. I remember the date exactly. This was only shortly after 4.0 had just come out. I had some acquaintances who were using PHP 3 but I had never tried it.

For awhile I loved PHP. Being used to the fixed-memory limitations of the data structures of C/C++ and mysterious segfaults I often couldn't understand, I found PHP to be incredibly liberating.

stupid mysql tricks

what do you know, screencasting works!

The new way to do show processlist

We all know and love the show processlist command to see what our queries are doing.

However, most of the show commands are being deprecated in favor of new information_schema virtual tables. (I believe the technical term is actually "system view".)

The new information_schema equivalent of show processlist is:

select * from information_schema.processlist \G

*************************** 1. row ***************************
     ID: 259584
   USER: readonly
   HOST: 192.168.0.186:44761
     DB: bln
COMMAND: Query
   TIME: 222
  STATE: Writing to net
   INFO: SELECT /*!40001 SQL_NO_CACHE */ * FROM `person5`
*************************** 2. row ***************************
     ID: 259468
   USER: readonly
   HOST: 192.168.0.108:43492
     DB: bln
COMMAND: Query
   TIME: 0
  STATE: executing
   INFO: select * from information_schema.processlist

The ID column appears to be the connection_id of the thread that is running the query.

The advantage of this method is that we can filter the results with a where clause, like so:

select * from information_schema.processlist where command <> 'Sleep' and id <> connection_id();

which will show only those queries that are not in sleep mode and excludes the current process.

Mysql sounds like operator

MySQL has an operator SOUNDS LIKE .

a sounds like b

is equivalent to

soundex(a) == soundex(b)

Cute, though not something I would use in production code.

Put last word in quotes

Really useful tidbit from the readline manual:

Put the following in your .inputrc:

$if Bash
# quote the current or previous word
"\C-g": "\eb\"\ef\""
$endif

Now cntrl-g will surround the previous or current argument with quote characters.

Fix up screen good

Awhile ago, I set up screen on another machine.

I was setting up my screen config for a new machine, and I had completely forgotten how I had fixed it up.

Here's what I did:

• don't bother with aliasing 'screen' to 'TERM=screen screen'
• to fix backspace-delete, instead put 'bindkey -k kD stuff ^H' in .screenrc (note ^H should be entered as cntrl-V-cntrl-H to get a literal)
• to get color shell prompts in bash:
• there should be a line in .bashrc that looks like case "$TERM" in xterm-color) color_prompt=yes;; esac
• add screen) color_prompt=yes;; into the body of the case statement
• to get window titles added correctly, for bash:
• add this to .screenrc: shelltitle '$ |bash'
• add this to .bashrc: PS1=$PS1'\[\033k\033\\\]'
• to get window titles added correctly for zsh, add the following to .zshrc:
• preexec () { echo -ne "\ek${1%% *}\e\\"}
• precmd () { echo -ne "\ekzsh\e\\" }
• put 'shell zsh' in .screenrc to start with zsh automatically
• startup_message off to disable the startup message
• a status line config that I like: '%{kw}[%{kw}%?%-Lw%?%{=b wR}(%{k}%n*%f %t%?(%u)%?%{=b wR})%{= wk}%?%+Lw%?%?%= %{= }]%{g}[%l]'

I found some of these example configs helpful, especially for deciphering the caption/hardstatus lines:

• http://aperiodic.net/phil/configs/screenrc
  
• http://ubuntuforums.org/showthread.php?t=356470
  
• http://chad.glendenin.com/screen/
  
• http://michael-prokop.at/computer/config/.screenrc
  
• http://aperiodic.net/screen/string_escapes

make python understand ~ for home directories

Python does not automatically expand ~ for user home directories. There is a function in os.path that does that for you:

mypath = os.path.expanduser('~/somepath')

mypath == '/home/nick/somepath'

any and all

Python has two builtin functions for evaluating iterables for truth/falseness: any() and all(). any() returns true if any member of the iterable evaluates to True, while all returns True only if every single member of the iterable evaluates to True.

>>> any([True, False, False])
True
>>> any([False, 0])
False
>>> all([True, 0])
False
>>> all([1])
True
>>> all([1, 1, True])
True

There are identically named subquery operators in sql, however they are only really useful for correlated subqueries which are poorly optimized and generally to be avoided in the current versions of mysql.

Commands that can see themselves

You can see the contents of the current command line in /proc/self/cmdline . Of course this is only supported by systems that support /proc, i.e. not a BSD-based system like OS X.

cat /proc/self/cmdline  --> cat /proc/self/cmdline

I'm not sure what this is useful for.

The different permutations of grep

bzegrep (1) - search possibly bzip2 compressed files for a regular expression
bzfgrep (1) - search possibly bzip2 compressed files for a regular expression
bzgrep (1) - search possibly bzip2 compressed files for a regular expression
egrep (1) - print lines matching a pattern
fgrep (1) - print lines matching a pattern
grep (1) - print lines matching a pattern
pgrep (1) - look up or signal processes based on name and other attributes
rgrep (1) - print lines matching a pattern
zegrep (1) - search possibly compressed files for a regular expression
zfgrep (1) - search possibly compressed files for a regular expression
zgrep (1) - search possibly compressed files for a regular expression

According to passing the string 'grep' to the always useful 'apropos' command

commandline fun tr and seq

Uppercase on the command line:

tr '[:lower:]' '[:upper:]'

Strip all newlines:

tr -d '\n'

Strip all non-printing characters:

tr -dc '[:print:]'

print from 1 to 100:

seq 100

print from 2 to 50, incrementing by 3:

seq 2 3 50

print from 1 to 10, all on the same line:

seq 10 | xargs echo

Parse any date easily

Use dateutil.parser.parse, possibly with the fuzzy flag set to True.

>>> from dateutil.parser import parse

>>> parse("Jan. 1st, 1991", fuzzy=True)

datetime.datetime(1991, 1, 1, 0, 0)

This is a huge boon over the tedious and underfeatured time.strptime and similar. Unhelpfully, the documentation is non-existent.

How to swap two tables without downtime

You can swap two tables in the same schema without messing up any readers by combining rename table operations into a single statement.

rename my_table to my_table_old, my_table_new to my_table;

I haven't checked this but the statement should use a name lock so that attempts to access either table by other clients will be blocked until the statements complete.

See how python compiles some code

use the dis module to get the bytecode of any python callable.

Example:

import dis
dis.dis(lambda: x is True)



  1           0 LOAD_GLOBAL              0 (var)
              3 LOAD_GLOBAL              1 (True)
              6 COMPARE_OP               8 (is)
              9 RETURN_VALUE        

This is another way to get the efficiency of different parts of code

See how python encodes a string internally

Fun fact: you can see how python encodes a string internally by asking it to encode a unicode object to the 'unicode_internal' codec.

Example:


>>> u'\N{SNOWMAN}'.encode('utf16')
'\xfe\xff&\x03'

>>> u'\N{SNOWMAN}'.encode('unicode_internal')

'&\x03'

As you can see, it's UTF16 without a BOM.

Fun stuff with unicode in python

Fun fact: you can interpolate a unicode character into a unicode string literal by using \N{name of character}

Example:

>>> print u'this sign is \N{VULGAR FRACTION ONE HALF} off'.encode('utf8')
this sign is ½ off

You can tell the name of any given character by using the unicodedata module

>>> import unicodedata
>>> unicodedata.name(u'\xbd')
'VULGAR FRACTION ONE HALF'

The name of the famous snowman character is simply enough 'SNOWMAN.

>>> unicodedata.name(unichr(9731))
'SNOWMAN'

So you can represent this character in python simply by u'\N{SNOWMAN}'

Use vim's multiple paste buffers

Ever find yourself avoiding cutting something in vim because you don't want to lose the contents of your paste buffer?

Fear not! Vim has been silently saving away the last 16 things you've cut into multiple paste buffers, which you could have been accessing all this time had you only heard the first thing of it.

Type :reg to see the contents of your paste buffers. To paste something that you cut 3 cuts ago, just use "3p

Hash images

Now, this is what I call cool: findimagedupes.

It's a image hashing program, that outputs a "perceptual hash" of image files. Unlike a cryptological hash, a small change in an image should result in a small change, so you can use this to tell how similar two slightly different images are.

You can also use it to recursively search directories for duplicate images.

It's in apt for debian and ubuntu.