Video 7-2: §7.2 Character Encodings

0:00 <Untitled Chapter 1>

0:03 in the first part we'll look at character frequencies and how to exploit the fact that e is much more frequent in natural language than some other letters and and other scripts will have similar properties and before as a prerequisite to that we'll talk a bit about character encodings

0:22 so in the simplest form an encoding is a function that Maps our input alphabet The Source alphabet to well somehow the coded alphabet where we can't just do letter by letter because the source alphabet could be say the English Latin alphabet with uh 26 letters or maybe lowercase uppercase many more uh and the coded alphabet often we want in this case uh binary so we need to do one character to a whole string of characters in The General case

0:58 then we call what we get if we apply this encoding function to a letter we call that the code word for that letter okay so think about that as bits

1:11 then we can have different codes the simplest one are fixed length every character in The Source alphabet is mapped to the same number of bits in the or the same number of characters in the coded alphabet but that's not necessarily needed we can also have encodings that give different lengths

1:29 Fixed-length codes food-length codes are the simplest type of character encodings

1:32 a simple well-known fixed length code is the ASCII character um the ASCII coding period encoding which is listed here so you see there's not all all special letters no emoji and all that it's from 1963. I it actually specifies seven bit for each character it's often used as eight bits where the extra bit is either not used at all or used for extended ASCII where there was this this terrible time where you always had to tell your text editor oh this file is so and so encoded and then Windows had their own and Mac had their own and everyone had their own um not very convenient and luckily no longer such a big deal because most people use something else but uh it's a fixed length code every character has exactly the same number of bits

2:30 Fixed-length codes - Discussion

2:33 and that's that's all I want to say about these there's they're boring so they're as fast and as easy as it gets but they do almost nothing in terms of compression and they're very inflexible if you if you invent a new Emoji you out of luck so let's briefly touch on the other alternative if we have variable length code words and that's also um an old idea I guess people know Morse code what you may not realize is that it's not just one Morse code again people have different Coatings the idea of having to tell the other party what code you use much predates computers here's the international Morse code um that's how Wikipedia shows it at least and that that shows something that in the uh ASCII table was was there but not very interesting um you often need different representations are convenient for encoding and for decoding so here for encoding you just have an alphabetic list of all the characters and it tells you the code word this pattern of of Dot and dash long and short you know um for decoding that's quite inconvenient if you have to if I tell you uh dot dot dot what character is that you have to look at it's SOS right so uh starting from that so you kind of know

3:58 it's s but finding that in this table is computationally annoying that's a technical term yeah I don't know much more convenient is a try representation of the same thing where you can say okay this is Dash or dot I went dot dot dot here's the s okay

4:20 I don't want to discuss Morse code in too much detail it's just um nice historic example I want to point out one thing how many characters are there in the coded alphabet well the morse code and the question has two two purposes one is hammer down these uh the the notation for alphabets there's also a second part so let me go back to that representation

4:26 Clicker Question

5:37 okay nobody voted for zero correct for for one one letter I guess the one letter alphabet you can pretty much ignore it's uh not much you can do with that

6:09 it's quite a variety of answers I'm looking forward to this okay three more all right there we go we have answers for most numbers right now I think the correct answer here is quite clear if it was only two so there's that there's Dot and dash obviously uh these two but then uh that wouldn't be enough to unambiguously send something you also have the the gap so in in Morse code you send letter by letter through the sequence of of dots and dashes but then uh

7:09 you have to know when one character ends and because we talked about try is last last week's uh you should be familiar with reading twice what I did here was I put the letters that these correspond to in the internal nodes that's a clear signal that there's something dangerous going on if I just see one dot how do I know whether that's an e or the beginning of an A or one of these others so I really need this third character that tells me stop right here okay so Morse code is a bit weird um in the computer world but it was not a problem in in the acoustic version now what do files most of the time use text files these days Unicode encoding or specifically utf-8 character encoding and I don't know how many of you wondered how that works here's how it works I guess that's the last bit we can do today uh so let me go over this very quickly uh the eight stands for normally use eight bits but there's way more characters than 256 so that cannot possibly be the only thing you can do so what happens is there's characters that are encoded with a single byte these are the Askew characters but then for others you need two three or four

7:41 Variable-length codes - UTF-8

8:29 bytes and there's a special symbol a special start of the byte that tells you which version you have and so characters have variable lengths some take one byte some take up to four now that's that's all fine but think about what that means for jumping to the so-and-so character in the text you can no longer do that because you don't know how many characters you skipped if they have variable length and so [Music] um most most things that deal with Unicode they Define code points or bytes as the unit of things to jump over instead of actual characters it's quite tricky if you think about it uh to work with these with these uh types of strings but that's what the world has has settled for because we care about the space it's all about reducing the space where you need utf-8 most of the time