binaries and text

every file on your computer is a sequence of bytes. a byte is just a number from 0 to 255. the file itself does not come with a tiny label saying “i am text” or “i am an image”. software has to decide what those numbers mean.

that is the whole trick behind text and binary files:

• a text file interprets its bytes as characters using an encoding such as utf-8
• a binary file interprets its bytes according to some other format, such as png, mp3, zip, or a program’s own data structure

both are made of bytes. “text” and “binary” describe how we read those bytes.

the same bytes can mean different things

put the word hello in a file and inspect it:

$ printf hello > hello.txt
$ xxd hello.txt
00000000: 6865 6c6c 6f                             hello

the left side shows the bytes in hexadecimal. 68 represents h in utf-8 (and ascii), 65 represents e, and so on. a text editor knows that mapping, so it shows hello instead of five numbers.

now take the bytes at the beginning of a png:

89 50 4e 47 0d 0a 1a 0a

those bytes are a file signature. a png reader recognizes them and starts decoding image dimensions, colors, pixels, and metadata. a text editor tries to turn the same bytes into characters and mostly produces nonsense. the bytes did not change; the interpretation did.

what makes text readable

an encoding is an agreement between bytes and characters. ascii covers basic english characters. utf-8 covers those same characters plus writing systems, symbols, and emoji from across unicode.

for example, A is one byte in utf-8:

41

the character é takes two:

c3 a9

this is why “one character equals one byte” is a bug waiting to happen. the string café has four characters but five bytes in utf-8. many emoji use four bytes, and some visible symbols are made from several unicode characters.

an encoding mismatch is what gives you text like café. the file may be perfectly intact; it was simply decoded using the wrong rules.

utf-8 is the sensible default for new text files. when opening text in code, specify it instead of depending on whatever default the operating system happens to use:

with open("notes.txt", "r", encoding="utf-8") as file:
    notes = file.read()       # str: decoded characters

with open("photo.png", "rb") as file:
    photo = file.read()       # bytes: the original byte values

in python, text mode decodes bytes into a str. binary mode returns bytes unchanged. writing works in reverse: text mode encodes a string, while binary mode expects bytes.

formats are more useful than extensions

extensions are hints, not proof. renaming photo.png to homework.txt does not turn its pixels into prose. many formats identify themselves using a signature near the beginning of the file:

• png starts with 89 50 4e 47 0d 0a 1a 0a
• jpeg usually starts with ff d8 ff
• pdf starts with %PDF-
• zip commonly starts with 50 4b 03 04, which looks like PK in ascii

some familiar files are containers for other files. a .docx, for example, is a zip archive containing xml, images, and metadata. it has text inside it, but the document as a whole must be handled as a zip-based binary format. this is why the text/binary distinction is useful, but not a perfect taxonomy.

on unix-like systems, a few commands make unknown files less mysterious:

file mystery.dat             # make an educated guess about the format
xxd mystery.dat | head       # inspect the first few bytes
strings mystery.dat | head   # find readable runs of text inside it

file checks the contents instead of trusting the name. xxd gives you an exact view of the bytes. strings is handy when a binary contains error messages, paths, or metadata, but its output is only a clue—not a safe representation of the whole file.

how binary files get corrupted

opening a binary file in a text editor does not normally hurt it. saving it can.

a text editor may try to decode the bytes, replace sequences it considers invalid, change the character encoding, or convert line endings. once it writes those changes back, the original byte sequence is gone and the program expecting a png, zip, or executable may reject it.

line endings are a smaller version of the same problem. unix text files normally end lines with the byte 0a (LF). windows traditionally uses 0d 0a (CRLF). text-aware tools may translate between them. that is helpful for prose and source code, but disastrous if those bytes are part of a binary structure.

when copying or hashing binary data, work with bytes all the way through:

from pathlib import Path

data = Path("source.png").read_bytes()
Path("copy.png").write_bytes(data)

if the copy is supposed to be exact, compare hashes:

sha256sum source.png copy.png

the two hashes should match. on macos, use shasum -a 256 if sha256sum is not installed.

binary data inside text systems

sometimes binary data has to travel through something that only accepts text, such as json, an email body, or an environment variable. base64 solves this by representing arbitrary bytes with ordinary text characters.

that does not make the data human-readable, compressed, or encrypted. it is just a reversible transport encoding, and it makes the data roughly one-third larger.

base64 < tiny.png > tiny.png.b64
base64 --decode < tiny.png.b64 > restored.png   # common on linux
base64 -D < tiny.png.b64 > restored.png         # macos

if a protocol already supports raw bytes—an http response body or a multipart file upload, for example—base64 is often unnecessary overhead.

choosing a format

use text when people should be able to inspect, edit, diff, or repair the data with ordinary tools. source code, configuration, logs, csv, json, and markdown benefit from that transparency.

use a binary format when compact size, exact types, fast parsing, compression, or media-specific structure matters. images, audio, video, archives, databases, and compiled programs usually belong here.

neither one is automatically better. json is easy to debug but can be verbose and cannot directly represent raw bytes. a binary serialization format can be smaller and preserve types precisely, but it needs compatible tooling. choose based on who must read the data and what the system needs to do with it.

the short version

files are bytes. formats give those bytes structure. encodings turn some of them into text.

when a file behaves strangely, do not just stare at its extension. ask four questions:

1. what format are these bytes supposed to follow?
2. if it is text, which character encoding does it use?
3. is any tool silently converting the encoding or line endings?
4. do i need characters here, or do i need the original bytes?

that mental model is enough to explain most file corruption, mojibake, failed uploads, and “works on my machine” encoding bugs. everything else is just learning the rules of the particular format in front of you.