Real mode addressing doesn't access memory as a linear, contiguous block - it's segmented, so a memory location is accessed by two parts: a segment and the offset within it.
The links I gave above, and others, explain how it works - check out the segment registers and how they're used to access memory.
If you're referring to an offset into the vector table to grab/plant specific destination addresses then yes, it's *4 because each address in the table is 4 bytes long (seg:ofs).
But this offset is just an offset into a table - it's not the same thing as the offset in real mode memory addressing.
http://wiki.osdev.org/Interrupt_Vector_Table
_________________
Giraffe: a ruminant with a view.