The running key cipher is a variation of the Vigenère cipher. Each letter of the plaintext is shifted along some number of alphabet positions in one specified direction.
Like other polyalphabetic ciphers, the running key cipher had been quite widely used until the first part of the twentieth century when effective attacks to this kind of ciphers were discovered.
Encrypting using running key is about shifting plaintext letters along some numbers of alphabet positions. The numbers are determined by letters of a secret keyword (like in other substitution ciphers). To search for proper letters during encrypting and decrypting, one can use tabula recta, as during using for example the Trithemius cipher or the Vigenère cipher, both based on the same idea.
Instead of determining a secret keyword and them using it repeatedly during encrypting all messages, the running key cipher uses long, publicly available sets of letters - books and other similar long texts. Parties should agree which book exactly (and exactly which edition) they will use during the communication. The must determine the number of the first page used for encryption, the first row and the number of letter in the row.
All letters of the message are encrypted using subsequent letters found in the book. After encrypting some characters, one may jump to another, arbitrarily selected position in the book and continue taking key letters from new positions. It is possible to encode a number of a new page, a number of a new row and a number of the first letter in the row as subsequent letters. The letters can be appended to the plaintext and both can be encrypted together. The second party, after finding the letters and decoding them, jumps to the new position of the secret key letters. One may also provide information about changing the book using during encryption.
The running key cipher distinguishes from other polyalphabetic substitution ciphers. Instead of a relatively short keyword, used over and over again, is uses a secret key of the same length as plaintext size. If ciphertext characters were completely random, the cipher would provide perfect security as the OTP cipher. However in this case both plaintext and secret key consist of existing words and sentences, which makes ciphertext analysis much easier.
The intruder can try to guess parts of plaintext and match them in such a way, that receiving secret keys characters will create meaningful sequences, that make up words and sentences.
In order to increase cipher's security, the parties can take ciphering letters not from one sequence but from some different sequences (in different parts of the text) at the same time. The attacker would have to guess rules used for changing the sequences. In this case, the analysis is much more difficult because secret key letters don't create correct words.
Another idea to make cryptanalysis more difficult is about assigning a few words to each alphabet letter and using those words instead of keyword letters. The method is intended to make difficult distinction ciphertext letters from plaintext letters. Usually ciphertext doesn't consist of words, unlike plaintext and secret key sequences.
Effective and popular methods for improving the cipher and creating better secret key characters are about to using texts which contain unusual expressions (it was often used for example by KGB) or avoiding the use of tabula recta and replacing it by random combinations.