The primary reference is 時代別国語大辞典：上代編 by 三省堂. It contains only words that may be found in the Old Japanese lexicon; later words and meanings are not recorded here. This is fundamental for any researcher of OJ. It is expensive, but used copies may be found for much less. 甲乙 sounds are marked with a bar to the left or right of the appropriate kana.
The 13-volume 日本国語大辞典 also lists 甲乙 distinctions, but it contains words (and meanings) from all periods of Japanese so will be of only limited use for OJ studies. This is more expensive and takes substantial space to store.
The more recent 4-volume 古語大鑑 is excellent with extensive quotations and notes, but only the first volume has so far been released. Keep an eye out for this in the near future. It focuses on vocabulary up to the 14th century.
If you only have a causal, non-professional interest, then 岩波古語辞典 is your best bet. It is quite a bit more comprehensive than other single volume dictionaries. Also to note is that one of the editors is the eminent Ōno Susumu. OJ words are listed in an ad-hoc romanization using diaeresis to mark one of 甲乙 pairs. If I remember correctly, it also uses f for ハ-sounds. I do not really care for this spelling, but it is easy enough to re-interpret as necessary. Do read the preface because it has some quarks, such as listing verbs in the 連用形 rather 終止形 in other dictionaries. (There is a good reason for this, though.)
That said, knowing if a syllable is a 甲 or 乙 is more a matter of spelling. Going from there to the "original pronunciation" is a more complicated matter. Like all historic or dead languages, no one knows what OJ sounded like. The best that can be done now is to reconstruct it based on available evidence. That study has advanced a significantly over the years to the point that we are quite confident about many issues. But do not forget that it is still a reconstruction, subject to change. If you desire more information on the subject, the most up to date and comprehensive analysis may be found in Old Japanese: A Phonetic Reconstruction by Marc Miyake.