High entropy ceramics are an emerging class of material which, like their metal alloy counterparts, are defined by containing multiple species (typically four or more) in roughly equi-molar proportions on a crystal lattice. These materials are unique within the broader classification of high entropy alloys in that they can have multiple sublattices. High entropy carbides, nitrides and oxides, for example, are in a rock salt structure containing an fcc sublattice of C, N or O atoms, respectively, with a second fcc sublattice containing a random population of cations.  After a brief introduction to these materials, this talk will focus on our first principles and molecular modeling studies of their structure, bonding, thermal-mechanical and opto-electronic properties.  For high entropy oxides, the talk will include how charge compensation leads to their ability to accommodate a variety of aliovalent elements with different ionic radii, how charge transfer is manifest in phonon thermal transport, and how a unique Jahn-Teller distortion arising from the random placement of Cu on the cation sublattice can influence their optical adsorption and lattice constants. For the high entropy carbides, our calculations suggest that a number of properties such as structure, binding energy and bulk modulus can be well approximated by averages of the binary constituent structures, and that their vacancy formation energies are bounded by those of their constituent structures. Predicted properties of high-entropy di-borides will also be discussed.