From DMG -> ISO type in terminal (changing imafile.dmg to your filename)

hdiutil convert /path/imagefile.dmg -format UDTO -o /path/convertedimage.iso

From ISO -> DMG type in terminal (changing imafile.dmg to your filename)

hdiutil convert /path/imagefile.iso -format UDRW -o /path/convertedimage.dmg