Como saber qual é o sistema operacional utilizando java

Para saber qual o sistema operacional basta
colocar a linha:
System.out.println(System.getProperty("os.name"));

Para saber todas as propriedades de um:
System.out.println(System.getProperties());

O Resultado do System.out.println(System.getProperties()); aqui foi:
{
java.runtime.name=Java(TM) SE Runtime Environment,
sun.boot.library.path=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64,
java.vm.version=20.1-b02,
java.vm.vendor=Sun Microsystems Inc.,
java.vendor.url=http://java.sun.com/,
path.separator=:,
java.vm.name=Java HotSpot(TM) 64-Bit Server VM,
file.encoding.pkg=sun.io,
sun.java.launcher=SUN_STANDARD,
user.country=BR,
sun.os.patch.level=unknown,
java.vm.specification.name=Java Virtual Machine Specification,
user.dir=/media/00B0DD30B0DD2CC2/Documents and Settings/dm/Meus documentos/REC/trabalhos/tb1rec/t,
java.runtime.version=1.6.0_26-b03,
java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment,
java.endorsed.dirs=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/endorsed,
os.arch=amd64,
java.io.tmpdir=/tmp, line.separator=
,
java.vm.specification.vendor=Sun Microsystems Inc.,


os.name=Linux,


sun.jnu.encoding=UTF-8,
java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib,
java.specification.name=Java Platform API Specification,
java.class.version=50.0,
sun.management.compiler=HotSpot 64-Bit Tiered Compilers,
os.version=2.6.32-40-generic,
user.home=/home/dm,
user.timezone=,
java.awt.printerjob=sun.print.PSPrinterJob,
file.encoding=UTF-8,
java.specification.version=1.6,
java.class.path=.,
user.name=dm,
java.vm.specification.version=1.0,
sun.java.command=TCPServer,
java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre,
sun.arch.data.model=64,
user.language=pt,
java.specification.vendor=Sun Microsystems Inc.,
java.vm.info=mixed mode,
java.version=1.6.0_26,
java.ext.dirs=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ext:/usr/java/packages/lib/ext,
sun.boot.class.path=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/charsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes,
java.vendor=Sun Microsystems Inc.,
file.separator=/,
java.vendor.url.bug=http://java.sun.com/cgi-bin/bugreport.cgi,
sun.io.unicode.encoding=UnicodeLittle,
sun.cpu.endian=little,
sun.desktop=gnome,
sun.cpu.isalist=}

Mas digamos que você deseja saber é a versão da JVM (Java Virtual Machine - Máquina Virtual Java)
Basta colocar a linha:
System.out.println(System.getProperty("java.version"));
em algum lugar do seu programa