Wednesday, February 27, 2013

Kafka on Windows for people with zero idea about it

So I'm new to Scala and SBT and I just want to get Apache Kafka working on my Windows machine as well, how did I go about it? well it was fairly simple once I started using my eyes.

Step 1 - Grab the latest version of Apache Kafka from:
http://kafka.apache.org/downloads.html
However if you are not in the mood to rewrite all .sh files to .bat just grab a zip which includes the bat file from:
https://github.com/apache/kafka/tree/trunk/bin/windows
(I did this because I just wanted to get go go going)

Step 2 - Unzip and follow the Kafka Quickstart guide (http://kafka.apache.org/quickstart.html)
> cd kafka-<VERSION>
> ./sbt update
> ./sbt package

Step 3 - Do not start the server yet, the dependencies are not your classpath, so no point in starting the server at all instead I just added the dependencies to a dir where it just happened to be picking jars from
add the following jars if not already present to kafka-<VERSION>/core/lib dir:

jline-0.9.94
kafka-0.7.2
log4j-1.2.15
metrics-annotation-3.0.0-c0c8be71
metrics-core-3.0.0-c0c8be71
netty-3.2.2.Final
scala-library
slf4j-api-1.6.1
slf4j-log4j12-1.6.1
zkclient-20120522
zookeeper-3.4.5

obviously some of the jars were already there, I kept on adding the ones I needed by hit and trial

Step 4 - Start the zookeeper server as per the kafka quickstart guide
Step 5 - Before starting the kafka server take a look at the /conf/server.properties file, it aught to have "brokerid" property defined in it, it isn't present add it with value 0 from now.

Step 5 - Start Kafka server, and you are good to go.

Note: Wrote this in case someone wanted get started with Kafka quickly on a Windows machine, obviously its a bit of a hack but then again it was meant to be.



















Posted by



at



























Labels:
,


















13 comments:








  1. pavan927June 28, 2013 at 1:53 AM
    i am try to install kafka 0.8 ..i follow's ur document but no success plz guide me
    ReplyDelete
  2. pavan927June 28, 2013 at 1:55 AM
    ./sbt package giving warnings

    bin/zookeeper-server-start.sh config/zookeeper.properties

    evamtec@evam /usr/local/kafka-0.8
    $ bin/zookeeper-server-start.sh config/zookeeper.properties
    java.lang.NoClassDefFoundError: org/apache/zookeeper/server/quorum/QuorumPeerMain
    Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.server.quorum.QuorumPeerMain
            at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
            at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    Could not find the main class: org.apache.zookeeper.server.quorum.QuorumPeerMain.  Program will exit.
    Exception in thread "main" Please build the project using sbt. Documentation is available at http://kafka.apache.org/
    ReplyDelete
    Replies
    1. MeJuly 9, 2013 at 5:49 AM
      Hi, did you add the stated libraries to the /core/lib folder? if not let me see if I can upload my working server and you can try using that.
      Delete
    2. MeJuly 9, 2013 at 8:04 AM
      Also try running just sbt in your kafka root directory, there are some dependencies that it needs to resolve.
      Delete
    3. JorgeJuly 14, 2013 at 6:48 PM
      If you downloaded the scripts from http://kafka.apache.org, then you must change a little bit the .bat files. In particular, the problems are in the script "kafka-run-class.bat":

      1) In line 24, you must change this:
      set BASE_DIR=%CD%\..

      with this:
      set BASE_DIR=%CD%\..\..

      2) At the bottom of the file you must change this:
      set CLASSPATH=%CLASSPATH%;"%1"

      with this:
      set CLASSPATH=%CLASSPATH%;"%*"

      3) In lines 51, 55 and 59, the "for" sentences must be changed from this (line 51):

      for %%i in (%BASE_DIR%\core\target\scala-2.8.0\*.jar) do (

      to this:

      for %%i in ("%BASE_DIR%\core\target\scala-2.8.0\*.jar") do (

      You must do this same last change for lines 55 and 59 as well.

      Hope this can help you.
      Delete
    4. MeAugust 7, 2013 at 2:16 AM
      thanks, i think that was the problem
      Delete
  3. UnknownJuly 23, 2013 at 12:38 PM
    I ran all three commands in order to make it run in windows. Thank you for getting me strarted.

    sbt update
    sbt package
    sbt assembly-package-dependency
    ReplyDelete
  4. UnknownJuly 23, 2013 at 1:03 PM
    Even then after making the changes in bat file for kafka-08.8 beta version..getting the following exception
    java.io.IOException: Cannot run program "javac": CreateProcess error=2, The system cannot find the file specified

    I have JAVA_HOME as C:\Program Files (x86)\Java\jre6
    ReplyDelete
    Replies
    1. MeAugust 7, 2013 at 2:17 AM
      hope this is solved now but try setting JAVA_HOME to the JDK path rather than jre
      Delete
  5. UnknownJuly 23, 2013 at 1:05 PM
    When i ran sbt package getting the below exception..I was pretty sure this is related to Path variable only but tried all options.
    ReplyDelete
  6. UnknownJuly 23, 2013 at 1:35 PM
    Got it...
    ReplyDelete
  7. AparajithFebruary 13, 2014 at 9:01 AM
    I have the proper Kafka package downloaded and did all steps as specified here. In fact I performed steps described to edit the kafka-run-class.bat file as well. Am still getting the error:
    C:\Cambria\kafka_2.8.0-0.8.0>bin\zookeeper-server-start.bat config\zookeeper.properties
    C:\Cambria\kafka_2.8.0-0.8.0\..\..
    Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

    Any suggestions?
    ReplyDelete
    Replies
    1. BartMay 2, 2014 at 3:19 PM
      I still have the similar problem. Besides, I need to resolve the proxy problem. I have spent a whole day on it. 

      did you try to remove the comment out ./gradlew copyDependantLibs
      what's your files in:

      $base_dir/perf/build/libs//kafka-perf_${SCALA_VERSION}*.jar;

      the build doesn't generate any jar in perf for me. 

      Thanks. 
      Delete




























        

      









Subscribe to:
Post Comments (Atom)