tag:blogger.com,1999:blog-20125739498834478092024-03-14T14:45:12.471+01:00void-main-argsPersonal notes on anything about programming.voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.comBlogger47125tag:blogger.com,1999:blog-2012573949883447809.post-5679703593194680172012-02-27T07:05:00.000+01:002012-02-27T08:58:52.798+01:00Having Fun with Monoid in Scalaz SevenIt's been some times since my last blog on <a href="http://voidmainargs.blogspot.com/2011/11/scalaz-seven-functor-feels-like-seven.html">Scalaz Seven</a> that talks about Functor. You may have guessed that my second post on the series would be Applicative Functor. Well, sorry, I can't write in order yet, I prefer to talk about Monoid in Scalaz Seven instead. OK Let's have fun.<br />
<div>
<br /></div>
<div>
<b>Having Fun With |+|</b></div>
<div>
To start with, let's have some fun with |+| operator.</div>
<div>
<br /></div>
<div>
Let's start</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">scala> 6 |+| 7</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">res0: Int = 13</span></div>
</div>
<div>
<br />
Well, not that interesting. It's just an addition. What about<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> 6 + "9"</span><br />
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">res2: String = 69</span></div>
</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><br class="Apple-interchange-newline" /></span></div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">scala> 6 |+| "9"</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><console>:14: error: type mismatch;</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> found : java.lang.String("9")</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> required: Int</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> 6 |+| "9"</span></div>
</div>
<div>
<br />
Not bad. |+| somehow protects you from adding integer to String. OK, not bad, but, not that fun. What about this:<br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> some(6) |+| some(9)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res9: Option[Int] = Some(15)</span></div>
<div>
<br /></div>
<div>
All right, that starts to be interesting. Give me more:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> some(6) |+| some(9) |+| some(10)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res10: Option[Int] = Some(25)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> some(6) |+| some(9) |+| some(10) |+| none[Int] |+| some(6)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res11: Option[Int] = Some(31)</span></div>
<div>
<br /></div>
<div>
Not bad at all. Want something more than that. Some String ?<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> "Hello" |+| "World"</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res25: java.lang.String = HelloWorld</span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> some("Hello") |+| some("World")</span></div>
<span style="font-family: 'Courier New', Courier, monospace;"></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res26: java.lang.String = Some(HelloWorld)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
What else do you have ? List ?<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;"></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> List(2,4) |+| List(4, 5)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res28: List[Int] = List(2, 4, 4,5)</span><br />
<br />
Cool. Boolean?<br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val b = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">b: Boolean = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val c = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">c: Boolean = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> b |+| c</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><console>:24: error: value |+| is not a member of Boolean</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> b |+| c</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> ^</span><br />
<br />
Ouch. Why ? Well, because |+| can be interpreted in conjunction or disjunction (note that, actually this works in Scalaz 6, not sure if scalaz seven will make it work as in scalaz 6). To fix this, let's do the following:<br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val a = Conjunction(true)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">a: scalaz.package.@@[Boolean,scalaz.Tags.Conjunction] = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val b = Conjunction(true)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">b: scalaz.package.@@[Boolean,scalaz.Tags.Conjunction] = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val c = Conjunction(false)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">c: scalaz.package.@@[Boolean,scalaz.Tags.Conjunction] = false</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> a |+| b</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res34: scalaz.package.@@[Boolean,scalaz.Tags.Conjunction] = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> a |+| c</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res35: scalaz.package.@@[Boolean,scalaz.Tags.Conjunction] = false</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> a |+| b |+| c</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res36: scalaz.package.@@[Boolean,scalaz.Tags.Conjunction] = false</span><br />
<br />
All right. That makes sense. What about Disjunction ? Well, just do the same.<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val e = Disjunction(true)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">e: scalaz.package.@@[Boolean,scalaz.Tags.Disjunction] = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val f = Disjunction(true)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">f: scalaz.package.@@[Boolean,scalaz.Tags.Disjunction] = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val g = Disjunction(false)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">g: scalaz.package.@@[Boolean,scalaz.Tags.Disjunction] = false</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> val h = Disjunction(false)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">h: scalaz.package.@@[Boolean,scalaz.Tags.Disjunction] = false</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> e |+| f</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res37: scalaz.package.@@[Boolean,scalaz.Tags.Disjunction] = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> e |+| h</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res38: scalaz.package.@@[Boolean,scalaz.Tags.Disjunction] = true</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> g |+| h</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res39: scalaz.package.@@[Boolean,scalaz.Tags.Disjunction] = false</span><br />
<br />
OK. That's cool, but it starts to be boring. Doesn't it? Give me something more spectacular. What about Tuple? All right, what about Tuple?<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> (6, "Hello", List(4, 2)) |+| (7, "Hello", List(5))</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">res40: (Int, java.lang.String, List[Int]) = (13,HelloHello,List(4, 2, 5))</span><br />
<br />
So, |+| actually "sums" each corresponding element of two tuples. That's quite cool. What if they are nested? Will it work ?<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">scala> (some(6), some("Hello"), (some(5), some(3))) |+| </span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> (some(7), none[String], (some(6), none[Int]))</span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">res43: (Option[Int], Option[java.lang.String], (Option[Int], Option[Int])) = </span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">(Some(13),Some(Hello),(Some(11), Some(3)))</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
Fantastic! Pretty cool.<br />
<br />
[ I heard somebody there "you see, this Scalaz guy loves using funny operator. This time, they use |+|. OMG, That's not readable". All right, all right. You can actually change all |+| above with its alias called mappend, hope you're happier now].<br />
<br />
<b>Monoid and Semigroup Behind the Scene</b><br />
<b><br /></b><br />
Behind the scene, what happen is that Scalaz offers a quite simple but powerful abstraction, called Monoid. Believe me, Monoid is something quite simple: it's a type class with two functions: <i>append</i> and <i>zero: </i><br />
<br />
<br />
<pre style="background-color: white;"><a href="http://www.blogger.com/blogger.g?blogID=2012573949883447809" name="l1"><span class="ln">1 </span></a><span class="s0">
<a href="http://www.blogger.com/blogger.g?blogID=2012573949883447809" name="l2"><span class="ln">2 </span></a> </span><span class="s1" style="color: navy; font-weight: bold;">trait </span><span class="s0">Monoid[A] {
<a href="http://www.blogger.com/blogger.g?blogID=2012573949883447809" name="l3"><span class="ln">3 </span></a> </span><span class="s1" style="color: navy; font-weight: bold;">val</span><span class="s0"> zero: A
<a href="http://www.blogger.com/blogger.g?blogID=2012573949883447809" name="l4"><span class="ln">4 </span></a> </span><span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">append(s1 : A, s2 : => A):A
<a href="http://www.blogger.com/blogger.g?blogID=2012573949883447809" name="l5"><span class="ln">5 </span></a> } </span></pre>
<br />
<br />
With the following contract (or a law, if you wish)<br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">zero append x = x</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">x append zero = x</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">(x append y) append z = x append (y append z)</span><br />
<br />
<br />
Scalaz provides several instances of Monoid (we have seen some of them in action above):<br />
<br />
<ul>
<li>Int, Short, BigInt, BigDecimal, Byte, Short ...</li>
<li>Boolean conjunction and disjunction</li>
<li>List and Stream</li>
<li>String</li>
<li>Either.LeftProjection and Either.RightProjection.</li>
<li>...</li>
</ul>
<div>
What about Option and Tuple ? Actually, Scalaz also provides some derived Monoid, like:</div>
<div>
<ul>
<li>Option[A] is a monoid if A is monoid </li>
<li>Tuple[A, B, C, D] is a monoid if A, B, C, and D is monoid</li>
<li>Map [A,B] is a monoid if B is a monoid</li>
<li>...</li>
</ul>
<div>
Map is an interesting example. Let's try this:</div>
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">scala> val m = Map("UO" -> BigDecimal(40.2),</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> | "US" -> BigDecimal(50.1),</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> | "YR" -> BigDecimal(10.1))</span></div>
</div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">scala> val n = Map("UO" -> BigDecimal(10.2),</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> | "US" -> BigDecimal(40.0),</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> | "YZ" -> BigDecimal(10.5))</span></div>
</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">scala> m |+| n</span></div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">res47: scala.collection.immutable.Map[java.lang.String,scala.math.BigDecimal] = Map(UO -> 50.4, US -</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">> 90.1, YZ -> 10.5, YR -> 10.1)</span></div>
</div>
<div>
<div>
<div>
<div>
<br /></div>
<div>
As you can see, |+| adds every corresponding element in m and n. If there's no corresponding element, for example "YR" in m and "YZ" in n, then it just puts the element in the result map.</div>
</div>
<div>
<div>
<br /></div>
<div>
All examples so far use append function of Monoid, but not zero. A type class with only append is called Semigroup. The function zero is useful when we want to fold a list of monoid, for example using suml function also provided by scalaz:</div>
<div>
<br /></div>
</div>
</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">scala> val xs = List(some(2,4), some(1, 3), some(2, 10))</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">xs: List[Option[(Int, Int)]] = List(Some((2,4)), Some((1,3)), Some((2,10)))</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">scala> xs.suml</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">res48: Option[(Int, Int)] = Some((5,17))</span></div>
</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<b>Conclusion</b><br />
Scalaz provides a cool thing called Semigroup and Monoid. With that, we can benefit the operator |+| and sum for their instances like Integer, String, List, and so on. But the main benefit is that Scalaz provides implementation of Monoid for Option, Tuple, and Map. You don't need to know what Monoid or Semigroup are to benefit all this. But, why should you avoid them ? It's a very simple stuff.<br />
<br />
I hope you enjoy this very express introduction to Scalaz Seven Monoid. Who said Scalaz is complex ?<br />
I'm preparing more on this subject. So, stay tuned.<br />
<br />
<br /></div>
</div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com51tag:blogger.com,1999:blog-2012573949883447809.post-55396033970038483572012-02-22T01:13:00.002+01:002012-02-22T01:14:05.122+01:00Gatling Tool at Riviera Scala ClojureThis evening, at <a href="http://www.meetup.com/riviera-scala-clojure">Riviera Scala Clojure</a>, we received Stéphane Landelle and Romain Sertelon, Gatling Tools developers.<br />
<br />
It's been quite some times I look for stress test tool that is intended for developers, instead of testers. You know, soemething like Caliper, but for stress tests.For a simple reason, I would like to profile my application and identify the problem on the load before even going to QA testing. Profiling can be done using a tool like Yourkit. But for load test, I haven't found anything convincing.<br />
<br />
Of course, I used JMeter in the past. But I really want to have something that can be coded, but stays simple. JMeter, well...., it's not that.<br />
<br />
So, Gatling caught my attention. It has already caught my attention since Stéphane mentioned it in Paris Scala User Group mailing list back in December. So, here they are, presenting the product in front of Riviera Scala Clojure group.<br />
<br />
<b>Performance, DSL, and Reporting</b><br />
One thing that makes Gatling different is its performance. The performance of Gatling is claimed to be superior than JMeter. This is because Gatling uses Akka and AsyncHttp that allow optimization of concurrent execution of scenarios. The argument really makes sense to me, and I would really love to see it in real. Check their wiki to see the performance benchmark compared to JMeter: <a href="https://github.com/excilys/gatling/wiki/Benchmarks">https://github.com/excilys/gatling/wiki/Benchmarks</a><br />
<br />
The second important point in Gatling is DSL. The DSL is intended to allow programmatic scenario to be expressed easily. DSL in Gatling, is nothing more than a fluent API, so, no need to worry, it's Scala. Check <a href="https://github.com/excilys/gatling/wiki/Benchmarks">here </a>to see the examples of the API.<br />
<br />
You may notice that the API is quite simple -- maybe too simple? I found the DSL too imperative, but maybe this is a pragmatic choice by Gatling developer. I would have preferred the DSL to be a little bit more functional with use of pattern matching, partial function, and function composition, and other functional programming concepts instead of sequencing, the use of doIf, and loop.<br />
<br />
The reporting is actually an interesting part of Gatling. Unfortunately, we were not lucky enough to have a full blown presentation on it, only an after presentation discussion. <br />
<br />
<b>Conclusion </b><br />
Gatling is an interesting tool to see if you"re interested in having a lightweight load test tool like me. Its performance benchmark is quite promising. I have had a look at the codes, played with it, and it looks quite promising. The wiki page contains quite extensive information. But, I give you a puzzle: please find me in the wiki a scala file that shows the scenario example. Go ahead, if you can find it in 30 minutes, you're better than I am :-)<br />
<br />
<br />voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com2tag:blogger.com,1999:blog-2012573949883447809.post-68064673887408041032012-01-18T13:27:00.000+01:002012-01-18T13:27:49.531+01:00Against SOPA.<span style="font-size: large;">To show my support all protest movements against SOPA, I close for today my blog. Will be back tomorrow.</span><br />
<br />voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-6759627406124998042011-12-31T16:30:00.001+01:002012-01-19T07:15:03.803+01:00On 2011 and 2012<b>The good things</b><br />
<br />
Joining <a href="http://www.amadeus.com/">Amadeus</a> is the best thing that happened to me this year. It was a little bit complicated, but I'm happy that it's finally done. The travel industry has a lot of very very interesting applications of computer science. And what's better than joining the leader in the field ?<br />
<i><br /></i><br />
<i>Needless to say, however, that the rest of the blog content will be all mines, not my employer.</i><br />
<br />
When I started the year 2011, I had used Scala for a year, mainly for my part-time master degree. 2012 is then my second year only with Scala. I introduced Scala two times this year, one for Amadeus and the other was for SII. In both occasions, the public showed strong interests in this language.<a href="http://manning.com/suereth/"> Scala in Depth </a>is the new book on Scala I read this year(well, the only one, actually). It's an interesting book, and can't wait to read the final version.<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="http://3.bp.blogspot.com/-kTqYTweKvJM/Tv87WMTQ_bI/AAAAAAAAAEw/ewycJjDt7v4/s1600/scala.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="113" src="http://3.bp.blogspot.com/-kTqYTweKvJM/Tv87WMTQ_bI/AAAAAAAAAEw/ewycJjDt7v4/s320/scala.jpg" width="320" /></a></div>
<br />
I started learning Clojure also in 2010 using Stuart Halloway, <a href="http://www.amazon.com/Programming-Clojure-Pragmatic-Programmers-Halloway/dp/1934356336/ref=sr_1_2?ie=UTF8&qid=1325349827&sr=8-2">Programming in Clojure</a>. But in 2011, reading The Joy of Clojure and Clojure in Action brought me to better understanding of this very interesting language. Clojure has also a very interesting community, many of them are very intelligent. Most of all, Clojure community do not seem to care on whether or not they become Java killer or Java replacements. In this regard, compared to Scala community, they are cooler.<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="http://1.bp.blogspot.com/-5xkAUci_UUw/Tv87k22adZI/AAAAAAAAAE8/NtwlBrJUC18/s1600/clojure.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-5xkAUci_UUw/Tv87k22adZI/AAAAAAAAAE8/NtwlBrJUC18/s1600/clojure.png" /></a></div>
<br />
What to say about Haskell? Simply the best mainstream language exist out there. Compared to 2010, I read a lot of papers on Haskell. And what superlative to be used to describe Monad Reader? Simply the best.<br />
<br />
If there is only one conference I would like to attend, I would name "Strange Loop". I read almost all slides, check almost all videos that made online. Of course, I didn't miss the best presentation of the year for me, which is <a href="http://www.infoq.com/presentations/Simple-Made-Easy">Rich Hickey's Simple Made Easy</a>. It's really my dream to be able to attend the conference some day.<br />
<br />
Participating in organization of <a href="http://www.rivieradev.fr/">RivieraDev </a>conference is another interesting moment. It's a small conference with a quality comparable to bigger European conferences. I loved the high and low, non stop thinking about the contents and the corresponding speakers were very interesting experience. Would love to do it again in 2012.<br />
<br />
This year, I also started to be interested in machine learning . I started with <a href="http://www.manning.com/owen/">Mahout in Action</a>, followed by <a href="http://www.manning.com/pharrington/">Machine Learning in Action</a>, two excellent books from Manning. I also read <a href="http://www.manning.com/alag/">Collective Intelligence in Action</a> and <a href="http://www.manning.com/marmanis/">Algorithms of Intelligent Web</a>, also from Manning. Finally, without surprise, I've also finished Standford ML-Class. That was an excellent class and surely will help me better understand the subject. If you didn't attend this class, please do for the next term.<br />
<br />
Finally, with Nicolas Bousquet and Tobo Atchou, we manage to found the long-term project of organizing user group around Scala and e. We named it Riviera Scala Clojure, following the name pattern given by Riviera Ruby and Riviera User Group. I love this group, and am optimistic on its future.<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="http://4.bp.blogspot.com/-J0gIKiK45U8/Tv89zj2OaXI/AAAAAAAAAFI/g_RKYvq0Nc0/s1600/logoriviera.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="131" src="http://4.bp.blogspot.com/-J0gIKiK45U8/Tv89zj2OaXI/AAAAAAAAAFI/g_RKYvq0Nc0/s200/logoriviera.png" width="200" /></a></div>
<br />
<br />
<b>The bad things</b> <br />
<br />
If I have to mention one bad thing this year, it would be a repetitive attack against Scala. Some of them were really unfounded. I really wonder why Scala had such difficult moments this year. Maybe this is because there are more and more developers use Scala and they expect to have the same supports as Java. Or maybe, because functional programming is indeed not an easy thing to understand. Well, I found that the Scala leaders (Typesafe, for example) have reacted quite positively.<br />
<br />
Other thing that I regret is that I didn't manage to go to either Devoxx or Fosdem, two European interesting conferences. Not to mention Scala Days 2011 that was not in Europe.<br />
<br />
I also regret the fact that I didn't manage to learn more on distributed systems like Hadoop, <a href="http://www.spark-project.org/">Spark</a>, and two interesting projects that address BSP like <a href="http://incubator.apache.org/giraph/">Apache Giraph</a> and <a href="http://incubator.apache.org/hama/">Hama</a>. I didn't really have enough time to learn those interesting things. I would say that, put aside Mahout, I didn't really add my knowledge on Hadoop (and its ecosystem) from what I have learned in 2010.<br />
<br />
Somewhere in October, I was in nostalgic mode, and started re-learn Prolog. Unfortunately, it was very quickly abandoned due to also the lack of time.<br />
<br />
<b>2012</b><br />
I would like to "make a comeback" to Hadoop and Lucene world :-). I will start my personal project around Lucene starting this year. I also hope that the project can benefit something from Spark and/or Mahout.<br />
<br />
Indeed, I think I will do a lot of things around text processing, machine learning, and all those things. Without surprise, I will register to Standford <a href="http://www.nlp-class.org/">NLP</a> and <a href="http://www.pgm-class.org/">PGM </a>classes.<br />
<br />
I also expect that in 2012, I'll learn a lot on building distributed system. That's why I will put some efforts on Akka's world, or Actor's world in general. This includes of course Erlang. Yes, I would like to seriously learn Erlang in 2012.<br />
<br />
I've learned functional programming a lot from Haskell and Clojure -- in addition to Scala. I plan to learn Erlang, and to complete the figure of functional programming, I would like to learn OCaml, one of the most important language in functional programming.<br />
<br />
That said, of course, Scala, Clojure, Haskell, and yes: Java (I still love my aging Java) will be my main concerns in 2012. <br />
<br />
If I manage to have everything in time, I will attend <a href="http://days2012.scala-lang.org/">Scala Days 2012</a>. Hopefully on or two other meetings in Scala/Clojure in Europe, and why not participating in French Scala Days organization or something similar? (If you plan to organize one and look for volunteers, you know who to contact :-)<br />
<br />
Riviera Scala Clojure will make sure that I consume my free time. I hope the group to become bigger and will give significant contribution to Scala or Clojure community in general.<br />
<br />
<b>Happy new year !</b><br />
<br />voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-45381067835200285172011-11-23T02:55:00.001+01:002012-01-19T07:15:03.692+01:00Scalaz Seven Functor Feels like Seven Samurai<span style="font-size: small;">Jason Zaugg and all his scalaz folks are working on scalaz seven at the moment. Although they have started quite a couple of times ago, I only had a chance to play with the librar</span>y only today.<br />
<br />
In case you don't know scalaz, it's actually a popular but often misunderstood library written in scala. It's not haskell standard library ported to scala, but it's highly inspired by the haskell standard library though. I recommend you to have a look here <a href="https://github.com/scalaz/scalaz">scalaz</a> . One recommendation: take your time, don't be hurry to understand, otherwise you will end up associate scalaz to banana [1], to rabbit [2], or to ejb2 [3]. Of course, scalaz is not ready for production, because you know ... you cannot persist a scalaz object using Hibernate. <br />
<br />
I start with one of the simplest scalaz type class, called Functor. It sounds scarry, right ? Don't worry, think of Functor as a way to lift a function that maps A to B to something that map "container" of A to "container" of B. If you have a function that maps, from int to int, say, a functor allows mapping from List of integer to a list of integer by mapping each element of the list.<br />
<br />
Say, you have a function incr:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">val incr:Int=>Int= x => x + 1</span><br />
<br />
a Functor[List] allows you to map every element in a list using that function like this:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">
val xs = List(5, 6, 7)</div>
<div style="font-family: "Courier New",Courier,monospace;">
Functor[List].map(xs, incr) // 6, 7, 8</div>
<br />
Well, yeah, is that all? No ! I can also have this:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">
val eitX:Either[String, Int] = Right(6)</div>
<div style="font-family: "Courier New",Courier,monospace;">
val eitY:Either[String, Int] = Left("Err") </div>
<div style="font-family: "Courier New",Courier,monospace;">
Functor[Either].map(eitX, incr) // Right(7)</div>
<div style="font-family: "Courier New",Courier,monospace;">
Functor[Either].map(eitY, incr) // Left("Err")</div>
<br />
<br />
It starts to be interesting right, because you don't have map for Either in Scala standard library. Fine, is it great ? No, it sucks. Function[Either] or Function[List] suck. We don't want it. Fortunately, scalaz comes wi th some magics that simplify the thing<br />
<br />
(wait a minute? Scalaz ? Simplifies something? You must be kidding ?) <br />
<br />
With the super complex highly intelligent Scalaz (as it is perceived by many), indeed life is simpler, we can directly write:<br />
<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">
val eitX:Either[String, Int] = Right(6)</div>
<div style="font-family: "Courier New",Courier,monospace;">
val eitY:Either[String, Int] = Left("Err") </div>
eitX.map(incr) // Right(7)<br />
eitY.map(incr) // Left("Err")<br />
<br />
That works, because Either, List, Option are instances of Functor (you know, the container that allows you to lift a function).<br />
<br />
Is that all ? No. There are something even more interesting. Imagine if we have a List of Option, is it a functor ? Oh yes, it turns out that the composition of functors is a functor. Let's check:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">
val listCompOpt = Functor[List].compose(Functor[Option])</div>
<div style="font-family: "Courier New",Courier,monospace;">
val ys = List(Some(1), Some(3), None, Some(6))<br />
val zs = listCompOpt.map(ys)(incr) // List(Some(2), Some(4), None, Some(7))</div>
<br />
Yeah! What about the product ?<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">
val listOptProdF = Functor[List].product(Functor[Option])</div>
<div style="font-family: "Courier New",Courier,monospace;">
val ts = (List(1,2),Some(1))</div>
<div style="font-family: "Courier New",Courier,monospace;">
val us = (List(3,7, 2, 1),None)</div>
<div style="font-family: "Courier New",Courier,monospace;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace;">
listOptProdF.map(ts)(incr) // (List(2, 3),Some(2))</div>
<span style="font-family: "Courier New",Courier,monospace;">listOptProdF.map(us)(incr) // (List(4, 8, 3, 2),None)</span><br />
<br />
Oh, Cool.<br />
<br />
OK, to finish this post, scalaz-seven (at least the version I was playing with) provides a couple of interesting functions like strengthL, strengthR, and fpair.<br />
<br />
Here they are:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">val xEit:Either[String, Int] = Right(3)</span><br />
<div style="font-family: "Courier New",Courier,monospace;">
val yEit:Either[String, Int] = Right(4)</div>
<div style="font-family: "Courier New",Courier,monospace;">
println(xEit.strengthL("Help")) // Right("Help",4)</div>
<div style="font-family: "Courier New",Courier,monospace;">
println(yEit.strengthL("Help")) // Right(("Help",3))</div>
<div style="font-family: "Courier New",Courier,monospace;">
val xOpt:Option[Int] = Some(4)</div>
<div style="font-family: "Courier New",Courier,monospace;">
val yOpt:Option[Int] = None</div>
<div style="font-family: "Courier New",Courier,monospace;">
println(xOpt.strengthR("my godness")) // Some(4, "my godness") println(yOpt.strengthR("My godness")) // None<br />
<br />
println(xOpt.fpair) // Some(4,4)<br />
println(xEit.fpair) // Right(3,3)</div>
<div style="font-family: "Courier New",Courier,monospace;">
<br /></div>
I hope to be able to come with other posts in this series. No promise, but stay tuned.<br />
<br />
----------------<br />
[1] Banana in this article context is not a scala or a java library, it is a fruit name. In case you're not familiar with it, check this <a href="http://en.wikipedia.org/wiki/Banana">article</a>. You may wonder how could one wrongly associate scalaz with banana. Well, that may happen, who knows. <br />
<br />
[2] Rabbit in this article context is not a scala or a java library, not even a cool javascript library, it is a name of an animal. Again, <a href="http://en.wikipedia.org/wiki/Rabbit">wikipedia</a> is helpful in case you're not familiar with it. Well, again, it may happen that you associate scalaz to rabbit, who knows.<br />
<br />
[3] <a href="http://www.oracle.com/technetwork/java/javaee/ejb/index.html">EJB2</a> is a server side component, usually managed by the application server. It is used mainly in enterprise application, and by the way the 'E' is Enterprise. You should not wonder why you might associate scalaz to EJB2, it <a href="http://blog.joda.org/2011/11/scala-feels-like-ejb-2-and-other.html">happened</a> !voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com7tag:blogger.com,1999:blog-2012573949883447809.post-17305727807891284582011-10-21T22:01:00.001+02:002012-01-19T07:15:03.644+01:00RivieraDev Day 2 Wrap Up - With Epilog TooAnother interesting day today. There were less participants than yesterday, although the talks are actually at least as interesting as yesterday.<br />
<br />
<b>Hibernate OGM by Emmanuel Bernard</b><br />
We might wonder (well, I'm still wondering even after the talk) why we need JPA for NoSQL. Even worse, a JPQL for NoSQL, quite contradictory isn't it? <br />
<br />
That was actually Emmanuel Bernard's presentation I attended this morning (in French). At the beginning, the goal of OGM is to use JPA for Infinispan. But, the Hibernate team found out that the JPA could be generalized to NoSQL in general. That's why they start to implement Hibernate OGM to No SQL databases.<br />
<br />
Emmanuel explained four types of NoSQL databases: key value, big table (e.g. HBase), graph database (Neo4J), or document based (MongoDB) one. At the moment, OGM supports Neo4J and MongoDB that are apparently keen to participate in OGM development.<br />
<br />
All in all, I find the presentation not that interesting and not that convincing. A good presentation to start the morning though.<br />
<br />
<b>Scala by Fredrik Ekholdt</b><br />
Fredrik explained a lot of things on Scala in this talk. Too many, in my opinion. For some, that might be interesting, because we can see a lot of things in once, but for some others that might be too hard to follow, hence not interesting.<br />
<br />
The presentation started with quick introduction to Scala, before going into detail to trait and loan pattern, higher order function, duck typing, functional programming and data structure, parallel collection, implicit, and pimp my library. All in 50 minutes presentation !<br />
<br />
I couldn't really judge the content, since nothing really new in his presentation today, after all it's a Scala introduction talk. I think it should have been better if Fredrik focused on some important features and not to go to many things.<br />
<br />
<b>Play! by Nicolas Leroux and Nicolas Martignoles (A little GQuery)</b><br />
Well, I'm sorry not to be able to say a lot for this presentation. I had to leave the session doing some buffet lunch preparation after the first 20 minutes.<br />
<br />
But, from the part that I saw, I really think that Play! is a good idea and helpful. I would love to see the framework more detail. Even better, Play! 2.0 core will become Scala, what can be better than that ?<br />
<br />
Oh, yes. I could also spend some 5 minutes or so in Manolo's presentation on GQuery. It looks like to be an interesting thing to see. Also, at the end of the talk, he announced the release of GQuery 1.1 today ! Thanks to choose RivieraDev to make the announce.<br />
<br />
<b>Ceylon by Stephane Epardaud and Cucumber by Frederick Ros </b><br />
To be honest, the presentation on three JVM languages are not that impressive: Scala, Kotlin, and Ceylon. The presenters went too detail into the detail of languages, but less in philosophical choices (if any). But, that was not the reason I left the session after 30 minutes: it was planned that I would see the two presentations.<br />
<br />
Ceylon as a language is quite boring, actually. Nothing specially mind-blowing in Ceylon (Scala: well, it's mind-blowing, reified generic in Kotlin is actually interesting) I have an impression Ceylon is there to reduce a little bit the frustration ones have with Java, but still keep all the rest, especially the possibility to work with famous java frameworks (yeah, yeah: hibernate). Oh, I have to say nevertheless that Union Class is insteresting.<br />
<br />
I missed the beginning of Cucumber presentation. When I arrived in the room, Frederick was doing some demos. Then, he presented some interesting use cases of Cucumber, including the BDD, or even the use of Cucumber to monitor production (not sure to understand what this meant though). There were only around twenty people in the room, but the discussion that followed the presentation was quite excellent (that might be the benefit of small public). One question I love much was "Is there really helpful for somebody without computer science all this specification written in natural language, but still expressed at the end in programming language? " That's exactly my doubt on BDD though. I still feel the "jump to solution, without real requirements analysis" is in the BDD approach. Is it really the best way to make "life specification?" . BDD is still programmer oriented from my point of view.<br />
<br />
<b>Coffee Script by Bodil Stokke</b><br />
This talk is the best talk of the day. It actually saved the day that at the end relatively less interesting than Thursday.<br />
<br />
Bodil Stokke presented the Coffee Script excellently. She showed really the essence of the coffee script: make clean solution, a lot of use of spaces, nice syntaxes, and all those things. Nothing really mind-blowing in coffee script though, it's just a better syntax than java script. There's no, for example, the concept of isolate, that quite interesting in dart.<br />
<br />
It was not the content who made Bodil's presentation interesting though. It was the way she presented the subject. The jokes about the programming language creator were very funny. From Bodil's presentation we can see however that Coffee Script is a serious attempt to improve java script. The syntaxes are indeed very nice, like destructuring [sum, diff] = (a,b) -> [a + b, a -b].<br />
<br />
All in all, the presentation was excellent, she succeded to show how nice coffee script was. But honestly, if you stick to use Java Script in a discipline way (see eloquent java script book), I don't think we really need Coffee Script. Dart is clearly something else -- unfortunately, it does not have nice syntaxes of Coffee Script though.<br />
<br />
<br />
<b>OPA by Louis Gesbert</b><br />
For me, the mind-blowing approach of web development is incarnated by OPA. They really attack the fundamental problem of web development: heterogeneity of the system, starting from client, server, and the database. They are really heterogen, for example, the programming languages are different, the presentation of the data is different. For example relational database is yet another form of representation of data in the system that uses object oriented and java script in the client.<br />
<br />
Those heterogeneity is indeed the problem that every body seems to try to solve, starting from Dart that wants to be a language that can be used in big application as well as a client one, node js, or even GWT.<br />
<br />
So, OPA tries to provide a comprehensive solution that unifies the way to develop web application. Not surprisingly, functional programming language is used for this purpose. Indeed, with functional programming, you can define what the system is, and not how the system should be architected. For example, the automatic slicing client/server applications provided in OPA is a very cool idea.<br />
<br />
In his presentation, however, Louis was not really able to show the concrete solution that OPA tried to propose. I did not really get how the problem of heterogeneity could really be solved by OPA. Maybe I should check OPA more to know better the product.<br />
<br />
Oh, yes. It was quite ashamed to see very few attendees were in this session. Maybe it was too hard to stay in a conference on Friday evening.<br />
<br />
<b>Epilog</b><br />
Finally, I really think that RivieraDev was a good conference, with nice programs. The programs can still be improved , but the important things might be the number of attendees , especially on the second day. But, small number of participants could also be positive as I saw in Cucumber session that I found to have really interesting discussions.<br />
<br />
I'm quite disappointed with Ceylon, Scala, and Kotlin presentations. Not on the content, but how they were presented. I don't really care about the exact syntaxes or features, because in one hour, I could not do anything. In one hour presentation, one should attack more on philosophical point of view of the language, instead of syntax things.<br />
<br />
Surprisingly, all Java scripts related sessions were quite interesting. I love Dart and CoffeeScript sessions, and heard a lot of nice things on JQuery. I had also a good impression on GQuery presentation. I would have been happier however if a real Java Script supporter to defend the languages. I still believe that it is possible to write Java Script cleanly (like it is the case in Eloquent Java Script book). Maybe, next year, the supporter of "clean" and plain Java Script should be invited.<br />
<br />
See you next year.voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-28507703212295774962011-10-21T06:51:00.000+02:002012-01-19T07:15:03.663+01:00RivieraDev Day 1 Wrap UpIt was an interesting day we had on Thursday at Riviera Dev. <br />
<br />
The conference started with a keynote from Stephane Epardaud, Inria, and some sponsors. Nothing much interesting in the keynote content, except that now I know that Inria puts (even) more efforts on programming languages. This is very cool. <br />
<br />
<b>Dart, by Florian Loitsch</b><br />
Then, the talks started. At the first session, Florian Loitsch from Google presented Dart. I was sooo sceptical about Dart, but I decided to come to his session anyway. To my surprise, his presentation reduced my scepticism on Dart. Not on the infamous "optional typing" on which I still have my reserves, but on the fact that Dart is not actually that boring. The Isolate concept, interface with factory, and couple other things are very neat. Florian actually did an excellent job in his presentation to demo all these things live.<br />
<br />
I had also some interesting discussions with him at lunch time about Dart, the role of Gilad Bracha, how Dart team works, and so on. All in all, his talk was almost my favorite talk at the conference.<br />
<br />
<br />
<b>Kotlin, by Dmitry Jemerov</b><br />
All right. After Dart, another language, Kotlin by Dmitry Jemerov from JetBrain. His talk was actually interesting, but because his session must compete with NodeJS session, the number of participants are not that many, but still he had a lot of audiences.<br />
<br />
Dmitry did not present many new things on Kotlin compared to the Kotlin web site or to Strange Loop presentation on the language. One thing I like in Kotlin is actually reified generics and also on String interpolation, that quite neat. I wonder how type check costs to Kotlin performance though. One other interesting thing from his presetnation was the IDEA demo on Kotlin. It was so short unfortunately, he could have made longer and slower demo though.<br />
<br />
<b>GWT, by Nicolas de Loof</b> <br />
Nicolas de Loof was a good presenter. His presentation was fluid, and by the way, it was the only French talk I attended.<br />
<br />
In his talk, he presented GWT from very high level point of view. Why GWT, the environments around GWT, and so on and so forth. For me, the talk was too high level though, I expected something more technical or strategic, like GWT after dart (oops, I almost said: GWT after dark). But after all, it was a quite interesting session.<br />
<br />
<b>Java 7 / 8, by Simon Ritter</b><br />
This talk was supposed to be the talk of the day. And indeed, it was talk of the day in term of audience. The INRIA amphi was quite full.<br />
<br />
He started with long history about Oracle and Sun, JCP and all those things. Then, quickly re, viewed Java 7 features, and finally Java 8. There were not many things in Java 8 though ..., oops sorry, there were a lot of things new in Java 8 of course, but I heard most of them. He showed examples on "closures", which was quite nice, and he talked about the possibility to have parallel collection in Java 8 (sounds familiar, isn't it?).<br />
<br />
<b>How GitHub Uses GitHub to Build GitHub</b> <br />
The star of the day is Zach Holman. His talk (was it a rant ? ) was astonishing and inspiring. <br />
<br />
He started with agile-bashing (I love a presenter that starts his presentation by bashing agile) and then started to explain on how GitHub work, something he called working asynchronously. Then, about the crazy practices of branching of branch of branch of branch, and how he sees the things should have been done simpler. He also explained about pull requests as a communication tool, about Hubot, and all those things.<br />
<br />
His rant was finally quite depressing for those who work with a lot of processes and less real things. The idea of Zach is to put to maximum the possibility of doing real things instead of those process stuffs like meetings, complicated problem report, and all other things that finally counter-productive.<br />
<br />
<b>Speaker Dinners / Open Café</b><br />
Open Café at the end of the day was also interesting. I discussed a lot with Henri Gomez (DevOps), Nicolas Leroux (Play!), Bodil Stokke (Coffee Script), Fredrik Ekholdt (Scala, TypeSafe) and my colleague Nicolas Bousquet. We had a lot of interesting things discussed, including what differences between Norwegians and Swedish :-) (Fredrik and Bodil are Norwegians) , Scala and TypeSafe, and all other things.<br />
<br />
<b>Summary</b> <br />
That was an interesting inspiring day. I hope to have similar experience today. Hope to see another depressing presentation again :-)voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-65015978427644282302011-09-26T07:24:00.010+02:002012-01-19T07:15:03.796+01:00Semantic Tableaux in Less than 90 Lines of ScalaThis week challenge for me was to write a code that can check a propositional logic formula like<br />
(¬(p ⋀ q) ↔ (¬p ∨ ¬q)) ∧ ( ¬r ∧ q) and check if the formula is valid or satisfiable. A formula is valid when it is always true under any interpretations of all its atoms. A formula is satisfiable when there is some interpretation of its atoms that can make the proposition true.<br />
<br />
The complete code for this post is available <a href="https://gist.github.com/1239945">here</a>.<br />
<br />
<b><span class="Apple-style-span" style="font-size: large;">Semantic Tableaux</span></b><br />
For example, p ⋀ q is satisfiable, since if p and q are both true, then the formula is true. (p ∨ ¬p) ⋀ (q ∨ ¬q) is valid, because regardless the interpretation of p and q, the formula is always true.<br />
<br />
One way to implement the satisfiability and validity check is by creating truth table. But, the use of truth table is always exponential in number of atoms. There is fortunately a simple technique to check the validity and satisfiability a propositional logic formula. The technique is called Semantic Tableaux. I need to get my logic book I used back to the time of 2nd year of university (Ben-Ari, 1992 ) to remind me how it works. Basically, there are 9 rules, 5 α rules, and 4 <span class="Apple-style-span" style="font-size: 12px;">β </span>rules. <br />
<br />
The following are the 9 rules:<br />
<br />
α Rules<br />
<br />
<table border="2"><tbody>
<tr> <th>α</th> <th>α1</th> <th>α2</th> </tr>
<tr> <td>¬ ¬A</td> <td>A</td> <td></td> </tr>
<tr> <td>A1 ∧ A2</td> <td>A1</td> <td>A2</td> </tr>
<tr> <td>¬(A1 ∨ A2)</td> <td>¬A1</td> <td>¬A2</td> </tr>
<tr> <td>¬(A1 → A2)</td> <td>A1</td> <td>¬A2</td> </tr>
<tr> <td>(A1 ↔ A2)</td> <td>A1 → A2 </td> <td>A2 → A1</td> </tr>
</tbody></table><br />
β Rules<br />
<table border="2"><tbody>
<tr> <th>β</th> <th>β1</th> <th>β2</th> </tr>
<tr> <td>B1 ∨ B2</td> <td>B1</td> <td>B2</td> </tr>
<tr> <td>¬(B1 ∧ B2)</td> <td>¬B1</td> <td>¬B2</td> </tr>
<tr> <td>B1 → B2</td> <td>¬B1</td> <td>B2</td> </tr>
<tr> <td>¬(B1 ↔ B2)</td> <td>¬(B1 → B2)</td> <td>¬(B2 → B1)</td> </tr>
</tbody></table><br />
I will not explain the Semantic Tableaux algorithms in detail. You can check a more detail in (Ben-Ari ) or in (<a href="http://imps.mcmaster.ca/courses/CAS-701-02/contributions/sem-tableaux.pdf">Issawi, 92</a>). Here is an example of tableau creation for ¬(p ⋀ q) ↔ (¬p ∨ ¬q) :<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg877m-R6RpFLRKSc75VT5379Dtyv9RdfnMZo2W0x4_4N9YePYa_PCk1KqvMm78QWPX_FmdmbJy06FV7cBIThNMaikG5WqQm1yiL2Cm0dHtLMgJKnqwmlB0BmDVVt4lxHin1T6Nf8jguxFj/s1600/semantictableau.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg877m-R6RpFLRKSc75VT5379Dtyv9RdfnMZo2W0x4_4N9YePYa_PCk1KqvMm78QWPX_FmdmbJy06FV7cBIThNMaikG5WqQm1yiL2Cm0dHtLMgJKnqwmlB0BmDVVt4lxHin1T6Nf8jguxFj/s640/semantictableau.jpg" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixdUyRTTloK2WqHxYRwz6tLPW38mgG88zMuQGIBxJ9wNtmyAEQirh9igXsIbdBgoKD4DFPN-R7SoqGNvQJ31TtMhrkTu27h9ClGpTQCSRg_RjHHR_Ou79PaBZuGovhz65SH23pVvclrFL6/s1600/semantictableau.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br />
</a></div><b><span class="Apple-style-span" style="font-size: large;">Code </span></b><br />
Ok, let's start coding now. The target is to have the following application working:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0" style="color: navy; font-weight: bold;">object </span><span class="s1">Run {
</span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">main(args:Array[String]):Unit = {
</span> <span class="s0" style="color: navy; font-weight: bold;">import </span><span class="s1">Formulas._
</span> <span class="s0" style="color: navy; font-weight: bold;">val </span><span class="s1">formula1 = (¬(</span><span class="s2" style="color: green; font-weight: bold;">'r</span><span class="s1">) ∧ </span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s1">) ∧ (¬(</span><span class="s2" style="color: green; font-weight: bold;">'p </span><span class="s1">∧ </span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s1">) ↔ (¬(</span><span class="s2" style="color: green; font-weight: bold;">'p</span><span class="s1">) ∨ ¬(</span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s1">) ))
</span> println(isSatisfiable(formula1) + <span class="s2" style="color: green; font-weight: bold;">"," </span><span class="s1">+ isValid(formula1)) </span><span class="s3" style="color: grey; font-style: italic;">// true, false</span><span class="s1">
</span>
<span class="s0" style="color: navy; font-weight: bold;">val </span><span class="s1">formula2 = ¬(</span><span class="s2" style="color: green; font-weight: bold;">'p </span><span class="s1">∧ </span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s1">) ↔ (¬(</span><span class="s2" style="color: green; font-weight: bold;">'p</span><span class="s1">) ∨ ¬(</span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s1">) )
</span> println(isSatisfiable(formula2) + <span class="s2" style="color: green; font-weight: bold;">"," </span><span class="s1">+ isValid(formula2)) </span><span class="s3" style="color: grey; font-style: italic;">// true, true</span><span class="s1">
</span> }
}</pre><br />
<br />
I used a sealed class Formula to represent all formulas. Atom, Conjunction, Disjunction, Implication, Equivalence, and Xor are classes that extends the trait. In addition, I introduced class ¬ to represent the negation. Here is how it looks like.<br />
<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0" style="color: navy; font-weight: bold;">sealed abstract class </span><span class="s1">Formula
</span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">Atom(symbol:Symbol) </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">Formula
</span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">Conjunction(p:Formula, q:Formula) </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">Formula
</span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">Disjunction(p:Formula, q:Formula) </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">Formula
</span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">Implication(p:Formula, q:Formula) </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">Formula
</span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">Equivalence(p:Formula, q:Formula) </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">Formula
</span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">Xor(p:Formula,q:Formula) </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">Formula
</span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">¬(p:Formula) </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">Formula</span></pre><br />
Note that Atom class takes a symbol as its property, so that it allows us to write Atom('p) for example.<br />
<br />
I let the Formula class empty in the example above just to make the examples clear. The content of the class is actually the boolean operation to a formula, like ∧,∨, →, ↔, and ⊕. I benefit from Scala that allows special characters to be used as function name. Here is the class Formula looks like:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0" style="color: navy; font-weight: bold;">sealed abstract class </span><span class="s1">Formula {
</span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">∧(q:Formula) = Conjunction(</span><span class="s0" style="color: navy; font-weight: bold;">this</span><span class="s1">, q)
</span>
<span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">∨(q:Formula) = Disjunction(</span><span class="s0" style="color: navy; font-weight: bold;">this</span><span class="s1">,q)
</span>
<span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">→(q:Formula) = Implication(</span><span class="s0" style="color: navy; font-weight: bold;">this</span><span class="s1">,q)
</span>
<span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">↔(q:Formula) = Equivalence(</span><span class="s0" style="color: navy; font-weight: bold;">this</span><span class="s1">,q)
</span>
<span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">⊕(q:Formula) = Xor(</span><span class="s0" style="color: navy; font-weight: bold;">this</span><span class="s1">, q)
</span>}</pre><pre></pre><br />
With this, we can write<span class="Apple-style-span" style="background-color: white; font-family: monospace; white-space: pre;"><span class="s0">(Atom(</span><span class="s1" style="color: green; font-weight: bold;">'p</span><span class="s0">) → Atom(</span><span class="s1" style="color: green; font-weight: bold;">'q</span><span class="s0">)) ∨ Atom(r)</span></span>. Not bad, but better to have directly <span class="Apple-style-span" style="background-color: white; font-family: monospace; white-space: pre;"><span class="s0">(</span><span class="s1" style="color: green; font-weight: bold;">'p </span><span class="s0">→ </span><span class="s1" style="color: green; font-weight: bold;">'q</span><span class="s0">) ∨ r . </span></span> For that purpose, implicit comes to rescue:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0" style="color: navy; font-weight: bold;">implicit def </span><span class="s1">symbolToAtom(sym:Symbol) = Atom(sym)</span></pre><br />
We can now define how the validity and satisfiability work. This code illustrates the example of the application for testing our program:<br />
<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s1" style="color: navy; font-weight: bold;">object </span><span class="s0">Run {
</span> <span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">main(args:Array[String]):Unit = {
</span> <span class="s1" style="color: navy; font-weight: bold;">import </span><span class="s0">Formulas._
</span> <span class="s1" style="color: navy; font-weight: bold;">val </span><span class="s0">formula1 = (¬(</span><span class="s2" style="color: green; font-weight: bold;">'r</span><span class="s0">) ∧ </span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s0">) ∧ (¬(</span><span class="s2" style="color: green; font-weight: bold;">'p </span><span class="s0">∧ </span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s0">) ↔ (¬(</span><span class="s2" style="color: green; font-weight: bold;">'p</span><span class="s0">) ∨ ¬(</span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s0">) ))
</span> println(isSatisfiable(formula1) + <span class="s2" style="color: green; font-weight: bold;">"," </span><span class="s0">+ isValid(formula1)) </span><span class="s3" style="color: grey; font-style: italic;">// true, false</span><span class="s0">
</span>
<span class="s1" style="color: navy; font-weight: bold;">val </span><span class="s0">formula2 = ¬(</span><span class="s2" style="color: green; font-weight: bold;">'p </span><span class="s0">∧ </span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s0">) ↔ (¬(</span><span class="s2" style="color: green; font-weight: bold;">'p</span><span class="s0">) ∨ ¬(</span><span class="s2" style="color: green; font-weight: bold;">'q</span><span class="s0">) )
</span> println(isSatisfiable(formula2) + <span class="s2" style="color: green; font-weight: bold;">"," </span><span class="s0">+ isValid(formula2)) </span><span class="s3" style="color: grey; font-style: italic;">// true, true</span><span class="s0">
</span> }
}</pre><br />
<br />
<br />
Now let's see the implementation of the 9 rules (you remember 5 α rules, and 4 <span class="Apple-style-span" style="font-size: 12px;">β </span>rules ?). It's quite simple actually. An application of rule actually retuns a list of leaf, and a leaf is actually a list of formula. Here it is:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0" style="color: navy; font-weight: bold;"> type </span><span class="s1">Leaf = Set[Formula]
</span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">applyRule(f:Formula):List[Leaf] =
</span> f <span class="s0" style="color: navy; font-weight: bold;">match </span><span class="s1">{
</span> <span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">f </span><span class="s0" style="color: navy; font-weight: bold;">if </span><span class="s1">isLiteral(f) => List(Set(f))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">¬(¬(a)) => List(Set(a))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">Conjunction(a,b) => List(Set(a,b))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">¬(Disjunction(a,b)) => List(Set(¬(a), ¬(b)))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">¬(Implication(a,b)) => List(Set(a, ¬(b)))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">Disjunction(a,b) => List(Set(a), Set(b))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">¬(Conjunction(a,b)) => List(Set(¬(a)), Set(¬(b)))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">(Implication(a,b)) => List(Set(¬(a)), Set(b))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">Equivalence(a,b) => List( Set(a,b) , Set(¬(a), ¬(b)))
</span>
<span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">¬(Equivalence(a,b)) => List(Set(a,¬(b)),Set(¬(a), b))
</span> }</pre><br />
Nothing should be surprising except the <span class="Apple-style-span" style="background-color: white; font-family: monospace; white-space: pre;"><span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">f </span><span class="s0" style="color: navy; font-weight: bold;">if </span><span class="s1">isLiteral(f) => List(Set(f)). </span></span> Just disregard this line at the moment, it's only useful in semantic tableau generation. Another surprise maybe for the equivalence rules. The two equivalence rules in the code above are actually equivalent to the one in the rule table (leave as an exercise :-) ). Note how close the rules definition to its coding implementation. Isn't it nice ?<br />
<br />
The most interesting part is of course the implementation of semantic tableau generation. In this naive implementation, the semantic tableau generation is embarrassingly simple, barely 10 lines of codes.<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0"> </span><span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">semanticTableau(f:Formula):List[Leaf] = {
</span>
<span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">combine(rec:List[Leaf], f:Formula):List[Leaf] =
</span> <span class="s1" style="color: navy; font-weight: bold;">for </span><span class="s0">( a <- applyRule(f); b <- rec) </span><span class="s1" style="color: navy; font-weight: bold;">yield </span><span class="s0">(a ++ b)
</span>
<span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">openLeaf(leaf:Leaf):List[Leaf] =
</span> <span class="s1" style="color: navy; font-weight: bold;">if </span><span class="s0">(leaf forall isLiteral)
</span> List(leaf)
<span class="s1" style="color: navy; font-weight: bold;">else</span><span class="s0">
</span> leaf.foldLeft(List(Set.empty:Leaf))(combine) flatMap(openLeaf)
openLeaf(Set(f))
}</pre><br />
Although short, the code is actually very dense. As you can see, the generation of semantic tableau above uses an auxiliary method openLeaf that does exactly that: opening a leaf. Opening a leaf is a recursive function that stops when all formulas in the leaf is a literal. A literal is either an atom or a negation of an atom. The else part is much more complex. But here is the idea: for each formula in a leaf, we apply one of the 9 rules defined above. The result of the application is combined using the combine function above. This will end up with a list of Leaf. For each leaf, then we recursively calls openLeaf using flatMap. This may be unclear, but try to play with the codes, hopefully it'll be clearer.<br />
<br />
Finally, we need to implement validity and satisfiability check. A formula is f said to be valid if the semantic tableau for <span class="Apple-style-span" style="font-family: monospace; white-space: pre;">¬</span><span class="Apple-style-span" style="white-space: pre;">f is closed. A semantic tableau is closed when all its leaves are closed, and finally a leaf is closed when it contains a formula in form of p and </span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;">¬</span>p. Here is the implementation:<br />
<pre><span class="s0"> </span><span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">isClosedLeaf(f:Leaf):Boolean =
</span> <span class="s1" style="color: navy; font-weight: bold;">if </span><span class="s0">(f.isEmpty) </span><span class="s1" style="color: navy; font-weight: bold;">false</span><span class="s0">
</span> <span class="s1" style="color: navy; font-weight: bold;">else </span><span class="s0">{
</span> (f.head <span class="s1" style="color: navy; font-weight: bold;">match </span><span class="s0">{
</span> <span class="s1" style="color: navy; font-weight: bold;">case </span><span class="s0">Atom(_) => f.tail.exists( _ == ¬(f.head))
</span> <span class="s1" style="color: navy; font-weight: bold;">case </span><span class="s0">¬(Atom(a)) => f.tail.exists( _ == Atom(a))
</span> <span class="s1" style="color: navy; font-weight: bold;">case </span><span class="s0">_ => </span><span class="s1" style="color: navy; font-weight: bold;">false</span><span class="s0">
</span> }) || isClosedLeaf(f.tail)
}
<span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">isOpenLeaf(f:Leaf) = !isClosedLeaf(f)
</span>
<span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">isValid(f:Formula):Boolean = semanticTableau(¬(f)) forall isClosedLeaf
</span>
<span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">isSatisfiable(f:Formula):Boolean = semanticTableau(f) exists isOpenLeaf</span></pre><br />
We're done.<br />
<br />
<b><span class="Apple-style-span" style="font-size: large;">Summary</span></b><br />
In this post, I showed my weekend hacking to implement naive satisfiability and validity checking of a propositional formula using semantic tableau. The implementation uses intensively Scala concepts like operator overriding, implicit, and fold and flatMap. All this concepts are helpful to implement the semantic tableaux techniques in relatively short Scala codes (less than 90 lines). The use of operator helps readibility of the implementation (imagine how it looks like if I didn't use operators ∧,∨, →, ↔, and ⊕).<br />
<br />
<b><span class="Apple-style-span" style="font-size: large;">Reference</span></b><br />
<span class="Apple-style-span" style="font-size: large;"><span style="font-size: small;">Ben-Ari, M (1992) <i>Mathematical Logic for Computer Science</i>. Prentice Hall.</span></span>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com3tag:blogger.com,1999:blog-2012573949883447809.post-3015540145203329242011-09-06T00:21:00.003+02:002012-01-19T07:15:03.667+01:00Fix from FoldMy two previous posts, <a href="http://voidmainargs.blogspot.com/2011/08/fold-again-fold-left-using-fold-right.html">Fold Right from Fold Left</a> and <a href="http://voidmainargs.blogspot.com/2011/08/folding-stream-with-scala.html">Folding Stream with Scala</a> are actually my interpretation to (Hutton, 1999) paper. Now, I would like to continue with another paper (Pope, 2010 ?) that also talks about fold. Note that, Pope's work cites of Hutton's work.<br />
<br />
This post ends my fold trilogy. It has been an exciting and fun to play with. I would love to continue with scalaz Fold (Foldr, Foldl, FoldMap Foldable are interesting), but I think I have to stop having fun :-)<br />
<br />
<span class="Apple-style-span" style="background-color: #999999; font-size: large;">Yet Another dropWhile Implementation</span><br />
<br />
In <a href="http://voidmainargs.blogspot.com/2011/08/folding-stream-with-scala.html">Folding Stream with Scala</a> , I implemented dropWhile using fold using (Hutton, 1999) paper. Pope proposes two more implementations, both work very well with infinite stream.<br />
<br />
First, a reminder of fold implementation:<br />
<pre><span class="s2" style="color: navy; font-weight: bold;">def </span>foldr[A, B](combine:(A, =>B) => B, base:B)(xs:Stream[A]): B = {
<span class="s2" style="color: navy; font-weight: bold;">if </span><span class="s0">(xs.isEmpty) base
</span> <span class="s2" style="color: navy; font-weight: bold;">else </span><span class="s0">combine(xs.head, foldr(combine, base)(xs.tail))
</span> } </pre><br />
And here is the solution:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="Apple-style-span" style="background-color: white;"><span class="s0"> </span><span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">dwHo[A](pred:A=>Boolean, xs:Stream[A]):Stream[A]=>Stream[A] = {
</span> <span class="s1" style="color: navy; font-weight: bold;">val </span><span class="s0">id =(s:Stream[A])=>s
</span> <span class="s1" style="color: navy; font-weight: bold;">val </span><span class="s0">tail= (s:Stream[A])=>s.tail
</span>
<span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">combine(next:A, rec: =>Stream[A]=>Stream[A]) = {
</span> <span class="s1" style="color: navy; font-weight: bold;">if </span><span class="s0">(pred(next)) (rec compose tail)
</span> <span class="s1" style="color: navy; font-weight: bold;">else </span><span class="s0">id
</span> }
foldr(combine, id)(xs)
}</span></pre>Example:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0"> scala> </span><span class="s1" style="color: navy; font-weight: bold;">val </span><span class="s0">xs = Stream.range(</span><span class="s2" style="color: blue;">20</span><span class="s0">, </span><span class="s2" style="color: blue;">120</span><span class="s0">)
</span> xs: scala.collection.immutable.Stream[Int] = Stream(<span class="s2" style="color: blue;">20</span><span class="s0">, ?)
</span> scala> dwHo( (_:Int)<<span class="s2" style="color: blue;">100</span><span class="s0">, xs)(xs).toList
</span> res33: List[Int] = List(<span class="s2" style="color: blue;">100</span><span class="s0">, </span><span class="s2" style="color: blue;">101</span><span class="s0">, </span><span class="s2" style="color: blue;">102</span><span class="s0">, </span><span class="s2" style="color: blue;">103</span><span class="s0">, </span><span class="s2" style="color: blue;">104</span><span class="s0">, </span><span class="s2" style="color: blue;">105</span><span class="s0">, </span><span class="s2" style="color: blue;">106</span><span class="s0">, </span><span class="s2" style="color: blue;">107</span><span class="s0">, </span><span class="s2" style="color: blue;">108</span><span class="s0">, </span><span class="s2" style="color: blue;">109</span><span class="s0">, </span><span class="s2" style="color: blue;">110</span><span class="s0">, </span><span class="s2" style="color: blue;">111</span><span class="s0">, </span><span class="s2" style="color: blue;">112</span><span class="s0">, </span><span class="s2" style="color: blue;">113</span><span class="s0">, </span><span class="s2" style="color: blue;">114</span><span class="s0">, </span><span class="s2" style="color: blue;">1</span><span class="s2" style="color: blue;">15</span><span class="s0">, </span><span class="s2" style="color: blue;">116</span><span class="s0">, </span><span class="s2" style="color: blue;">117</span><span class="s0">, </span><span class="s2" style="color: blue;">118</span><span class="s0">, </span><span class="s2" style="color: blue;">119</span><span class="s0">) </span></pre>It also works with infinite Stream:<span class="Apple-style-span" style="background-color: white; font-family: monospace; white-space: pre;"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=301554014520332924" name="l37"> </a></span><br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="Apple-style-span" style="background-color: white;"><span class="s0"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=301554014520332924" name="l38">scala> </a></span><span class="s1" style="color: navy; font-weight: bold;">val </span><span class="s0">ys = Stream.from(</span><span class="s2" style="color: blue;">1</span><span class="s0">)
</span>ys: scala.collection.immutable.Stream[Int] = Stream(<span class="s2" style="color: blue;">1</span><span class="s0">, ?)
</span>scala> dwHo( (_:Int)<<span class="s2" style="color: blue;">5</span><span class="s0">, ys)(ys).take(</span><span class="s2" style="color: blue;">10</span><span class="s0">).toList
</span>res38: List[Int] = List(<span class="s2" style="color: blue;">5</span><span class="s0">, </span><span class="s2" style="color: blue;">6</span><span class="s0">, </span><span class="s2" style="color: blue;">7</span><span class="s0">, </span><span class="s2" style="color: blue;">8</span><span class="s0">, </span><span class="s2" style="color: blue;">9</span><span class="s0">, </span><span class="s2" style="color: blue;">10</span><span class="s0">, </span><span class="s2" style="color: blue;">11</span><span class="s0">, </span><span class="s2" style="color: blue;">12</span><span class="s0">, </span><span class="s2" style="color: blue;">13</span><span class="s0">, </span><span class="s2" style="color: blue;">14</span><span class="s0">)</span></span></pre>Under the hood, here is what happen:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0">foldr combine id [</span><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=301554014520332924" name="l44"><span class="s1" style="color: blue;">1.</span><span class="s0">.]
</span></a> =combine <span class="s1" style="color: blue;">1 </span><span class="s0">(foldr combine id [</span><span class="s1" style="color: blue;">2.</span><span class="s0">.]) =
</span> =(foldr combine id [<span class="s1" style="color: blue;">2.</span><span class="s0">.]) . tail
</span> =(combine <span class="s1" style="color: blue;">2 </span><span class="s0">(foldr combine id [</span><span class="s1" style="color: blue;">3.</span><span class="s0">.]) . tail
</span> =(foldr combine id [<span class="s1" style="color: blue;">3.</span><span class="s0">.]) .tail . tail
</span> =(combine <span class="s1" style="color: blue;">3 </span><span class="s0">(foldr combine id [</span><span class="s1" style="color: blue;">4.</span><span class="s0">.]) . tail . tail
</span> =(foldr combine id [<span class="s1" style="color: blue;">4.</span><span class="s0">.]) .tail . tail . tail
</span> =(combine <span class="s1" style="color: blue;">4 </span><span class="s0">(foldr combine id [</span><span class="s1" style="color: blue;">5.</span><span class="s0">.]) . tail . tail . tail
</span> =(foldr combine id [<span class="s1" style="color: blue;">5.</span><span class="s0">.]) . tail . tail . tail . tail
</span> =(combine <span class="s1" style="color: blue;">5 </span><span class="s0">(foldr combine id [</span><span class="s1" style="color: blue;">6.</span><span class="s0">.]) . tail . tail . tail
</span> =id . tail . tail . tail . tail</pre>And (id.tail.tail.tail.tail)[1..] = [5..].<br />
<br />
<span class="Apple-style-span" style="background-color: #999999; font-size: large;">Fix from Fold</span><br />
The most interesting part of the (Pope, 2010) is not on dropWhile though, but on an implementation of a function using fold. The function is called <i>fix</i>, also known as Y combinator. Check this page to have an idea of what fix function is.<br />
<br />
Basically, using fix, we can encode recursion function. The following is an example of factorial function using fix in haskell:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="Apple-style-span" style="background-color: white;"><span class="s0">Prelude> :m Control.Monad.Fix
Prelude Control.Monad.Fix> fix (\rec n -> </span><span class="s1" style="color: navy; font-weight: bold;">if </span><span class="s0">n == </span><span class="s2" style="color: blue;">0 </span><span class="s0">then </span><span class="s2" style="color: blue;">1 </span><span class="s1" style="color: navy; font-weight: bold;">else </span><span class="s0">n * rec (n-</span><span class="s2" style="color: blue;">1</span><span class="s0">)) </span><span class="s2" style="color: blue;">5</span><span class="s0"> </span></span></pre><span class="Apple-style-span" style="background-color: white; font-family: monospace; white-space: pre;"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=301554014520332924" name="l51"><span class="s2" style="color: blue;">120</span></a></span><br />
In Scala, fix function is implemented in a much trickier way. The difficulties come from the Scala strictness. Here is the implementation of fix I found after googling a little bit: <br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">fix[A](f: (A=>A)=>(A=>A)): A=>A = f(fix(f))(_)</span></pre><pre><span class="s1">
</span></pre>Example:<br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s0">scala> fix[Long](f=>x=> </span><span class="s1" style="color: navy; font-weight: bold;">if </span><span class="s0">(x == </span><span class="s2" style="color: blue;">0</span><span class="s0">) </span><span class="s2" style="color: blue;">1 </span><span class="s1" style="color: navy; font-weight: bold;">else </span><span class="s0">x * f(x - </span><span class="s2" style="color: blue;">1</span><span class="s0">))(</span><span class="s2" style="color: blue;">5</span><span class="s0">) </span></span><span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s0">res5: Long = </span><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=301554014520332924" name="l67"><span class="s2" style="color: blue;">120</span></a></pre><pre><span class="Apple-style-span" style="color: blue;">
</span></pre><br />
One question that may arise is whether fold can be implemented using fix. Well, apparently, yes, after all, it's a recursion, but I haven't tried it yet. A more interesting question would be if fix can be implemented using fold. The answer is yes, and that's the essence of (Pope, 2010) article. Here is the implementation of fix using foldr in Scala:<br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">fix[A](f: (A=>A)=>(A=>A)): A=>A = {
</span> <span class="s1" style="color: navy; font-weight: bold;">def </span><span class="s0">combine( a:(A=>A)=>(A=>A), b: =>A=>A):A=>A = f(b)(_)
</span> foldr(combine, <span class="s1" style="color: navy; font-weight: bold;">null</span><span class="s0">)(Stream.continually(</span><span class="s1" style="color: navy; font-weight: bold;">null</span><span class="s0">))
</span>}</pre><pre><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=301554014520332924" name="l73">
</a></pre><br />
Give a try:<span class="Apple-style-span" style="background-color: white; font-family: monospace; white-space: pre;"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=301554014520332924" name="l66"> </a></span><br />
<span class="Apple-style-span" style="background-color: white;"></span><br />
<pre><span class="Apple-style-span" style="background-color: white;"><span class="s0">scala> fix[Long](f=>x=> </span><span class="s1" style="color: navy; font-weight: bold;">if </span><span class="s0">(x == </span><span class="s2" style="color: blue;">0</span><span class="s0">) </span><span class="s2" style="color: blue;">1 </span><span class="s1" style="color: navy; font-weight: bold;">else </span><span class="s0">x * f(x - </span><span class="s2" style="color: blue;">1</span><span class="s0">))(</span><span class="s2" style="color: blue;">5</span><span class="s0">)
</span>res6: Long = <span class="s2" style="color: blue;">120</span></span></pre><br />
Youpi..., isn't it awesome? This shows how expressive fold is, since it can now be used to implement (any?) recursion functions.<br />
<br />
<b>References</b><br />
<span class="Apple-style-span" style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18px;"></span><br />
<div><ul style="line-height: 1.4; margin-bottom: 0.5em; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 2.5em; padding-right: 2.5em; padding-top: 0px;"><li style="margin-bottom: 0.25em; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-indent: 0px;">Hutton G (1999) A tutorial on the universality and expressiveness of fold. Available from <a href="http://www.cs.nott.ac.uk/~gmh/fold.pdf" style="color: #2288bb; text-decoration: none;">http://www.cs.nott.ac.uk/~gmh/fold.pdf</a> </li>
</ul></div><div><ul style="line-height: 1.4; margin-bottom: 0.5em; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 2.5em; padding-right: 2.5em; padding-top: 0px;"><li style="margin-bottom: 0.25em; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-indent: 0px;">Pope B (2010 ?) Getting a Fix from the Right Fold. Available from: <a href="http://www.haskell.org/wikiupload/1/14/TMR-Issue6.pdf" style="color: #2288bb; text-decoration: none;">http://www.haskell.org/wikiupload/1/14/TMR-Issue6.pdf</a> . There is one other dropWhile solution in this paper that I dumped here: <a href="https://gist.github.com/1195982">https://gist.github.com/1195982</a>.</li>
</ul></div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com1tag:blogger.com,1999:blog-2012573949883447809.post-13927082631900799572011-08-29T13:24:00.002+02:002012-01-19T07:15:03.640+01:00Fold Again: Fold Left using Fold RightIn my <a href="http://voidmainargs.blogspot.com/2011/08/folding-stream-with-scala.html">previous post</a>, I have shown how fold can be used to implement other collection methods, like filter, map, length, reverse, or even dropWhile and break. <br />
<br />
We wonder now whether it is possible to implement fold left using fold right. The article (Hutton, 1999) shows indeed that it is possible. The objective of this post is then to show it in Scala and discuss a little bit about fold universality.<br />
<br />
Before implementing fold left using fold right, let's implement something simpler. Let's implement suml, a function that is similar to sum we discussed in the <a href="http://voidmainargs.blogspot.com/2011/08/folding-stream-with-scala.html">previous post</a>. Instead of summing from right to left, we want suml to sum from left to right. <br />
<br />
The following illustrates the difference between sum and suml:<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #9966ff;">sum</span><span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">2</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span> <span style="color: red;">1</span> <span style="color: black;"><b>+</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">2</span> <span style="color: black;"><b>+</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">3</span> <span style="color: black;"><b>+</b></span> <span style="color: red;">4</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span><span style="color: #9966ff;">suml</span><span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">2</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span> <span style="color: black;"><b>(</b></span><span style="color: black;"><b>(</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span> <span style="color: black;"><b>+</b></span> <span style="color: red;">2</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>+</b></span> <span style="color: red;">3</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>+</b></span> <span style="color: red;">4</span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span>
</span></pre><br />
<br />
As described in (Hutton, 1999), it turns out that we can't directly implement suml using fold. What possible is to define suml_ that returns a function Int=> Int. Here is the implementation:<br />
<br />
<pre><span style="color: black;"><span style="background-attachment: scroll; background-color: #dbdbdb; background-image: none; background-position: 0% 0%; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">suml_</span><span style="color: black;"><b>(</b></span>xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-attachment: scroll; background-color: #dbdbdb; background-image: none; background-position: 0% 0%; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x:<span style="color: #009966;"><b>Int</b></span>, g: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>(</b></span>acc:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #9966ff;">g</span><span style="color: black;"><b>(</b></span>acc <span style="color: black;"><b>+</b></span>x<span style="color: black;"><b>)</b></span>
<span style="background-attachment: scroll; background-color: #dbdbdb; background-image: none; background-position: 0% 0%; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: black;"><b>(</b></span>x:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>x<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-attachment: scroll; background-color: #dbdbdb; background-image: none; background-position: 0% 0%; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-attachment: scroll; background-color: #dbdbdb; background-image: none; background-position: 0% 0%; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">sum</span><span style="color: black;"><b>(</b></span>xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: #9966ff;">suml_</span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">0</span><span style="color: black;"><b>)</b></span>
</span></pre><br />
When suml_(Stream(1, 3, 4, 5)) is called, it returns actually <br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">((_:Int) + 5) compose ((_:Int) + 4) </div><div style="font-family: "Courier New",Courier,monospace;">compose ((_:Int) + 3) compose ((_:Int)+ 1) </div><div style="font-family: "Courier New",Courier,monospace;">compose ( (x:Int)=>x)</div>Note that, the returned function is actually an Int=>Int function, and when it receives 0 as its input, it returns (((1 + 3) + 4) + 5) = 13. Also note a special function, the identity function (x:Int)=>x.<br />
<br />
OK, Great. What about foldl? Well, it is the generalization of suml above. Here is the foldl implementation:<br />
<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> foldl_<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><b>(</b></span>f: <span style="color: black;"><b>(</b></span>A,B<span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B, xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: A, g: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B<span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B<span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>(</b></span>acc:B<span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: #9966ff;">g</span><span style="color: black;"><b>(</b></span><span style="color: #9966ff;">f</span><span style="color: black;"><b>(</b></span>x,acc<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: black;"><b>(</b></span>a:B<span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>a<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: #006699;"><b>def</b></span> foldl<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><b>(</b></span>f:<span style="color: black;"><b>(</b></span>A,B<span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B, base:B, xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #9966ff;">foldl_</span><span style="color: black;"><b>(</b></span>f,xs<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>base<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span>
</span></pre><br />
Let's give a shot:<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #9966ff;">foldl</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>+</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span>, <span style="color: red;">0</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span>, <span style="color: red;">5</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span>res1: <span style="color: #009966;"><b>Int</b></span> <span style="color: black;"><b>=</b></span> <span style="color: red;">13</span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #9966ff;">foldl</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>*</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span>, <span style="color: red;">1</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span>, <span style="color: red;">5</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span>res2: <span style="color: #009966;"><b>Int</b></span> <span style="color: black;"><b>=</b></span> <span style="color: red;">60</span>
</span></pre><br />
Excellent. But, all this looks like a magic, right? Is there a systematic way to derive a implementation of a function using fold? Fortunately, yes. Here we come to the most interesting part of (Hutton, 1999).<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Fold Universality and Fusion Property of Fold</span><br />
<br />
Two important concepts explained in (Hutton, 1999) is fold universality and fusion property of fold.<br />
<br />
The fold universality states that the two Scala code below are equivalent:<br />
<b>Code 1</b><br />
<pre><span style="color: black;"><span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> g<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, f:<span style="color: black;"><b>(</b></span>A, B<span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B, v:B<span style="color: black;"><b>)</b></span>: B <span style="color: black;"><b>=</b></span>
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span>xs.isEmpty<span style="color: black;"><b>)</b></span> v <span style="color: #006699;"><b>else</b></span>
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #9966ff;">f</span><span style="color: black;"><b>(</b></span>xs.head, <span style="color: #9966ff;">g</span><span style="color: black;"><b>(</b></span>xs.tail, f, v<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
</span></pre><br />
<b>Code 2</b><br />
<pre><span style="color: black;"><span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> g<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><b>(</b></span>xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, f:<span style="color: black;"><b>(</b></span>A,B<span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B, v:B<span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>f,v<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span>
</span></pre><br />
Or more concise, in haskell symbols:<br />
<pre><span style="color: black;"><span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span>g[] <span style="color: black;"><b>=</b></span> v
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: black;"><b><</b></span><span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> g <span style="color: black;"><b>=</b></span> fold f v
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span>g(x:xs) <span style="color: black;"><b>=</b></span> f x (g xs)
</span></pre><br />
<br />
And the fusion property of fold states the following:<br />
<pre><span style="color: black;"><span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">1 </span></span>h w <span style="color: black;"><b>=</b></span> v
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">2 </span></span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> h . fold g w <span style="color: black;"><b>=</b></span> fold f v
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">3 </span></span>h(g x y) <span style="color: black;"><b>=</b></span> f x (h y)
</span></pre><br />
Let's have examples.<br />
<br />
First, let's see how universal property of fold is useful to derive an implementation of a function using fold.<br />
<br />
We will start with the recursive definition of foldl:<br />
<pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">1 </span></span>foldl <span style="color: #9900cc;">[]</span> f v <span style="color: black;"><strong>=</strong></span> v
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">2 </span></span>foldl <span style="color: black;"><strong>(</strong></span>x:xs<span style="color: black;"><strong>)</strong></span> f v <span style="color: black;"><strong>=</strong></span> foldl xs f <span style="color: black;"><strong>(</strong></span>f v x<span style="color: black;"><strong>)</strong></span>
</span></pre><br />
In scala:<br />
<pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">1 </span></span><span style="color: #006699;"><strong>def</strong></span> foldl<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><strong>(</strong></span>xs:<span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[A]</span>, f:<span style="color: black;"><strong>(</strong></span>B,A<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>></strong></span>B, v:B<span style="color: black;"><strong>)</strong></span>: B <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">2 </span></span> <span style="color: #006699;"><strong>if</strong></span> <span style="color: black;"><strong>(</strong></span>xs.isEmpty<span style="color: black;"><strong>)</strong></span> v
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">3 </span></span> <span style="color: #006699;"><strong>else</strong></span> <span style="color: #9966ff;">foldl</span><span style="color: black;"><strong>(</strong></span>xs.tail, f, <span style="color: #9966ff;">f</span><span style="color: black;"><strong>(</strong></span>v, xs.head<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">4 </span></span><span style="color: black;"><strong>}</strong></span>
</span></pre><br />
Unfortunately, foldl does not match directly with universal property definition. We need then an auxiliary method foldl_ :<br />
<br />
<pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">1 </span></span><span style="color: #006699;"><strong>def</strong></span> foldl_<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><strong>(</strong></span>xs:<span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><strong>)</strong></span> <span style="color: black;"><strong>=</strong></span> foldl<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><strong>(</strong></span>xs:<span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[A]</span>, <span style="color: black;"><strong>(</strong></span>_:<span style="color: black;"><strong>(</strong></span>B,A<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>></strong></span>B<span style="color: black;"><strong>)</strong></span>, <span style="color: black;"><strong>(</strong></span>_:B<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>
</span></pre><br />
That is, foldl function without the last two parameters.<br />
<br />
We're ready to use universal property now:<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">foldl_ [] = v</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">foldl_ (x:xs) = f x (foldl_ xs)</span><br />
<br />
Here, v is the identity function =id.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">{Functions}</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">foldl_ (x:xs) g a = f x (foldl_ xs) g a</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><=></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">{Definition of foldl}</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">foldl_ xs g (g a x) = f x (foldl_ xs) g a</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><=></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">{Generalizing foldl_ xs = h}</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">h g (g a x) = f x h g a</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><=></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">f = (λx h -> (λa -> h(g a x)))</span><br />
<br />
So, we got:<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">foldl xs f v = foldr((λx h -> (λa -> h(g a x)))) id xs v</span><br />
<br />
When translated to Scala, we obtain the code explained at the beginning of the post.<br />
<br />
***<br />
<br />
Now, go for fusion property. Recall our map definition defined in the previous post (I modified a little bit):<br />
<br />
<pre><span style="color: black;"><span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> map<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><b>(</b></span>f1:A<span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B<span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x:A, xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[B]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #9966ff;">f<span style="color: black;"><b>1</b></span></span><span style="color: black;"><b>(</b></span>x<span style="color: black;"><b>)</b></span> #:: xs
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: #0099ff;"><b>Stream</b></span>.empty<span style="color: black;"><b>)</b></span>
<span style="background-attachment: scroll; background-clip: initial; background-color: #dbdbdb; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: black;"><b>}</b></span>
</span></pre><span class="Apple-style-span">Note that the combine function can be represented as </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">λx xs->f1 x : xs .</span><br />
<br />
We would like to (a little bit informally) prove that <span style="font-family: 'Courier New', Courier, monospace;">map(f1) compose map(f2) </span> = <span style="font-family: 'Courier New', Courier, monospace;">map(f1 compose f2)</span>.<br />
From the equation, we can substitute:<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">h = map(f1)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">g = λx xs->f2(x):xs</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">w = v = Stream.empty = []</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">f = f1 compose f2</span><br />
<br />
But,<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">map(f1) [] = []</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">map(f1) (g(x,xs)) = map(f1) ( f2(x):xs )</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> = (f1 compose f2)(x):map(f1)(xs)</span><br />
<br />
So, the equation <span style="font-family: 'Courier New', Courier, monospace;">map(f1) compose map(f2) </span>= <span style="font-family: 'Courier New', Courier, monospace;">map(f1 compose f2)</span>indeed holds.voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-26082665548105922182011-08-15T23:48:00.005+02:002012-01-19T07:15:03.659+01:00Folding Stream with Scala<b><span class="Apple-style-span" style="font-size: large;">Introduction</span></b><br />
Couple of weeks ago, I read a well known paper on expressiveness of fold (Hutton, 1999). The paper is a very interesting paper that explains how fold is a very powerful construct. <br />
<br />
The codes in the paper are written in Haskell and therefore assumes laziness. This means that the execution of function in Haskell is delayed until it is really needed. This is not the case for Scala, where the operation is executed eagerly, or in other word, it's strict instead of lazy.<br />
<br />
Stream is the implementation of lazy list in Scala. Using Stream, the elements are only evaluated when they are needed. So, we may expect that Stream is close to Haskell's list.<br />
<br />
The use of scala Stream and fold right are the topic of this post.<br />
<br />
<b><span class="Apple-style-span" style="font-size: large;">Fold Right</span></b><br />
Let's back to fold. The paper I mentioned above (Hutton, 1999) showed how expressive fold construct, especially foldRight. The objective of this post is to show the same things as the paper, but in Scala. <br />
<br />
Let's see the problem we may encounter. The following Haskell code works:<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span>Prelude<span style="color: black;"><b>></b></span> foldr <span style="color: black;"><b>(</b></span><span style="color: black;"><b>|</b></span><span style="color: black;"><b>|</b></span><span style="color: black;"><b>)</b></span> <b>False </b><span style="color: black;"><b>(</b></span>repeat True<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span>True
</span></pre><br />
But not the following Scala code:<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">continually</span><span style="color: black;"><b>(</b></span><span style="color: #cc00cc;">true</span><span style="color: black;"><b>)</b></span>.<span style="color: #9966ff;">foldRight</span><span style="color: black;"><b>(</b></span><span style="color: #cc00cc;">false</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>_ <span style="color: black;"><b>|</b></span><span style="color: black;"><b>|</b></span> _<span style="color: black;"><b>)</b></span>
</span></pre><br />
because the code above throws StackOverflowError.<br />
<br />
The reason of the error is because Stream.foldRight is not actually lazy. When I read the article and wondered how the non-laziness of Stream.foldRight could be solved, I received Tony Morris' tweet telling that scalaz fixes the non-laziness. The following is the reimplementation of lazy foldRight for Stream inspired by Foldr scalaz implementation:<br />
<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">1 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">2 </span></span><span style="color: #006699;"><b>def</b></span> foldr<span style="color: #9900cc;">[A,</span><span style="color: #9900cc;"> </span><span style="color: #9900cc;">B]</span><span style="color: black;"><b>(</b></span> combine: <span style="color: black;"><b>(</b></span>A, <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B<span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> B, base: B <span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span>: B <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">3 </span></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span>xs.isEmpty<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">4 </span></span> base
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;">5 </span></span> <span style="color: #006699;"><b>else</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">6 </span></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>xs.head, <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, base<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs.tail<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">7 </span></span><span style="color: black;"><b>}</b></span>
</span></pre><br />
The following code works without StackOverflowError:<br />
<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: <span style="color: #009966;"><b>Boolean</b></span>, y: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Boolean</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> x <span style="color: black;"><b>|</b></span><span style="color: black;"><b>|</b></span> y
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span><span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: #cc00cc;">false</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">continually</span><span style="color: black;"><b>(</b></span><span style="color: #cc00cc;">true</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
</span></pre><br />
The key of the foldr implementation above is in thetype definition of combine parameter: (A, =>B) instead of (A,B). This makes the second parameter evaluated lazily. <br />
<br />
<b><span class="Apple-style-span" style="font-size: large;">Some Fold Right Uses</span></b><br />
The following shows some familiar functions implemented using foldr.<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">sum</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: <span style="color: #009966;"><b>Int</b></span>, y: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> x <span style="color: black;"><b>+</b></span> y
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: red;">0</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">product</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: <span style="color: #009966;"><b>Int</b></span>, y: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> x <span style="color: black;"><b>*</b></span> y
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: red;">1</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span><span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">and</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Boolean]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: <span style="color: #009966;"><b>Boolean</b></span>, y: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Boolean</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> x <span style="color: black;"><b>&</b></span><span style="color: black;"><b>&</b></span> y
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: #cc00cc;">true</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 13 </span></span><span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">or</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Boolean]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 14 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: <span style="color: #009966;"><b>Boolean</b></span>, y: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Boolean</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> x <span style="color: black;"><b>|</b></span><span style="color: black;"><b>|</b></span> y
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 15 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: #cc00cc;">false</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 16 </span></span><span style="color: black;"><b>}</b></span>
</span></pre><br />
<b><span class="Apple-style-span" style="font-size: large;">Using Fold Right to Implement Scala Collection Functions</span></b><br />
As in (Hutton, 1999), the following section shows how Fold Right can be used to implement some functions. At the first time, let's see length, filter, reverse, flatten, and map implementations.<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> map<span style="color: #9900cc;">[A,B]</span><span style="color: black;"><b>(</b></span>f:A<span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span>B, xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x:A, xs: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[B]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #9966ff;">f</span><span style="color: black;"><b>(</b></span>x<span style="color: black;"><b>)</b></span> #:: xs
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #006699;"><b>val</b></span> base:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[B]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span>.empty
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, base<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span><span style="color: #006699;"><b>def</b></span> length<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: A, len: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> len <span style="color: black;"><b>+</b></span> <span style="color: red;">1</span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, <span style="color: red;">0</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span><span style="color: #006699;"><b>def</b></span> reverse<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: A, xs: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">concat</span><span style="color: black;"><b>(</b></span>xs, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span>x<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span> <span style="color: #006699;"><b>val</b></span> base:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span>.empty
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 13 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, base<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 14 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 15 </span></span><span style="color: #006699;"><b>def</b></span> filter<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>p:A<span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Boolean</b></span>, xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 16 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: A, xs: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span><span style="color: #9966ff;">p</span><span style="color: black;"><b>(</b></span>x<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span> x #:: xs <span style="color: #006699;"><b>else</b></span> xs
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 17 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine , <span style="color: #0099ff;"><b>Stream</b></span>.empty<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 18 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 19 </span></span><span style="color: #006699;"><b>def</b></span> filterNot<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>p:A<span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Boolean</b></span>, xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 20 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: A, xs: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span><span style="color: #9966ff;">p</span><span style="color: black;"><b>(</b></span>x<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span> xs <span style="color: #006699;"><b>else</b></span> x #:: xs
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 21 </span></span> <span style="color: #006699;"><b>val</b></span> base:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span>.empty
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 22 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, base<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 23 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 24 </span></span><span style="color: #006699;"><b>def</b></span> flatten<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>xss: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Stream[A]]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 25 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, ys: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> xs #::: ys
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 26 </span></span> <span style="color: #006699;"><b>val</b></span> base: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span>.empty
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 27 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, base<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xss<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 28 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 29 </span></span><span style="color: #006699;"><b>def</b></span> partition<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>p:A<span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Boolean</b></span>, xs:<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 30 </span></span> <span style="color: black;"><b>(</b></span><span style="color: #9966ff;">filter</span><span style="color: black;"><b>(</b></span>p, xs<span style="color: black;"><b>)</b></span>, <span style="color: #9966ff;">filterNot</span><span style="color: black;"><b>(</b></span>p, xs<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 31 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 32 </span></span>
</span></pre><br />
Note that, the functions like flatten, filter, and map, work with infinite Streams. See the following examples:<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #9966ff;">map</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>+</b></span><span style="color: red;">1</span>, <span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">from</span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span>res2: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">2</span>, ?<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span>scala<span style="color: black;"><b>></b></span> res2.<span style="color: #9966ff;">take</span><span style="color: black;"><b>(</b></span><span style="color: red;">4</span><span style="color: black;"><b>)</b></span>.toList
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span>res3: <span style="color: #0099ff;"><b>List</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>List</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">2</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span>, <span style="color: red;">5</span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #9966ff;">filter</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>%</b></span> <span style="color: red;">2</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>=</b></span><span style="color: red;">0</span>, <span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">from</span><span style="color: black;"><b>(</b></span><span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>.<span style="color: #9966ff;">take</span><span style="color: black;"><b>(</b></span><span style="color: red;">5</span><span style="color: black;"><b>)</b></span>.toList
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span>res5: <span style="color: #0099ff;"><b>List</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>List</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">10</span>, <span style="color: red;">12</span>, <span style="color: red;">14</span>, <span style="color: red;">16</span>, <span style="color: red;">18</span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #9966ff;">partition</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>%</b></span> <span style="color: red;">2</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>=</b></span> <span style="color: red;">0</span>, <span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">from</span><span style="color: black;"><b>(</b></span><span style="color: red;">20</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span>res6: <span style="color: black;"><b>(</b></span>scala.collection.immutable.<span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">20</span>, ?<span style="color: black;"><b>)</b></span>,<span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">21</span>, ?<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 13 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #9966ff;">flatten</span><span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">continually</span><span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>.<span style="color: #9966ff;">take</span><span style="color: black;"><b>(</b></span><span style="color: red;">10</span><span style="color: black;"><b>)</b></span>.toList
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 14 </span></span>res13: <span style="color: #0099ff;"><b>List</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>List</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span>, <span style="color: red;">1</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span>, <span style="color: red;">1</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span>, <span style="color: red;">1</span><span style="color: black;"><b>)</b></span>
</span></pre><br />
<span class="Apple-style-span" style="font-size: large;"><b>Implementing dropWhile and break</b></span><br />
One of the most interesting part of (Hutton, 1999) is the dropWhile implementation using foldr. The implementation is shown in the following code:<br />
<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> dropWhile<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>pred: A <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: #009966;"><b>Boolean</b></span>, xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: A, xs: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span><span style="color: #9966ff;">pred</span><span style="color: black;"><b>(</b></span>x<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> <span style="color: black;"><b>(</b></span>xs._1, x #:: xs._2<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span> <span style="color: #006699;"><b>else</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: black;"><b>(</b></span>x #:: xs._2, x #:: xs._2<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: #006699;"><b>val</b></span> base:<span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span>.empty, <span style="color: #0099ff;"><b>Stream</b></span>.empty<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, base<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span> _1
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span><span style="color: black;"><b>}</b></span>
</span></pre><br />
Instead of directly producing the result, a tuple of Streams is returned. The first Stream is the result and the second one is used for bookkeeping purpose. <br />
The following illustrates an execution of dropWhile:<br />
<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #9966ff;">dropWhile</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b><</b></span><span style="color: red;">5</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>,<span style="color: red;">2</span>,<span style="color: red;">7</span>,<span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span><span style="color: red;">10</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span><span style="color: red;">9</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: red;">8</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: red;">7</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span><span style="color: red;">6</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span><span style="color: red;">5</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span><span style="color: red;">4</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span><span style="color: red;">3</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span><span style="color: red;">7</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">7</span>, <span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">7</span>,<span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span><span style="color: red;">2</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">7</span>, <span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">2</span>, <span style="color: red;">7</span>,<span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span><span style="color: red;">1</span> <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span> <span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span><span style="color: red;">7</span>, <span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span>, <span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">2</span>, <span style="color: red;">7</span>,<span style="color: red;">3</span>,<span style="color: red;">4</span>,<span style="color: red;">5</span>,<span style="color: red;">6</span>,<span style="color: red;">7</span>,<span style="color: red;">8</span>,<span style="color: red;">9</span>,<span style="color: red;">10</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
</span></pre><br />
What surprising is that the solution actually works for infinite Stream too as illustrated in the following code:<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #9966ff;">dropWhile</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b><</b></span><span style="color: red;">5</span>, <span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">from</span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>.<span style="color: #9966ff;">take</span><span style="color: black;"><b>(</b></span><span style="color: red;">10</span><span style="color: black;"><b>)</b></span>.toList
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span>res17: <span style="color: #0099ff;"><b>List</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>List</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">5</span>, <span style="color: red;">6</span>, <span style="color: red;">7</span>, <span style="color: red;">8</span>, <span style="color: red;">9</span>, <span style="color: red;">10</span>, <span style="color: red;">11</span>, <span style="color: red;">12</span>, <span style="color: red;">13</span>, <span style="color: red;">14</span><span style="color: black;"><b>)</b></span>
</span></pre><br />
The last interesting function implemented here is break. The equivalent of Haskell's break in Scala collection is called span (Update, Oct 24 thanks to oxbow). The break function creates a tuple of two lists separated at condition boundary. Example:<br />
<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span>scala<span style="color: black;"><b>></b></span> <span style="color: #006699;"><b>val</b></span> <span style="color: black;"><b>(</b></span>a,b<span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: #9966ff;">break</span><span style="color: black;"><b>(</b></span> <span style="color: black;"><b>(</b></span>_:<span style="color: #009966;"><b>Int</b></span><span style="color: black;"><b>)</b></span><span style="color: black;"><b><</b></span><span style="color: red;">5</span>, <span style="color: #0099ff;"><b>Stream</b></span>.<span style="color: #9966ff;">from</span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span>a: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, ?<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span>b: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>Stream</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">5</span>, ?<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span>scala<span style="color: black;"><b>></b></span> a.toList
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span>res22: <span style="color: #0099ff;"><b>List</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>List</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span>, <span style="color: red;">2</span>, <span style="color: red;">3</span>, <span style="color: red;">4</span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span>scala<span style="color: black;"><b>></b></span> b.<span style="color: #9966ff;">take</span><span style="color: black;"><b>(</b></span><span style="color: red;">5</span><span style="color: black;"><b>)</b></span>.toList
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span>res23: <span style="color: #0099ff;"><b>List</b></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><b>=</b></span> <span style="color: #0099ff;"><b>List</b></span><span style="color: black;"><b>(</b></span><span style="color: red;">5</span>, <span style="color: red;">6</span>, <span style="color: red;">7</span>, <span style="color: red;">8</span>, <span style="color: red;">9</span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span>
</span></pre><br />
The implementation of break here, of course, highly inspired from dropWhile one:<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><b>def</b></span> break<span style="color: #9900cc;">[A]</span><span style="color: black;"><b>(</b></span>pred: A<span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: #009966;"><b>Boolean</b></span>, xs: <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><b>def</b></span> <span style="color: #9966ff;">combine</span><span style="color: black;"><b>(</b></span>x: A, xs: <span style="color: black;"><b>=</b></span><span style="color: black;"><b>></b></span><span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span><span style="color: #9966ff;">pred</span><span style="color: black;"><b>(</b></span>x<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> <span style="color: black;"><b>(</b></span>xs._1, x #:: xs._2, x #:: xs._3<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span> <span style="color: #006699;"><b>else</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: black;"><b>(</b></span>x #:: xs._3, <span style="color: #0099ff;"><b>Stream</b></span>.empty, x #:: xs._3<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: #006699;"><b>val</b></span> base:<span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span>, <span style="color: #0099ff;"><b>Stream</b></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>=</b></span> <span style="color: black;"><b>(</b></span><span style="color: #0099ff;"><b>Stream</b></span>.empty, <span style="color: #0099ff;"><b>Stream</b></span>.empty, <span style="color: #0099ff;"><b>Stream</b></span>.empty<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span> <span style="color: #006699;"><b>val</b></span> result <span style="color: black;"><b>=</b></span> <span style="color: #9966ff;">foldr</span><span style="color: black;"><b>(</b></span>combine, base<span style="color: black;"><b>)</b></span><span style="color: black;"><b>(</b></span>xs<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: black;"><b>(</b></span>result._2, result._1<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span><span style="color: black;"><b>}</b></span>
</span></pre><br />
<b><span class="Apple-style-span" style="font-size: large;">Summary </span></b><br />
In this blog post, I showed the result of playing around with Stream and fold right. First, I rewrote the implementation of foldRight, showed the use of foldRight, expressed some collection functions using foldRight, and finally provided dropWhile and break.<br />
<br />
The blog post does not have intention to overcome the non-laziness of Scala language. Scala is not a lazy language, and I think it's not appropriate to use lazy evaluation as the main stream of the codes.<br />
<br />
Pope (Pope, 2010 ?) in Monad Reader 6 discussed further the implementation of dropWhile. He presented two further solutions based on higher order function composition. He also presented the fold implementation using what is so called fix function. See <a href="http://en.wikibooks.org/wiki/Haskell/Fix_and_recursion">http://en.wikibooks.org/wiki/Haskell/Fix_and_recursion</a> to have an idea of what Fix function is.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><b>References</b></span><span class="Apple-style-span" style="font-size: large; font-weight: bold;"> </span><br />
<div><ul><li>Hutton G (1999) A tutorial on the universality and expressiveness of fold. Available from <a href="http://www.cs.nott.ac.uk/~gmh/fold.pdf">http://www.cs.nott.ac.uk/~gmh/fold.pdf</a> </li>
</ul></div><div><ul><li>Pope B (2010 ?) Getting a Fix from the Right Fold. Available from: <a href="http://www.haskell.org/wikiupload/1/14/TMR-Issue6.pdf">http://www.haskell.org/wikiupload/1/14/TMR-Issue6.pdf</a> </li>
</ul></div><div><br />
</div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com4tag:blogger.com,1999:blog-2012573949883447809.post-36839654728460448782011-07-13T01:09:00.004+02:002012-01-19T07:15:03.675+01:00A List containing elements with consecutive types A and BJust wondering how we can create a "heterogenous" list of two types, where an object of the first type must be inserted after the second.<br />
<br />
Examples: <br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">5 :: "help" :: 8 :: "ldf"</span> to create a list with type (Int, String)<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">"help" :: 8 :: "ldf" :: 9</span> to create a list with type (String, Int)<br />
<br />
Here is what I got:<br />
<script src="https://gist.github.com/1079163.js">
</script>Wondering if I can do better.<br />
<br />
-voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-19603068551768920582011-01-24T18:56:00.010+01:002012-01-19T07:15:03.648+01:00Implementing Bird's Make Century PearlBook of Bird (2000) contains 30 pearls of functional programming. I haven't finished this very inspiring book yet, but I would love to share one of the pearl, the pearl number 6, making century.<br />
<br />
The problem that the pearl solves is given a number (1, 2, 3, 4, 5, 6, 7, 8, 9) return all expressions containing + and * that make 100. In this case, the output is:<br />
<br />
1 * 2 * 3 + 4 + 5 + 6 + 7 + 8 * 9<br />
1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 * 9<br />
1 * 2 * 3 * 4 + 5 + 6 + 7 * 8 + 9<br />
12 + 3 * 4 + 5 + 6 + 7 * 8 + 9<br />
1 + 2 * 3 + 4 + 5 + 67 + 8 + 9<br />
1 * 2 + 34 + 5 + 6 * 7 + 8 + 9<br />
12 + 34 + 5 * 6 + 7 + 8 + 9<br />
<br />
The following is the specification written using BDD library in Scala:<br />
<pre><span class="s0" style="color: navy; font-weight: bold;">class </span><span class="s1">MakeNumberSpec </span><span class="s0" style="color: navy; font-weight: bold;">extends </span><span class="s1">FlatSpec </span><span class="s0" style="color: navy; font-weight: bold;">with </span><span class="s1">ShouldMatchers {
</span><span class="s2" style="color: green; font-weight: bold;">"Make Number" </span><span class="s1">should </span><span class="s2" style="color: green; font-weight: bold;">"return all expressions that make a number" </span><span class="s1">in {
</span><span class="s1"> </span><span class="s0" style="color: navy; font-weight: bold;">val </span><span class="s1">xs = (</span><span class="s3" style="color: blue;">1 </span><span class="s1">to </span><span class="s3" style="color: blue;">9</span><span class="s1">).toList
solutions(xs, </span><span class="s3" style="color: blue;">100</span><span class="s1">).map(displayE(_)) should equal(
List(</span><span class="s2" style="color: green; font-weight: bold;">"1 * 2 * 3 + 4 + 5 + 6 + 7 + 8 * 9"</span><span class="s1">,
</span><span class="s2" style="color: green; font-weight: bold;">"1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 * 9"</span><span class="s1">,
</span><span class="s2" style="color: green; font-weight: bold;">"1 * 2 * 3 * 4 + 5 + 6 + 7 * 8 + 9"</span><span class="s1">,
</span><span class="s2" style="color: green; font-weight: bold;">"12 + 3 * 4 + 5 + 6 + 7 * 8 + 9"</span><span class="s1">,
</span><span class="s2" style="color: green; font-weight: bold;">"1 + 2 * 3 + 4 + 5 + 67 + 8 + 9"</span><span class="s1">,
</span><span class="s2" style="color: green; font-weight: bold;">"1 * 2 + 34 + 5 + 6 * 7 + 8 + 9"</span><span class="s1">,
</span><span class="s2" style="color: green; font-weight: bold;">"12 + 34 + 5 * 6 + 7 + 8 + 9"</span><span class="s1">))
}</span></pre><pre><span class="s1"></span>}</pre><br />
The complete solution described in this blog is available at my github <a href="https://github.com/anrizal06/rizier/blob/master/src/main/scala/org/rizier/functions/MakeNumber.scala">here</a>.<br />
<br />
Let's first define the expression, term, and factor. A factor is a list of digit, like List(1, 2) is interpreted as 12. Then, term is a multiplication of factors, like 4 * 15, and expression is sum of terms.<br />
<br />
It might be tempted to introduce class Expression, Term, or Factor. But I prefer to make the solution simple by using type alias instead:<br />
<pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l18"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">type </span><span class="s1">Factor = List[Int]
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l19"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">type </span><span class="s1">Term = List[Factor]
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l20"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">type </span><span class="s1">Expr = List[Term] </span></pre><br />
To display factor, term, and expression, I introduced displayF, displayT, and displayE as follow:<br />
<br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s1"></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s0" style="color: navy; font-weight: bold;">def </span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s1">displayF(f: Factor) = f.reduceLeft(</span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s3" style="color: blue;">10 </span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s1">* _ + _).toString</span></span><br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s1"></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s0"></span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">displayT(t: Term) = t.map(displayF(_)).mkString(</span><span class="s4" style="color: green; font-weight: bold;">" * "</span><span class="s1">)</span></span><br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s1"></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s0"></span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">displayE(e: Expr) = e.map(displayT(_)).mkString(</span><span class="s4" style="color: green; font-weight: bold;">" + "</span><span class="s1">) </span></span><br />
<br />
Let's test first the displayE method:<br />
<br />
<pre><span class="s1"> scala> displayE(List(List(List(</span><span class="s3" style="color: blue;">1</span><span class="s1">), List(</span><span class="s3" style="color: blue;">2</span><span class="s1">), List(</span><span class="s3" style="color: blue;">3</span><span class="s1">)),
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l51"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> List(List(</span><span class="s3" style="color: blue;">4</span><span class="s1">)), List(List(</span><span class="s3" style="color: blue;">5</span><span class="s1">)), List(List(</span><span class="s3" style="color: blue;">6</span><span class="s1">)), List(List(</span><span class="s3" style="color: blue;">7</span><span class="s1">)),
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l52"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> List(List(</span><span class="s3" style="color: blue;">8</span><span class="s1">), List(</span><span class="s3" style="color: blue;">9</span><span class="s1">)))) </span></pre><pre><span class="Apple-style-span" style="font-family: 'Times New Roman'; white-space: normal;"><pre><span class="s3" style="color: blue;"> 1 </span><span class="s1">* </span><span class="s3" style="color: blue;">2 </span><span class="s1">* </span><span class="s3" style="color: blue;">3 </span><span class="s1">+ </span><span class="s3" style="color: blue;">4 </span><span class="s1">+ </span><span class="s3" style="color: blue;">5 </span><span class="s1">+ </span><span class="s3" style="color: blue;">6 </span><span class="s1">+ </span><span class="s3" style="color: blue;">7 </span><span class="s1">+ </span><span class="s3" style="color: blue;">8 </span><span class="s1">* </span><span class="s3" style="color: blue;">9</span></pre></span></pre><br />
<br />
The book (Bird, 2010) provides an excellent theory of the solution described here. I will try to be more practical here.<br />
<br />
The problem can be attacked by exhaustive search, which means for each two consecutive number, either we add '+', '*' or simply concatenating the numbers.<br />
For example, for 8 and 9 we can have 89 by concatenation, 8 + 9 by summation, and 8 * 9 by multiplication. Given the description of the problem, there will be then 3<sup>n</sup> expression candidates to evaluate. For the problem of make century here, it is not that big, only 3<sup>8</sup> = 6561 candidates. <br />
<br />
To do the exhaustive search, we need the following function that expands an expression to a list of expressions when a number is introduced to the list of candidates so far. Three possibilities: concatenating, summing, and multiplying. The code is (with xs is the factor, xss is the term, and xsss is the rest of the expression).<br />
<br />
<pre><span class="s0" style="color: navy; font-weight: bold;"> case </span><span class="s1">(xs::xss):: xsss =>
List( ((x::xs)::xss)::xsss, // concatenating
List(x) :: xs::xss) :: xsss, // multiplying
( List(List(x)) :: (xs::xss):: xsss)) // sum </span></pre><br />
<br />
We need of course to take into account the basic case, which is inserting to an empty expression. The empty expression itself is List(List(Nil)) . The complete function becomes:<br />
<br />
<br />
<pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l28"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">glue(expr: Expr, x: Int) : List[Expr]=
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l29"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> expr </span><span class="s0" style="color: navy; font-weight: bold;">match </span><span class="s1">{
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l30"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">(Nil::Nil)::Nil => List(List(List(List(x))))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l31"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">(xs::xss):: xsss =>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l32"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> List( ((x::xs)::xss)::xsss,
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l33"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> (List(x) :: xs::xss) :: xsss,
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l34"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> ( List(List(x)) :: (xs::xss):: xsss))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l35"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">_ => Nil // or throw IllegalArgumentException ?
}</span></pre><br />
Let's do some tests:<br />
<br />
<pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l40"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> scala> glue(List(List(Nil)), </span><span class="s3" style="color: blue;">9</span><span class="s1">).map(displayE(_))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l41"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> res3: List[String] = List(</span><span class="s3" style="color: blue;">9</span><span class="s1">)
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l42"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l43"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> scala> glue( List(List(List(</span><span class="s3" style="color: blue;">9</span><span class="s1">))), </span><span class="s3" style="color: blue;">2</span><span class="s1">).map(displayE(_))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l44"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> res4: List[String] = List(</span><span class="s3" style="color: blue;">29</span><span class="s1">, </span><span class="s3" style="color: blue;">2 </span><span class="s1">* </span><span class="s3" style="color: blue;">9</span><span class="s1">, </span><span class="s3" style="color: blue;">2 </span><span class="s1">+ </span><span class="s3" style="color: blue;">9</span><span class="s1">)
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l45"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l46"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> scala> glue( List(List(List(</span><span class="s3" style="color: blue;">2</span><span class="s1">), List(</span><span class="s3" style="color: blue;">9</span><span class="s1">))), </span><span class="s3" style="color: blue;">1</span><span class="s1">).map(displayE(_))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l47"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> res5: List[String] = List(</span><span class="s3" style="color: blue;">12 </span><span class="s1">* </span><span class="s3" style="color: blue;">9</span><span class="s1">, </span><span class="s3" style="color: blue;">1 </span><span class="s1">* </span><span class="s3" style="color: blue;">2 </span><span class="s1">* </span><span class="s3" style="color: blue;">9</span><span class="s1">, </span><span class="s3" style="color: blue;">1 </span><span class="s1">+ </span><span class="s3" style="color: blue;">2 </span><span class="s1">* </span><span class="s3" style="color: blue;">9</span><span class="s1">)
</span></pre><div><span class="s1"><br />
</span></div><br />
The exhaustive solution can then be implemented by continuously applying the glue for each insertion of new number taken into account. This is done by using foldRight of flatMap functions. Then, of course we filter only the one that makes the number to be kept. The following is the code that executes the exhaustive search:<br />
<br />
<pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l75"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">valueF(f: Factor) = f.reduceLeft(</span><span class="s3" style="color: blue;">10 </span><span class="s1">* _ + _)
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l76"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">valueT(t: Term) = t.foldLeft(</span><span class="s3" style="color: blue;">1</span><span class="s1">)( _ * valueF(_))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l77"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">valueE(e: Expr) = e.foldLeft(</span><span class="s3" style="color: blue;">0</span><span class="s1">)(_ + valueT(_))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l79"><span class="ln" style="color: black; font-style: normal; font-weight: normal;">
</span></a></span></pre><pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l79"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">val </span><span class="s1">emptyExpression: Expr = List(List(Nil))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l81"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">solutions(xs: List[Int], n: Int): List[Expr] = {
</span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">good(exp: Expr) = valueE(exp) == n
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l83"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l84"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> xs.foldRight(List(emptyExpression))(
(x, ys) => ys.flatMap(glue(_,x))).
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l86"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> filter(good)
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l87"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> } </span></pre><br />
The code can be optimized using a simple enhancement, that is, by using an ok method that has the property good(candidate) => ok(candidate). In this case, we can use ok definition as follow:<br />
<br />
<pre><span class="s0" style="color: navy; font-weight: bold;"> def </span><span class="s1">ok(exp: Expr) = valueE(exp) <= n</span></pre><br />
and use it as the the filter of the glue operation. The implementation of solutions method becomes:<br />
<br />
<pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l49"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">solutions(xs: List[Int], n: Int): List[Expr] = {
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l50"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l51"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">ok(exp: Expr) = valueE(exp) <= n
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l52"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">good(exp: Expr) = valueE(exp) == n
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l53"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l54"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> xs.foldRight(List(emptyExpression))(
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l55"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> (x, ys) => </span>ys.flatMap(<span class="Apple-style-span" style="color: red;">glue(_,x).filter(ok)</span>)).filter(good) </pre><pre><pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l58"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> } </span></pre></pre><br />
The solution above can still be optimized by observing that value method executes the same expressions several times. To avoid recalculation, we propagate the last factor, term, and expression.<br />
The following is the improved code, a little bit obscure, I admit.<br />
<br />
First, the factor, term, and expression is represented by Intermediate class:<br />
<br />
<pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l103"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">case class </span><span class="s1">Intermediate(k: Int, f: Int, t: Int, e: Int) {
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l104"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">value = f * t + e
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l105"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> }
</span></pre><div><span class="s1"><br />
</span><br />
<span class="s1">And, finally the modified glue and solution:</span><br />
<span class="s1"></span><br />
<pre><span class="s1"><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l107"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">glue(expr: Expr, x: Int, im: Intermediate):
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l108"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> List[(Expr,Intermediate)]= {
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l109"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l110"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> expr </span><span class="s0" style="color: navy; font-weight: bold;">match </span><span class="s1">{
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l111"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">(Nil::Nil)::Nil =>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l112"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">val </span><span class="s1">exp = List(List(List(x)))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l113"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> List( (exp, Intermediate(</span><span class="s3" style="color: blue;">10</span><span class="s1">, x, </span><span class="s3" style="color: blue;">1</span><span class="s1">, </span><span class="s3" style="color: blue;">0</span><span class="s1">)) )
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l114"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l115"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">(xs::xss):: xsss =>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l116"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> List( ( ((x::xs)::xss)::xsss,
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l117"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> Intermediate(</span><span class="s3" style="color: blue;">10</span><span class="s1">*im.k, im.k * x + im.f , im.t, im.e)),
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l118"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> ( (List(x) :: xs::xss) :: xsss,
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l119"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> Intermediate(</span><span class="s3" style="color: blue;">10</span><span class="s1">, x, im.f * im.t, im.e)),
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l120"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> ( (List(List(x)) :: (xs::xss):: xsss),
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l121"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> Intermediate(</span><span class="s3" style="color: blue;">10</span><span class="s1">, x, </span><span class="s3" style="color: blue;">1</span><span class="s1">, im.f*im.t + im.e)))
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l122"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">case </span><span class="s1">_ => Nil
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l123"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l124"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> }
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l125"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> }</span></span></pre></div><br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l129"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"></span></a></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s0" style="color: navy; font-weight: bold;">def </span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span class="s1">solutions(xs: List[Int], n: Int): List[Expr] = { </span></span><br />
<pre><span class="s1"><a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l130"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l131"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">ok(exp: (Expr, Intermediate)) = exp._2.value <= n
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l132"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> </span><span class="s0" style="color: navy; font-weight: bold;">def </span><span class="s1">good(exp: (Expr, Intermediate)) = exp._2.value == n
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l133"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l134"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> xs.foldRight(List( (emptyExpression, Intermediate(</span><span class="s3" style="color: blue;">0</span><span class="s1">,</span><span class="s3" style="color: blue;">0</span><span class="s1">,</span><span class="s3" style="color: blue;">0</span><span class="s1">,</span><span class="s3" style="color: blue;">0</span><span class="s1">))))(
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l135"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> (x: Int, ys: List[(Expr, Intermediate)]) =>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l136"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> ys.flatMap( y =>
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l137"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> glue(y._1, x, y._2).filter(ok))).
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l138"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> filter(good).unzip._1
<a href="http://www.blogger.com/post-edit.g?blogID=2012573949883447809&postID=1960306855176892058" name="l140"><span class="ln" style="color: black; font-style: normal; font-weight: normal;"> </span></a> } </span></pre><pre><span class="s1">
</span></pre><pre><b>
</b></pre><pre></pre><b>Reference</b><br />
Bird R(2010), <i>Pearls of Functional Algorithm Design</i>. Cambridge University Press.voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-11374198588540245212010-12-29T09:54:00.008+01:002012-01-19T07:15:03.636+01:00Lazy Sieve Eratosthenes in Scala and Clojure<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Sieve Eratosthenes is a popular prime number generation algorithm. Start with 2 and cross out all even numbers until a limit number. Then, doing the same for 3 to cross out all factors of 3, like 3, 6, 9. The goal of the blog is to show some lazy sieve algorithms using clojure and scala and show the benchmark result of the algorithms. </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The complete code is available at <a href="https://github.com/anrizal06/rizier">https://github.com/anrizal06/rizier</a> .</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">It is easy to implement eager Sieve algorithms, using scala, this can be achieved as follow:</span><br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="color: #006699;"><strong>def</strong></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"> </span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="color: #9966ff;">sieve</span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="color: black;"><strong>(</strong></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;">limit: </span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="color: #009966;"><strong>Int</strong></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="color: black;"><strong>)</strong></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"> </span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="color: black;"><strong>=</strong></span></span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"> </span><span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><span style="color: black;"><strong>{</strong></span></span><br />
<pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 3 </span></span> <span style="color: #006699;"><strong>val</strong></span> composites <span style="color: black;"><strong>=</strong></span> <span style="color: #006699;"><strong>new</strong></span> <span style="color: #0099ff;"><strong>Array</strong></span><span style="color: #9900cc;">[Boolean]</span><span style="color: black;"><strong>(</strong></span>limit<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 5 </span></span> <span style="color: black;"><strong>(</strong></span><span style="color: red;">2</span> to math.<span style="color: #9966ff;">sqrt</span><span style="color: black;"><strong>(</strong></span>limit<span style="color: black;"><strong>)</strong></span>.toInt<span style="color: black;"><strong>)</strong></span>.<span style="color: #9966ff;">foreach</span><span style="color: black;"><strong>(</strong></span>i <span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>></strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: #006699;"><strong>if</strong></span> <span style="color: black;"><strong>(</strong></span><span style="color: black;"><strong>!</strong></span><span style="color: #9966ff;">composites</span><span style="color: black;"><strong>(</strong></span>i<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: black;"><strong>(</strong></span>i <span style="color: black;"><strong>*</strong></span> i until limit by i<span style="color: black;"><strong>)</strong></span>.<span style="color: #9966ff;">foreach</span><span style="color: black;"><strong>(</strong></span><span style="color: #9966ff;">composites</span><span style="color: black;"><strong>(</strong></span>_<span style="color: black;"><strong>)</strong></span> <span style="color: black;"><strong>=</strong></span> <span style="color: #cc00cc;">true</span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 8 </span></span> <span style="color: black;"><strong>}</strong></span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 10 </span></span> <span style="color: #006699;"><strong>for</strong></span> <span style="color: black;"><strong>{</strong></span> i <span style="color: black;"><strong><</strong></span><span style="color: black;"><strong>-</strong></span> <span style="color: red;">2</span> until limit
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: #006699;"><strong>if</strong></span> <span style="color: black;"><strong>(</strong></span><span style="color: black;"><strong>!</strong></span><span style="color: #9966ff;">composites</span><span style="color: black;"><strong>(</strong></span>i<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span> <span style="color: black;"><strong>}</strong></span> <span style="color: #006699;"><strong>yield</strong></span> i
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 13 </span></span><span style="color: black;"><strong>}</strong></span></span></pre><pre><span style="color: black;"><span style="color: black;"><strong>
</strong></span></span></pre><pre><div style="color: black; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">On of the problem with the solution above is that the limit parameter that must be used. The parameter is OK if we want to get prime numbers below a certain limit, but it is not OK when we want, for example to get the 1000th prime.</span></div><div style="color: black; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div><div style="color: black; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The solution for the problem above presented here is by using lazy evaluation provided in both scala and clojure. Using lazy algorithms, we generate a prime number, only when necessary. For example, if we want to get the 26th prime number, we only generate 26 prime number, not more.</span></div><div style="color: black; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><strong></strong></span></span></div><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div><div style="color: black;"></div><div style="color: black; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: pre;"></span></span></div><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The lazy algorithms for sieve are presented in the work of Melissa ONeill (2009) and the idea are adopted by for example clojure contrib API. The paper also showed that the naive functional solution (not exposed here) is very limited. Indeed, I made some tests with the solution and reached easily StackOverflow error for relatively small number.</span></div></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><b><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"></span></div><div style="color: black;"></div><div style="color: black; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: pre;"></span></span></div><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span>
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Not So Naive Solution</span></div></b></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div><div style="color: black;"></div><div style="color: black; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: pre;"></span></span></div><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span>
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Let's start first with the still naive, but optimized somehow, lazy sieve eratosthenes. First in clojure:</span></div></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><span class="Apple-style-span" style="color: #990066; font-family: Arial, Helvetica, sans-serif;"><div style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"></div></span><pre><span class="Apple-style-span" style="color: #990066; font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="color: black; font-family: 'Times New Roman'; white-space: normal;"><pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 1 </span></span><span style="color: blue;">(</span><span style="color: #9966ff;">defn</span> naive-primes
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #ff00cc;">"</span><span style="color: #ff00cc;">Implementation</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">of</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">lazy</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">but</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">naive</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">sieve</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">algorithms</span><span style="color: #ff00cc;">"</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #02b902;">[</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> <span style="color: blue;">(</span>letfn <span style="color: #02b902;">[</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 5 </span></span> <span style="color: blue;">(</span>next-prime <span style="color: #02b902;">[</span>x xs<span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 6 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>if</strong></span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>some</strong></span> <span style="color: #9966ff;">#(</span><span style="color: #66ccff;"><strong>zero?</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>rem</strong></span> x <span style="color: #66ccff;"><strong>%</strong></span><span style="color: blue;">)</span><span style="color: #9966ff;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>take-while</strong></span> <span style="color: #9966ff;">#(</span><span style="color: #66ccff;"><strong><=</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>*</strong></span> <span style="color: #66ccff;"><strong>%</strong></span> <span style="color: #66ccff;"><strong>%</strong></span><span style="color: blue;">)</span> x<span style="color: #9966ff;">)</span> xs<span style="color: blue;">)</span><span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>recur</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x <span style="color: red;">2</span><span style="color: blue;">)</span> xs<span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">cons</span> x <span style="color: blue;">(</span><span style="color: #9966ff;">lazy-seq</span> <span style="color: blue;">(</span>next-prime <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x <span style="color: red;">2</span><span style="color: blue;">)</span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>conj</strong></span> xs x<span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 10 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">cons</span> <span style="color: red;">2</span> <span style="color: blue;">(</span><span style="color: #9966ff;">lazy-seq</span> <span style="color: blue;">(</span>next-prime <span style="color: red;">3</span> <span style="color: #02b902;">[</span><span style="color: #02b902;">]</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">))</span><span style="color: blue;">)</span>
</span></pre></span></span></pre></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; white-space: normal;">
</span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; white-space: normal;">Line 10 starts the implementation by consing 2 to the lazy sequence of prime number starting from 3. Then, starting from 3, all odd numbers are examined if they are prime or not. This is done at line 6-7 of the code. If it is a composite number, then it recursively calls next prime to examine the next odd number (line 8) . Otherwise, if it is a prime, then the prime number is returned, and the list of prime numbers discovered so far is updated.</span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; white-space: normal;">To get the 100th prime, one can use </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; white-space: normal;">(nth (naive-primes) 100)</span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">The equivalent Scala program is the following:</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"><pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">primes</span><span style="color: black;"><strong>(</strong></span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 2 </span></span> <span style="color: #006699;"><strong>lazy</strong></span> <span style="color: #006699;"><strong>val</strong></span> ps <span style="color: black;"><strong>=</strong></span> <span style="color: red;">2</span> #:: <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span><span style="color: red;">3</span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span>p: <span style="color: #009966;"><strong>Int</strong></span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> p #:: <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span></span></pre><pre><span style="color: black;"><span style="color: black;"><strong></strong></span><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 5 </span></span> <span style="color: #0099ff;"><strong>Stream</strong></span>.<span style="color: #9966ff;">from</span><span style="color: black;"><strong>(</strong></span>p <span style="color: black;"><strong>+</strong></span> <span style="color: red;">2</span>, <span style="color: red;">2</span><span style="color: black;"><strong>)</strong></span>.
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: #9966ff;">find</span><span style="color: black;"><strong>(</strong></span>i<span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>></strong></span> ps.<span style="color: #9966ff;">takeWhile</span><span style="color: black;"><strong>(</strong></span>j <span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>></strong></span> j <span style="color: black;"><strong>*</strong></span> j <span style="color: black;"><strong><=</strong></span> i<span style="color: black;"><strong>)</strong></span>.
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 7 </span></span> <span style="color: #9966ff;">forall</span><span style="color: black;"><strong>(</strong></span>i <span style="color: black;"><strong>%</strong></span> _ <span style="color: black;"><strong>></strong></span> <span style="color: red;">0</span><span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>.get<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span> ps
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">10 </span></span> <span style="color: black;"><strong>}</strong></span> </span></pre></span></span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Stream is the Scala equivalent of Clojure lazy-seq. Note the interesting use of lazy val. To get the 100th prime number, one can use </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">primes().take(100).last</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">At<a href="http://stackoverflow.com/questions/4534230/why-stream-lazy-val-implementation-using-is-faster-than-listbuffer-one"> stack overflow</a> , I exposed another solution using mutable list as the alternative solution to the lazy stream above. It turned out that the mutable list solution was faster than the solution above.</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;"><b>With Wheel</b></span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">The Clojure Contrib library implements Sieve algorithm using wheel. The idea of the algorithm is to discard as many as possible the composite number that is factor of first prime numbers (3, 5, 7 in addition to 2). Indeed, the first solution already discards 2. To discard the numbers, a circular list of step -- or wheel -- is usd. For (2, 3, 5, 7), the wheel is </span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">(2 , 4 , 2 , 4 , 6 , 2 , 6 , 4 , 2 ,
4 , 6 , 6 , 2 , 6 , 4 , 2, 6 , 4 , 6 ,
8 , 4 , 2 , 4 , 2 , 4 , 8 , 6 , 4 , 6 ,
2 , 4 , 6, 2 , 6 , 6 , 4 , 2 , 4 , 6 ,
2 , 6 , 4 , 2 , 4 , 2 , 10 , 2 , 10).</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">To make the list circular, clojure has a very convenient method call cycle that can then be used to create the circular list:</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: helvetica, arial, freesans, clean, sans-serif; font-size: 11px; line-height: 14px;"><pre style="font-family: 'Bitstream Vera Sans Mono', Courier, monospace; font-size: 12px; font: normal normal normal 12px/normal Monaco, 'Courier New', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><div class="line" id="LC38" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 1em; padding-right: 0px; padding-top: 0px;"><span class="nb" style="color: #0086b3; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">cycle </span><span class="p" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">[</span><span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span></div><div class="line" id="LC39" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 1em; padding-right: 0px; padding-top: 0px;"><span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">8</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">8</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span></div><div class="line" id="LC40" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 1em; padding-right: 0px; padding-top: 0px;"> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">6</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">4</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">10</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">2</span> <span class="mi" style="color: #009999; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">10</span><span class="p" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">]</span>
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: normal; white-space: normal;">
</span>
<span class="p" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"></span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: normal; white-space: normal;">Scala does not have built-in cycle, but it is quite straight forward to implement cycle in scala:</span></div></pre></span></span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: helvetica, arial, freesans, clean, sans-serif; font-size: 11px; line-height: 14px;"><pre style="font-family: 'Bitstream Vera Sans Mono', Courier, monospace; font-size: 12px; font: normal normal normal 12px/normal Monaco, 'Courier New', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><div class="line" id="LC19" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 1em; padding-right: 0px; padding-top: 0px;"> <span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">def</span> <span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">cycle[A]</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">(</span><span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">xs</span><span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">:</span> <span class="kt" style="color: #445588; font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">List</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">[</span><span class="kt" style="color: #445588; font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">A</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">])</span><span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">:</span> <span class="kt" style="color: #445588; font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">Stream</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">[</span><span class="kt" style="color: #445588; font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">A</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">]</span> <span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">=</span> <span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">{</span></div><div class="line" id="LC20" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 1em; padding-right: 0px; padding-top: 0px;"> <span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">lazy</span> <span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">val</span> <span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">result</span><span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">:</span> <span class="kt" style="color: #445588; font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">Stream</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">[<span class="Apple-style-span" style="color: #445588;">A</span></span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">]</span> <span class="k" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">=</span> <span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">xs</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">.</span><span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">toStream</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">.</span><span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">append</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">(</span><span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">result</span><span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">)</span></div><div class="line" id="LC21" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 1em; padding-right: 0px; padding-top: 0px;"> <span class="n" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">result</span></div><div class="line" id="LC22" style="line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 1em; padding-right: 0px; padding-top: 0px;"> <span class="o" style="font-weight: bold; line-height: 1.4em; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">}</span></div></pre></span></span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">The complete implementation of the algorithm in Clojure is:</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"><pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: blue;">(</span><span style="color: #9966ff;">defn</span> wheel-primes
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #ff00cc;">"</span><span style="color: #ff00cc;">The</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">optimization</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">of</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">lazy</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">naive</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">sieve</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">algorithm</span><span style="color: #ff00cc;"> </span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">by</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">discarding</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">the</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">number</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">divisible</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">by</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">2</span><span style="color: #ff00cc;">,</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">3</span><span style="color: #ff00cc;">,</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">5</span><span style="color: #ff00cc;">,</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">and</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">7</span><span style="color: #ff00cc;">.</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">This</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">is</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">done</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">by</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">adding</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">the</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">next</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">number</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">to</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">be</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">examined</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">by</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">the</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">number</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">given</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">by</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">the</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">wheel</span><span style="color: #ff00cc;">.</span><span style="color: #ff00cc;">"</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: #02b902;">[</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">let</span> <span style="color: #02b902;">[</span>next-prime
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">fn</span> next-prime <span style="color: #02b902;">[</span>x xs <span style="color: #02b902;">[</span>f <span style="color: #0099ff;"><strong>&</strong></span> r<span style="color: #02b902;">]</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>if</strong></span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>some</strong></span> <span style="color: #9966ff;">#(</span><span style="color: #66ccff;"><strong>zero?</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>rem</strong></span> x <span style="color: #66ccff;"><strong>%</strong></span><span style="color: blue;">)</span><span style="color: #9966ff;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>take-while</strong></span> <span style="color: #9966ff;">#(</span><span style="color: #66ccff;"><strong><=</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>*</strong></span> <span style="color: #66ccff;"><strong>%</strong></span> <span style="color: #66ccff;"><strong>%</strong></span><span style="color: blue;">)</span> x<span style="color: #9966ff;">)</span> xs<span style="color: blue;">)</span><span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>recur</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x f<span style="color: blue;">)</span> xs r<span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">cons</span> x <span style="color: blue;">(</span><span style="color: #9966ff;">lazy-seq</span> <span style="color: blue;">(</span>next-prime <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x f<span style="color: blue;">)</span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>conj</strong></span> xs x<span style="color: blue;">)</span> r<span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 13 </span></span> wheel <span style="color: blue;">(</span><span style="color: #006699;"><strong>cycle</strong></span> <span style="color: #02b902;">[</span><span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">6</span> <span style="color: red;">2</span> <span style="color: red;">6</span> <span style="color: red;">4</span> <span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">6</span> <span style="color: red;">6</span> <span style="color: red;">2</span> <span style="color: red;">6</span> <span style="color: red;">4</span> <span style="color: red;">2</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 14 </span></span> <span style="color: red;">6</span> <span style="color: red;">4</span> <span style="color: red;">6</span> <span style="color: red;">8</span> <span style="color: red;">4</span> <span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">8</span> <span style="color: red;">6</span> <span style="color: red;">4</span> <span style="color: red;">6</span> <span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">6</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 15 </span></span> <span style="color: red;">2</span> <span style="color: red;">6</span> <span style="color: red;">6</span> <span style="color: red;">4</span> <span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">6</span> <span style="color: red;">2</span> <span style="color: red;">6</span> <span style="color: red;">4</span> <span style="color: red;">2</span> <span style="color: red;">4</span> <span style="color: red;">2</span> <span style="color: red;">10</span> <span style="color: red;">2</span> <span style="color: red;">10</span><span style="color: #02b902;">]</span><span style="color: blue;">)</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 16 </span></span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>concat</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 17 </span></span> <span style="color: #02b902;">[</span><span style="color: red;">2</span> <span style="color: red;">3</span> <span style="color: red;">5</span> <span style="color: red;">7</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 18 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">lazy-seq</span> <span style="color: blue;">(</span>next-prime <span style="color: red;">11</span> <span style="color: #02b902;">[</span><span style="color: #02b902;">]</span> wheel<span style="color: blue;">)))</span><span style="color: blue;">)</span><span style="color: blue;">)</span>
</span></pre></span></span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><pre style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The second solution is then close to the first one, except that instead of incrementing with 2, the next number to be examined is obtained by incrementing with the number in the wheel.</span></span></pre><pre style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;"></span></span></pre><pre style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The Scala version of the solution is:</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> </span></span></pre><pre style="color: #666666; line-height: 18px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre style="color: #666666; font-family: 'Trebuchet MS', Trebuchet, sans-serif; font-size: 13px; line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="color: black; font-family: monospace; white-space: pre;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span> <span style="color: #006699;"><strong>def</strong></span> cycle<span style="color: #9900cc;">[A]</span><span style="color: black;"><strong>(</strong></span>xs: <span style="color: #0099ff;"><strong>List</strong></span><span style="color: #9900cc;">[A]</span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[A]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 2 </span></span> <span style="color: #006699;"><strong>lazy</strong></span> <span style="color: #006699;"><strong>val</strong></span> result: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[A]</span> <span style="color: black;"><strong>=</strong></span> xs.toStream.<span style="color: #9966ff;">append</span><span style="color: black;"><strong>(</strong></span>result<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> result
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 5 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">wheel</span><span style="color: black;"><strong>(</strong></span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: #9966ff;">cycle</span><span style="color: black;"><strong>(</strong></span><span style="color: #0099ff;"><strong>List</strong></span><span style="color: black;"><strong>(</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: red;">2</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span> , <span style="color: red;">4</span> , <span style="color: red;">6</span> , <span style="color: red;">2</span> , <span style="color: red;">6</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span> ,
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: red;">4</span> , <span style="color: red;">6</span> , <span style="color: red;">6</span> , <span style="color: red;">2</span> , <span style="color: red;">6</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span>, <span style="color: red;">6</span> , <span style="color: red;">4</span> , <span style="color: red;">6</span> ,
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 9 </span></span> <span style="color: red;">8</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span> , <span style="color: red;">4</span> , <span style="color: red;">8</span> , <span style="color: red;">6</span> , <span style="color: red;">4</span> , <span style="color: red;">6</span> ,
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 10 </span></span> <span style="color: red;">2</span> , <span style="color: red;">4</span> , <span style="color: red;">6</span>, <span style="color: red;">2</span> , <span style="color: red;">6</span> , <span style="color: red;">6</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span> , <span style="color: red;">4</span> , <span style="color: red;">6</span> ,
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: red;">2</span> , <span style="color: red;">6</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span> , <span style="color: red;">4</span> , <span style="color: red;">2</span> , <span style="color: red;">10</span> , <span style="color: red;">2</span> , <span style="color: red;">10</span><span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 13 </span></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 14 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">from</span><span style="color: black;"><strong>(</strong></span>x: <span style="color: #009966;"><strong>Int</strong></span>, stepIt: Iterator<span style="color: #9900cc;">[Int]</span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 15 </span></span> x #:: <span style="color: #9966ff;">from</span><span style="color: black;"><strong>(</strong></span>x <span style="color: black;"><strong>+</strong></span> stepIt.next, stepIt<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 16 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 17 </span></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 18 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">primes</span><span style="color: black;"><strong>(</strong></span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 19 </span></span> <span style="color: #006699;"><strong>lazy</strong></span> <span style="color: #006699;"><strong>val</strong></span> ps <span style="color: black;"><strong>=</strong></span> <span style="color: red;">2</span> #:: <span style="color: red;">3</span> #:: <span style="color: red;">5</span> #:: <span style="color: red;">7</span> #:: <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span><span style="color: red;">11</span>, wheel.iterator<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 20 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span>p: <span style="color: #009966;"><strong>Int</strong></span>, it: Iterator<span style="color: #9900cc;">[Int]</span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 21 </span></span> p #:: <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span> <span style="color: #9966ff;">from</span><span style="color: black;"><strong>(</strong></span>p <span style="color: black;"><strong>+</strong></span> it.next, it<span style="color: black;"><strong>)</strong></span>.
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 22 </span></span> <span style="color: #9966ff;">find</span><span style="color: black;"><strong>(</strong></span>i <span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>></strong></span> ps.<span style="color: #9966ff;">takeWhile</span><span style="color: black;"><strong>(</strong></span>j <span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>></strong></span> j <span style="color: black;"><strong>*</strong></span> j <span style="color: black;"><strong><=</strong></span> i<span style="color: black;"><strong>)</strong></span>.
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 23 </span></span> <span style="color: #9966ff;">forall</span><span style="color: black;"><strong>(</strong></span>i <span style="color: black;"><strong>%</strong></span> _ <span style="color: black;"><strong>></strong></span> <span style="color: red;">0</span><span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>.get,
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 24 </span></span> it<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 25 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 26 </span></span> ps
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 27 </span></span> <span style="color: black;"><strong>}</strong></span></span></span></span></pre><pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="color: #666666; font-family: 'Times New Roman'; font-size: medium; font-weight: normal; line-height: 18px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></span></pre><pre style="font-size: medium; font-weight: normal; line-height: 18px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>
</b></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>
</b></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>
</b></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b><span class="Apple-style-span" style="font-family: monospace; font-weight: normal; line-height: normal; white-space: pre;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>Hash Map Based Solution</b></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">The problem with the first two solutions is that both solutions require the check whether a number is factor of the prime number discovered so far. For example, when examining 121, the first two algorithms verify if 121 are factor 3, 5, and 7. It turns out, that the check is not necessary, and this leads to the third solution of lazy sieve. </span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">Like the two solutions above, the algorithm receives a stream of odd number and returns a stream of prime number. The algorithm uses a hash map that stores the last discovered composite number. Each composite number that is stored in the hash map is a factor of prime numbers discovered so far. The key of the hash map is the composite number and the value is the double of the corresponding prime number. </span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">When a number is under examination, it is checked against the hash map. </span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; line-height: 18px; white-space: normal;">If the number is in the hash map, obviously, it is a composite number. In this case, the number is removed from the hash map and replaced by new composite number that is factor of the corresponding prime. The new composite number must not be in the hash map, therefore it continues to increment the number until discovering a brand new composite number.</span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">On the other hand, when the number is not in the hash map since the beginning, it is a prime. A new composite corresponding to the prime is then inserted to the hashmap; the number is the square of the prime. </span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"></span></span></pre></pre></span></b></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>
</b></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>
</b></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="color: #666666; font-size: 13px; line-height: 18px;"><b><span class="Apple-style-span" style="white-space: normal;"><pre style="display: inline !important; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="font-weight: normal; white-space: normal;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="font-weight: normal; white-space: normal;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman'; white-space: normal;"><pre style="display: inline !important;"><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: blue;">(</span><span style="color: #9966ff;">defn</span> hashtable-primes </span></pre></span></span></pre></span></b></pre></span></b></pre></span></b></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="color: #666666; font-family: monospace; font-size: 13px; white-space: pre;"><b><span class="Apple-style-span" style="white-space: normal;"><pre style="font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="font-weight: normal; white-space: normal;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="font-weight: normal; white-space: normal;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman'; white-space: normal;"><pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #ff00cc;">"</span><span style="color: #ff00cc;">An</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">implementation</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">of</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">lazy</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">sieve</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">algorithm</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">by</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">storing</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">in</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">a</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">hastable</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">the</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">next</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">composite</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">number</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">corresponding</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">to</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">a</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">prime</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">number</span><span style="color: #ff00cc;">"</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 4 </span></span> <span style="color: #02b902;">[</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 5 </span></span> <span style="color: blue;">(</span>letfn <span style="color: #02b902;">[</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: blue;">(</span>next-composite <span style="color: #02b902;">[</span>x step sieve<span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>if</strong></span> <span style="color: blue;">(</span>sieve x<span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: #cc0000;">;</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">the</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">composite</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">number</span><span style="color: #cc0000;"> has </span><span style="color: #cc0000;">already been</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">found</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">previously</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 9 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>recur</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x step<span style="color: blue;">)</span> step sieve<span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 10 </span></span> <span style="color: #cc0000;">;</span><span style="color: #cc0000;"> brand </span><span style="color: #cc0000;">new</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">composite</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">number</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>assoc</strong></span> sieve x step<span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 12 </span></span> <span style="color: blue;">(</span>next-prime <span style="color: #02b902;">[</span>x sieve<span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 13 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">let</span> <span style="color: #02b902;">[</span>step <span style="color: blue;">(</span>sieve x<span style="color: blue;">)</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 14 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>if</strong></span> <span style="color: blue;">(</span>sieve x<span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 15 </span></span> <span style="color: #cc0000;">;</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">found</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">a</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">composite</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 16 </span></span> <span style="color: blue;">(</span><span style="color: #009966;"><strong>recur</strong></span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x <span style="color: red;">2</span><span style="color: blue;">)</span> <span style="color: blue;">(</span>next-composite <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x step<span style="color: blue;">)</span> step <span style="color: blue;">(</span><span style="color: #006699;"><strong>dissoc</strong></span> sieve x<span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 17 </span></span> <span style="color: #cc0000;">;</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">found</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">a</span><span style="color: #cc0000;"> </span><span style="color: #cc0000;">prime</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 18 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">cons</span> x
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 19 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">lazy-seq</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 20 </span></span> <span style="color: blue;">(</span>next-prime <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>+</strong></span> x <span style="color: red;">2</span><span style="color: blue;">)</span> <span style="color: blue;">(</span><span style="color: #006699;"><strong>assoc</strong></span> sieve <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>*</strong></span> x x<span style="color: blue;">)</span> <span style="color: blue;">(</span><span style="color: #66ccff;"><strong>*</strong></span> x <span style="color: red;">2</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: #02b902;">]</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 21 </span></span> <span style="color: blue;">(</span><span style="color: #9966ff;">cons</span> <span style="color: red;">2</span> <span style="color: blue;">(</span><span style="color: #9966ff;">lazy-seq</span> <span style="color: blue;">(</span>next-prime <span style="color: red;">3</span> <span style="color: #02b902;">{</span><span style="color: #02b902;">}</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span><span style="color: blue;">)</span></span></pre></span></span></pre></span></b></pre></span></b></pre></span></b></span></span></span></pre><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></pre><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">The Scala version of the solution is the following. I used mutable map to gain better performance without really sacrifying the functional nature of the code:</span></span>
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span>
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="font-family: 'Times New Roman'; line-height: normal;"></span></span></span>
<pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="font-family: 'Times New Roman'; white-space: normal;"><pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span> <span style="color: #006699;"><strong>import</strong></span> scala.collection.mutable.Map
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 2 </span></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">primes</span><span style="color: black;"><strong>(</strong></span><span style="color: black;"><strong>)</strong></span> : <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> <span style="color: red;">2</span> #:: <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span><span style="color: red;">3</span>, Map<span style="color: black;"><strong>{</strong></span> <span style="color: red;">9</span> <span style="color: black;"><strong>-</strong></span><span style="color: black;"><strong>></strong></span> <span style="color: red;">6</span> <span style="color: black;"><strong>}</strong></span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 5 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: #006699;"><strong>private</strong></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span>p: <span style="color: #009966;"><strong>Int</strong></span>, pQ: Map<span style="color: #9900cc;">[Int,</span><span style="color: #9900cc;"> </span><span style="color: #9900cc;">Int]</span><span style="color: black;"><strong>)</strong></span>: <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 7 </span></span> p #:: <span style="color: #9966ff;">sieve</span><span style="color: black;"><strong>(</strong></span><span style="color: #9966ff;">nextPrime</span><span style="color: black;"><strong>(</strong></span>p <span style="color: black;"><strong>+</strong></span> <span style="color: red;">2</span>, pQ<span style="color: black;"><strong>)</strong></span>, pQ <span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span> <span style="color: #006699;"><strong>private</strong></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">nextCompositeNumber</span><span style="color: black;"><strong>(</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">10 </span></span> x:<span style="color: #009966;"><strong>Int</strong></span>, step: <span style="color: #009966;"><strong>Int</strong></span>, pQ: Map<span style="color: #9900cc;">[Int,</span><span style="color: #9900cc;"> </span><span style="color: #9900cc;">Int]</span><span style="color: black;"><strong>)</strong></span>: <span style="color: #009966;"><strong>Unit</strong></span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">11 </span></span> pQ.<span style="color: #9966ff;">get</span><span style="color: black;"><strong>(</strong></span>x<span style="color: black;"><strong>)</strong></span> <span style="color: #006699;"><strong>match</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;">12 </span></span> <span style="color: #006699;"><strong>case </strong></span><span style="color: #0099ff;"><strong>Some</strong></span><span style="color: black;"><strong>(</strong></span>_<span style="color: black;"><strong>)</strong></span> <span style="color: #006699;"><strong>=></strong></span> <span style="color: #9966ff;">nextCompositeNumber</span><span style="color: black;"><strong>(</strong></span>x <span style="color: black;"><strong>+</strong></span> step, step, pQ<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">13 </span></span> <span style="color: #006699;"><strong>case </strong></span><span style="color: #66ccff;"><strong>None</strong></span> <span style="color: #006699;"><strong>=></strong></span> <span style="color: #9966ff;">pQ</span><span style="color: black;"><strong>(</strong></span>x<span style="color: black;"><strong>)</strong></span> <span style="color: black;"><strong>=</strong></span> step
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">14 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">15 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">16 </span></span> <span style="color: #006699;"><strong>private</strong></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">nextPrime</span><span style="color: black;"><strong>(</strong></span>candidate: <span style="color: #009966;"><strong>Int</strong></span>, pQ: Map<span style="color: #9900cc;">[Int,</span><span style="color: #9900cc;"> </span><span style="color: #9900cc;">Int]</span><span style="color: black;"><strong>)</strong></span>: <span style="color: #009966;"><strong>Int</strong></span> <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;">17 </span></span> pQ.<span style="color: #9966ff;">get</span><span style="color: black;"><strong>(</strong></span>candidate<span style="color: black;"><strong>)</strong></span> <span style="color: #006699;"><strong>match</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">18 </span></span> <span style="color: #006699;"><strong>case </strong></span><span style="color: #0099ff;"><strong>Some</strong></span><span style="color: black;"><strong>(</strong></span>step<span style="color: black;"><strong>)</strong></span> <span style="color: #006699;"><strong>=></strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">19 </span></span> pQ <span style="color: black;"><strong>-</strong></span><span style="color: black;"><strong>=</strong></span> candidate
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">20 </span></span> <span style="color: #9966ff;">nextCompositeNumber</span><span style="color: black;"><strong>(</strong></span>candidate <span style="color: black;"><strong>+</strong></span> step, step, pQ<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">21 </span></span> <span style="color: #9966ff;">nextPrime</span><span style="color: black;"><strong>(</strong></span>candidate <span style="color: black;"><strong>+</strong></span> <span style="color: red;">2</span>, pQ<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;">22 </span></span> <span style="color: #006699;"><strong>case </strong></span><span style="color: #66ccff;"><strong>None</strong></span> <span style="color: #006699;"><strong>=></strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">23 </span></span> <span style="color: #9966ff;">pQ</span><span style="color: black;"><strong>(</strong></span>candidate <span style="color: black;"><strong>*</strong></span> candidate<span style="color: black;"><strong>)</strong></span> <span style="color: black;"><strong>=</strong></span> candidate <span style="color: black;"><strong>*</strong></span> <span style="color: red;">2</span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">24 </span></span> candidate
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">25 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;">26 </span></span> <span style="color: black;"><strong>}</strong></span></span></pre></span></span></span></pre></div><div></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>
</b></span></span>
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><b>Benchmark</b></span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">To complete the implementation, I did a simple benchmark, mainly with the Scala versions, but Clojure version showed also the similar results. For the implementation, I used Scala 2.8.1 and Clojure 1.2 with Oracle JVM 1.6.0_14 version. The benchmark was done in my tiny laptop with 3GB of memory, Windows, and Intel Pentium Dual Core. </span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">The result is shown in the following graph: </span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="color: #666666; font-size: 13px; white-space: pre;"></span></span></span>
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="color: #666666; font-size: 13px; white-space: pre;"><b><span class="Apple-style-span" style="white-space: normal;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjd2mYDuW_9s6ZOVmq3rdOzSCaB6pShE-7r0N50AZUbcuGHr3EkCuKAz0uF4EYIf8cLqy6nEO_hNmb5c63Ee-tpFVPG36bcPe2-E6E_jEVu99uBlrfiMQES_JDXvYplXQwD2fTDtQetrkUp/s1600/primes.jpg" imageanchor="1" style="color: #ff9900; margin-left: 1em; margin-right: 1em; text-decoration: underline;"><img border="0" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjd2mYDuW_9s6ZOVmq3rdOzSCaB6pShE-7r0N50AZUbcuGHr3EkCuKAz0uF4EYIf8cLqy6nEO_hNmb5c63Ee-tpFVPG36bcPe2-E6E_jEVu99uBlrfiMQES_JDXvYplXQwD2fTDtQetrkUp/s400/primes.jpg" style="border-bottom-style: none; border-color: initial; border-color: initial; border-left-style: none; border-right-style: none; border-top-style: none; border-width: initial; border-width: initial; position: relative;" width="400" /></a></div></span></b></span></span></span></pre></div><div></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">Where x-axis shows the limit in million and the y axis is the execution time in ms.
The blue line is for the naive solution, and magenta one for wheel solution, and the yellow one for the hash map one. </span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">The same curve, but not the same execution time, is also observed in the implementation using Clojure. However, The Clojure implementation execution time is longer than Scala.
The wheel solution improves the performance of the naive one, but the implementation that uses hashmap performs much better. </span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">The maximum limit I could manage to have in my machine using the Scala implementation of the hash map algorithm was 150 000 000 that represented more than 8 million primes . The time of execution was around 270 s.
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="white-space: pre;"></span></span></span>
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;"><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="white-space: normal;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="white-space: normal;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Reference</span></pre></span></b></pre><pre style="font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="white-space: normal;"><pre style="display: inline !important; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">ONeill M (2009) The Genuine of Sieve Erathosthenes. Available at: </span></pre></span></b></pre><pre style="color: #666666; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: monospace;"><b><span class="Apple-style-span" style="white-space: normal;"><pre style="display: inline !important; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><a href="http://www.cs.hmc.edu/~oneill/papers/Sieve-JFP.pdf" style="color: #ff9900; text-decoration: none;">http://www.cs.hmc.edu/~oneill/papers/Sieve-JFP.pdf</a> </pre></span></b></span></span></pre></span></b></pre></span></span></span></span></span></pre></pre></pre></pre></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px; white-space: normal;">
</span></span></div></pre></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><pre style="display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="color: #666666; font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="line-height: 18px;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b><span class="Apple-style-span" style="white-space: normal;"><div style="font-size: 13px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: monospace;"><b><span class="Apple-style-span" style="white-space: normal;"></span></b></span></span>
<pre style="display: inline !important; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></pre></div></span></b></pre></span></span></pre></pre></pre></pre></span></pre></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="white-space: normal;">
</span></span></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre><pre><div style="color: black; display: inline !important; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; white-space: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">
</span></div></pre>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com3tag:blogger.com,1999:blog-2012573949883447809.post-440269247951341102010-12-06T00:02:00.000+01:002012-01-19T07:15:03.684+01:00Hello Hadoop - Getting Started with Hadoop in Pseudo-Distributed Cluster Mode (Ubuntu)<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">In this blog, we will get started with Apache Hadoop by writing a map reduce job to process log files. We will start with the problem to solve, design the solution, download and install hadoop, and then execute the mapreduce solution that we design. Therefore, you can perceive this blog as "Hello Hadoop" blog.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Times,'Times New Roman',serif; font-size: large;"><b>Problem and Solution Sketch</b></span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The case that we will see is taken from a fictitious case from a web server for flight reservation. The web server logs the requests from its users, including the origin and destionation (airport code and city code), the date of departure and the date of the return. </span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The log lines look like the following: </span><br />
<i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">PARCDGFRAFRA-02022011/06022011 </span></i><br />
<i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">MRSMRSMUCMUC-10022011/20112011</span></i><br />
<i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">NCENCEMUCMUC-11012011/19012011</span></i><br />
<i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">NCENCELONLHR-21022011/03032011</span></i><br />
<div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The first three characters represent the city of origin, for example PAR to represent Paris. The second three characters represent the airport of origin, for example CDG to represent Charles de Gaulle airport. Then, the following six characters represent the city and airport of destinations. Starting from the 14th character, we have the date of departure in ddMMyy format, followed by date of the return flight in the same format.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We can have a lot of things to do with the logs, for example to know the distribution of the request for each city, the distribution of the request per stay duration, the distribution of the request for each date departure, and son on and so forth. For this example, we will just try to get the distribution of the requests per each origin. That is, we want to have the following consolidated output:</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><i>LYS 80</i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><i>MRS 42</i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><i>NCE 834</i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><i>PAR 838</i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><i>TLS 296</i></span><br />
<br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Hadoop distributes the files in the cluster. The files are then served as input of the mapper nodes which maps each line of the input to a (key, value) pair. </span><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Hadoop optimizes the computation by ensuring that the node maps the data that are local to it. </span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">For example, the line </span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<div style="font-family: 'Times New Roman'; margin: 0px;"><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span></i></div><div style="font-family: 'Times New Roman'; margin: 0px;"><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">PARCDGFRAFRA-02022011/06022011 </span></i></div><div><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span></i></div><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">may be mapped to (PAR, 1) in a mapreduce jobs that count the number of Paris city code and 1 is to be used as the count. The same line may also be mapped to (PAR, 4) when the line is mapped to pair of <origin city, duration of the stay>. The line may also be mapped to (PARFRA, 4 days) when the line is mapped to <origin and destination city, duration of the stay></span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The result of the mapping are then delivered to the reducer. Each (key, value) pair that has the same key value is delivered to the same reducer. The delivery to the same reducer is called shuffle and sort process.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">For our problem here, we will then map each line of the log to <origin city, 1>. We map then the following lines </span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<div style="font-family: 'Times New Roman'; margin: 0px;"><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">PARCDGFRAFRA-02022011/06022011 </span></i></div><div style="font-family: 'Times New Roman'; margin: 0px;"><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">MRSMRSMUCMUC-10022011/20112011</span></i></div><div style="font-family: 'Times New Roman'; margin: 0px;"><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">NCENCEMUCMUC-11012011/19012011</span></i></div><div style="font-family: 'Times New Roman'; margin: 0px;"><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">NCENCELONLHR-21022011/03032011</span></i></div><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">PARCDGFRAFRA-02022011/06022011 </span></i><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"><i><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span></i></span></span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">to (PAR, 1), (MRS, 1), (NCE, 1), (NCE, 1). The (PAR, [1, 1]), (MRS, [1]), and (NCE, [1, 1]) are then delivered to reducer. The reducer then reduces the three lines to </span><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">(PAR, 2), (MRS, 1), and (NCE, 2) which is what we need as the solution of the problem.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<div style="font-family: 'Times New Roman'; margin: 0px;"><span class="Apple-style-span" style="font-family: Times,'Times New Roman',serif; font-size: large;"><b>Installing Hadoop in Pseudo-Distributed Mode </b></span></div><div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We are now ready to install Hadoop in our environment. To do so, you will need to have the following configuration of machines:</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgK3EuAT-gOI32BbROEQQkXUtu8YUnACsEq7w75Tt_5lT1xV6kFChsczrUk_s2dXPhg4XMdfpxZTq4XbDfuL4yc04hmlvrZIKVKqthoFuJpZgh4-eXQq5wcGh-4EFQ1hyphenhyphenvelBbjUaQuE5wJ/s1600/rack.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgK3EuAT-gOI32BbROEQQkXUtu8YUnACsEq7w75Tt_5lT1xV6kFChsczrUk_s2dXPhg4XMdfpxZTq4XbDfuL4yc04hmlvrZIKVKqthoFuJpZgh4-eXQq5wcGh-4EFQ1hyphenhyphenvelBbjUaQuE5wJ/s320/rack.jpg" width="320" /></a></div><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">I'm joking of course. No, you will not need those machines, you can use your personal computer. I don't even use my desktop computer for this example, only my laptop computer. However, you will need Ubuntu to be installed in your machine.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We will download hadoop from apache site: </span><a href="http://hadoop.apache.org/">http://hadoop.apache.org/</a> . <span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We will use the 0.21.0 version. Make sure to execute md5sum against the downloaded file. Then, copy the file to the target directory. For example to /home/arizal.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">sudo mv /home/arizal/Downloads/hadoop-0.21.0.tar.gz /home/arizal</span><br />
<div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div><div style="font-family: Arial,Helvetica,sans-serif;">Then untar the file:</div><div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">tar xzf hadoop-0.21.0.tar.gz</span></div><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Check if the installation work fine by launching</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">bin/hadoop version.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">You might get the following error:</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">Error: JAVA_HOME is not set.</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">To fix the error, you need to export JAVA_HOME that tells the directory of the JDK. Example:</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">export JAVA_HOME=/yourjdkdirectory/jdk1.6.0_22</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">When everything is OK you will have something like:</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">Hadoop 0.21.0</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">Compiled by tomwhite on Tue Aug 17 01:02:28 EDT 2010</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">From source with checksum a1aeb15b4854808d152989ba76f90fac</span><br />
<br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Now, we need to configure ssh that will be used by the pseudo nodes like name nodes, data nodes, and so on. I will not discuss what they are in this blog, you will find them in hadoop site, or I might come back later on the subject in another blog.</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">To install ssh, you can do</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">sudo apt-get install ssh</span></div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Once installed, we will configure passwordless ssh connection as follow:</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">First, generate the key using ssh-keygen</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif; font-size: x-small;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;">ssh-keygen -t rsa -P ""</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;">Generating public/private rsa key pair.</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;"><br />
</span></span><br />
<div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;">Enter file in which to save the key (/home/arizal/.ssh/id_rsa): </span></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;"><br />
</span></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;">Your identification has been saved in /home/arizal/.ssh/id_rsa.</span></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;">Your public key has been saved in /home/arizal/.ssh/id_rsa.pub.</span></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;">The key fingerprint is:</span></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><span class="Apple-style-span" style="font-size: x-small;">...</span></span></div><div><div style="font-family: 'Times New Roman'; font-size: medium; margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Then, enable SSH access to the localhost using the generated key as follow:</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</span></div></div><div style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Test that the configuration works by launching:</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">ssh localhost</span><br />
<span class="Apple-style-span" style="font-size: x-small;"></span><br />
<div style="font-family: 'Times New Roman'; font-size: medium; margin: 0px;"><span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></span></div><div style="font-family: 'Times New Roman'; margin: 0px;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">If the configuration works, you will not be asked for password.</span></span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We will modify the conf/hadoop-env.sh to add JAVA_HOME to the file. </span></span></div><div style="font-size: medium; margin: 0px;"><span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></span></div><div style="font-family: 'Times New Roman'; font-size: medium; margin: 0px;"><span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">JAVA_HOME=/yourjdkdirectory/jdk1.6.0_22</span></span></span></div><div style="font-family: 'Times New Roman'; font-size: medium; margin: 0px;"><span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></span></div><div style="margin: 0px;"><div style="font-family: 'Times New Roman';"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We're almost there, be patient. Just couple of steps before coding. Promise.</span></span></div><div style="font-family: 'Times New Roman';"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We need to modify three files conf/core-site.xml, conf/hdfs-site.xml, and conf/mapred-site.xml as follow:</span></span></div><div style="font-family: 'Times New Roman';"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></span></div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><!-- core-site.xml --></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><configuration></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> <property></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> <name>fs.default.name</name></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> <value>hdfs://localhost/</value></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> </property></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"></configuration></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><!-- hdfs-site.xml --></span><br />
<span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><configuration></span></span><br />
<span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> <property></span></span><br />
<span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> <name>dfs.replication</name></span></span><br />
<span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> <value>1</value></span></span><br />
<span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> </property></span></span><br />
<span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"></configuration></span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><!-- mapred-site.xml --></span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"><configuration></span></span><br />
<span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"> <property></span></span><br />
<span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"> <name>mapred.job.tracker</name></span></span><br />
<span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"> <value>localhost:8021</value></span></span><br />
<span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"> </property></span></span><br />
<span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;"></configuration></span></span><br />
<span class="Apple-style-span" style="font-size: x-small;"> </span></div><div style="font-family: 'Times New Roman'; font-size: medium; margin: 0px;"><span class="Apple-style-span" style="font-size: x-small;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></span></div><div style="margin: 0px;"><span class="Apple-style-span"></span><br />
<div style="margin: 0px;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Then, we format HDFS file system. The formatting creates an empty file system.</span></span></div><div style="margin: 0px;"></div><div style="margin: 0px;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">bin/hadoop namenode -format</span></span></div><div style="font-family: Arial,Helvetica,sans-serif; font-size: medium;"><span class="Apple-style-span"><br />
</span></div><div><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">When it's OK, the output will look like the following:</span></span></div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326; compiled by 'tomwhite' on Tue Aug 17 01:02:28 EDT 2010</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">************************************************************/</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:49 INFO namenode.FSNamesystem: defaultReplication = 1</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:49 INFO namenode.FSNamesystem: maxReplication = 512</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:49 INFO namenode.FSNamesystem: minReplication = 1</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:49 INFO namenode.FSNamesystem: maxReplicationStreams = 2</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:49 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks = false</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:49 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:50 INFO namenode.FSNamesystem: fsOwner=arizal</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:50 INFO namenode.FSNamesystem: supergroup=supergroup</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:50 INFO namenode.FSNamesystem: isPermissionEnabled=true</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:50 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:51 INFO common.Storage: Image file of size 112 saved in 0 seconds.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:51 INFO common.Storage: Storage directory /tmp/hadoop-arizal/dfs/name has been successfully formatted.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10/12/05 21:38:51 INFO namenode.NameNode: SHUTDOWN_MSG: </span><br />
<div style="font-family: 'Courier New',Courier,monospace; font-size: small;"><br />
</div></div><br />
<div style="margin: 0px;"><div style="margin: 0px;"><span class="Apple-style-span"></span></div><div style="margin: 0px;"></div><div><div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Perfect. We're almost there. We start everything now. We can use</span></div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> bin/start-all.sh </span><br />
<div style="font-family: Arial,Helvetica,sans-serif;">but we will get this annoying warning message.</div><br />
<span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh.</span><br />
<br />
<div style="font-family: Arial,Helvetica,sans-serif; font-size: medium; margin: 0px;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Don't worry, ignoring the warning looks OK. But you can try launching start-dfs.sh or start-mapred.sh you will get something even more annoying error message:</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">Hadoop common not found.</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span></div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The fix for the "Hadoop common not found" can be found in the following JIRA entry:</span></div><div><div style="margin: 0px;"><a href="https://issues.apache.org/jira/browse/HADOOP-6953"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">https://issues.apache.org/jira/browse/HADOOP-6953</span></a></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The solution described there solved the problem for me.</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We can check the running nodes by using the jps command:</span></div><div style="margin: 0px;"></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">$jps</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">9530 DataNode</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">9325 NameNode</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10131 Jps</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">9750 SecondaryNameNode</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">10087 TaskTracker</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">9880 JobTracker</span></div><div style="font-family: Arial,Helvetica,sans-serif; font-size: medium;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"> Finally, to stop all the nodes, we can use</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"> bin/stop-all.sh</span><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif; font-size: small;">.</span></div><div style="font-family: Arial,Helvetica,sans-serif; font-size: medium; margin: 0px;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">OK. That's all about installation, as promised, we're now ready to code.</span></div><div style="margin: 0px;"><br />
</div><div style="margin: 0px;"></div><div style="margin: 0px;"><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Times,'Times New Roman',serif; font-size: large;"><b>Coding Mapper and Reducer </b></span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We can code the mapper and reducer now. You can use any IDE you like, I used NetBeans 6.9.1 for this purpose. </span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Make sure that hadoop-common-0.21.0.jar and hadoop-mapred-0.21.0.jar are in the project library.</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Then, code the mapper as follow:</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"></span></span><br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"><span class="Apple-style-span" style="font-family: inherit;"> 1 </span></span></span><span class="Apple-style-span" style="font-family: inherit;">
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 2 </span></span><span style="color: #6600cc;">/**</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 3 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">The</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">class</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">that</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">maps</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">an</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">input</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">map</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">task</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">into</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">set</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">integer</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">that</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">represents</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">existence</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">an</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">origin</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">in</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">line</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">HDFS</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 6 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">file</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 7 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 8 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@author</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">arizal</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 9 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*/</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span><span style="color: #006699;"><b>public</b></span> <span style="color: #0099ff;"><b>class</b></span> FlightOriginMapper
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: #006699;"><b>extends</b></span> Mapper<span style="color: black;"><b><</b></span>LongWritable, Text, Text, IntWritable<span style="color: black;"><b>></b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 12 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 13 </span></span> <span style="color: #6600cc;">/**</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">Maps</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">first</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">3</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">characters</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">line</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">to</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">1</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 14 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">The</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">line</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">PARCDGFRAFRA</span><span style="color: #6600cc;">-</span><span style="color: #6600cc;">02022011</span><span style="color: #6600cc;">/</span><span style="color: #6600cc;">06022011</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">maps</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">to</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">(</span><span style="color: #6600cc;">PAR</span><span style="color: #6600cc;">,</span><span style="color: #6600cc;">1</span><span style="color: #6600cc;">)</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 15 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@param</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">key</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">key</span><span style="color: #6600cc;">,</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">it</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">is</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">not</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">used</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">in</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">mapper</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 16 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@param</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">value</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">value</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">that</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">corresponds</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">to</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">line</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 17 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">file</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 18 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@param</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">context</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">execution</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">context</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 19 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@throws</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">IOException</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">exception</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">thrown</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">on</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">IO</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">problem</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 20 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@throws</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">InterruptedException</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">thrown</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">when</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">execution</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">is</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 21 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">interrupted</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 22 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*/</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 23 </span></span> <span style="color: #66ccff;"><b>@</b></span><span style="color: #66ccff;"><b>Override</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 24 </span></span> <span style="color: #006699;"><b>public</b></span> <span style="color: #0099ff;"><b>void</b></span> <span style="color: #9966ff;">map</span><span style="color: black;"><b>(</b></span>LongWritable key, Text value, Context context<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 25 </span></span> <span style="color: #006699;"><b>throws</b></span> IOException, InterruptedException <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 26 </span></span> String line <span style="color: black;"><b>=</b></span> value.<span style="color: #9966ff;">toString</span><span style="color: black;"><b>(</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 27 </span></span> <span style="color: #ff8400;">//</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">discard</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">any</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">line</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">whose</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">length</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">less</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">than</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">3.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 28 </span></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span>line.<span style="color: #9966ff;">length</span><span style="color: black;"><b>(</b></span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>></b></span> <span style="color: red;">3</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 29 </span></span> <span style="color: #ff8400;">//</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">pickup</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">the</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">origin</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 30 </span></span> String origin <span style="color: black;"><b>=</b></span> line.<span style="color: #9966ff;">substring</span><span style="color: black;"><b>(</b></span><span style="color: red;">0</span>, <span style="color: red;">3</span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 31 </span></span> <span style="color: #ff8400;">//</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">then</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">provides</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">the</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">output</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">of</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">(origin,</span><span style="color: #ff8400;"> </span><span style="color: #ff8400;">1)</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 32 </span></span> context.<span style="color: #9966ff;">write</span><span style="color: black;"><b>(</b></span><span style="color: #006699;"><b>new</b></span> <span style="color: #9966ff;">Text</span><span style="color: black;"><b>(</b></span>origin<span style="color: black;"><b>)</b></span>, <span style="color: #006699;"><b>new</b></span> <span style="color: #9966ff;">IntWritable</span><span style="color: black;"><b>(</b></span><span style="color: red;">1</span><span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 33 </span></span> <span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 34 </span></span> <span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 35 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 36 </span></span></span></span></pre></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"></span></span><br />
<pre><span style="color: black;">
</span></pre></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The code for reducer receives the iterator of pair (origin, [ list of integer ] ). The reducer then reduces the pair to (origin, size of list). </span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"><span class="Apple-style-span" style="font-family: inherit;"> 1 </span></span></span><span style="color: #6600cc;"><span class="Apple-style-span" style="font-family: inherit;">/**</span></span><span class="Apple-style-span" style="font-family: inherit;">
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 2 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">The</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">reducer</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">that</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">reduces</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">list</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">integer</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">corresponding</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 3 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">to</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">existence</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">an</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">origin</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">in</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">line</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">an</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">HDFS</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">file</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">The</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">reducer</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">may</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">also</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">be</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">used</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">as</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">combiner</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">to</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">compute</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">total</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">number</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">origin</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">appearance</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">computed</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">by</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">a</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">mapper</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 6 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 7 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@author</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">arizal</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 8 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*/</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 9 </span></span><span style="color: #006699;"><b>public</b></span> <span style="color: #0099ff;"><b>class</b></span> FlightOriginReducer
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span> <span style="color: #006699;"><b>extends</b></span> Reducer<span style="color: black;"><b><</b></span>Text, IntWritable, Text, IntWritable<span style="color: black;"><b>></b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 11 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 12 </span></span> <span style="color: #6600cc;">/**</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">Executes</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">reduction</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 13 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 14 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@param</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">key</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">key</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">mapped</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">values</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 15 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@param</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">values</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">values</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">mapped</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">by</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">mapper</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 16 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@param</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">context</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">context</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">of</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">execution</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 17 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@throws</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">IOException</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">exception</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">thrown</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 18 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">on</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">IO</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">operation</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 19 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@throws</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">InterruptedException</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">exception</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">thrown</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">when</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 20 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">reduction</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">is</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">interrupted</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 21 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*/</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 22 </span></span> <span style="color: #66ccff;"><b>@</b></span><span style="color: #66ccff;"><b>Override</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 23 </span></span> <span style="color: #006699;"><b>public</b></span> <span style="color: #0099ff;"><b>void</b></span> <span style="color: #9966ff;">reduce</span><span style="color: black;"><b>(</b></span>Text key, Iterable<span style="color: black;"><b><</b></span>IntWritable<span style="color: black;"><b>></b></span> values,
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 24 </span></span> Context context<span style="color: black;"><b>)</b></span> <span style="color: #006699;"><b>throws</b></span> IOException, InterruptedException <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 25 </span></span> <span style="color: #0099ff;"><b>int</b></span> sum <span style="color: black;"><b>=</b></span> <span style="color: red;">0</span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 26 </span></span> <span style="color: #006699;"><b>for</b></span> <span style="color: black;"><b>(</b></span>IntWritable value: values<span style="color: black;"><b>)</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 27 </span></span> sum <span style="color: black;"><b>=</b></span> sum <span style="color: black;"><b>+</b></span> value.<span style="color: #9966ff;">get</span><span style="color: black;"><b>(</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 28 </span></span> <span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 29 </span></span> context.<span style="color: #9966ff;">write</span><span style="color: black;"><b>(</b></span>key, <span style="color: #006699;"><b>new</b></span> <span style="color: #9966ff;">IntWritable</span><span style="color: black;"><b>(</b></span>sum<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 30 </span></span> <span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 31 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 32 </span></span></span></span></pre></div><div style="margin: 0px;"></div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Finally, we prepare the driver class that glues the mapper and reducer:</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"></span></span><br />
<pre><span style="color: black;"><span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"><span class="Apple-style-span" style="font-family: inherit;"> 1 </span></span></span><span style="color: #6600cc;"><span class="Apple-style-span" style="font-family: inherit;">/**</span></span><span class="Apple-style-span" style="font-family: inherit;">
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 2 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">The</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">driver</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">class</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">for</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">flight</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">origin</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">map</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">reduce</span><span style="color: #6600cc;">.</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 3 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@author</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">arizal</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 4 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*/</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span><span style="color: #006699;"><b>public</b></span> <span style="color: #0099ff;"><b>class</b></span> Main <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 6 </span></span> <span style="color: #6600cc;">/**</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 7 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 8 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*</span><span style="color: #6600cc;"> </span><span style="color: #02b902;">@param</span><span style="color: #02b902;"> </span><span style="color: #cc6600;">args</span><span style="color: #02b902;"> </span><span style="color: #6600cc;">the</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">command</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">line</span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">arguments</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 9 </span></span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;"> </span><span style="color: #6600cc;">*/</span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 10 </span></span> <span style="color: #006699;"><b>public</b></span> <span style="color: #006699;"><b>static</b></span> <span style="color: #0099ff;"><b>void</b></span> <span style="color: #9966ff;">main</span><span style="color: black;"><b>(</b></span>String[] args<span style="color: black;"><b>)</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 11 </span></span> <span style="color: #006699;"><b>throws</b></span> IOException, InterruptedException, ClassNotFoundException <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 12 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 13 </span></span> <span style="color: #006699;"><b>if</b></span> <span style="color: black;"><b>(</b></span>args.length <span style="color: black;"><b>!</b></span><span style="color: black;"><b>=</b></span> <span style="color: red;">2</span><span style="color: black;"><b>)</b></span> <span style="color: black;"><b>{</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 14 </span></span> System.err.<span style="color: #9966ff;">println</span><span style="color: black;"><b>(</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 15 </span></span> <span style="color: #ff00cc;">"</span><span style="color: #ff00cc;">Usage:</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">Flight</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"><input</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">path></span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;"><output</span><span style="color: #ff00cc;"> </span><span style="color: #ff00cc;">path></span><span style="color: #ff00cc;">"</span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 16 </span></span> System.<span style="color: #9966ff;">exit</span><span style="color: black;"><b>(</b></span><span style="color: black;"><b>-</b></span><span style="color: red;">1</span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 17 </span></span> <span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 18 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 19 </span></span> Job job <span style="color: black;"><b>=</b></span> <span style="color: #006699;"><b>new</b></span> <span style="color: #9966ff;">Job</span><span style="color: black;"><b>(</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 20 </span></span> job.<span style="color: #9966ff;">setJarByClass</span><span style="color: black;"><b>(</b></span>Main.<span style="color: #0099ff;"><b>class</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 21 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 22 </span></span> FileInputFormat.<span style="color: #9966ff;">addInputPath</span><span style="color: black;"><b>(</b></span>job, <span style="color: #006699;"><b>new</b></span> <span style="color: #9966ff;">Path</span><span style="color: black;"><b>(</b></span>args[<span style="color: red;">0</span>]<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 23 </span></span> FileOutputFormat.<span style="color: #9966ff;">setOutputPath</span><span style="color: black;"><b>(</b></span>job, <span style="color: #006699;"><b>new</b></span> <span style="color: #9966ff;">Path</span><span style="color: black;"><b>(</b></span>args[<span style="color: red;">1</span>]<span style="color: black;"><b>)</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 24 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 25 </span></span> job.<span style="color: #9966ff;">setMapperClass</span><span style="color: black;"><b>(</b></span>FlightOriginMapper.<span style="color: #0099ff;"><b>class</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 26 </span></span> job.<span style="color: #9966ff;">setReducerClass</span><span style="color: black;"><b>(</b></span>FlightOriginReducer.<span style="color: #0099ff;"><b>class</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 27 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 28 </span></span> job.<span style="color: #9966ff;">setOutputKeyClass</span><span style="color: black;"><b>(</b></span>Text.<span style="color: #0099ff;"><b>class</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 29 </span></span> job.<span style="color: #9966ff;">setOutputValueClass</span><span style="color: black;"><b>(</b></span>IntWritable.<span style="color: #0099ff;"><b>class</b></span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 30 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 31 </span></span> job.<span style="color: #9966ff;">waitForCompletion</span><span style="color: black;"><b>(</b></span><span style="color: #cc00cc;">true</span><span style="color: black;"><b>)</b></span>;
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 32 </span></span> <span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 33 </span></span><span style="color: black;"><b>}</b></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: black;"> 34 </span></span>
<span style="background-color: #dbdbdb; border-right: 2px solid black; margin-right: 5px;"><span style="color: #990066;"> 35 </span></span></span></span></pre><br />
<span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span><br />
<div style="margin: 0px;"></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">Put the three classes into a jar, for example mapreduce1.jar and we now have the program. </span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">To launch the program we need to prepare the input files. We need to launch the nodes ,e.g. using bin/hadoop/start-all.sh again. In this example, we assume that there are two log files: flight-1.log and flight-2.log. The two files are copied first to HDFS using the following commands:</span></div><div style="margin: 0px;"></div><div style="margin: 0px;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">bin/hadoop dfs -copyFromLocal /home/arizal/flightlog/flight-1.log flightlog</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">bin/hadoop dfs -copyFromLocal /home/arizal/flightlog/flight-2.log flightlog</span></div><div><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">To verify that the two files are really taken into account, use the dfs -ls as follow:</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">bin/hadoop dfs -ls /user/arizal/flightlog</span></div><div style="font-family: Arial,Helvetica,sans-serif; margin: 0px;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">That returns:</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">Found 2 items</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"></span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">-rw-r--r-- 1 arizal supergroup 64792 2010-12-05 22:32 /user/arizal/flightlog/flight-1.log</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">-rw-r--r-- 1 arizal supergroup 64792 2010-12-05 22:49 /user/arizal/flightlog/flight-2.log</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span></div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">At this point, we're ready to launch the mapreduce job. This can be done using the following command:</span><br />
<div style="margin: 0px;"></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">$ bin/hadoop jar /home/arizal/NetBeansProjects/mapreduce1/dist/mapreduce1.jar flightlog origin-out</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">The command tells hadoop to execute the main class inside mapreduce1.jar with two arguments flightlog and origin-out. The flightlog is used by the program as the path to be used and origin-out is used to be the output file path. From the preparation that we did before, we have two files inside flightlog directory (flight-1.log and flight-2.log). </span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">When the command is launched, we have:</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO input.FileInputFormat: Total input paths to process : 2</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.JobSubmitter: number of splits:2</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.Job: Running job: job_201012052231_0002</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.Job: map 0% reduce 0%</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.Job: map 50% reduce 0%</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.Job: map 100% reduce 0%</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.Job: map 100% reduce 100%</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.Job: Job complete: job_201012052231_0002</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">INFO mapreduce.Job: Counters: 33</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>FileInputFormatCounters</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>BYTES_READ=129584</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>FileSystemCounters</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>FILE_BYTES_READ=41806</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>FILE_BYTES_WRITTEN=83682</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>HDFS_BYTES_READ=129826</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>HDFS_BYTES_WRITTEN=41</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Shuffle Errors</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>BAD_ID=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>CONNECTION=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>IO_ERROR=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>WRONG_LENGTH=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>WRONG_MAP=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>WRONG_REDUCE=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Job Counters </span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Data-local map tasks=2</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Total time spent by all maps waiting after reserving slots (ms)=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Total time spent by all reduces waiting after reserving slots (ms)=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>SLOTS_MILLIS_MAPS=18785</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>SLOTS_MILLIS_REDUCES=6104</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Launched map tasks=2</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Launched reduce tasks=1</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Map-Reduce Framework</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Combine input records=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Combine output records=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Failed Shuffles=0</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>GC time elapsed (ms)=26</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Map input records=4184</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Map output bytes=33440</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Map output records=4180</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Merged Map outputs=2</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Reduce input groups=5</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Reduce input records=4180</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Reduce output records=5</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Reduce shuffle bytes=41812</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Shuffled Maps =2</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Spilled Records=8360</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>SPLIT_RAW_BYTES=242</span></div><div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">We can check the output in the origin-out directory using hadoop dfs -cat command:</span><br />
<div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">bin/hadoop dfs -cat /user/arizal/origin-out/part-r-00000</span></div><div style="margin: 0px;"></div><div style="font-family: Arial,Helvetica,sans-serif; font-size: medium; margin: 0px;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">In my case, it returns:</span></div><div style="margin: 0px;"></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">LYS 160</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">MRS 84</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">NCE 1668</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">PAR 1676</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;">TLS 592</span></div><div style="font-family: Arial,Helvetica,sans-serif; font-size: medium;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"></span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial,Helvetica,sans-serif;">OK. This ends this "Hello Hadoop" blog. I'll try to come back with couple other things. Perhaps Avro or Thrift. </span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace; font-size: x-small;"><br />
</span></div><div style="font-family: 'Times New Roman'; font-size: medium; margin: 0px;"></div></div></div></div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com7tag:blogger.com,1999:blog-2012573949883447809.post-50643651672294706152010-11-23T23:42:00.000+01:002012-01-19T07:15:03.709+01:00Devoxx 2010 Wrap Up (Epilog)<div style="font-family: Arial,Helvetica,sans-serif;">This post ends my blogs around Devoxx 2010 that has just ended on last Friday. Overall, it was an excellent conference, very informative, and refreshing too.</div><div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div><div style="font-family: Arial,Helvetica,sans-serif;">If there are two things to retain from the content of the conference -- at least for me -- it would be Scala and NoSQL. I believe that these two subjects have taken so much interests from many people who came to the conference. </div><div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div><div style="font-family: Arial,Helvetica,sans-serif;">The first Scala touch was on Scala lab session on 2nd day that were packed before 9.30. Couple of programmer fellows simply could not join the session because of that. The trends were confirmed in the evening, at BOF session where again, many conference attendees needed to stay out of the room. Two Scala 2.8 presentations the next day confirmed even more the popularity of Scala in the conference. Akka presentation by Viktor Klang might not have the same success to the others, but it was still a massive achievement for a specialized library outside the language itself. Not to mention that Scala was also mentioned by Brian Goetz during his session, by Josh Bloch/Bill Pugh in Java Puzzlers too. <br />
For the content of the presentation, I would mention Odersky presentation on the Scala 2.8 collection as my favorite presentations on the subject. The Odersky's presentation has shown the power of high kinded Scala programming language -- some are scary things -- but it shows how productive a Scala programmer might be.<br />
<br />
<div style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_N2iy2ISbG0Qy8eBmR1Xj_usxQVNSKoHxrI80n8TfcjFTxQ85dUWzmMAuhUxuU39Ya6_vE9ss-yZUJPY5yY6rOgMJBc66np-mrA4TS32B30XJsBM_BhnjZKxwt3GYfda8VyDyIdimO_Qe/s1600/languages.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_N2iy2ISbG0Qy8eBmR1Xj_usxQVNSKoHxrI80n8TfcjFTxQ85dUWzmMAuhUxuU39Ya6_vE9ss-yZUJPY5yY6rOgMJBc66np-mrA4TS32B30XJsBM_BhnjZKxwt3GYfda8VyDyIdimO_Qe/s320/languages.jpg" width="320" /></a> </div><br />
Fortunately, not all things about Scala were good. Stephen Colebourne, for example, said that Scala would not be the next JVM language. I'm not sure about the reason yet, but it might be because Scala has done too much than what the Java programmers wait. Other problem was when Scala seems to become victim on his popularity when tweets on Scail - a fictitious new Scala web framework - flooded the Internet. The joke came from Scala BOF, that made association Ruby => Rail, Groovy => Grail, Scala => Scail. I was also the victim of the jokes when I retweeted a tweet on the joke - Damn !<br />
<br />
The second subject of the conference was nosql. They were just fantastically represented. We had Hadoop, HBase, MongoDB, Cassandra, and Voldemort. We had even Mahout, one of the Hadoop sub project . I would like also to add Elastic Search to the same category. I attended almost all nosql sessions, except Cassandra and Voldemort that unfortunately were at the same time as Odersky's Scala Collection and Goetz's Project Lambda. On the other hand, I attended the two sessions of Tom White from Cloudera on Hadoop, the two sessions of Jonathan Gray on HBase (and Facebook). I would even dare to say that Tom White was one of my favorite speaker in the conference. I would say that I'm particularly impressed by the Nosql movements although still have some reserves on the subject.<br />
<br />
The nosql trends are popular because of its ability to scale out in the presence the world of very huge amount of data, pentabyte. Twitter, Facebook, Youtube, ... are the most visible use case of such application, but there are more than those guys. Twitter and Facebook had interesting presentations where Dimitri Ryaboy and Jonathan Gray showed an impressive list of on going projects on the subject (yes, Facebook announce on its HBase use just before the conference played in favor of nosql again) . No doubt, the movements will play important roles in the near future. I myself, am very interested in a specific conference on the subject, called Buzzwords that will take place in Berlin 2011.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgORf2QjKsYmldvIisnxCvVYEeIH8UwDsNfwLZmi4N7eXUe3YaYGNdPFVlj92Gch6QZpwxuKFLEhGX3KdguIC3O4bi4NPzn1U1Cu79jJWIytgH5Y6vM9MHDhgU5wuhNKz4LY2r2OLxAxEjT/s1600/mr.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgORf2QjKsYmldvIisnxCvVYEeIH8UwDsNfwLZmi4N7eXUe3YaYGNdPFVlj92Gch6QZpwxuKFLEhGX3KdguIC3O4bi4NPzn1U1Cu79jJWIytgH5Y6vM9MHDhgU5wuhNKz4LY2r2OLxAxEjT/s320/mr.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_N2iy2ISbG0Qy8eBmR1Xj_usxQVNSKoHxrI80n8TfcjFTxQ85dUWzmMAuhUxuU39Ya6_vE9ss-yZUJPY5yY6rOgMJBc66np-mrA4TS32B30XJsBM_BhnjZKxwt3GYfda8VyDyIdimO_Qe/s1600/languages.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br />
</a></div><br />
<br />
Of course, there were not only Scala and nosql in the conference.<br />
There were the future of Java where Mark Reinhold, Brian Goetz, and Dalibor Topic played important roles during the conference with their sessions on Java Modularity, Project Lambda, and Open JDK. Couple of interesting things from their presentations (note that, the cinema was full for Brian Goetz').<br />
<br />
And of course, the Java Puzzle session by Josh Bloch and Bill Pugh, who could forget such inspiring session. Thanks for entertaining and enlightening us. <br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1eDFzHLOQQXQ7ou9RajFtR5EI8gF75R8nhkJAEubUBRb_uOA9OA1BQKy7XeyJT_7asCS1hBtDFzJBI4mKQr6lL3MnyT0Nddpnoki8e24HgCOXT05Bt67paAwLvxXdiqnwdhbh8dXeztQY/s1600/puzzlers.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1eDFzHLOQQXQ7ou9RajFtR5EI8gF75R8nhkJAEubUBRb_uOA9OA1BQKy7XeyJT_7asCS1hBtDFzJBI4mKQr6lL3MnyT0Nddpnoki8e24HgCOXT05Bt67paAwLvxXdiqnwdhbh8dXeztQY/s320/puzzlers.jpg" width="320" /></a></div><br />
</div><div style="font-family: Arial,Helvetica,sans-serif;">I would like also to mention an interesting presentation on Apache Camel by Claus Ibsen. Camel is awesome. Finally, a pretty cool presentation was made by Talip Ozturk on Hazelcast.<br />
-<br />
<br />
While the contents were excellent, there were still many things that could be improved. First, the transport. Since there were not enough hotels around the conference, a lot of people come from hotels in the center of Antwerp. The trams from city center to conference venue were simply too small and overloaded in the morning. Buses specific for the conference would have been better.<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjF6URgQAbGxPp_T_z56VE2UgiLVbDXkU2_2ItQiMa1IKTcOlfN1bd36feFzPAxsTI9ZvnO_ZP3Xf1sPRi1SuBOL_YaIKZC6GH1ohchEg5rSXue8R6-YnvDqd-JS6uUtWB3LuAslhW-Estu/s1600/tram.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjF6URgQAbGxPp_T_z56VE2UgiLVbDXkU2_2ItQiMa1IKTcOlfN1bd36feFzPAxsTI9ZvnO_ZP3Xf1sPRi1SuBOL_YaIKZC6GH1ohchEg5rSXue8R6-YnvDqd-JS6uUtWB3LuAslhW-Estu/s320/tram.jpg" width="320" /></a></div><br />
<br />
</div><div style="font-family: Arial,Helvetica,sans-serif;">Finally, I would like to thank to SII, my employer who has financed my travel to the conference, and of course would love to come back to the conference next year. Thank you too for my colleagues at Amadeus who have shown the interests on the conference. Needless to say however, all opinions in this blog during the conference are fully mine, they are not from SII nor my client, Amadeus. </div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-47951563794294117782010-11-23T12:11:00.000+01:002012-01-19T07:15:03.701+01:00Notes on Devoxx 5th day<span style="font-family: Arial,Helvetica,sans-serif;">The fifth day notes is here, finally. I've been so lazy to write the 5th day notes, you know, kind of post-conference syndrome.</span><br />
<br />
<span style="font-family: Arial;">There were only 4 parallel sessions compared to the other days where there were 6, and the conference ended at 1 pm, compared to 7 pm the other days.</span><br />
<br />
<br />
<b><span style="font-family: Arial,Helvetica,sans-serif; font-size: large;">Keynote Panel on the Future of Java by Joshua Bloch, Mark Reinhold, Stephen Colebourne, Antonio Goncalves, Juergen Hoeller and Bill Venners</span></b><br />
<span style="font-family: Arial; font-size: large;"></span><br />
<br />
<ul><li><span style="font-family: Arial; font-size: large;"><span style="font-size: small;">Josh Bloch worked for Sun before, and he was the architect of the Java Collection, now he's working for Google. Author of Effective Java.</span></span></li>
<li><span style="font-family: Arial; font-size: large;"><span style="font-size: small;">Mark Reinhold, from Oracle, he's been in Java longer than Josh. </span></span></li>
<li><span style="font-family: Arial; font-size: large;"><span style="font-size: small;">Stephen Colebourne is a Java champion and active in Apache Software Foundation, he is kind of sitting between the two worlds: Oracle and ASF. During the conference he created an important buzz by telling that "Scala will <b>not</b> be the future Java".</span></span></li>
<li><span style="font-family: Arial; font-size: large;"><span style="font-size: small;">Antonio Goncalves represented the Java community. He's the Paris JUG leader.</span></span></li>
<li><span style="font-family: Arial; font-size: large;"><span style="font-size: small;">Juergen Hoeller is from Spring Framework. </span></span></li>
<li><span style="font-family: Arial; font-size: large;"><span style="font-size: small;">Bill Venners represented the new languages on JVM, e.g. Scala.</span></span></li>
</ul><span style="font-family: Arial; font-size: large;"><span style="font-size: small;">The panelists were asked couple of questions coming from Devoxx whiteboards or from a site specifically built for the panel. </span></span><br />
<br />
<span style="font-family: Arial;">There were couple of interesting questions like how Oracle would be perceived by Java community, Android, OSGi vs Jigsaw, around JCP and of course Doug Lea departure, impact of functional programming to Java, a modification in Java reflection, what Java does not have compared to .NET, whether Java should keep its backward compatibility, until what is a typical Java programmer.</span><br />
<br />
<span style="font-family: Arial;">The panel was really interesting, Stephen talked a lot, Bill talked a little. The others were in the average. I admired Mark Reinhold that was pretty available to answer the questions, while actually he could have avoided some. </span><br />
<br />
<span style="font-family: Arial;">If there are two things that I should retain from the panel, it would be TCK: No restriction on TCK would be lifted, that's final (bye ASF), and the wait and see of Java User Group regarding the possiblity to be included into Oracle User Group. I should also mention that Josh said "but Java has already functional programming capabilities". </span><br />
<br />
<span style="font-family: Arial;">At the end, the panel was quite informative, and it was nice to see Josh Bloch and Mark Reinhold sat side by side. The image would be good for Java community.</span><br />
<br />
<span style="font-family: Arial; font-size: large;"><b>Apache Camel by Claus Ibsen</b></span><br />
<br />
<span style="font-family: Arial;">As a user of JMS, JMS is damn complicated. The messaging should not be that complicated. That's exactly what Apache Camel proposes: simplifying integration of n-tier systems.It is done through Enterprise Integration Pattern, which is a catalog of patterns. Messaging, Remote Invocation, Message Filter are some random picks of the patterns.</span><br />
<br />
<span style="font-family: Arial;">Oh, yes. Apache Camel has Scala DSL and from the Akka presentation the day before, Akka supports Apache Camel too.</span><br />
<br />
<span style="font-family: Arial;">Quite a cool presentation although I didn't know Enterprise Integration Pattern that much, but this one is worth to have a look though.</span><br />
<br />
<span style="font-family: Arial; font-size: large;"><b>Elastic Search by Shay Banon</b></span><br />
<span style="font-family: Arial; font-size: large;"><br />
<span style="font-size: small;">Elastic Search is a free-text search engine, specially designed for cloud environment. It is a Lucene-powered system, and in this regard it's just like Solr or Hibernate Search. </span><br />
<span style="font-size: small;">Unlike Hibernate Search, however, elastic search works on document-oriented database.</span><br />
<span style="font-size: small;">The data model is represented in JSON, the query is also in JSON DSL.</span><br />
<span style="font-size: small;">Finally, the elastic search is distributed, so Shay explained the index replication algorithms (sharding, etc.).</span><br />
<br />
--<br />
<span style="font-size: small;">Devoxx ended by attending Elastic Search and Apache Camel sessions. Quite good presentations to finish the conference. I took some times to visit Diamant Museum in Antwerp downtown and then flied back to Nice from Brussels in the evening. </span><br />
<span style="font-size: small;"></span></span>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-23086105846316887422010-11-19T03:48:00.000+01:002012-01-19T07:15:03.653+01:00Notes on Devoxx 4th day<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">This blog might be my last daily note on Devoxx university and conference. Everything has an end, and devoxx must end tomorrow (Friday, November 19). I'm not sure to write notes on tomorrow sessions, and if I do, it will be for Saturday at the earliest.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">After having difficulties in public transport yesterday, this morning I left a little bit later with the risk of missing the first couple of minutes of keynote speaker. </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">So, here is my summary of the day. </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span><br />
<span class="Apple-style-span" style="font-size: large;"><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Future Roadmap of JEE (Keynote), Jerome Dochez, Linda de Michiel, Paul Sandoz</span></b></span><br />
<span class="Apple-style-span" style="font-size: large;"><span class="Apple-style-span" style="font-family: inherit;"><b><i><br />
</i></b></span></span><br />
<b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><i>JEE on Cloud (Jerome Dochez, JD)</i></span></b><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">When I arrived at the conference, Jerome Dochez was presenting JEE on cloud environment. He mentioned that the cloud support should not be revolutional, but should be an evolution. Programmers should not be asked to change a lot of things from what they have known so far.</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">At least two things he mentioned particularly: State Management and Better packaging.</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">He finished the JEE on Cloud presentation part by running a small and successful demo on GlassFish.</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>Modularity (JD)</i></b></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">There are important efforts on going on making JEE more modular, especially to leverage the development on Java Modularity in general (Jigsaw). Unfortunately, the dependencies to Jigsaw means that modularity on JEE would also be late. </span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Some points on modularity that I noted:</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Applications are made of modules (modules in term of Jigsaw)</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Dependencies are made of explicit instead of by convention of configuration<i style="font-weight: bold;">.</i></span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Versionings are built in.</span></li>
</ul><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>JSF (JD)</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">He mentioned two kinds of modifications: short and long term. Some of the short term ones:</span></div><div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Transient state saving</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">XML view cleanup</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Facelets cache API</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">XML free (oops, it was written fee in his slide, but yes, it is to removed XML taxes)</span></li>
</ul><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">He mentioned also about support of HTML 5. I'm not sure whether it is shorter or longer term.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>JMS (JD)</i></b></span></div></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">There has been almost no important modifications on JMS, and this time it will change, there will be important modifications on JMS. The modifications include ambiguities resolution, standardize couple of vendor extensions, integration with other specs and also to non-Java languages (which ones ?).</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>Web Tier (JD)</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">WebSocket support, Standard JSON API, and NIO2-based web container (I'm not sure to understand the relation between NIO2 and web container, but anyway ...). He mentioned Grizzly library in his presentation.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>JPA (Linda De Michiels)</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">There are couple of interesting things in her talks. I noted some interesting things only here:</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>Mapping:</i></b></span></div><div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Support on custom mappings.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Dynamic fetching plan. This is contrast to JPA today that requires fetching definition upfront using annotation (EAGER, LAZY).</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Better support for immutable attributes (read-only entities)</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">More flexible XML descriptors. Yeah, XML, why don't make it Java based ??</span></li>
</ul><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>API:</i></b></span></div></div><div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Additional event listeners and callbacks<i style="font-weight: bold;"> </i><i>Really ?? Are they really used in production ?)</i></span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Support for dynamic persistence unit.<b style="font-style: italic;"> </b><span style="font-style: italic;">This one is cool, no XML, right ?</span></span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Inspection on persistence unit.<b style="font-style: italic;"> </b><span style="font-style: italic;">Cool as well</span></span></li>
</ul><div><div><div style="font-family: 'Times New Roman';"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>Query:</i></b></span></div><div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Stored procedure support.<b style="font-style: italic;"> </b></span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Interoperate JPQL and criteria queries<b style="font-style: italic;">. </b><span style="font-style: italic;">For example, create criteria query from JPQL</span><b style="font-style: italic;">.</b></span></li>
</ul><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For me, all programmatic supports like dynamic fetching and persistence units are the most interesting improvements.</span></div></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i>JAX ??? (Paul Sandoz)</i></b></span></div><div><div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I did not really follow the last part of the presentation, no notes I could share here.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><i><br />
</i></b></span></div><div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: large;"><span class="Apple-style-span" style="font-family: inherit;"><b><i><br />
</i></b></span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: large;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>The Essence of Caching by Greg Luck</b></span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><div style="display: inline !important;"><div style="display: inline !important;"><b><i><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"></span></i></b><br />
<div style="display: inline !important;"><div style="display: inline !important;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><b><i><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"></span></i></b></span></span><br />
<div style="display: inline !important; font-family: 'Times New Roman';"><div style="display: inline !important;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><i>A side story -- this is the only presentation so far where the company I'm working for, Amadeus, is mentioned. Yes !! :-)</i></span></span></span></span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><i><br />
</i></span></span></span></span></div></div></div></div></div></div><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><b><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><i>Why caching ? </i></span></span></b></b></span></span></b><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Because of performance problem</span></span></span></span><br />
<b><i><br />
</i></b><br />
<div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><i><b></b></i></div><div style="display: inline !important;"><div style="display: inline !important;"><b><i><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"></span></i></b><br />
<div style="display: inline !important;"><div style="display: inline !important;"><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><b><b><i><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"></span></i></b></b></span></span></b><br />
<div style="display: inline !important; font-family: 'Times New Roman';"><div style="display: inline !important;"><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><b><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><i>Amdahl's Law</i></span></span></b></b></span></span></b></div></div></div></div></div></div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">What important to keep in mind on performance optimization is the Amdahl's Law: </span></span></span></span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Speedup = 1 / ( ( 1 - f) + ( f / s) ) where f is the proportion of program being sped up. Illustration: Making 10% of portion of the system 20 times faster makes 1.105 overall improvement.</span></span></span></span><br />
<br />
<div style="font-family: 'Times New Roman'; font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-weight: normal;">The law is important in deciding the part of the system to be sped up. For example, if the problem of downloading a page is on the downloading of its content, there is no point of improving the server side code. Maybe CDN is needed in this case.</span></div><div style="font-family: 'Times New Roman'; font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-weight: normal;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Performance Problem Sources</b></span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-weight: normal;"> </span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><br />
<ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Rendering</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Program</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Marshalling/unmarshalling data.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Database</span></li>
</ul><br />
<div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Caching solved the problem mainly by offloading some data to a cache, e.g. to memory-based cache.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Cache Efficiency</b> </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Cache efficiency = Cache Hits/Total Hits. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Needs to take into account pareto principle.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Cache Coherency</b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">To handle cache coherency, one of the simplest solution is by applying TTL + LRU</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">. But there are other strategy: Eternal item + invalidation strategy, write through pattern.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Cache in Clustered Environment</b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">A problem called N* problem is inherent in cache in clustered enviroment. The problem is by also clustering the caching. This introduces another problem: bootstrap problem and cache coherency again.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">CAP Theorem</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">CAP = Consistency, Availability, Partition Tolerance => there must be trade off of the three in clustered environment.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The session was very interesting and informative, all that only in one hour. The number of audience in this session was quite important.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><div style="font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: large;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Akka by <i>V</i><span class="Apple-style-span">iktor Klang</span></span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><div style="font-family: 'Times New Roman'; font-size: medium; font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I saw couple of presentation of Akka online, and this morning I saw it live. Even with the pretty distracting Devoxx template, Akka presentation was still excellent. Great job by <i>V</i>iktor and Akka team. </span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Akka is designed taking into account that it is hard to make concurrent program right. Akka comes with two solutions: Actor and STM. Akka has Scala and Java implementation. In general Scala implementation is better, but Akka has succeeded in removing a lot of boiler plate codes almost inherent in Java.</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Actor</span></b></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Actor is a higher level abstraction on thread. It has an important property of shares nothing. So, one actor does not share anything with any other actor, so actors work in isolation (<i>unlike Clint Eastwood, George Clooney, ..., that cannot work in isolation, although they might share nothing too</i>). The communication between actors are through message passing. Each actor has mailbox where the message is queued.</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Three different types of message sending: sending fully one way, one way but with implicit future, and one way with explicit future. In Scala, they are represented by !, !!, !!! methods.</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For fault tolerance, Akka uses "let it crash semantics" stolen from Erlang. It also has a notion of supervisor hierarchy.</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Actor can be remote, and there are two types of remote actor: client-managed remote actor and server-managed remote actor. Client-managed actor is handy, but it of course cannot be deployed in untrusted enviroment. </span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Remote actor implementation is based by Netty and uses ProtoBuf.</span></div><div style="font-weight: normal;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Software Transactional Memory (STM)</span></b></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Akka supports STM. It provides couple of transactional data structure like transactional maps, transactional lists, and so on.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Java implementation uses a library called Multiverse.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The combination of Actor and STM called Transactor.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Miscellaneous</span></b></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Akka has a couple of interesting add-ons like Spring add-on, Camel , MongoDB, CouchDB, and couple of other things.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Viktor's presentation was awesome. Akka is awesome.</span></div><div style="font-size: medium;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-size: medium;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: large;">Data Management at Twitter Scale System by Dimitri Ryaboy</span></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Dimitri presentation was not only about Hadoop how and all its ecosystems are used in Twitter. Hadoop is appropriate for offline processing, but the presentation is not only about offline processing. It's also about online. The presentation was dense and presented very quickly that I had sometimes difficulties to follow. But, here are some notes:</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Twitter is 95 million tweets, with 3000 TPS = Tweets per seconds. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">First problem, UUID Generator. Twitter uses Snowflake: </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">https://github.com/twitter/snowflake. The issue is how to make the generator scalable. The UUID is not necessarily sorted, it should only be roughly sorted (k-sorted).</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Second problem, Sharding. Twitter uses Gizzard </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><a href="https://github.com/twitter/gizzard">https://github.com/twitter/gizzard</a> , a Scala-based framework for sharding. Sharding is, by the way, storing data accross multiple nodes. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Gizzard maps a range of tweet to a particular shard. A shard is mapped to a replication tree (<i>hmm..., not really sure to understand this, but, I write it here anyway</i>). Shard can be physical when it refers to a particular backend. It can also be logical, when it refers to other shards.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Third problem, Fault tolerance. I lost a little bit in this section. Not much I have in my note. The only thing I have in my note is the system must be tolerant of eventual consistency and stay CALM (Consistency as Logical Monotonicity). Hmm... this is pretty puzzle for now. But anyway, that was about fault tolerance.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Fourth problem, is about timeline treatment (message vector cache). To display timelines means that billions of tweets must be filtered to only show messages from the people one follows The solution: Haplochairus.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Because it is a cache: cache effeciency.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">FlockDB is the solution used for social graph store. It is basically a customized distributed index database. It is used to handle, e.g. intersection operations on @ . </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><a href="https://github.com/twitter/flockdb">https://github.com/twitter/flockdb</a> .</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Cassandra is used for geo database like nearby search and realtime data analysis.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Gaglia for monitoring.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For offline processing, Hadoop is used. Hadoop is appropriate for some analysis that cannot be achieved using SQL. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Elephant-Bird is used to work with data in Hadoop, and finally HBase is used to address mutability and random access in Hadoop.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Excellent presentation from Twitter engineer. Love it very much.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: large;">Hadoop, HBase, and Hive in Production at Facebook by Jonathan Gray</span></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The previous presentation came from Twitter and this one is from Facebook. First, Jonathan explained why HDFS / Hadoop. Basically, the choice came from the fact that traditional database processing was slower than the debit of incoming data: need 24 hours to process one day data. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The use of Hadoop introduced some other problems: difficult to write mapreduce jobs. Solution: Hive. So Hive is the datawarehouse solution for Facebook. But Hive is itself still not user friendly: fear of command line. HiPAL is introduced for querying using web UI.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Current limitation: name node is still a single point of failure. The high availability solution today is not enough because it takes </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">hours </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">for backup name node to fail over. Facebook is now working on something called AvatarNode. Jonathan claimed 10s of failover using AvatarNode.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Other limitation is a non-optimized map-reduce. Facebook is working on Hive optimization. Other problem is about better scheduling, called fairshares scheduler, that controls the task by its priority/nature. Jonathan claimed that queries at Facebook to be less than 10 minutes now.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"></span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">HBase is used because it's linearly scalable, fast indexed units, and integration with Hadoop. It is also suitable for realtime analysis because it has optimized increment. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Why Cassandra was not selected for Facebook Messaging ? There is a probblem of consistency at Cassandra that does not suit messaging requirements. HBase, is good. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: large;">Modularity in Java by Mark Reinhold</span></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Mark presentation this afternoon was really interesting. Unfortunately, I was so tired to write notes that I missed many points in his presentation. Instead of writing something completely wrong. Although my notes have many erros given the times used to write ones, but it should be OK for a blog. I'm afraid that if I write something here, it would completely be wrong, even for a blog. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">At the end of his presentation, Mark took some times explaining "Why not OSGi" that should answer many questions on the subject. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">At the end of his presentation, Mark Reinhold gave us one URL to follow: </span><a href="http://openjdk.java.net/projects/jigsaw/"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">http://openjdk.java.net/projects/jigsaw</span>/</a> . <span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> He said that the project was not that active because of JDK 7 delivery deadline, but it will come back soon.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: large;">Java Puzzler by William Pugh and Josh Bloch</span></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Before starting hist presentation, Mark started with a joke "The reason I'm here is to make sure to have the best place for Java Puzzle session". That should describe how popular this session is. Indeed, it was the most popular session so far at devoxx. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The two speakers came with 6 brand new puzzles around couple of subjects. Well, I will not write them here because it will not be fun and even if it is fun, it will take some times. The subjects they took are around generics, collections, raw type ,big decimal, and couple of other things. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">One thing that I keep in mind is: "Do not ignore warning of typing" issued by the compiler. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I answered 3 of 6 puzzle correctly, and considered one of them as cheating puzzle :-) , so 3 of 5, not bad, heh ? Yeah, not bad. But, to be honest, only one of them that I answered correctly with the correct explanation too.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">--</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">OK. Java Puzzler completed my day. It was just teriffic day. The best day at devoxx. Quite sad that it will end tomorrow... Back to Nice in the afternoon.</span></div></div></div></div></div></div><div style="font-family: 'Times New Roman'; font-style: normal;"><ul></ul></div></div></div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-44096322310815039222010-11-18T00:55:00.000+01:002012-01-19T07:15:03.688+01:00Notes on Devoxx 3rd day<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">We entered the conference today, and yes, it was a long queue everywhere: transportation, breakfast, vestiaire, session, everywhere. </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">But, the sessions were very interesting, at least majority of them.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Here is my summary.</span><br />
<br />
<b><span class="Apple-style-span" style="font-size: x-large;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Introduction by Stephan Janssen</span></span></b><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Cool introduction by Stephan Janssen and on Parleys. We can have the video of all presentations through Parleys for only 79 Euros. With that, you can watch Parleys presentation in the toilet, if you have iPad of course.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: x-large;">Keynotes by Mark Reinhold</span></span></b><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Mark Reinhold presented a lot of cool things on Java. I tweeted something that I regreted afterward, I tagged his presentation was disappointing. It was indeed disappointing for me, because I expected to have some new information on Java evolution. He didn't present anything new, but his presentation was an excellent presentation on Java evolution and roadmap. You cannot not have better presentation on the roadmap than his presentation.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> He started his presentation with a statement that Oracle wanted Java to exist until 2030. He then explained some axis of improvements: productivity, performance, universality, modularity, integration, servicability. </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For productivity, he cited some of the works on Project Coin, like the infamous <> symbol and Automatic Resource Management (ARM). </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For performance, he mentioned about some changes on the language by introducing the results from project lambda like lambda expression, method extension, and some of other things on the subject. But, as Brian Goetz confirmed later on, this must be done in the context of performance improvements in the context of multicore.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For universality, he mentioned about support of JVM for other languages. I saw more or less the same cloud of languages before in SophiaConf, with Scala and Groovy in relatively small size. I found the presentation on the subject was very short. There will be a lot sessions on new language on JVM at Devoxx, but no presentation on the corresponding JSR.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For modularity, he presented quickly the jigsaw project with the concept of simplifying classpaths with jmod. jmod install, jmod add-repo can be used. A little bit inspired by ruby gem, I suppose.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">For serviceability, he talked a little on a quite sensitive subject: JVM convergence.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">He mentioned two interesting things: about the possibility of having reification in Java and also value class (= Scala case class?). </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Overall, nothing really new in the presentation, but I think that was a comprehensive presentation that Java developers expect to have directly from the source.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: x-large;">The State of the Web by Dion Almaer and Ben Galbraith</span></b></span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">You know what? This was the most entertaining presentation so far in the conference. Small problem: it was so entertaining, so well choreographed, that I didn't really get the contents. </span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I didn't even take time to take some notes, the presentation was just too entertaining with graphisms and stuffs, really good for your eyes (but maybe not for your brain). </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">So, just couple Math.random points that I vaguely retained from the presentation.</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> </span><br />
<br />
<ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Application is content.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Mobile application becomes very important.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">HTML 5 is great.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Web needs app store model.</span></li>
</ul><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Not much really, so I should stop here too.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: x-large;">JPA by Linda de Michiel</span></b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: x-large;"></span></b></span><br />
<div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I think I made a mistake on coming to the session after reading back to back at least 2 times JPA book by Schincariol and Keith. All in Linda's presentation are in the book. Sorry, but I could not really tell much on the presentation, just read the book. It was an extremely dense presentation though.</span></span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: x-large;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></span></b></span></div></div><div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-size: x-large;">The State of Hadoop by Tom White</span></b></span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">In term of style, Tom White's presentation on Hadoop was completely opposite of Almaer & Galbraith presentation: Tom's style is very monotonic and without rythms. But, guess what? I got his contents better.</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Tom started by explaining the replication mechanisms in HDFS, followed by on how read and write are done. He made quick overview on the algorithms of the read and write that optimizes the bandwidth usage.</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Then, the explanation moved to failure modes, especially the differences between data node crash and name node crash. When data node crashes, client will read from other replicas and name node instructs data node to replicate. Name node crash is a much important issue because it means down time. There is an ongoing effort on high availability name node: </span><a href="https://issues.apache.org/jira/browse/HDFS-1064">https://issues.apache.org/jira/browse/HDFS-1064</a> .</div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The presentation continued to map reduce overview. Input => Map => Shuffle => Reduce => Output. Tom used an illustration of unix pipeline to illustrate the concept. Pretty cool.</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Tack tracker and job tracker failure modes were then explained. </span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">He showed also some examples on Hadoop use.</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> </span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The presentation moved forward to ecosystem. He showed an overly complicated project graphs showing the relations among the Hadoop projects. Very complex, but then he simplified (he mapped and reduced, right ?).</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Fundamental Projects</b></span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Some projects are fundamental projects. They are HDFS (the file system), map reduce frameworks, zoo keeper, and avro. Zoo keeper is a coordintion service for distributed application, including leader selection algorithms and distributed locking. Avro is data serialization library.</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The main challenges on the fundamental projects are the API update impacts and multi programming language support, typically Python.</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Component for Analysis</b></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Some projects in Hadoop are intended for analysis. We have Pig, Hive, Cascading, Mahout. Pig and Hive are the data retrieval components using SQL like language or a pretty independent query language (Pig). Mahout is a library for machine learning that implements some of map/reduce algorithms.</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Howl project is intended to share table among services.</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Components for Data Loading</b></span></div><div style="font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><div style="font-family: 'Times New Roman'; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Some projects are intended to facilitate data load components like Sqoop and Flume. </span></div><div style="font-family: 'Times New Roman'; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-weight: normal;"><b></b></span></span></div><div style="display: inline !important; font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-weight: normal;"><b></b></span></span></b></span></div><div style="display: inline !important; font-family: 'Times New Roman'; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Components for Coordination</span></b></span></b></span></div><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><div style="font-family: 'Times New Roman'; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Oozie and Whirr are examples of components that handle coordination.</span></div><div style="font-family: 'Times New Roman'; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="font-family: 'Times New Roman'; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I found the presentation extremely fluid and informative. Unfortunately, like the other Tom White's session, the audience is not that responsive. I wonder why.</span></div><div style="font-family: 'Times New Roman'; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><div style="font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: x-large;">Lambda Project by Brian Goetz</span></span></div><div><div style="font-family: 'Times New Roman'; font-size: medium; font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><b></b></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">One thing that one must always keep in mind, lambda project is intended not to make Java more concise, but to make the modification that is applicable for parallelization. With this in mind, not every cool things that language like Scala propose would be available in Java.</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Nothing really new in Brian's presentation: </span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Use of SAM instead of function type.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Some starter SAMs are to be included in the JDK.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Method references that allow do Collections.sort(persons, #Person.getLastName). </span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I did not hear the word "defender methods" anymore, I heard a lot extension methods instead.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Exception transparency.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Interface conflicts handling.</span></li>
</ul><div><div style="font-family: 'Times New Roman'; font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: x-large;">What's new in Scala 2.8 by Dick Wall and Bill Venners</span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Dick Wall and Bill Venners presented new things included in 2.8. They presented using live coding. Very interesting way of presenting things, although some errors made the presentation long. But, this duo is simply my favorite in the conference so far. Dick Wall and Bill Venners for President !</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Tooling. Dick showed IntelliJ behavior on implicit method.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The presentation started with REPL. One new thing I learnt was :sh to invoke shell command. Other cool things on REPL were also presented.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Default value for case class. Interesting example on copy that comes together with case class. <i>I should give this a try</i>.</span></li>
<li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">tailrec </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">annotation to make sure that the recursion that we think tail recursive is indeed tail recursive. I have used the annotation in the codes inside this blog though. Fibonacci and Factorial were used as examples. Classical ones. </span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Nested package.</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">@specialized</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">continuation, scarry things, to get a way from this unless you know what you're doing, too advanced.</span></li>
</ul><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Nothing much to say , the demo was just great.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><div style="font-weight: bold; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: x-large;">Scala Collection and Parallelization by Martin Odersky</span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">That was not the title. The real title is more enigmatic: <i>Future Proofing Collections from Mutable to Persistent to Parallel</i>. But it's actually this: scala collection and parallelization.</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Martin started with the slide that he called "If I have to keep one slide, this is the one". Basically, he explained his concept of scalable language that can be agile, but type safe and performant. This looks contradictive, but this can be reached by combining Object Orientation and Functional Programming. </span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The focus of the presentation was on Collection, because Collection is heavily used in the codes, and a lot of problems are on the collection. Collections in Scala, especially in 2.8 are:</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><ul><li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Object oriented</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Generic</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Persistent</span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Higher Order </span></li>
<li><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Uniform Return type principle: Function should return collections of the same type as the (left hand side) operand. That is, map of List should return the same class of List. If it is set, it should return the same class of Map, and so on.</span></li>
</ul><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikFxmrt-iuDG5AEcIx9DS0bgI7F-w44938tzuOgXS-xCQ9_0AKe3_TBgoJ7P2D-z82uzQAOUi53Bo1JMTIf1lfdtcPbBOsYDooPMDpN-EH2aCkCgwoqzH3V_srvhd7dZ_Q9Brox_medmWA/s1600/uniform.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="98" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikFxmrt-iuDG5AEcIx9DS0bgI7F-w44938tzuOgXS-xCQ9_0AKe3_TBgoJ7P2D-z82uzQAOUi53Bo1JMTIf1lfdtcPbBOsYDooPMDpN-EH2aCkCgwoqzH3V_srvhd7dZ_Q9Brox_medmWA/s320/uniform.jpg" width="320" /></a></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Concurrency is hard. Actors and STM are two good tools to address concurrency, but they are not enough. In concurrent world, for safety and performance, immutable collections are needed. That's exactly what Scala proposes. Scala proposed immutable collections, it also proposed parallelization (<i>well, not in 2.8 though</i>). </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">He took an example coming from a paper of Lutz Prechelt that compared several programming languages using a case study of phone code. Martin showed how Scala implemented the solution of phone code. It was a code of more or less 30 lines of codes, excellent. To take advantage of concurrency, the collection used in the example can be changed to parallel computation and par method can be used. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><b>Bit Rot is Dark Side</b></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">So, Scala was, at least around 18.30 Antwerp time, was good. But, there was a dark side: the implementation of </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><i>Uniform Return Type</i> principle. It turned out that implementing the principle represented an important challenge. The duplications are needed. The function </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">filter</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">, for example, needs to be reimplemented in every class. What a mess !</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">We then reached something Martin called as Bit Rot: lots of duplication methods, inconsistencies, and broken window effect. </span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Scala does not let you down on this. Martin Odersky presented the solution of higher kinded types that one could encode in Scala, thanks to </span><b><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">implicit </span></b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">construct of Scala. Martin Odersky presented how this encoded, especially in the Scala collection code -- to implement <i>Uniform Return Type </i>principle.</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The complete solution included in the presentation was very hard to grasp (I might want to blog my understanding on the solution some day). I was just wondering, why Martin Odersky insisted to present this at the Devoxx conference. Shouldn't it better for him to get into detail on parallelization of par and parallel collections instead ?</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">But anyway, the dark side of his presentation gave me a homework to do. Hopefully I would be able to solve it on time, right Professor Odersky ?</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: x-large;"><b>Frites et Mayo</b></span></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The day terminated -- for me , there were still couple of BOFs running in the evening -- with Frites et Mayonnaise to consume. Great ones, with very long queues for everybody. Before the conference, I could not imagine to be in the same queue as Mark Reinhold (and yes, he was 4 people behind me, not sure he got the frites though).</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><div style="font-family: 'Times New Roman';"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: x-large;"><b>Play Framework Meet-up at Axxess</b></span></span></div><div style="font-family: 'Times New Roman';"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Before going back to the hotel to web-caming with my children, I took some times to attend Play Framework Meet-up at Axxess. Met couple of interesting people there, of course Peter Hilton and Nicolas Leroux, and somebody from Alfresco who will present Activiti tomorrow (18 November). They are nice people.</span></div><div style="font-family: 'Times New Roman';"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">I have played with Play couple times ago, and I think of playing it again after the conference. </span></div><div><span class="Apple-style-span" style="color: #444444; font-family: sans-serif; font-size: small;"><span class="Apple-style-span" style="font-size: 12px; line-height: 19px;"><br />
</span></span></div></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><div style="font-family: Arial, Helvetica, sans-serif;"><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: x-large;"><br />
</span></span></b></div></div><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div></div></div></div></div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com1tag:blogger.com,1999:blog-2012573949883447809.post-33269981887473219872010-11-17T02:00:00.000+01:002012-01-19T07:15:03.670+01:00Notes on Devoxx 2nd day (2 - Evening)The reason I cut the notes into two was because there were a lot of things to say on evening session. So, here they are.<br />
<br />
<b><span class="Apple-style-span" style="font-size: x-large;">Scala BOF</span></b><br />
In my previous post, I said that I did not like to bash other programming language -- after all, for a programming language lover, you can't do that. But,...., I could not avoid comparing Groovy and Scala BOFs. The latter had simply double in term of audience... It could have been even better if the place was bigger, some people simply could not join the room anymore.<br />
<br />
All right. Dick Wall and Bill Venners were there, and in addition, Martin Odersky was there too.<br />
<br />
<ul><li>Help with implicit. Martin said that IntelliJ showed and helped with code completion. Great, maybe I would give a try then.</li>
<li>Code completion for implicit in REPL. Why not ? Technically feasible, just the question of resources.</li>
<li>Code completion in REPL again: why didn't it work for jar loaded by :cp ? No answers.</li>
<li>Compilation time. Martin said we should expect that compilation time would not be drastically improved because the Scala compiler did a lot of things, especially on type checking, type erasure, translation to Java, and so on. The best we can expect is reducing by order of two. Otherwise, one may think of using better build tool, why not sbt. Otherwise, a classical answer: Scala will pay off with less codes (well...).</li>
<li>SBT news : well, it's fine. on going.</li>
<li>Debugger: anybody could tell the experiences on debugger ? No answer (I myself did not use debugger for my project, I used plain old println).</li>
<li>Scala ecosystem, industry adoption: Scala works with Java very fine. Industry started to adopt Scala. Play! was mentioned a couple of times. Dick Wall mentioned an interesting project called Squeryl, ORM in Scala. Bill mentioned about ScalaTest. The biggest missing things are tooling like Check Style, PMD, Code Coverage.</li>
<li>Binary compatibility issues on 2.8 when migrating to 2.7: yes, it would be taken into account so that it never happened again. But, unfortunately, it will likely to happen on 2.9 with the introduction of parallel collection. Martin considered that should be acceptable for a language like Scala that is still relatively young, he compared with Java Collection that will be final hard to change.</li>
<li>(A small incident happened when we discussed this, somebody somewhere outside the room yelled noisily ... didn't know to whom .... We took that a joke: that must be a Java Collection developer)</li>
<li>Tooling again: check style. Dick Wall told that FindBug might work although it complained too much by default at the moment.</li>
<li>Best practice: it is important, to control the variation of codes, especially in big company where one code may move from one team to another. However, it might be challenging, given two trends so far: Scalaz trend and Java-ish trend. But, the three agreed to consider this issue as important.</li>
<li>Training. Scala did have some good trainings, Martin, Bill, and Dick are Scala trainers. One of the trainee was there too. </li>
<li>Big company behind Scala. Martin said that there might be advantage of having that kind of company, but Ruby and Python were the two cases that both did not have big company behind them, but they did succeed.</li>
<li>Code generation tool, like Grail for Groovy, Rail for Ruby. Dick argued that those two examples might not be applicable for Scala. The thing is to find other killer-app for Scala, and it's not necessarily a tool like Grail or Rail. The killer app might be Akka or Lift (Play! framework was again mentioned)</li>
<li>2.9 version. Parallel collection.</li>
<li>Target of Scala, only JVM ? No .NET as well (funded by Microsoft directly). Is there any limitation imposed by this choice? Type erasure for example? Not really. Type erasure is not a limitation on itself either, only .NET does not erase type. Manifest introduced in 2.8 should be helpful for type erasure.</li>
<li>Scala Solution. Scala solution is intended to provide professional development tool (again, tooling) and for longer future maybe on middleware. </li>
<li>Is there any design that you regret ? (Long reflection by Martin..., I expected XML literal, but not..., it was :) postfix operator.</li>
</ul><div>Like Scala Lab in the morning, this session was fantastic. Thanks to the three people. Scala Rocks !</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: x-large;">Hazelcast by Talip Ozturk</span></b></div><div><b><span class="Apple-style-span" style="font-size: x-large;"><br />
</span></b></div><div><div>It was an interesting presentation on Hazelcast by Talip Ozturk. He is a nice guy. </div></div><div><div><ul><li>How share data structures (map, queue, list) across several JVMs</li>
<li>Multicast is used to discovery, followed by TCP/IP communication.</li>
<li>Security: data can be encrypted.</li>
<li>Demo on how map updated in one node affected other node.</li>
<li>Also on locking/unlocking</li>
<li>Interesting presentation on internal mechanism of Hazelcast.</li>
<li>Executor service is useful to avoid unnecessary distributed computation.</li>
<li>Update are synchronous.</li>
</ul><div>Pretty interesting things Talip presented here. It was very powerful tool, but damn simple. Maybe I'll have a try. </div></div></div><div><br />
</div><div><br />
</div><div><br />
</div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com1tag:blogger.com,1999:blog-2012573949883447809.post-74675067613232040872010-11-17T01:04:00.000+01:002012-01-19T07:15:03.813+01:00Notes on Devoxx 2nd day (Morning - Afternoon)<b><span class="Apple-style-span" style="font-size: x-large;">Scala Hands On by Dick Wall and Bill Venners</span></b><br />
This session is my favorite session so far. There were not so many new things I learnt from the session. After all: it was a very short course on beginning Scala.<br />
<br />
Couple of things I learnt at the session though :<br />
<ul><li>I always took for granted things like <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">args.foreach( s => println(s.reverse)) </span>. I did not notice that <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">args.foreach(s => println(s.reverse())</span> did not actually compile. After the session I knew why. </li>
<li>extending case class came into my mind several time during my project. I didn't know whether it worked. At the class, I saw that a case class extends another case class led to deprecation warning: it simply wont work smoothly.</li>
<li>I discovered some limitations on code completion in Scala REPL.</li>
</ul><div>Overall, the session was very entertaining, nobody wanted a break ! It was just great. The duo Bill and Dick was just fantastic. I also took advantage of observing Java learning Scala. It was a good experience, especially for my talk on Scala at the office soon.</div><div><br />
</div><div>By the way, the lab was full of people. I didn't come late, but I sat far behind, got difficulties to read what was on the screen.</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: x-large;">Lunch</span></b></div><div>It was a pretty quick lunch with Pasta and Shrimp. It was great ! But after 3 hours at the lab without power, the priority was to plug my laptop. So, short lunch.</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: x-large;">Programming in Pain by Enno Runne</span></b></div><div>It was a quickie session by Enno Rune on Java and Scala comparisons. To be honest, comparing Java and Scala to tell that Scala is superior is not that useful and not the approach I support when promoting my favorite programming language (Scala). Why should one bash Java to show that Scala is great. Scala IS great. That's all. No need agressive comparison to Java. No need to tell Java is painful to live with once you knew Scala. </div><div><br />
</div><div>But anyway, the content of the presentation is actually quite great. You can have it here: </div><div>http://www.slideshare.net/ennorunne/programming-in-pain</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: x-large;">HBase by Michael Stack and Jonathan Gray</span></b></div><div><b><br />
</b></div><div>I have some serious difficulties on understanding the session. I hope my notes here make sense:</div><div><ul><li>HBase is based on Hadoop, hence HFDS.</li>
<li>Based on Big Table paper on Google that everyone should read, even Michael's son read that (I tried to read the paper between session, I realized how brilliant Michael's son would be).</li>
<li>Some names on HBase customer, the most popular one of course is Facebook.</li>
<li>But HFDS lack of random read/write capabilities, and HBase is basically adding the random read/write capbilities to Hadoop. </li>
<li>The read/write is achieved by writing items and then compacting later (??)</li>
<li>Guarantee: atomic row writing, regardless the number of column families.</li>
<li>Compared to Relational Database, HBase is more up-scalable, up to PBytes of data. Best when there are big number of columns, and still perform well in presence of sparse columns.</li>
<li>HBase data is structured in rosw and column , like Relational DB.</li>
<li>A row has a row key ~ primary key and the table is ordered by the row key.</li>
<li>Each row has timestamp that is provided by HBase, but application can also use their own timestamp.</li>
<li>HBase may store multiple rows of data.</li>
<li>HBase provides a shell, - JIRB based.</li>
<li>Unfortunately no demo of the shell because of network problem.</li>
<li>But, some examples of create table, get, put, scan , ..</li>
<li>Syntax is Ruby heavy.</li>
<li>Then, a long-heavy-boring-hard-to-understand presentation on HBase architecture, but ... anyway.</li>
<li>Some terms: Region ~ Partition, RegionServer.</li>
<li>Region split mechanism and recovery.</li>
<li>Back to programming again: Java API presentation: create table, put, get , scan, mapreduce</li>
</ul><div>Pause. 10 minutes after pause, session restarted, but with less than 50 % audience. To be honest, even the presentation content was good, there were a lot improvement needed to make the contents understandable easily.</div><div><ul><li>Interesting API on concurrency: locking, checkAndPut, increment, filters.</li>
<li>A little presentation on deployment: nothing really interesting though.</li>
<li>Monitoring with Ganglia was mentioned. JMX too.</li>
<li>More on backup, tuning.</li>
<li>Compactions can be configured.</li>
</ul>Overall, it was quite informative session, but it was somehow quite hard to grasp. Maybe the subject is difficult, maybe the presentation was not that clear. But, yes, like other no sql things, this one needs to be explored further.</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: x-large;">Double Pasta</span></b></div><div>Since I planned to stay until 9, at least, I went down and found that some pastas were left. Yes, they were the same pasta as for lunch. It didn't matter. I re-took one, with some clementines, and hop, ready to go till late evening.</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: x-large;">Java EE Tooling by Ludovic Champenois</span></b><span class="Apple-style-span" style="font-size: x-large;"> </span></div><div>The idea of the presentation was comparing three IDEs: Eclipse, IntelliJ, and of course NetBeans (Ludovic is one of GlassFish architect) in term of their supports on Java EE 6. Java EE 6 is btw, annotation based EJB/Servlet/CDI/JPA 2.0. </div></div><div>So, he demonstrated features like code completion, contextual help, menus, wizards in the three IDEs for the supports of JEE previously mentioned. It was pretty cool demo. He clicked here and there, opened wizard here and there, launching http page here and there. Well, pretty cool to see. </div><div>At the end, he showed a comparison matrix on the three, and NetBeans was the best, followed by IntelliJ, and finally Eclipse that had no support of CDI at all.</div><div><br />
</div><div>O, yes. He showed some scarry long annotations of JPA. What was that ??? It was very fast and I was at the last row at the cinema. But it was kind of 100 lines of JPA annotations out there. </div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: x-large;">Excel on JVM by Peter Arrenbrecht</span></b></div><div>Peter presented his tool formulacompiler <a href="http://www.formulacompiler.org/">http://www.formulacompiler.org</a> that compiles Excel to Java byte codes. </div><div><ul><li> Excel is an excellent tool for modelling, but not for computation. </li>
<li>So, let the user model in Excel and compile the Excel to Java classes, and then you can build any application from that.</li>
<li>Cool idea, isn't it? </li>
<li>The result of compilation is a jar file, so that you can use it in any applications without reparsing the excel or the java code file.</li>
<li>Some profilings against Apache PIO was shown. Pretty cool indeed. </li>
</ul><div>I wouldn't say that it was my favorite presentation. But Peter did present the subject well. </div><div><br />
</div><div>That was my morning and afternoon sessions. </div></div><div><br />
</div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-65529743319769229192010-11-15T23:25:00.000+01:002012-01-19T07:15:03.799+01:00Notes on Devoxx 1st day<b>Hadoop by Tom White. </b><br />
It was an interesting presentation on Hadoop, MapReduce, Hive, and Pig. My presence in the session was a pretty much last minute change, because for some reasons I found Language in Action Lab less interesting than Hadoop. I'm not disappointed.<br />
<br />
The first half of his presentation is about Hadoop and MapReduce:<br />
<ul><li>Hadoop is based on HDFS (Hadoop File System) and map reduce algorithms.</li>
<li>HDFS is a 100% Java-based file system that handles replication, distribution, and access to the files.</li>
<li>HDFS contains meta data nodes that store the information about file and data noted, the nodes that contain the data.</li>
<li>So, data in Hadoop are distributed among several nodes. The data themselves are stored in a block of 64MB or 128MB (?)</li>
<li>The map reduce algorithms are basically algorithms that produce set of values from large datasets. The data are first mapped (they can also be filtered) to a set of key values in every node. Then, the results are aggregated and reduced by the reducer. </li>
<li>The mechanisms to send the output of map to the reducer are sort and shuffle mechanisms.</li>
<li>Combiner is an object that reduces the values resulted from the map calculation, but instead of doing the reduction in other nodes, combiner does the reduction in the same node as mapper. Of course, not all reductions can be done in one node, since the node must be independent and must not assume anything about the other nodes.</li>
<li>Then, Tom shows some codes on Hadoop: driver codes, map, reduce codes. Interesting ones.</li>
</ul><div>The second half is about Hive and Pig:</div><div><ul><li>Hive is actually a query language much like SQL-likethat is used on top of Hadoop Map/Reduce.</li>
<li>Hive supports some map/reduce specific extension to the SQL and it also allows user defined function to be used in the query language.</li>
<li>Map/Reduce is considered to be complex, and Tom showed an example on the system that has complex map reduce architecture (not really sure to understand the complexity though).</li>
<li>Hive codes are compiled into Map/Reduce.</li>
<li>Hive has a construct called Serde.</li>
<li>Hive has a managed storage area.</li>
<li>Hive has an interesting support to multi table inserts (multi table inserts that share the same FROM clause).</li>
<li>Hive is used by Facebook that launches ~7000 hives a day (?).</li>
<li>Pig is like Hive, but it is not intended to look like SQL.</li>
<li>Unlike Hive, Pigs have no schema.</li>
<li>Pig looks like LINQ to Entity Framework (This is my feeling, not the presenter).</li>
</ul><div>Overall, it was an interesting presentation. The presentation was scheduled to last 3 hours, but it had completed after around 2.5 hours. The underflow was for some reasons: the public were not that reactive, maybe because it was the very first session of the conference, the presenter did not have too many contents to present. When the public is not reactive, the presenter might should have shown couple of more contents though. </div></div><div>The demo was not that appealing. To be honest, I didn't see that much demo, and Tom had difficulties in running some of his scripts because he had difficulties reading the screen. </div><div>Other than that, I really think it was a nice presentation to start Devoxx with. Would like to play around with Hadoop, Hive, and Pig some day.</div><div><br />
</div><div><b>MongoDB by Alvin Richards</b></div><div><b><br />
</b></div><div>Unlike Hadoop that I discovered at Devoxx, I played around already with MongoDB, a document-oriented database. The presentation however gave pretty much different views of MongoDB. </div><div><br />
</div><div>Here are some notes:</div><div><ul><li>MongoDB is a document-oriented database implemented in C++ with driver in many languages, including Java, Erlang, Scala, ... , Smalltalk. </li>
<li>The presentation started by the roles of Relational Database and argued that the separation between data and logic was one of the main contributions of the relational database to the database technology. MongoDB continues the philosophy.</li>
<li>Using "blog system" as examples, MongoDB -- or document-oriented database in general introduces instead what is called "pre joining the data" or embedding and linking.</li>
<li>Tintin Destination Moon example was used all along the presentation, pretty fun.</li>
<li>MongoDB supported automatic id generation that guarantees uniqueness and immutability of the id.</li>
<li>Indexing is supported.</li>
<li>Some operations: db.posts.save, db.posts.find, db.posts.ensureIndex( {author:1}), db.system.indexes.find.</li>
<li>Map reduce algorithms are built-into the MongoDB.</li>
<li>GroupBy support.</li>
<li>Support of "class" inheritance, example: rectangle, circle. Null is used for attributes that are not applicalble to a class (like radius for rectangle).</li>
<li>Quick explanations on OneToMany and ManytoMany. I didn't get it completely in detail though.</li>
<li>Replication support. Alvin explained the algorithms, the voting, and those things.</li>
<li>Sharding.</li>
<li>The sharding can be done by range and are configurable: it is assumed that the owner of the data know the best sharding strategy.</li>
<li>Config servers hold meta data.</li>
<li>Mongos does the switching: routing requests to the right servers.</li>
<li>To code using MongoDB in Java, one can use raw mongo DB driver or Morphia.</li>
<li>Raw Mongo Driver works with lower level representation of mongo DB data (JSON) while Morphia work with classes.</li>
<li>Morphia is an annotation oriented system, just like JPA (hmm........, for me it's quite strange).</li>
<li>@Id, @Entity, @Transient ..., are examples of Morphia annotations.</li>
<li>Relationship is also configurable using annotations</li>
<li>Durability strategy is interesting. One can configure that the write acknowledge comes when the memory of the master is updated, but can also configure that the write acknowledge comes when the disk of the master and slaves are updated. Some strategy in betweens are possible (memory of the slaves, disk on master, disk on slaves, ...)</li>
<li>Reading on replication is also configurable. You can configure to read only the master or you may only want to read the data replicated consistently in all slaves.</li>
<li>Sharding is transparent to Java code.</li>
<li>Backup strategy: mongodump / mongostore, fcync + lock.</li>
<li>fsync flushes buffers to disk.</li>
<li>Slave delay is a the recommended strategy to cope with database administration errors.</li>
<li>Recommended configurations: memory (should be huge), file system , I/O ...</li>
<li>MongoDB does not support transaction, the only supports is atomic write, either it writes or it does not. </li>
<li>No view support yet, but maybe it's coming.</li>
<li>Limited security support (I wonder whether encryption is supported).</li>
<li>No attempt on standardisation yet. For me, this can be a show stopper for a bigger company for adopting MongoDB though.</li>
</ul><div>For me, the presentation is the best presentation of the day. I loved the rich contents the presenter proposed. The publics were very reactive, many interesting questions were thrown and the answeres were pretty good. Again, it's a nice presentation, and it confirms my intention to play around MongoDB. </div></div><div><br />
</div><div><div><b>Spring Tools by Christian Dupuis</b></div></div><div>Compared to the other two, the presentation was much shorter: 30 mins. What could you do in 30 minutes. Well, not that much. That's a problem with the presentation: the presenter threw soo much contents, each is mentioned in 2-3 minutes. The demo was not that convincing, well, it was a little bit disappointing.</div><div><ul><li>The idea of the presentation is that Spring tools support framework and languages (including Maven and AspectJ).</li>
<li>It was nice to see aspect visualisation though.</li>
<li>Spring Roo is part of the distribution of STS.</li>
<li>STS is actually a specific distribution of Eclipse.</li>
<li>SpringInsight is a powerful profiling tool for enterprise application. </li>
</ul><div>Indeed, SpringInsight quick presentation was interesting and that saved a little bit the presentation from my point of view.</div></div><div><br />
</div><div><b>Apache Mahout by Isabel Drost</b></div><div>Apache Mahout is a tool for machine learning. It is built on Apache Hadoop and came from Lucene. The presentation was more on machine learning than from codes point of view, not even the relation with Hadoop was clear from the presentation. But, the presentation was very clear, and that was enough, it is always better to have things clear than complete but unclear.</div><div><br />
</div><div><ul><li>To have a successful machine learning -- especially from text -- first, it must remove noises, followed by converting the text to vector. </li>
<li>Model is then developed.</li>
<li>Followed by train a prediction model.</li>
<li>Finally, evaluate the prediction.</li>
<li>Some applications: recommendation builing, spam filtering, ...</li>
<li>Current release: 0.4</li>
<li>To start, one can use Amazon EC2 or from a simple desktop.</li>
<li>Berlin Buzzwords 2011, May/June.</li>
<li>FOSDEM : Data analysis forum.</li>
</ul><div><div><b><br />
</b></div><div><b>Groovy BOF</b></div></div></div><div>To be honest, I was not sure what I was looking for in this BOF. The idea was just to listen discussions on a language that runs in JVM -- just like Scala. And yes, I didn't understand quite many of it. Oh, yes. I got an impression that GWT was something different and pretty opposite of Groovy. Pretty interesting. There were also discussions on Maven support vs Groovy build system. </div><div>Yeah, maybe needs to have a look a little bit on Groovy before making sense all of those discussions.</div><div><br />
</div><div><b>jPatterns</b></div><div>Finally, I prefered to leave after Groovy BOF, I was just too hungry to stay longer. So, no jPatterns. </div><div><br />
-<br />
<br />
</div><div>Tomorrow should be exciting. I'll try to get into Scala LAB, Quickies (Programming in Pain), HBase (yes, that's again a schedule change), J2EE Tooling, Excel in JVM, Scala BOF, Hazelcast, and eventually Asynchronity in Java Web Services.</div><div><b><br />
</b></div><div><br />
</div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-52010099158246905442010-11-09T00:34:00.000+01:002012-01-19T07:15:03.790+01:00Euler Problem 97: Find last 10 digits of a big prime number<span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">First, I'm glad that I've managed to solve 39 problems by now. Since the last problem, I don't find really interesting example to blog, but this problem 97 deserves a blog. Not really because Scala is good in answering this, but because the problem itself is interesting.</span></span><br />
<span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br />
</span></span><br />
<span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The problem is expressed in a little bit history of prime number, but finally the real problem is here: "What is the 10 last digits of the following prime number: <span class="Apple-style-span">28433</span><span class="Apple-style-span"><img alt="×" border="0" height="9" src="http://projecteuler.net/images/symbol_times.gif" style="border-bottom-width: 0px; border-color: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; vertical-align: middle;" width="9" /></span><span class="Apple-style-span">2</span><span class="Apple-style-span"><sup>7830457</sup></span><span class="Apple-style-span">+ 1. </span></span></span><span class="Apple-style-span"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Looks intimidating, right ? Try give a try calculating 28433 * (2: BigInt).pow(7830457) won't work, simply because its too large. Another thing is needed.</span></span></span><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: small;">The other thing is the modular exponentiation, excellently explained here: <a href="http://en.wikipedia.org/wiki/Modular_exponentiation">http://en.wikipedia.org/wiki/Modular_exponentiation</a> . Basically, the techniques are very useful to compute b<span class="Apple-style-span"><sup><i>p </i></sup></span>mod m by observing that (a*b) mod m = ( (a mod m) * (b mod m) ) mod m. </span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: small;"><br />
</span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: small;">Taking that into account, b<span class="Apple-style-span"><sup><i>2 </i></sup></span>mod m can be calculated as ( (b mod m) * (b mod m) ) mod m, and </span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: small;">b<span class="Apple-style-span"><sup><i>3 </i></sup></span>mod m =( ( b mod m) * ( b<span class="Apple-style-span"><sup><i>2 </i></sup></span>mod m) )mod m and so on. </span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: small;"><br />
</span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-size: small;">When this is translated to Scala it becomes:</span></span><br />
<div style="font-size: medium;"><br />
</div></div><pre><span style="color: black;"><span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 1 </span></span>dev <span style="color: #9966ff;">div</span><span style="color: black;"><strong>(</strong></span>b: <span style="color: #009966;"><strong>Long</strong></span>, exp: <span style="color: #009966;"><strong>Long</strong></span>, m: BigInt<span style="color: black;"><strong>)</strong></span>: BigInt <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 2 </span></span> <span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">div</span><span style="color: black;"><strong>(</strong></span>b: <span style="color: #009966;"><strong>Long</strong></span>, counter: <span style="color: #009966;"><strong>Long</strong></span>,
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 3 </span></span> m: BigInt, result: BigInt<span style="color: black;"><strong>)</strong></span>: BigInt <span style="color: black;"><strong>=</strong></span> <span style="color: black;"><strong>{</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 4 </span></span> <span style="color: #006699;"><strong>if</strong></span> <span style="color: black;"><strong>(</strong></span>counter <span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>=</strong></span> <span style="color: red;">0</span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;"> 5 </span></span> result
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 6 </span></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 7 </span></span> <span style="color: #006699;"><strong>else</strong></span> <span style="color: #9966ff;">div</span><span style="color: black;"><strong>(</strong></span>b, counter <span style="color: black;"><strong>-</strong></span> <span style="color: red;">1</span>, m, <span style="color: black;"><strong>(</strong></span>result <span style="color: black;"><strong>*</strong></span> b<span style="color: black;"><strong>)</strong></span> <span style="color: black;"><strong>%</strong></span> m<span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 8 </span></span> <span style="color: black;"><strong>}</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: black;"> 9 </span></span> <span style="color: #9966ff;">div</span><span style="color: black;"><strong>(</strong></span>b, exp, m, <span style="color: red;">1</span><span style="color: black;"><strong>)</strong></span>
<span style="background-attachment: initial; background-clip: initial; background-color: #dbdbdb; background-image: initial; background-origin: initial; border-right-color: black; border-right-style: solid; border-right-width: 2px; margin-right: 5px;"><span style="color: #990066;">10 </span></span><span style="color: black;"><strong>}</strong></span>
</span></pre><div style="font-size: medium; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-size: 13px;"></span></span><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">With this in our hand, we can then get the last 10 digit of<span class="Apple-style-span"> <span class="Apple-style-span">2</span><span class="Apple-style-span"><sup>7830457 </sup></span></span></span></span><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">which nothing less than<span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span"> </span></span><span class="Apple-style-span"><i> </i><span class="Apple-style-span">2</span><span class="Apple-style-span"><sup>7830457</sup></span></span></span></div><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">mod 10000000000. Using div method above, we can get then the last 10 digits of the term. We can multiply the result with 28433 and then add 1 to get the answer to the problem.</span></span><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">OK. That was my 39th solved problem.</span></span></div></div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-45632708118899635872010-11-03T08:28:00.001+01:002012-01-19T07:15:03.696+01:00Problem: Euler Task 12: Number of factors of triangle number.<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The following problem (euler 12) is a good example of the use of Stream collection in Scala. The use of Stream is explained in <a href="http://www.scala-lang.org/docu/files/collections-api/collections_14.html">http://www.scala-lang.org/docu/files/collections-api/collections_14.html</a> .</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Basically, using Stream, we can create a list that computes its elements lazily. For example, to create a list of fibonacci number we can use </span><br />
<table cellpadding="0" cellspacing="1"><tbody>
<tr><td align="left" colspan="99" nowrap="true"><br />
</td></tr>
</tbody></table><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">def fibFrom(a: Int, b: Int): Stream[Int] = </span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> a #:: fibFrom(b, a + b)</span></div><div><br />
</div><div><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The problem is available here: </span><a href="http://projecteuler.net/index.php?section=problems&id=12">http://projecteuler.net/index.php?section=problems&id=12</a> :</div><div style="margin: 0px;"><br />
</div><div style="margin: 0px;">"<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 16px;"><i>The sequence of triangle numbers is generated by adding the natural numbers. So the 7</i></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 16px;"><sup><i>th</i></sup></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 16px;"><i> triangle number would be 1 + 2 + 3 + 4 + 5 + 6 + 7 = 28. The first ten terms would be:</i></span></div><div style="font-family: 'Trebuchet MS', sans-serif; font-size: 16px; text-align: center;"><i>1, 3, 6, 10, 15, 21, 28, 36, 45, 55, ...</i></div><div style="font-family: 'Trebuchet MS', sans-serif; font-size: 16px;"><i>Let us list the factors of the first seven triangle numbers:</i></div><blockquote style="font-family: 'courier new'; font-size: 16px;"><b><i> 1</i></b><i>: 1<br />
<b> 3</b>: 1,3<br />
<b> 6</b>: 1,2,3,6<br />
<b>10</b>: 1,2,5,10<br />
<b>15</b>: 1,3,5,15<br />
<b>21</b>: 1,3,7,21<br />
<b>28</b>: 1,2,4,7,14,28</i></blockquote><div style="font-family: 'Trebuchet MS', sans-serif; font-size: 16px;"><i>We can see that 28 is the first triangle number to have over five divisors.</i></div><div style="font-family: 'Trebuchet MS', sans-serif; font-size: 16px;"><i>What is the value of the first triangle number to have over five hundred divisors?</i>"</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin: 0px;"></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The triangle number here is a good candidate for Stream collection. We can define the triangle function as follow:</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"></span></span></div><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span style="color: black;"><span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;"> 1 </span></span><span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">triangle</span><span style="color: black;"><strong>(</strong></span>a: <span style="color: #009966;"><strong>Int</strong></span>, b: <span style="color: #009966;"><strong>Int</strong></span><span style="color: black;"><strong>)</strong></span> : <span style="color: #0099ff;"><strong>Stream</strong></span><span style="color: #9900cc;">[Int]</span> <span style="color: black;"><strong>=</strong></span>
<span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;"> 2 </span></span> a #:: <span style="color: #9966ff;">triangle</span><span style="color: black;"><strong>(</strong></span>a <span style="color: black;"><strong>+</strong></span> b <span style="color: black;"><strong>+</strong></span> <span style="color: red;">1</span>, b <span style="color: black;"><strong>+</strong></span> <span style="color: red;">1</span><span style="color: black;"><strong>)</strong></span></span></span></pre><br />
<br />
<div><br />
</div><div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Which means the first element of triangle number list is a and the next element is (a + b + 1), followed by ( a + b + 1 + b + 1 + 1), and so on .</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">triangle(1, 1).take(7).foreach(println) </span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">prints the first 7 triangle numbers.</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">The following function computes the number of factors of an integer:</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"></span></span></div><pre><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><span style="color: black;"><span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;">1 </span></span><span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">nfactor</span><span style="color: black;"><strong>(</strong></span>n: <span style="color: #009966;"><strong>Int</strong></span><span style="color: black;"><strong>)</strong></span>: <span style="color: #009966;"><strong>Int</strong></span> <span style="color: black;"><strong>=</strong></span>
<span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;">2 </span></span> <span style="color: black;"><strong>(</strong></span><span style="color: red;">1</span> to <span style="color: #9966ff;">csqrt</span><span style="color: black;"><strong>(</strong></span>n<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span>.<span style="color: #9966ff;">foldLeft</span><span style="color: black;"><strong>(</strong></span><span style="color: red;">0</span><span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>(</strong></span>
<span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;">3 </span></span> _ <span style="color: black;"><strong>+</strong></span> <span style="color: #9966ff;">incrfactor</span><span style="color: black;"><strong>(</strong></span>n, _<span style="color: black;"><strong>)</strong></span><span style="color: black;"><strong>)</strong></span></span></span></pre><br />
<div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">where incrfactor is a function that receives two parameters (n, x) and returns 0 if x does not divide n, 2 if x divides n and n/x != x , 1 otherwise (because when n/x = x ,we have only found one factor = x , otherwise if we have found 2: x and n/x).</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><pre><span style="color: black;"><span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;">1 </span></span><span style="color: #006699;"><strong>def</strong></span> <span style="color: #9966ff;">incrfactor</span><span style="color: black;"><strong>(</strong></span>n: <span style="color: #009966;"><strong>Int</strong></span>, x: <span style="color: #009966;"><strong>Int</strong></span><span style="color: black;"><strong>)</strong></span> <span style="color: black;"><strong>=</strong></span>
<span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;">2 </span></span> <span style="color: #006699;"><strong>if</strong></span> <span style="color: black;"><strong>(</strong></span>n <span style="color: black;"><strong>%</strong></span> x <span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>=</strong></span> <span style="color: red;">0</span><span style="color: black;"><strong>)</strong></span> <span style="color: black;"><strong>(</strong></span>
<span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;">3 </span></span> <span style="color: #006699;"><strong>if</strong></span> <span style="color: black;"><strong>(</strong></span>x <span style="color: black;"><strong>=</strong></span><span style="color: black;"><strong>=</strong></span> n<span style="color: black;"><strong>/</strong></span>x<span style="color: black;"><strong>)</strong></span> <span style="color: red;">1</span> <span style="color: #006699;"><strong>else</strong></span> <span style="color: red;">2</span></span></pre><pre><span style="color: black;"><span style="color: red;"><span class="Apple-style-span" style="color: black;"><span style="background-clip: initial; background-color: #dbdbdb; background-origin: initial; border-right: black 2px solid; margin-right: 5px;"><span style="color: black;">4 </span></span> </span></span></span><span style="color: black;"><strong>)</strong></span> <span style="color: #006699;"><strong>else</strong></span> <span style="color: red;">0</span><span class="Apple-style-span" style="color: red;"> </span></pre><pre><b>
</b></pre><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">With all these on our hands, we're ready to answer the problem. The answer to our problem is</span></div><div style="margin: 0px;"><br />
</div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">triangle(1,1).find( nfactor(_) > 500)</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div style="margin: 0px;"><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">which returns the first triangle number that has more than 500 factors. In my computer, the code above needs 6 s.</span></div></div>voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0tag:blogger.com,1999:blog-2012573949883447809.post-37964413608644385562010-11-03T07:52:00.000+01:002012-01-19T07:15:03.680+01:00Problem: Euler Task 14: Which starting number, under one million, produces the longest chain?I pick this problem 14 because it provides an interesting example of tuple used in combination with fold. Let's see how it works nicely.<br />
<br />
First, the problem:<br />
<br />
<span style="font-family: inherit;"><em>"The following iterative sequence is defined for the set of positive integers:</em></span><br />
<div style="margin-left: 50px;"><span style="font-family: inherit;"><var>n</var><em> <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> <var>n</var>/2 (<var>n</var> is even)<br />
<var>n</var> <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 3<var>n</var> + 1 (<var>n</var> is odd)</em></span></div><span style="font-family: inherit;"><em>Using the rule above and starting with 13, we generate the following sequence:</em></span><br />
<div style="text-align: center;"><span style="font-family: inherit;"><em>13 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 40 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 20 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 10 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 5 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 16 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 8 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 4 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 2 <img alt="→" border="0" height="7" src="http://projecteuler.net/images/symbol_maps.gif" style="vertical-align: middle;" width="15" /> 1</em></span></div><span style="font-family: inherit;"><em>It can be seen that this sequence (starting at 13 and finishing at 1) contains 10 terms. Although it has not been proved yet (Collatz Problem), it is thought that all starting numbers finish at 1.</em></span><br />
<span style="font-family: inherit;"><em>Which starting number, under one million, produces the longest chain?"</em></span><br />
<br />
Now, we're able to attack the problem. We will need a function that receives a term and returns the length of the chain. Let's say <span style="font-family: 'Courier New', Courier, monospace;">def chainLength(n : Int) : Int . </span><span style="font-family: 'Times New Roman';">Unfortunately, this will not work out since the term can be bigger than Integer.MAX_VALUE, which is the case when I first tested the problem. So, we'll need instead </span><span style="font-family: 'Courier New', Courier, monospace;">chainLength(n : Long). W</span><span style="font-family: 'Times New Roman';">e're ready for the code then. We have basically four cases, when n <=0, n == 1, n is odd, n is even. We'll have then something like :</span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">def chainLength(n :Long): Int = {<br />
if (n <= 0) {<br />
throw new IllegalArgumentException("Negative term: " + n)<br />
} else if ( n == 1 ){<br />
1<br />
} else </span><span style="font-family: 'Courier New', Courier, monospace;">if (n% 2 == 0) {<br />
1 + chainLength(n/2) </span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> } else {<br />
1 + chainLength(n *3 + 1)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> }<br />
}</span><br />
<br />
Almost OK. But the recursion is not tail recursion. We need to make it tail recursive by introducing an accumulator as the second parameter.<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">def chainLength(n :Long, len: Int): Int = {<br />
if (n <= 0) {<br />
throw new IllegalArgumentException("Negative term: " + n)<br />
} else if ( n == 1 ){<br />
len<br />
} else </span><span style="font-family: 'Courier New', Courier, monospace;">if (n% 2 == 0) {<br />
chainLength( n/2, len + 1) </span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> } else {<br />
chainLength( n *3 + 1, len + 1)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> }<br />
}</span><br />
<br />
To get the chainLength for 13 we call then chainLength(13, 1)<br />
<br />
Great. The only thing missing is now to find the lowest starting term between 2 to 1000000.<br />
Here we're going to use combination of tuple (a, b) where a is the term and b is the length of sequence for the term. Here is the final code:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">(2 to 1000000).foldLeft( (2,0) )( (x,y) => { <br />
val len = chainLength(y, 1)<br />
if (len > x._2 ) (y, len)<br />
else x<br />
})</span><br />
<br />
The code took me less than 6 seconds in my machine.voidmainargshttp://www.blogger.com/profile/00911109433888554531noreply@blogger.com0