1 #+TITLE: Real World Clojure
3 #+EMAIL: tmarble@info9.net
10 [[file:~/src/software-passion/Clojure-glyph.svg]]
12 My journey in using Clojure for a client that is
13 developing a multiplayer game server hosting service.
16 ** Overview: Topics to cover
18 Press *f9* to see the list of topics
20 **** Note's on what I put in my .emacs.d/custom.el
22 (require 'org-tree-slide)
24 (global-set-key (kbd "<f1>") 'show-all)
25 (global-set-key (kbd "<f5>") 'text-scale-decrease)
26 (global-set-key (kbd "<f6>") 'text-scale-increase)
27 (global-set-key (kbd "<f8>") 'org-tree-slide-mode)
28 (global-set-key (kbd "<f9>") 'org-tree-slide-content)
29 (global-set-key (kbd "<f10>") 'hide-sublevels)
31 ** What is that presentation tool?
35 This is *org-tree-slide* from https://github.com/takaxp/org-tree-slide
37 For more on org mode see http://orgmode.org/org.html
39 Yes I will share my "slides" on my website http://tmarble.info9.net
41 [[file:~/src/software-passion/org-mode.png]]
45 My great grandmother immigrated to the USA from Sweden around 1900
47 [[file:~/src/software-passion/asimn.png]]
55 [[file://home/tmarble/Pictures/Tom/Mugshot2011/Tom-2011-200.jpg]]
57 *** Sun: technical presales during the dot.com era
59 [[file:~/src/software-passion/e10k.jpg]]
63 *** Sun: Java Performance
65 [[file:~/src/software-passion/specjvm2008.png]]
69 Early 2006: DLJ with Debian and Canonical (Ubuntu)
71 [[file:~/src/software-passion/dlj.png]]
75 JavaOne 2006: Rich Green announces that Sun will open source Java
78 - How organize community governance
79 - Copyright, Patent and Trademark licensing
80 - Infrastructure tools
83 First OpenJDK Ambassador (I went to a lot of conferences)
88 ApacheCon 2006: Sun unBOF/Party
89 [[file:~/src/software-passion/apachecon2006.jpg]]
90 Copyright 2006 Ted Leung: https://secure.flickr.com/photos/twleung/268116213/
92 *** Left Sun to do a startup
94 [[file:~/src/software-passion/tom-think.jpg]]
98 The global financial meltdown of 2008 happened (etc.)
102 And so I... got into consulting!
108 Probabilistic Model Verification (Electrical Engineering)
110 Software for Smart Grid + Renewable Energy
116 Using Linux since 1996
118 Using Debian since 2003
120 Helped Debian Java Packaging Team since DLJ in 2006
122 World with Debian and Oracle on meshing Jigsaw with *apt*
124 [[file:~/src/software-passion/gsoc11.jpg]]
135 - Organized by Karen Sandler, Bradley Kuhn, Richard Fontana and myself
136 - Check out the Wiki http://info9.net/wiki/fosdem/LegalIssuesDevRoom/
137 - Check out the oggcast http://faif.us/
142 - Developing a multiplayer game server hosting service
143 - Comprised of very young developers
144 - Is in stealth mode (sorry!)
146 I have been given the authority to
147 - Make significant choices about architecture
148 - Green light to open source generic bits
150 (this is why i like consulting :)
152 I'm not the only one (especially in Nordic countries)!
156 John McCarthy is old school:
157 [[file:~/src/software-passion/John_McCarthy.jpg]]
166 (def mylist '(1 2 3))
170 (defn myadd [a b] (+ a b))
172 Clojure is defined in terms of the evaluation of data structures
173 and not in terms of the syntax of character streams/files.
175 ** macros: code transformations at compile time
177 Macros offer hooks for syntactic abstraction
178 and there is very little syntax.
185 (if and# (and ~@rest) and#))))
187 Allows code transformation *before* the reader does evaluation
188 defn is a macro that makes defining functions a little simpler.
190 *** defining functions uses the defn macro
192 Clojure supports arity overloading in a single function object,
193 self-reference, and variable-arity functions using &:
199 ([x y & more] (+ (argcount x y) (count more))))
211 ** Very easy to work with code (because it's data)
213 LISP is the language of choice when writing
214 Domain Specific Languages (DSL's).
215 - Mentioned by Theo (JRuby) and Morton (APL) today!
217 Example from ILC '09 at MIT
218 - Alex Fukunaga (Tokyo University) spoke on The Satisfyability Problem
219 - A DSL for SAT algorithms
220 - Used a biological evolution inspired algorithm
224 The Read Eval Print Loop
226 Interactive code development
228 Instead of just dump a stack trace and die on an error...
229 you can edit data and functions (they look the same)
230 and continue your program!
234 Artificial Intelligence
236 Scientific Computing Lisp
238 SciCL augments Common Lisp with an extensive library of aggregate-wise (“AG-wise”)
239 operations on arrays, providing the essential functionality of languages such as APL,
240 Fortran 90, IDL and Matlab.
242 http://www.siginf.com/
246 ** Many enterprise deployments already use Java
248 Clojure adds a jar to the CLASSPATH
249 (lowers the barrier to customer approval)
251 Embraces the power of the JVM
253 (defn #^Properties as-properties
254 "Convert any seq of pairs to a java.utils.Properties instance.
255 Uses as-str to convert both keys and values into strings."
258 (let [p (Properties.)]
260 (.setProperty p (as-str k) (as-str v)))
263 Leverages the wealth of existing Java libraries
265 ** Need the benefits of LISP and
267 Need to deal with concurrency using native threads and locking.
269 Without the downsides of Java
270 - Skip the boilerplate (to not fetishize complexity)
271 - Multi-methods instead of the "Kingdom of Nouns (OOP)"
272 - Unmoderated mutation simply "has to go"
273 (makes concurrency very difficult)
275 ** Functional Programming
277 Immutable data + first-class functions, supporting recursion
281 Emphasizes recursive iteration instead of side-effect based looping
283 user> (let [my-vector [1 2 3 4]
284 my-map {:fred "ethel"}
285 my-list (list 4 3 2 1)]
288 (assoc my-map :ricky "lucy")
293 -> ([1 2 3 4 5] {:ricky "lucy", :fred "ethel"} (5 4 3 2 1) [1 2 3 4] {:fred "ethel"} (4 3 2 1))
295 ** Software Transactional Memory
297 Core data structures are immutable and can easily be shared between threads
299 Mutation *is* possible using locks to avoid conflicts
301 - dosync, ref, set, alter, et al, supports sharing changing
302 state between threads in a synchronous and coordinated manner.
304 - The agent system supports sharing changing state between threads
305 in an asynchronous and independent manner.
307 - The atoms system supports sharing changing state between threads
308 in a synchronous and independent manner.
310 - The dynamic var system supports isolating changing state within
313 ** No spec, one implementation
315 Disadvantages: All eggs in one basket
317 Advantages: Clojure works *everywhere*
318 Innovation happens quickly
319 Core data structures are extensible abstractions
321 [[file:~/src/software-passion/rich.jpg]]
325 Embraces the power of the JVM
329 Note: also runs on the CLR and on JavaScript (*)
331 *** Java - multiplatform
333 Sun originally wanted Java to enable customers to use SPARC
335 Today many Enterprises run on Intel architectures
337 But what about tomorrow?
339 *** ARM looks very good for size, cost, heat
341 Maybe we will see ARM in the data center?
343 [[file:~/src/software-passion/arm-datacenter.png]]
345 http://news.softpedia.com/news/Ubuntu-and-HP-Will-Power-ARM-Data-Centers-231827.shtml
347 *** We are seeing ARM everywhere in embedded devices
350 - SoC is a Broadcom BCM2835. This contains an ARM1176JZFS, with floating point, running at 700Mhz
351 - Videocore 4 GPU. The GPU is capable of BluRay quality playback, using H.264 at 40MBits/s.
352 - It has a fast 3D core accessed using the supplied OpenGL ES2.0 and OpenVG libraries.
355 - (Model B adds a 2nd USB port, Ethernet)
357 [[file:~/src/software-passion/raspi_blue_white.png]]
359 http://www.raspberrypi.org/
361 *** Java for the "Internet of Things"
366 [[file:~/src/software-passion/arm-1mm.png]]
368 *** Java as assembly language
370 For these reasons Clojure is one of many vibrant,
371 alternative languages on the JVM which include:
381 ** Bleeding Edge OpenJDK features
383 NOT yet truly being used by Clojure
387 Bring Doug Lea's Fork/Join framework into Clojure
389 Primary example *pmap*
390 - using the shortest map/reduce tutorial ever
391 - WAIT, Morton did this in one line in APL :)
393 user> (def mylist '(1 2 3 4 5 6))
395 user> (map even? mylist)
396 (false true false true false true)
397 user> (reduce 'or (map even? mylist))
400 David Liebke: "From Concurrency to Parallelism"
401 http://incanter.org/downloads/fjclj.pdf
403 *** Tail Call Optimization
405 Save space on the stack:
420 replace arguments with (2 3), jump to "fact"
421 replace arguments with (1 6), jump to "fact"
422 replace arguments with (0 6), jump to "fact"
426 NOTE: Clojure does have *recur* and *trampoline* but
427 the JVM itself lacks a generic optimization for TCO
428 (but there is an older patch in the MVLM repo).
430 https://en.wikipedia.org/wiki/Tail_call
436 Enables the HotSpot VM to *see* into your "JVM Language" code
439 Why Clojure Doesn't Need Invokedynamic (Unless You Want It to be More Awesome)
440 http://blog.headius.com/2011/10/why-clojure-doesnt-need-invokedynamic.html
442 *** Modularization (Jigsaw)
445 Finer grained dependencies
446 Smaller footprint (embedded)
448 [[file:~/src/software-passion/graph.png]]
450 * The Tools I am using
454 Finding dependencies: =mvn dependency:tree -DoutputFile=dependency.txt=
456 my-website:my-website:jar:0.1.0-SNAPSHOT
457 +- org.clojure:clojure:jar:1.3.0:compile
458 \- noir:noir:jar:1.2.2-SNAPSHOT:compile
459 +- compojure:compojure:jar:1.0.0-RC2:compile
460 | +- org.clojure:core.incubator:jar:0.1.0:compile
461 | +- org.clojure:tools.macro:jar:0.1.0:compile
462 | +- clout:clout:jar:1.0.0:compile
463 | \- ring:ring-core:jar:1.0.1:compile
464 | +- commons-io:commons-io:jar:1.4:compile
465 | +- commons-fileupload:commons-fileupload:jar:1.2.1:compile
466 | \- javax.servlet:servlet-api:jar:2.5:compile
467 +- org.clojure:tools.namespace:jar:0.1.0:compile
468 | \- org.clojure:java.classpath:jar:0.1.0:compile
469 +- clj-json:clj-json:jar:0.4.3:compile
470 | \- org.codehaus.jackson:jackson-core-asl:jar:1.5.0:compile
471 +- ring:ring:jar:1.0.1:compile
472 | +- ring:ring-devel:jar:1.0.1:compile
473 | | \- ns-tracker:ns-tracker:jar:0.1.1:compile
474 | +- ring:ring-jetty-adapter:jar:1.0.1:compile
475 | | +- org.mortbay.jetty:jetty:jar:6.1.25:compile
476 | | | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
477 | | \- org.mortbay.jetty:jetty-util:jar:6.1.25:compile
478 | \- ring:ring-servlet:jar:1.0.1:compile
479 +- hiccup:hiccup:jar:0.3.7:compile
480 +- clj-stacktrace:clj-stacktrace:jar:0.2.3:compile
481 +- ring-reload-modified:ring-reload-modified:jar:0.1.1:compile
482 +- net.java.dev.jets3t:jets3t:jar:0.8.1:compile
483 | +- commons-codec:commons-codec:jar:1.3:compile
484 | +- commons-logging:commons-logging:jar:1.1.1:compile
485 | +- commons-httpclient:commons-httpclient:jar:3.1:compile
486 | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
487 \- org.mindrot:jbcrypt:jar:0.3m:compile
492 https://github.com/technomancy/leiningen
494 Use the REPL *swank-clojure*
495 https://github.com/technomancy/swank-clojure
498 $ lein plugin install swank-clojure 1.4.0
499 $ lein plugin install lein-localrepo 0.3
500 $ lein plugin install lein-noir 1.2.1
502 =lein localrepo help=
504 Public Repos: http://clojars.org/
506 Private Repos: https://github.com/technomancy/s3-wagon-private
508 Lein directly from git: https://github.com/tobyhede/lein-git-deps
512 Amazing NoSQL Database: http://redis.io
514 With a Clojure binding! https://github.com/mmcgrana/clj-redis
516 Redis utterly killed it in 2010 – check out the growth in share of developer conversation
517 http://www.redmonk.com/jgovernor/2012/03/15/redis-utterly-killed-it-in-2010-check-out-the-growth-in-share-of-developer-conversation/
523 Continuous Integration Server: http://jenkins-ci.org/
525 Amazing Plugins: https://wiki.jenkins-ci.org/display/JENKINS/Plugins
529 - Dependency Graph Viewer
531 - Pathignore (essential for big git repo)
534 - Build Result Trigger
540 KK slides from February at MonkiGras in London
541 http://www.slideshare.net/kohsuke/building-developer-community
545 Git push triggers Jenkins
546 Updates the one (master) workspace
547 Projects started based on updated paths
551 - Triggers native Mac OS X build on Mac slave
552 - Triggers native Windows build on Windows slave
554 Deploying Noir application
555 - shuts down dev website
561 http://trac.edgewall.org/
563 - Tickets (bugs, tasks), Reports, Browse code, Timeline, Wiki
564 - Can now use git (yeah!)
565 - Integration with Jenkins
566 http://trac-hacks.org/wiki/XmlRpcPlugin
570 Let's talk about Noir http://webnoir.org
572 * Why Open Source Matters
576 Free as in Free Speech
578 Knowing the shape of the solutions: Ease of integration
580 No marketing: just code (extra credit: build in tests and Jenkins)
582 Fewer bugs (recent Coverity study)
584 Education, credentials and employment
585 - Employers *will* google you
586 - Many directly ask for pointers to FLOSS contributions
588 ** Where are you going to deploy that code?
592 Are you really going to deploy to Windows?
593 - you have to name your machines #FAIL
594 - you have to Remote Desktop in and click-to-admin #FAIL
595 - no anticipated downtime until 2016 :)
597 You can't deploy to Mac OS X
598 - X Serve died a long time ago
600 You want to deploy to Linux
606 ** permissive vs. restrictive licensing
608 BSD (MIT AL2) vs. GPL (MPL)
610 Permissive is necessary, but sometimes not enough to
611 hold a community together.
613 Jeremy Allison: Why Samba Switched to GPLv3
614 2011 Linux Collaboration Summit
615 http://faif.us/cast/2011/may/10/0x0F/
617 NOTE: proprietary (dual) licensing with contributor
618 license agreements is now considered harmful
620 ** Open Source and Web Services
622 What if you want to build a strong community
623 around a web service?
625 In the "cloud" the GPL is just like BSD.
627 The answer? The AGPL (Affero General Public License)
630 The GNU Affero General Public License is a modified version
631 of the ordinary GNU GPL version 3. It has one added requirement:
632 if you run the program on a server and let other users communicate
633 with it there, your server must also allow them to download
634 the source code corresponding to the program that it's running.
636 What? I'm going to build a business on AGPL? Is that CRAZY?
638 It is being done now: http://status.net
639 "Enterprise Social Software is OPEN for business."
641 ** Where is the value?
646 - Hardware is effectively free
647 - The best software in life is Free
648 - Savoir Faire (brainpower) is expensive
649 - Data are like diamonds: they vary in clarity, quality and value
651 New business models need to maximize productivity
652 around managing and improving quality of data.
654 (NOTE: China doesn't care about intellectual property anyway)
658 Commitment to quality and building everything from source
660 Package inter-dependencies are core to the system
661 - Windows needs Maven, Gems, cygwin, etc.
662 - Mac needs MacPorts, etc.
664 Very predictable, easy to administer & automate, secure, stable
666 One of the two major Linux families (.deb and .rpm)
667 and the foundation of many derivatives (e.g. Ubuntu)
669 [[file:~/src/software-passion/debconf11.jpg]]
671 http://wiki.debconf.org/wiki/DebConf11/Pictures/GroupPhoto
673 * Challenges and Next Steps
675 ** The state of Clojure Contrib (is a challenge)
677 "Modularization of Contrib"
679 http://dev.clojure.org/display/doc/Clojure+Contrib
681 Wait, why isn't there a project.clj (for lein)?
682 - officially must use mvn (!) (lein originally could not deploy
685 The idea is that everything that hasn't been modularized yet
686 is supposedly either low quality or in low demand
688 Using clojars: change groupID to highlight it's non-canonical
690 Also it's tricky to find out what the *real* disposition of
691 stuff is.. I wanted java-utils (moved to clojure.java.io)
693 ** My client will expand capacity from private to public cloud
695 Expand service from customer hosted into EC2
696 With auto provisioning of resources (up/down)
705 Fully bi-directional pipes (no more AJAX polling)!
707 ** Redis binding change
709 clj-redis / Jedis / Apache connection library
710 - missing functionality
713 My be replaced with redis-clojure
715 ** Keeping an eye on Datomic
717 [[file:~/src/software-passion/datomic.png]]
719 ** Keeping an eye on ClojureScript One
723 ClojureScript One + the remote REPL + browser testing + no CSS reloads
725 Connect With Your Creation Through a Real-Time Editor
726 http://www.webmonkey.com/2012/03/connect-with-your-creation-through-a-real-time-editor/
727 http://www.chris-granger.com/2012/02/20/overtone-and-clojurescript/
729 ** Experiment with exposing bleeding edge JVM features in the Clojure
731 It only takes about 30 min to build the JDK on an 8 core machine
733 Tighter Debian Clojure packaging (Jigsaw)
737 LISP is incredibly powerful (don't be afraid of the parens)
739 Clojure is the best LISP now (because of the JVM)
741 Java means future proof for platforms in the cloud
742 and the "Internet of Things".
744 Open Source isn't just free, it's key to a
745 strong business model (and probably saving the planet).
747 Software "best practice" tools are available for Clojure now
749 There are *still* many optimizations waiting to be made
751 The #1 reason to use Clojure: productivity.
754 This presentation: Copyright @ 2012 Informatique, Inc.
755 under a Creative Commons Share Alike USA 3.0 license
756 https://creativecommons.org/licenses/by-sa/3.0/us/
758 [[file:~/src/software-passion/by-sa-3.0.png]]
761 Clojure: Copyright 2008-2012 Rich Hickey http://clojure.org
763 NOTE: Tom will code in Clojure / Jigsaw / Debian / ARM for food!
764 http://tmarble.info9.net
768 file:~/src/software-passion
770 ** Command line processing and configuration files
773 https://github.com/clojure/tools.cli
775 connected to SSH agent (has at least one identity)
776 tmarble@noir 102$ lein search tools.cli
777 == Results from central - Showing page 1 / 1 total
778 [org.clojure/tools.cli "0.1.0"]
779 [org.clojure/tools.cli "0.1.0"]
782 ** Pretty Print HTML and XML
784 I created a future-contrib package:
785 file:~/src/maas/clojure/future-contrib/project.clj
787 See file:~/src/maas/clojure/future-contrib/src/future_contrib/core.clj
789 Demonstrate example with file:~/src/clojuremn/example.xml
793 Demonstrates command line processing and configuration files
795 see file:~/src/maas/clojure/redis2xml/project.clj
797 see: file:~/.redis2xml
799 also try command line:
801 =redis-cli -a NoOneWillEverGuess -n 3=
803 ./bin/redis2xml -v -n 3 -f -i ~/src/clojuremn/example.xml
807 See file:~/src/noir-examples/my-website