--- /dev/null
+#+TITLE: Real World Clojure
+#+AUTHOR: Tom Marble
+#+EMAIL: tmarble@info9.net
+#+STARTUP: content
+
+* Overview
+
+Real World Clojure
+
+[[file:~/src/software-passion/Clojure-glyph.svg]]
+
+My journey in using Clojure for a client that is
+developing a multiplayer game server hosting service.
+
+
+** Overview: Topics to cover
+
+Press *f9* to see the list of topics
+
+**** Note's on what I put in my .emacs.d/custom.el
+
+(require 'org-tree-slide)
+
+(global-set-key (kbd "<f1>") 'show-all)
+(global-set-key (kbd "<f5>") 'text-scale-decrease)
+(global-set-key (kbd "<f6>") 'text-scale-increase)
+(global-set-key (kbd "<f8>") 'org-tree-slide-mode)
+(global-set-key (kbd "<f9>") 'org-tree-slide-content)
+(global-set-key (kbd "<f10>") 'hide-sublevels)
+
+** What is that presentation tool?
+
+Emacs!
+
+This is *org-tree-slide* from https://github.com/takaxp/org-tree-slide
+
+For more on org mode see http://orgmode.org/org.html
+
+Yes I will share my "slides" on my website http://tmarble.info9.net
+
+[[file:~/src/software-passion/org-mode.png]]
+
+** Tack!
+
+My great grandmother immigrated to the USA from Sweden around 1900
+
+[[file:~/src/software-passion/asimn.png]]
+
+
+* Background
+
+** About Tom
+
+tmarble
+[[file://home/tmarble/Pictures/Tom/Mugshot2011/Tom-2011-200.jpg]]
+
+*** Sun: technical presales during the dot.com era
+
+[[file:~/src/software-passion/e10k.jpg]]
+
+
+
+*** Sun: Java Performance
+
+[[file:~/src/software-passion/specjvm2008.png]]
+
+*** Sun: DLJ
+
+Early 2006: DLJ with Debian and Canonical (Ubuntu)
+
+[[file:~/src/software-passion/dlj.png]]
+
+*** Sun: OpenJDK
+
+JavaOne 2006: Rich Green announces that Sun will open source Java
+
+Core Strategy Team
+- How organize community governance
+- Copyright, Patent and Trademark licensing
+- Infrastructure tools
+- Pick license
+
+First OpenJDK Ambassador (I went to a lot of conferences)
+- FOSDEM
+- FISL
+- OSCON
+
+ApacheCon 2006: Sun unBOF/Party
+[[file:~/src/software-passion/apachecon2006.jpg]]
+Copyright 2006 Ted Leung: https://secure.flickr.com/photos/twleung/268116213/
+
+*** Left Sun to do a startup
+
+[[file:~/src/software-passion/tom-think.jpg]]
+
+*** But
+
+The global financial meltdown of 2008 happened (etc.)
+
+Didn't work out :(
+
+And so I... got into consulting!
+
+*** Consulting
+
+Cybersecurity
+
+Probabilistic Model Verification (Electrical Engineering)
+
+Software for Smart Grid + Renewable Energy
+
+Clojure
+
+*** Tom and Debian
+
+Using Linux since 1996
+
+Using Debian since 2003
+
+Helped Debian Java Packaging Team since DLJ in 2006
+
+World with Debian and Oracle on meshing Jigsaw with *apt*
+
+[[file:~/src/software-passion/gsoc11.jpg]]
+
+*** FOSDEM
+
+Java track
+- Oracle
+- Distros
+- Developers
+- Users
+
+Legal Issues Track
+- Organized by Karen Sandler, Bradley Kuhn, Richard Fontana and myself
+- Check out the Wiki http://info9.net/wiki/fosdem/LegalIssuesDevRoom/
+- Check out the oggcast http://faif.us/
+
+** About my client
+
+The client
+- Developing a multiplayer game server hosting service
+- Comprised of very young developers
+- Is in stealth mode (sorry!)
+
+I have been given the authority to
+- Make significant choices about architecture
+- Green light to open source generic bits
+
+(this is why i like consulting :)
+
+I'm not the only one (especially in Nordic countries)!
+
+* Why Lisp?
+
+John McCarthy is old school:
+[[file:~/src/software-passion/John_McCarthy.jpg]]
+
+
+** homoiconic
+
+*code is data*
+
+List
+
+ (def mylist '(1 2 3))
+
+Function
+
+ (defn myadd [a b] (+ a b))
+
+Clojure is defined in terms of the evaluation of data structures
+and not in terms of the syntax of character streams/files.
+
+** macros: code transformations at compile time
+
+Macros offer hooks for syntactic abstraction
+and there is very little syntax.
+
+(defmacro and
+ ([] true)
+ ([x] x)
+ ([x & rest]
+ `(let [and# ~x]
+ (if and# (and ~@rest) and#))))
+
+Allows code transformation *before* the reader does evaluation
+defn is a macro that makes defining functions a little simpler.
+
+*** defining functions uses the defn macro
+
+Clojure supports arity overloading in a single function object,
+self-reference, and variable-arity functions using &:
+
+(defn argcount
+ ([] 0)
+ ([x] 1)
+ ([x y] 2)
+ ([x y & more] (+ (argcount x y) (count more))))
+
+-> #'user/argcount
+(argcount)
+-> 0
+(argcount 1)
+-> 1
+(argcount 1 2)
+-> 2
+(argcount 1 2 3 4 5)
+-> 5
+
+** Very easy to work with code (because it's data)
+
+LISP is the language of choice when writing
+Domain Specific Languages (DSL's).
+- Mentioned by Theo (JRuby) and Morton (APL) today!
+
+Example from ILC '09 at MIT
+- Alex Fukunaga (Tokyo University) spoke on The Satisfyability Problem
+- A DSL for SAT algorithms
+- Used a biological evolution inspired algorithm
+
+** REPL
+
+The Read Eval Print Loop
+
+Interactive code development
+
+Instead of just dump a stack trace and die on an error...
+you can edit data and functions (they look the same)
+and continue your program!
+
+** Lisp successes
+
+Artificial Intelligence
+
+Scientific Computing Lisp
+
+ SciCL augments Common Lisp with an extensive library of aggregate-wise (“AG-wise”)
+ operations on arrays, providing the essential functionality of languages such as APL,
+ Fortran 90, IDL and Matlab.
+
+ http://www.siginf.com/
+
+* Why Clojure?
+
+** Many enterprise deployments already use Java
+
+Clojure adds a jar to the CLASSPATH
+(lowers the barrier to customer approval)
+
+Embraces the power of the JVM
+
+(defn #^Properties as-properties
+ "Convert any seq of pairs to a java.utils.Properties instance.
+ Uses as-str to convert both keys and values into strings."
+ {:tag Properties}
+ [m]
+ (let [p (Properties.)]
+ (doseq [[k v] m]
+ (.setProperty p (as-str k) (as-str v)))
+ p))
+
+Leverages the wealth of existing Java libraries
+
+** Need the benefits of LISP and
+
+Need to deal with concurrency using native threads and locking.
+
+Without the downsides of Java
+- Skip the boilerplate (to not fetishize complexity)
+- Multi-methods instead of the "Kingdom of Nouns (OOP)"
+- Unmoderated mutation simply "has to go"
+ (makes concurrency very difficult)
+
+** Functional Programming
+
+Immutable data + first-class functions, supporting recursion
+
+Dynamic polymorphism
+
+Emphasizes recursive iteration instead of side-effect based looping
+
+user> (let [my-vector [1 2 3 4]
+ my-map {:fred "ethel"}
+ my-list (list 4 3 2 1)]
+ (list
+ (conj my-vector 5)
+ (assoc my-map :ricky "lucy")
+ (conj my-list 5)
+ my-vector
+ my-map
+ my-list))
+-> ([1 2 3 4 5] {:ricky "lucy", :fred "ethel"} (5 4 3 2 1) [1 2 3 4] {:fred "ethel"} (4 3 2 1))
+
+** Software Transactional Memory
+
+Core data structures are immutable and can easily be shared between threads
+
+Mutation *is* possible using locks to avoid conflicts
+
+- dosync, ref, set, alter, et al, supports sharing changing
+ state between threads in a synchronous and coordinated manner.
+
+- The agent system supports sharing changing state between threads
+ in an asynchronous and independent manner.
+
+- The atoms system supports sharing changing state between threads
+ in a synchronous and independent manner.
+
+- The dynamic var system supports isolating changing state within
+ threads.
+
+** No spec, one implementation
+
+Disadvantages: All eggs in one basket
+
+ Advantages: Clojure works *everywhere*
+ Innovation happens quickly
+ Core data structures are extensible abstractions
+
+[[file:~/src/software-passion/rich.jpg]]
+
+** Java
+
+Embraces the power of the JVM
+- Multiplatform
+- Performance
+
+Note: also runs on the CLR and on JavaScript (*)
+
+*** Java - multiplatform
+
+Sun originally wanted Java to enable customers to use SPARC
+
+Today many Enterprises run on Intel architectures
+
+But what about tomorrow?
+
+*** ARM looks very good for size, cost, heat
+
+Maybe we will see ARM in the data center?
+
+[[file:~/src/software-passion/arm-datacenter.png]]
+
+http://news.softpedia.com/news/Ubuntu-and-HP-Will-Power-ARM-Data-Centers-231827.shtml
+
+*** We are seeing ARM everywhere in embedded devices
+
+Raspberry Pi = $25
+- SoC is a Broadcom BCM2835. This contains an ARM1176JZFS, with floating point, running at 700Mhz
+- Videocore 4 GPU. The GPU is capable of BluRay quality playback, using H.264 at 40MBits/s.
+- It has a fast 3D core accessed using the supplied OpenGL ES2.0 and OpenVG libraries.
+- 256 MB RAM
+- One USB port
+- (Model B adds a 2nd USB port, Ethernet)
+
+[[file:~/src/software-passion/raspi_blue_white.png]]
+
+http://www.raspberrypi.org/
+
+*** Java for the "Internet of Things"
+
+Tiny ARM chip
+IPv6
+
+[[file:~/src/software-passion/arm-1mm.png]]
+
+*** Java as assembly language
+
+For these reasons Clojure is one of many vibrant,
+alternative languages on the JVM which include:
+
+- JRuby
+- Scala
+- Jython
+- IKVM.NET
+- Gosu
+- Smalltalk
+- JavaScript
+
+** Bleeding Edge OpenJDK features
+
+NOT yet truly being used by Clojure
+
+*** Fork/Join
+
+Bring Doug Lea's Fork/Join framework into Clojure
+
+Primary example *pmap*
+- using the shortest map/reduce tutorial ever
+- WAIT, Morton did this in one line in APL :)
+
+ user> (def mylist '(1 2 3 4 5 6))
+ #'user/mylist
+ user> (map even? mylist)
+ (false true false true false true)
+ user> (reduce 'or (map even? mylist))
+ true
+
+David Liebke: "From Concurrency to Parallelism"
+http://incanter.org/downloads/fjclj.pdf
+
+*** Tail Call Optimization
+
+Save space on the stack:
+
+ call factorial (3)
+ call fact (3 1)
+ call fact (2 3)
+ call fact (1 6)
+ call fact (0 6)
+ return 6
+ return 6
+ return 6
+ return 6
+ return 6
+
+ call factorial (3)
+ call fact (3 1)
+ replace arguments with (2 3), jump to "fact"
+ replace arguments with (1 6), jump to "fact"
+ replace arguments with (0 6), jump to "fact"
+ return 6
+ return 6
+
+NOTE: Clojure does have *recur* and *trampoline* but
+the JVM itself lacks a generic optimization for TCO
+(but there is an older patch in the MVLM repo).
+
+https://en.wikipedia.org/wiki/Tail_call
+
+*** Invoke Dynamic
+
+JSR 292
+
+Enables the HotSpot VM to *see* into your "JVM Language" code
+and optimize it!
+
+Why Clojure Doesn't Need Invokedynamic (Unless You Want It to be More Awesome)
+http://blog.headius.com/2011/10/why-clojure-doesnt-need-invokedynamic.html
+
+*** Modularization (Jigsaw)
+
+Better startup time
+Finer grained dependencies
+Smaller footprint (embedded)
+
+[[file:~/src/software-passion/graph.png]]
+
+* The Tools I am using
+
+** Maven
+
+Finding dependencies: =mvn dependency:tree -DoutputFile=dependency.txt=
+
+my-website:my-website:jar:0.1.0-SNAPSHOT
++- org.clojure:clojure:jar:1.3.0:compile
+\- noir:noir:jar:1.2.2-SNAPSHOT:compile
+ +- compojure:compojure:jar:1.0.0-RC2:compile
+ | +- org.clojure:core.incubator:jar:0.1.0:compile
+ | +- org.clojure:tools.macro:jar:0.1.0:compile
+ | +- clout:clout:jar:1.0.0:compile
+ | \- ring:ring-core:jar:1.0.1:compile
+ | +- commons-io:commons-io:jar:1.4:compile
+ | +- commons-fileupload:commons-fileupload:jar:1.2.1:compile
+ | \- javax.servlet:servlet-api:jar:2.5:compile
+ +- org.clojure:tools.namespace:jar:0.1.0:compile
+ | \- org.clojure:java.classpath:jar:0.1.0:compile
+ +- clj-json:clj-json:jar:0.4.3:compile
+ | \- org.codehaus.jackson:jackson-core-asl:jar:1.5.0:compile
+ +- ring:ring:jar:1.0.1:compile
+ | +- ring:ring-devel:jar:1.0.1:compile
+ | | \- ns-tracker:ns-tracker:jar:0.1.1:compile
+ | +- ring:ring-jetty-adapter:jar:1.0.1:compile
+ | | +- org.mortbay.jetty:jetty:jar:6.1.25:compile
+ | | | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
+ | | \- org.mortbay.jetty:jetty-util:jar:6.1.25:compile
+ | \- ring:ring-servlet:jar:1.0.1:compile
+ +- hiccup:hiccup:jar:0.3.7:compile
+ +- clj-stacktrace:clj-stacktrace:jar:0.2.3:compile
+ +- ring-reload-modified:ring-reload-modified:jar:0.1.1:compile
+ +- net.java.dev.jets3t:jets3t:jar:0.8.1:compile
+ | +- commons-codec:commons-codec:jar:1.3:compile
+ | +- commons-logging:commons-logging:jar:1.1.1:compile
+ | +- commons-httpclient:commons-httpclient:jar:3.1:compile
+ | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
+ \- org.mindrot:jbcrypt:jar:0.3m:compile
+
+** Leiningen
+
+Leiningen is awesome
+ https://github.com/technomancy/leiningen
+
+Use the REPL *swank-clojure*
+ https://github.com/technomancy/swank-clojure
+
+Get...
+$ lein plugin install swank-clojure 1.4.0
+$ lein plugin install lein-localrepo 0.3
+$ lein plugin install lein-noir 1.2.1
+
+=lein localrepo help=
+
+Public Repos: http://clojars.org/
+
+Private Repos: https://github.com/technomancy/s3-wagon-private
+
+Lein directly from git: https://github.com/tobyhede/lein-git-deps
+
+** Redis
+
+Amazing NoSQL Database: http://redis.io
+
+With a Clojure binding! https://github.com/mmcgrana/clj-redis
+
+Redis utterly killed it in 2010 – check out the growth in share of developer conversation
+http://www.redmonk.com/jgovernor/2012/03/15/redis-utterly-killed-it-in-2010-check-out-the-growth-in-share-of-developer-conversation/
+
+
+
+** Jenkins
+
+Continuous Integration Server: http://jenkins-ci.org/
+
+Amazing Plugins: https://wiki.jenkins-ci.org/display/JENKINS/Plugins
+
+The ones that I use:
+- Trac Publisher
+- Dependency Graph Viewer
+- IM
+- Pathignore (essential for big git repo)
+- SSH Slaves
+- Thin Backup
+- Build Result Trigger
+
+Fun ones
+- Gravatar
+- Emotional Jenkins
+
+KK slides from February at MonkiGras in London
+ http://www.slideshare.net/kohsuke/building-developer-community
+
+** Using Jenkins
+
+Git push triggers Jenkins
+Updates the one (master) workspace
+Projects started based on updated paths
+
+Java Client
+- Builds on Linux
+- Triggers native Mac OS X build on Mac slave
+- Triggers native Windows build on Windows slave
+
+Deploying Noir application
+- shuts down dev website
+- updates code
+- restarts website
+
+** Trac
+
+http://trac.edgewall.org/
+
+- Tickets (bugs, tasks), Reports, Browse code, Timeline, Wiki
+- Can now use git (yeah!)
+- Integration with Jenkins
+ http://trac-hacks.org/wiki/XmlRpcPlugin
+
+** Noir
+
+Let's talk about Noir http://webnoir.org
+
+* Why Open Source Matters
+
+Free as in Free Beer
+
+Free as in Free Speech
+
+Knowing the shape of the solutions: Ease of integration
+
+No marketing: just code (extra credit: build in tests and Jenkins)
+
+Fewer bugs (recent Coverity study)
+
+Education, credentials and employment
+- Employers *will* google you
+- Many directly ask for pointers to FLOSS contributions
+
+** Where are you going to deploy that code?
+
+The "cloud".
+
+Are you really going to deploy to Windows?
+- you have to name your machines #FAIL
+- you have to Remote Desktop in and click-to-admin #FAIL
+- no anticipated downtime until 2016 :)
+
+You can't deploy to Mac OS X
+- X Serve died a long time ago
+
+You want to deploy to Linux
+- Cost effective
+- Legal
+- More reliable
+- More automatable
+
+** permissive vs. restrictive licensing
+
+BSD (MIT AL2) vs. GPL (MPL)
+
+Permissive is necessary, but sometimes not enough to
+hold a community together.
+
+Jeremy Allison: Why Samba Switched to GPLv3
+2011 Linux Collaboration Summit
+ http://faif.us/cast/2011/may/10/0x0F/
+
+NOTE: proprietary (dual) licensing with contributor
+license agreements is now considered harmful
+
+** Open Source and Web Services
+
+What if you want to build a strong community
+around a web service?
+
+In the "cloud" the GPL is just like BSD.
+
+The answer? The AGPL (Affero General Public License)
+
+From the FSF
+ The GNU Affero General Public License is a modified version
+ of the ordinary GNU GPL version 3. It has one added requirement:
+ if you run the program on a server and let other users communicate
+ with it there, your server must also allow them to download
+ the source code corresponding to the program that it's running.
+
+What? I'm going to build a business on AGPL? Is that CRAZY?
+
+It is being done now: http://status.net
+"Enterprise Social Software is OPEN for business."
+
+** Where is the value?
+
+Productivity!
+
+Right Now
+- Hardware is effectively free
+- The best software in life is Free
+- Savoir Faire (brainpower) is expensive
+- Data are like diamonds: they vary in clarity, quality and value
+
+New business models need to maximize productivity
+around managing and improving quality of data.
+
+(NOTE: China doesn't care about intellectual property anyway)
+
+** Why Debian
+
+Commitment to quality and building everything from source
+
+Package inter-dependencies are core to the system
+- Windows needs Maven, Gems, cygwin, etc.
+- Mac needs MacPorts, etc.
+
+Very predictable, easy to administer & automate, secure, stable
+
+One of the two major Linux families (.deb and .rpm)
+and the foundation of many derivatives (e.g. Ubuntu)
+
+[[file:~/src/software-passion/debconf11.jpg]]
+
+http://wiki.debconf.org/wiki/DebConf11/Pictures/GroupPhoto
+
+* Challenges and Next Steps
+
+** The state of Clojure Contrib (is a challenge)
+
+"Modularization of Contrib"
+
+ http://dev.clojure.org/display/doc/Clojure+Contrib
+
+Wait, why isn't there a project.clj (for lein)?
+- officially must use mvn (!) (lein originally could not deploy
+ to remote mvn repos)
+
+The idea is that everything that hasn't been modularized yet
+is supposedly either low quality or in low demand
+
+Using clojars: change groupID to highlight it's non-canonical
+
+Also it's tricky to find out what the *real* disposition of
+stuff is.. I wanted java-utils (moved to clojure.java.io)
+
+** My client will expand capacity from private to public cloud
+
+Expand service from customer hosted into EC2
+With auto provisioning of resources (up/down)
+
+** Websockets
+
+jQuery
+Atmosphere
+Jetty
+Noir
+
+Fully bi-directional pipes (no more AJAX polling)!
+
+** Redis binding change
+
+clj-redis / Jedis / Apache connection library
+- missing functionality
+- times out
+
+My be replaced with redis-clojure
+
+** Keeping an eye on Datomic
+
+[[file:~/src/software-passion/datomic.png]]
+
+** Keeping an eye on ClojureScript One
+
+ClojureScript
+ClojureScript One
+ClojureScript One + the remote REPL + browser testing + no CSS reloads
+
+Connect With Your Creation Through a Real-Time Editor
+http://www.webmonkey.com/2012/03/connect-with-your-creation-through-a-real-time-editor/
+http://www.chris-granger.com/2012/02/20/overtone-and-clojurescript/
+
+** Experiment with exposing bleeding edge JVM features in the Clojure
+
+It only takes about 30 min to build the JDK on an 8 core machine
+
+Tighter Debian Clojure packaging (Jigsaw)
+
+* Conclusion
+
+LISP is incredibly powerful (don't be afraid of the parens)
+
+Clojure is the best LISP now (because of the JVM)
+
+Java means future proof for platforms in the cloud
+and the "Internet of Things".
+
+Open Source isn't just free, it's key to a
+strong business model (and probably saving the planet).
+
+Software "best practice" tools are available for Clojure now
+
+There are *still* many optimizations waiting to be made
+
+The #1 reason to use Clojure: productivity.
+
+
+This presentation: Copyright @ 2012 Informatique, Inc.
+under a Creative Commons Share Alike USA 3.0 license
+https://creativecommons.org/licenses/by-sa/3.0/us/
+
+[[file:~/src/software-passion/by-sa-3.0.png]]
+
+
+Clojure: Copyright 2008-2012 Rich Hickey http://clojure.org
+
+NOTE: Tom will code in Clojure / Jigsaw / Debian / ARM for food!
+http://tmarble.info9.net
+
+* Q/A + Live Hacking
+
+file:~/src/software-passion
+
+** Command line processing and configuration files
+
+tools.cli
+ https://github.com/clojure/tools.cli
+ awesome, right?
+connected to SSH agent (has at least one identity)
+tmarble@noir 102$ lein search tools.cli
+ == Results from central - Showing page 1 / 1 total
+[org.clojure/tools.cli "0.1.0"]
+[org.clojure/tools.cli "0.1.0"]
+tmarble@noir 103$
+
+** Pretty Print HTML and XML
+
+I created a future-contrib package:
+file:~/src/maas/clojure/future-contrib/project.clj
+
+See file:~/src/maas/clojure/future-contrib/src/future_contrib/core.clj
+
+Demonstrate example with file:~/src/clojuremn/example.xml
+
+** redis2xml
+
+Demonstrates command line processing and configuration files
+
+see file:~/src/maas/clojure/redis2xml/project.clj
+
+see: file:~/.redis2xml
+
+also try command line:
+
+=redis-cli -a NoOneWillEverGuess -n 3=
+
+./bin/redis2xml -v -n 3 -f -i ~/src/clojuremn/example.xml
+
+** Example Noir site
+
+See file:~/src/noir-examples/my-website
+
+
+
projects / wiki.git / blobdiff