Presentation: Clean Code

Wed, 24 Mar 2021 00:00:00 +0000

Presentation: Clean Code

This presentation was created for Prof. Dutt’s software engineering classes for the Spring 2021 semester.

Clean Code GitHub presentation slides

Presentation: Computer Science - A breadth-first search

Sun, 14 Mar 2021 00:00:00 +0000

Northern Virginia MEDLIFE chapter - Computer Science presentation

The MEDLIFE NoVA high school chapter (Medicine, education, and development for low income families everywhere) provided virtual presentations on different career fields. MEDLIFE NoVA asked me to target a presentation on Computer Science to a 6th grade level. That felt a lot more challenging than the more technical presentations that I normally give. Computer Science is such a broad field that it was hard to do it justice in 45 minutes with questions. And there were some great questions after the talk…

Computer Science - A breadth-first search GitHub presentation slides

Presentation: Ammonite - Succinct Scala Shell Scripting

Thu, 24 Oct 2019 00:00:00 +0000

Ammonite: Succinct Scala Shell Scripting

A presentation on the Ammonite Scala shell for the Baltimore Scala meetup on October 24th, 2019.

Ammonite Shell GitHub presentation slides

Ammonite Scala Shell

Sat, 17 Aug 2019 00:00:00 +0000

Ammonite Shell

The Ammonite Shell is a Scala-based shell with built-in libraries and short-cuts to provide the ability to do fully typed shell programming in Scala.

val l = (0 to 1000)
show(l, height=50)

# Multi-line edit:
def foo = {
  val a = 10
  val b = 15
  val c = 30
}
# history shows multi-line (can use up arrow)

# History search:
enter search term, hit Up/down arrow

Also: Ctrl-r, start search, Ctrl-Shift-r cycle through search results

# Import 3rd party library
import $ivy.`com.example::scala:1.2.3`

Examples

Update sbt version for all projects under current directory

# Start Ammonite with 10GB of memory
JAVA_OPTS="-Xmx10g" amm

val allFiles = ls.rec(wd)
val sbtVersionFiles = allFiles.filter{ f =>
   val segmentCount = f.segmentCount
   if (segmentsCount >= 2) {
     val parentDir = f.getSegment(segmentCount - 2)
     val lastSegment = f.last
     parentDir == "project" && lastSegment == "build.properties"
   } else {
     false
   }
}
sbtVersionFiles.foreach { versionFile =>
  //also: write (fail if file exists)
  //write.append
  write.over(versionFile, "sbt.version=1.2.8\n")
}

Scala Build Tooling

Fri, 09 Aug 2019 00:00:00 +0000

Faster Scala Builds with bloop and coursier

One common complaint about Scala is the long time lag for compiles, running tests and starting up Scala programs. Another challenge about dependency management in general is the use of libraries depending on other libraries etc. that all must be downloaded.

Enter the Scala Center’s bloop build server to reduce compile, test and run times and the coursier sbt plugin to parallelize library dependency downloads with an all Scala dependency management tool.

Bloop server

The Bloop server as supports the build server protocol (bsp) that allows integration with the IntelliJ Idea IDE and integrates with sbt, Maven, Gradle and Mill.

Ubuntu Install

# Installs to $HOME/.bloop
curl -L https://github.com/scalacenter/bloop/releases/download/v1.3.2/install.py | python

# Add bloop to PATH in .bashrc
PATH=$HOME/.bloop:....

# Add autocompletion ~/.bash_profile
. $HOME/.bloop/bash/bloop

Start/Stop via systemd

systemctl --user enable $HOME/.bloop/systemd/bloop.service
systemctl --user daemon-reload
journalctl --user-unit bloop
systemctl --user status bloop
systemctl --user start bloop # also stop/restart

sbt integration

# ${PROJECT_ROOT}/project/plugins.sbt
# https://scalacenter.github.io/bloop/setup
addSbtPlugin("ch.epfl.scala" % "sbt-bloop" % "1.3.2")

sbt
> bloopInstall #re-run when changing dependencies or build-related artifacts, modules etc.

Now compile, test run on bloop server

Coursier sbt plugin

# Edit or create ${PROJECT_ROOT}/project/project/plugins.sbt
// https://github.com/coursier/sbt-coursier/releases
// Setup: https://get-coursier.io/docs/sbt-coursier
// also see project/plugins.sbt
addSbtPlugin("io.get-coursier" % "sbt-coursier" % "2.0.0-RC3-1")

# Edit or create ${PROJECT_ROOT}/project/plugins.sbt
// https://github.com/coursier/sbt-coursier/releases
// Setup: https://get-coursier.io/docs/sbt-coursier
// also see project/project/plugins.sbt
addSbtCoursier
classpathTypes += "maven-plugin"
coursierUseSbtCredentials := true

Checksum problems

When using the 1.x version of the coursier sbt plugin I ran into some problems with a 3rd party artifact checksum not being correct. Following the comments on coursier ticket 319 I added the following but haven’t needed it since in other projects:

coursierChecksums := Nil
coursierArtifactsChecksums := Nil

Presentation: Data Engineering with Apache Spark

Thu, 02 May 2019 00:00:00 +0000

Data Engineering with Apache Spark

A presentation on doing data engineering with Apache Spark for the Baltimore Scala meetup on May 2nd, 2019.

Data Engineering with Apache Spark GitHub presentation slides

Reactive Summit 2018 - 1

Sun, 28 Oct 2018 00:00:00 +0000

New or forgotten resources referenced in talks

Online courses

Free Lightbend Reactive Architecture courses (for certification):
- Introduction to Reactive Principles
- Domain Driven Design
- Reactive Microservices
- Building Scalable Systems
- Not released yet:
  - Distributed Message Patterns
  - Reactive Architecture: CQRS & Event Sourcing

Papers/Web content

Pelkonen, Franklin, Teller et al., Gorilla: A Fast, Scalable, In-Memory Time Series Database, VLDB, 2015
Tyler Akidau, The world beyond batch
- Streaming 101, 2015
- Streaming 102, 2016
Knoldus Blog
- e.g. node.js testing with Gatling
- Welcome, Akka Typed!!

Books

Libraries

Big Data

Spark Notebook

Performance monitoring, metrics

FiloDB time series / event / operational database
- Great keynote on this by Evan Chan, Apple, @Evanfchan
Riemann distributed systems monitoring
Alexey Ragozin, jvm-tools/Swiss Java Knife
- “Small set of tools for JVM troublshooting, monitoring and profiling.”
Java Microbenchmark Harness
- Kevin Peters, Performance measurement with JMH - Java Microbenchmark Harness, 2017
Influxdata Telegraf
- “plugin-driven server agent for collecting & reporting metrics”

Reactive Frameworks

RSocket (originally from Netflix)
- Polyglot, reactive streams binary protocol

Around containers

Scala Process

Tue, 29 Aug 2017 00:00:00 +0000

Code uses java.nio.file.Files to list all files in the current directory. Using Scala JavaConverters we convert the Java iterator to a Scala buffer and the use Process to call unzip on each file.

import scala.collection.JavaConverters._
import scala.sys.process._
import java.nio.file._
val files = Files.list(Paths.get(".")).iterator.asScala
files.foreach { f => s"unzip $f"! }

Gimp Python-Fu

Tue, 20 Dec 2016 00:00:00 +0000

Background

A folder full of jpeg images of different sizes that I needed to resize to 300x300. Not wanting to open each image in Gimp - Tools - Transform Tools - Scale I was looking for a more automated solution. Gimp’s Python-Fu, Google and Stackoverflow with kind souls sharing their expertise to the resuce:

Gimp Python-Fu

In Gimp go to Filters - Python-Fu - Console
Hit Browse to find available Python methods - searched for scale and found gimp_image_scale, which seemed to do what I wanted.

With some more Googling I devised the following, which solves my problem. Pop into console and hit enter. See main sources below for proper credit.

import os

def get_files(path):
     for (dir, _, filenames) in os.walk(path):
         for filename in filenames:
             yield os.path.join(dir, filename)

             
list_files = get_files('/my_various_images')


for filename in list_files:
    image = pdb.file_jpeg_load(filename, filename)
    pdb.gimp_image_scale(image, 300, 300)
    new_img = pdb.gimp_image_duplicate(image)
    layer = pdb.gimp_image_merge_visible_layers(new_img, CLIP_TO_IMAGE)
    pdb.gimp_file_save(new_img, layer, filename + '.scaled.jpg', '?')
    pdb.gimp_image_delete(new_img)

Main sources

Presentation: Spark 2.0 Performance Improvements

Mon, 13 Jun 2016 00:00:00 +0000

On June 13th I was fortunate to present the latest Apache Spark - Project Tungsten updates, which further improve Spark performance through whole-stage code generation and vectorization.

See presentation at my github repo at SparkPerformance.pdf. The src directory of that repo also contains the corresponding example files.