Blog

Vaclav Kosar

Software And Machine Learning Blog

ELECTRA - How to Train BERT 4x Cheaper

Reducing training flops 4x by GAN-like discriminative task compared to RoBERTa-500K transformer model.

Expire-Span: Scaling Transformer by Forgetting

Reducing computational costs by differentiably dropping memorized embeddings from self-attention context.

Scout Mindset Book Summary

Short summary of a book from Julia Galef on clear thinking.

Quilt Data Versioning Review & How-to

How to version data using Quilt data for Python on AWS S3.

Wav2vec: Semi and Unsupervised Speech Recognition

Audio Word2vec - Quantize phonemes, transform, GAN the text.

PID Controller: A Simple Control Loop Mechanism

Proportional–integral–derivative controller calculates feedback to reduce the error in the next step.

DreamCoder: Wake & Sleep Program Learning

Learning to code by growing function library, fantasising coding tasks, and training neural search.

Automatically Expanding Taxonomy

Arborist model finds parents for unseen textual nodes using triplet-loss, StarSpace embeddings, & shortest path.

Submodularity in Ranking, Summarization, and Self-attention

Diminishing returns with a budget constraint in problems of coverage and results diversification.

Feed-Forward, Self-Attention & Key-Value

Transformer model queries values of keys in the self-attention and in the feed-forward memories.

Lambda Networks Transform Self-Attention

Is Lambda Layer similar to self-attention in a Transformer? What gives LambdaNet its power? LambdaResNet beats EfficientNet but does it loose to Performer?

Performers FAVOR+ Faster Transformer Attention

The Performer model attention approximation has linear time & space complexity in input token count in contrast to vanilla Transformer's square complexity and outperforms Linformer.

Double Descent Contrary to Bias-Variance Trade-Off

Increasing model's parameter count leads to multiple test loss peaks and achieving global minima in the overparameterized regime.

Brutalist and Modernist Architectures Collide at Sunshine Plaza in Prague

Take a tour of the surprising merge of a modern and soviet era design on a Prague's public square with this photo album.

Word Mover's Embedding: Cheap WMD For Documents

What is Word Mover's Embedding for documents and how it approximates Word Mover's Distance between documents.

Transfigure Stress into Energy by Drawing on Research

Your pounding heart and blush will announce a flashing opportunity instead of an impending fight or flight after applying research from this post.

OpenAI's Glow - Flow-Based Model Teardown

Get interpretable latent representations by composing non-linear invertible functions and maximizing the exact log-likelihood.

BentoML vs Cortex - ML Serving Showdown

To find the best model serving tool, compare open-source MLOps platforms BentoML and Cortex.

StarSpace Embedding - United and universal spaces of vectors

To embed variety of entities into single vector space, this paper describes general-purpose neural embedding model.

Thinkpad P52 Disassembly For Repaste, RAM Upgrade, Or Anything Else

Repaste, max RAM, or install antenna into Thinkpad P52 with these links and tips for full disassembly.

Handle Exception And Close Resource Using Python Context Manager With-Statement

Wrap you resource creation and closure into context manager with-statement and catch exceptions there.

I read papers on a podcast

To improve my pronunciation and speech, I read mostly Machine Learning scientific papers on a podcast.

Learn faster with a generated quiz

Reduce your effort of creating and revising learning material using a free AI-powered tool.

Thinkpad P53 vs P52 Thermals: Any Improvement?

Is Thinkpad P53 the cooler brother of P52?

Constant 1D Kalman Filter Is Exponential Or Cumulative Average

In one dimension and with constant measurement uncertainty and process noise, the filter converges to cumulative average in case of zero process noise or exponential average in case of non zero process noise.

FastText Vector Norms And OOV Words Visualizations

Visualize vector norms of FastText embedding and evaluate use of FastText word vector norm multiplied with number of word n-grams for detecting non-english OOV words.

Highly Compressed Richard Hamming's Lectures

Get inspired by Hamming's lectures compressed into tiny downloadable files.

Thinkpad P52 vs ZBook 15 G5 vs Precision 7530

This is my experience working on the best mobile workstation of 2019 with specs matching Thinkpad P52 and ZBook 15 G5.

My First Contribution To Major Oss Project

Our Presentation At Spark N AI Summit

Me and Marek Novotny had opportunity to present our POC and future plans for Spark Structured Streaming data lineage.

Debounce In Bash To Fix Lenovo Touchpad And Trackpoint Lost Sync

Another functional programming tip for Bash.

How To Create Custom Ubuntu Web Link App

Turn any web page into an Ubuntu application and prevent the web owner from tracking you around the web.

Modern Config Injection In Maven Plugins

Simple constructor injection of config params in Maven Plugins can be done.

Walking Desk: Cheap And Tiny

A review of my motor-less walking desk setup.

Boundary Control Entity Architecture: The Pattern to Structure Your Classes

BCE is a file structure for source code files sometimes with additional dependency restrictions. Example included!

What Are RxJS And Spring Integration Similarities?

Understand the frameworks better by comparing their concepts.

Spring Integration Highlights - The message driven

Get familiar with Spring Integration implementation of Enterprise Integration Patterns and compare it to Java 8 Streams.

Fish Roe vs Fish Oil

Healthiness and price of a salty delicacy versus oily softgels. EPA, DHA, Neu5Gc.

Restore Missing Punctuation with Keras Convolutional Text Punctuator

Simple deep neural network machine learning project for restoring punctuation in a text.

Easy Online Independence: 1. Mail Backup

This is first post of a series called "Easy Online Independence" aiming to provide cheap and easy way to increase your independence from current online service giants like Google and Facebook.

Generic Class Name Signals Low Cohesion: Why to avoid non-specific class names like util, and utils.

Even when most classes in project have nice specific name, somehow "Util" suffixed classes seem to be excluded from intuitive naming strategies.

How to Structure Code: Localize Related, Inline over Extract, Specific over Generic

A short alternative view on code structure principles based on several years of personal experience, Carmack, Jonathan Blow, and Adam Bien's posts.

Youtube Reader: Save Data Using Only Punctuated Captions

Android app that uses neural network to punctuate Youtube video captions to be used with any text to speech software.

Is $15 USB Microscope Enough For You?

See yeast cells and pond water critters paying just $15 for an USB microscope.

Linux Text To Speech Comparison: Flite Vs Pico2Wave Vs Festival

Comparison of open-source text to speech software in terms of pleasantness, comprehensibility, and modularity.

Try This Sped-Up Classical Music Attuned For Today's Sped-Up Age

Are you attracted to complexity and nobility of classical music, but deterred by its slow pace?

The Drone Hackathoning - Experience From A Hackathon

On a hackathon I implemented trivial image processing algorithm to locate white piece of paper on grey carpet floor to be used for drone horizontal stabilization.

Functional ForEach In Bash

Don't you hate verbosity of Bash's while-do statements when writing in-line scripts? No worries, you can improve on that!

Obsolete Git Branch Remover Maven Plugin

Having many branches left behind, abandoned, never deleted? How do you deal with them? We had same problem and I developed a automated DevOps solution.

How To Boost Your Jog Morale Using Military Cadence And Run Farther

Run beyond your max with this professional mind hack.

GitFlow Incremental Builder - Speed up your multi-module Maven build

This open-source Maven plugin allows you to build only those modules that changed compared to a reference Git branch and and all their Maven dependents.

Hamiltonians with constant spectral intervals and time-dependent perturbation

On quantum systems determined by time-dependent Hamilton operators. Family of quantum systems, whose Hamilton operators take form H(t) = H 0 + V (t), where V (t) is perturbation and H 0 is self-adjoint with pure-point spectrum and constant gaps between eigenvalues in spectrum σ(H 0 ).

Feynman summation in finite-dimensional quantum mechanics

A summary and enhancement of existing literature regarding finite-dimensional quantum mechanics. In the later parts Feynman’s path summation is discussed.

Simulation of Soft Photon Calorimeter

Understand how an electromagnetic calorimeter works.

Transverse momentum spectra and correlations in the blast wave model with resonances

This work provides a review of theories of properties of high energy density matter originating in heavy-ion high energy collisions (GeV/nucleus).